Expectation

The definition of expectation or expected value of a random variable X : S, P → ℝ is supposed to formalize the intuitive idea of the ”average value” of X. As we know, the elementary notion of the average of a collection of values is the sum of the values divided by the number of values under consideration; furthermore in calculus one defined the average of a function f(x) on an interval [a, b] to be the integral, (b − a)^{−1} ∫_{a}^{b} f(x) dx. (One can prove rather easily that this integral is the limit of the elementary average of the values of f at n equally spaced points in [a, b] as n → ∞.)

It is implicit in the above definitions that the various values under consideration are weighted equally in taking the average (e.g., with n values, one weights each with a factor 1/n.) The expectation of a random variable is quite similar to the average of a function and the general definition involves an integral; the main difference is that one wants to weight the values of X with the probabilities that these values occur.

There are several ways to proceed. One can give a rather ad hoc definition of expectation for various types of random variables which is rather elementary, but for which certain useful properties are not so easy to deduce or understand initially. This is the route taken in our text. On the other hand, one can give a somewhat “fancier” definition for which these useful properties are more transparent, and then observe that in particular cases the fancy definition is just the one given in the text. I will follow this latter course here.

The general definition of the expectation of X, denoted E(X) is

E(X) = ∫_{S} X(s) P (ds).

Of course, one has to define what this means; the idea is that it should be an integral (or sum) of values of X weighted with the probabilities that these values occur. I won’t give a complete definition here, but will rather indicate how the definition might be formulated; this will be enough for our purposes.

Suppose first that X is a function which has value 1 on some event A and is equal to 0 elsewhere, i.e. X = 1_{A}, the characteristic or indicator function of A. The we define

∫_{S} X(s) P (ds) = ∫_{S} 1_{A}(s) P (ds) = P (A).

More generally if X is a finite linear combination of characteristic functions, X = ∑_{i} c_{i} · 1_{A_{i}}, then

∫_{S} X(s) P (ds) = ∫_{S}(∑_{i} c_{i} · 1_{A_{i}}(s)) P (ds) = ∑_{i} c_{i} · P (A_{i}).

This last formula is true for an infinite sum (series) also, provided the sums involved converge absolutely.

A function of form X = ∑_{i} c_{i} · 1_{A_{i}} is called a simple function. We have thus defined the integral of a random variable which is a simple function. For more general random variables X, one proceeds as follows. Find a sequence of random variables X_{n} which are simple functions and for which lim_{n→∞} X_{n} = X. Then define

∫_{S} X(s) P (ds) = lim_{n→∞} ∫_{S} X_{n}(s) P (ds).

(Of course, it takes some work to show that this makes sense for a certain class of random variables X, that the definition is independent of the particular sequence chosen, etc, etc, but this can be done, and it is not necessary to see all the details in order to have an understanding of the resulting ideas.)

An property of the integral which follows easily from the definition is the following: if X and Y are random variables on S equipped with P, and C and D are constants, then

∫_{S}(C · X(s) + D · Y (s)) P (ds) = C · ∫_{S} X(s) P (ds) + D · ∫_{S} Y (s) P (ds).

This can also be stated:

E(CX + DY) = CE(X) + DE(Y).

This is a very important and useful property! (and it is less obvious if one uses the definition of expectation that is given in the text.)

Now suppose that S is a finite sample space with P. (The discussion to follow also applies if S is countable, provided the sums discussed converge properly.) Let X be a random variable on S. Then X has only finitely many values, so X is actually a simple function. (X = ∑_{s∈S} X(s) · 1_{{s}}.) Therefore we have that

E(X) = ∑_{s∈S} X(s)P (s).

We can often rearrange the above sum to get a possibly more compact expression. Namely, it may be that the set {x_{1}, x_{2}, ..., x_{i}, ...} of values of X is considerably smaller than S itself; X may have the same value, x_{i}, at many points of S. Then we could write

E(X) = ∑_{s∈S} X(s)P (s) = ∑_{i}(x_{i} · ∑_{{s:X(s)=x_{i}}} P (s)) = ∑_{i} x_{i} · P (X = x_{i}).

This is the definition given in the text for the expectation of a discrete random variable.

Suppose next that X is a continuous random variable with pdf f(x). Assume for simplicity that f is continuous. We can express the expectation of X in this case using f(x) by reasoning as follows. Suppose we pick reals x_{1} < x_{2} < ... < x_{i} < ... with (Δx)_{i} = x_{i+1} −x_{i} small. Then we could approximate X by the simple function which has the values x_{i} on the set {s : x_{i} ≤ X(s) < x_{i+1}} which has probability ∫_{x_{i}}^{x_{i+1}} f(x) dx ≈ f(x_{i})(Δx)_{i}. The integral of this simple function is

∑_{i} x_{i} ∫_{x_{i}}^{x_{i+1}} f(x) dx ≈ ∑_{i} x_{i}f(x_{i})(Δx)_{i}.

These latter sums are an approximation to the integral ∫_{−∞}^{∞} xf(x) dx, so we see that in this case

E(X) = ∫_{−∞}^{∞} xf(x) dx.

This is the definition given in the text for the expectation of a continuous random variable.

The formulas derived above indicate that the expectation, E(X), of a random variable X depends only on the distribution of X, and this is indeed true (and not difficult to show, although we omit the proof.) Nowm we know that X on {S, P } has the same distribution as x (the identity function) on {ℝ, P_{X}}. Thus, these have the same expectation:

E(X) = ∫_{S} X(s) P (ds) = ∫_{−∞}^{∞} x P_{X}(dx).

(The last integral is just a special case of the integral of a random variable discussed above.) We can view this as a general formula including the special cases discussed above for discrete random variables or continuous random variables with a pdf. In fact, if P_{X} = ∑_{i} p(x_{i})δ_{x_{i}} (which is the case if X is discrete), then

∫_{−∞}^{∞} x P_{X}(dx) = ∑_{i} x_{i}p(x_{i}) = ∑_{i} x_{i} · P (X = x_{i});

and if P_{X}(dx) = f(x)dx, then

∫_{−∞}^{∞} x P_{X}(dx) = ∫_{−∞}^{∞} xf(x) dx.

Another notation which is commonly used is to write dF_{X}(x) for P_{X}(dx). Here F_{X}(x) = P (X ≤ x) = ∫_{−∞}^{x} P_{X}(ds). For this reason P_{X}(dx) is considered to be a ”generalized” derivative of F_{X}. (Note that if X has a pdf f(x), then F_{X}(x) = ∫_{−∞}^{x} f(s)ds, so dF_{X}(x)/dx = f(x), or dF_{X}(x) = f(x)dx in the usual sense if f is continuous.) Thus we have

E(X) = ∫_{S} X(s) P (ds) = ∫_{−∞}^{∞} x P_{X}(dx) = ∫_{−∞}^{∞} x dF_{X}(x).

Another fact of considerable importance and usefulness is that if h(X) is a funtion of X, then the expectation of h(X) can be computed using the distribution of X. Precisely,

E(h(X)) = ∫_{−∞}^{∞} h(x) P_{X}(dx) = ∫_{−∞}^{∞} h(x) dF_{X}(x).

For the special cases that X is discrete or continuous with pdf f, this amounts to the formulas

E(h(X)) = ∑_{i} h(x_{i})P (X = x_{i}) or E(h(X)) = ∫_{−∞}^{∞} h(x)f(x) dx.

This is often quite convenient, because computing the distribution of h(X) can be quite complicated. However, this not necessary if one just wants E(h(X)).

Further discussion and examples will be given in class.