## The Method Of Ordinary Least Squares

The method of ordinary least squares is attributed to Carl Friedrich Gauss, a German mathematician. Under certain assumptions (discussed in Section 3.2), the method of least squares has some very attractive statistical properties that have made it one of the most powerful and popular methods of regression analysis. To understand this method, we first explain the least-squares principle.

Recall the two-variable PRF:

However, as we noted in Chapter 2, the PRF is not directly observable. We

CHAPTER THREE: TWO-VARIABLE REGRESSION MODEL 59

estimate it from the SRF:

where Yi is the estimated (conditional mean) value of Yi.

But how is the SRF itself determined? To see this, let us proceed as follows. First, express (2.6.3) as

which shows that the Ui (the residuals) are simply the differences between the actual and estimated Y values.

Now given n pairs of observations on Y and X, we would like to determine the SRF in such a manner that it is as close as possible to the actual Y. To this end, we may adopt the following criterion: Choose the SRF in such a way that the sum of the residuals J2 Ui = J](Yi — Yi) is as small as possible. Although intuitively appealing, this is not a very good criterion, as can be seen in the hypothetical scattergram shown in Figure 3.1.

If we adopt the criterion of minimizing YI Ui, Figure 3.1 shows that the residuals U2 and U3 as well as the residuals U1 and U4 receive the same weight in the sum (U1 + U2 + U3 + U4), although the first two residuals are much closer to the SRF than the latter two. In other words, all the residuals receive