Gauss-Markov Theorem

Given the assumptions of the classical linear regression model, the least-squares estimators, in the class of unbiased linear estimators, have minimum variance, that is, they are BLUE. In other words Gauss-Markov theorem holds the properties of Best Linear Unbiased Estimators.

Following are some of the assumptions which should be taken into consideration for the mathematical derivation of the Gauss-Markov Theorem :

Properties of \(k_i\);

1. \(\sum_{}^{}{k_i}=\; 0\)

2. \(\sum_{}^{}{k_ix_i}=\; \sum_{}^{}{k_iX_i}\; =\; 1\)

3. \(\sum_{}^{}{k_i^{2}}=\; \frac{1}{\sum_{}^{}{x_i^{2}}}\)

Assumptions of \(u_i\);

1. \(E\left( u_i \right)\; =\; 0\)

2. \(Var\left( u_i \right)\; = E\left( u_i^{2} \right)\; =\; \sigma _{u}^{2}\; =\;Constant\)

3. \(u_i\; \sim \; N\left( 0, \;\sigma _{u}^{2} \right)\)

4. \(E\left( u_i,u_j \right)\; =\; 0\)

5. \(E\left( u_i,\; X_i \right)\; =\; X_i E\left(u_i \right)\; =\; 0\)

Properties of BLUE (Best Linear Unbiased Estimators)

Lets take the regression line \(Y\; =\; \alpha \; +\; \beta X_{i}\; +\; u_{i}\)

1. Linearity i.e., \(\hat{\beta}\)

Lets take the regression line \(Y\; =\; \alpha \; +\; \beta X_i\; +\; u_{i}\)

As we know,

\(\hat{\beta} = \frac{\sum_{}^{}{x_{i}y_{i}}}{\sum_{}^{}{x_{i^{}}^{2}}}\),

\(=\frac{\sum_{}^{}{\left( X- \bar{X} \right)\left( Y-\bar{Y} \right)}}{\sum_{}^{}{x_{i}^{2}}}\),

\(= \frac{\sum_{}^{}{\left( X- \bar{X} \right)Y\; -\; \sum_{}^{}{\left( X-\bar{X} \right)\bar{Y}}}}{\sum_{}^{}{x_{i}^{2}}}\),

\(=\; \frac{\sum_{}^{}{x_i\; Yi}}{\sum_{}^{}{x_{i}^{2}}}\),

\(=\; \sum_{}^{}{k_{i}Y_{i}}\)         because     \([\frac{x_{i}}{\sum_{}^{}{x_{i}^{2}}}=\; k_{i}]\)

2. Unbiasedness i.e., \(\hat\beta\)  is unbiased,

From Linearity,

\(\hat{\beta}=\; \sum_{}^{}{k_{i}Y_{i}}\),

\(=\; \sum_{}^{}{k_{i}\; \left(\alpha \; +\; \beta X_{i}\; +\; u_{i} \right)}\),

\(=\; \alpha \sum_{}^{}{k_{i}}+\; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}}\),

\(=\; \; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}}\), because \([\sum_{}^{}{k_i}=\; 0]\),

Taking expectations both sides,

\(E\left( \hat{\beta} \right)\; =\; \beta \),         because        \([\sum_{}^{}{k_ix_i}=\; 1 \;,\; E\left( u_i \right)\; =\; 0 ]\),

3. Best i.e, \(\hat\beta\) is best or \(\hat\beta\) have minimum variance

As we know,

\(Var\left(\hat \beta \right)\; =\; E\left\{ \hat \beta \; -\beta \right\}^{2}\),

\(=\;E\; \left\{ \beta \; +\; \sum_{}^{}{k_{i}u_{i}\; +\; \beta } \right\}^{2}\),

\(=\;E\; \left\{ \sum_{}^{}{k_{i}u_{i}} \right\}^{2}\),

\(=\; E\left\{ k_{1}^{2}u_{1}^{2}+k_{2}^{2}u_{2}^{2}+k_{3}^{2}u_{3}^{2}……….. \right\} + 2E\left\{ k_{1}k_{2}u_{1}u_{2}+ k_{2}k_{3}u_{2}u_{3} +…….. \right\}\),

\(= E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\}\; +\; 2E\left\{ k_{i}k_{j}u_{i}u_{j} \right\}\),

\(=\; E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\}\),

\(=\; \sum_{}^{}{k_{i}^{2}\; E\left( u_{i}^{2} \right)}\),  beacause       \([E\left( k_{i}k_{j}u_{i}u_{j} \right)\; =\; 0]\),

\(= \sigma _{ui}^{2} \;\sum_{}^{}{k_{i}^{2}\; }\),

Now suppose there is another estimator,

\(var\left( \beta^\ast \right)\; =\; \sum_{}^{}{u_{i}^{2}}\sigma _{ui}^{2}\),

Where \(k_{i}\; \neq \; w_{i}\),

Let \(w_{i}\; =\; k_{i}\; +\; c_{i}\),

So, \(Var\; \left( \beta^\ast \right)\; =\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)^{2}\; \sigma _{ui}^{2}}\),

\(=\; \sigma _{ui}^{2}\; \sum_{}^{}{\left\{ k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right\}}\),

\(=\; \sigma _{ui}^{2}\; \sum_{}^{}{k_{i}^{2}}+\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}}\),  As \([\sum k_{i}c_{i} = 0]\),

\(=\; Var\left( \hat{\beta} \right)\; +\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}}\),

So, \(Var\; \left( \hat\beta \right)\; <\; Var\left( \beta^\ast \right)\),

Now for \(\alpha\) ;

1. Linearity i.e.; \(\alpha\) is linear,

As we know,

\(\hat\alpha \; =\; \bar{Y}\; -\; \hat\beta \bar{X}\),

\(=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\hat\beta \bar{X}\),

\(=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\; \sum_{}^{}{k_{i}Y_{i}\bar{X}}\),

\(=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i\),

So, \(\hat\alpha\) is linear.

2. Unbiasedness i.e.; \(\hat\alpha\)  is unbiased,

From linearity,

\(\hat\alpha=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i\),

\(=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\; \bar{X} \right\}\; \left( \alpha \; +\; \beta X_{i}\; +\; u_{i} \right)}\),

\(=\; \sum_{}^{}{\left\{ \frac{\alpha }{n}\; -\; \alpha k_{i}\bar{X}\; +\; \frac{\beta X_{i}}{n}\; -\; \beta X_{i}k_{i}\bar{X}\; +\; \frac{u_{i}}{n}\; -\; u_{i}k_{i}\bar{X} \right\}}\),

\(=\; \frac{n\alpha }{\alpha }\; -\; \alpha \bar{X}\sum_{}^{}{k_{i}\; +\; \beta \; \sum_{}^{}{\frac{X_{i}}{n}\; -\; \beta \bar{X}\sum_{}^{}{k_{i}X_{i}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}-\sum_{}^{}{k_{i}u_{i}\bar{X}}}}}\; \),

\(=\; \alpha \; +\; \beta \bar{X}\; -\; \beta \bar{X}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}\; -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}\),

Now taking expectations both sides,

\(E\left( \hat\alpha \right)\; =\; \alpha \; +\; E\left( ui \right)\; \frac{1}{n}\),

\(E\left( \hat\alpha \right)\; =\; \alpha \; \),

3. Best i.e.; \(\hat\alpha\) is best or \(\hat\alpha\) have minimum variance,

From unbiased,

\(\hat\alpha =\; \alpha \; +\;\; \frac{\sum{u_i}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}\),

We know,

\(var\left( \alpha \right)\; =\; E\left\{ \hat\alpha -\alpha \right\}^{2}\),

\(=\; E\left\{ \alpha +\frac{\sum_{}^{}{u_{i}}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}-\alpha \right\}^2\),

\(=\; E\left\{ \sum_{}^{}{\left( \frac{1}{n}\; -\; k_{i}\bar{X} \right)^{2}}u_{i}^{2} \right\}\),

\(=\; \sigma _{ui}^{2}\left\{ \sum_{}^{}{\left( \frac{1}{n^{2}}\; +\; k_{i}^{2}\bar{X}\; -2\; \frac{1}{n}ki\bar{X} \right)^{}} \right\}\),

\(=\; \sigma _{ui}^{2}\left\{ \frac{n}{n^{2}}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}}-\; \frac{2}{n}\bar{X}\sum_{}^{}{ki} \right\}\),

\(=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}} \right\}\),     As \([\sum_{}^{}{k_i}=\; 0 ]\),

Lets take another estimator \(\alpha^\ast\),

\(Var\left( \alpha^\ast \right)\; =\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{w_{i}^{2}} \right\}\),

Where \(k_{i}\; \neq \; w_{i}\) and let \(w_i=k_i\;+\;c_i\),

Now,

\(Var\left( \beta \right)\; =\; \sigma _{ui}^{2}\; \left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)} \right\}\),

\(=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right)} \right\}\),

\(=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{k_{i}^{2}\; +\; \bar{X}\; \sum_{}^{}{c_{i}^{2}}} \right\}\),           As \([\sum k_{i}c_{i} = 0]\),

\(=\;\sigma _{ui}^{2}\; \frac{1}{n}+\; \sigma _{ui}^{2}\bar{X}\sum_{}^{}{k_{i}^{2}\; +\; \sigma _{ui}^{2}\bar{X}\sum_{}^{}{c_{i}^{2}}}\),

\(=\;\sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{c_{i}^{2}} \right\}\; +\; \sigma _{ui}^{2}\; \bar{X}\sum_{}^{}{c_{i}^{2}}\),

So, \(var\left( \hat\alpha \right)\; <\; var \left( \alpha^\ast \right)\)

Leave a Reply