# Gauss-Markov Theorem

Given the assumptions of the classical linear regression model, the least-squares estimators, in the class of unbiased linear estimators, have minimum variance, that is, they are BLUE. In other words Gauss-Markov theorem holds the properties of Best Linear Unbiased Estimators.

Following are some of the assumptions which should be taken into consideration for the mathematical derivation of the Gauss-Markov Theorem :

Properties of $$k_i$$;

1. $$\sum_{}^{}{k_i}=\; 0$$

2. $$\sum_{}^{}{k_ix_i}=\; \sum_{}^{}{k_iX_i}\; =\; 1$$

3. $$\sum_{}^{}{k_i^{2}}=\; \frac{1}{\sum_{}^{}{x_i^{2}}}$$

Assumptions of $$u_i$$;

1. $$E\left( u_i \right)\; =\; 0$$

2. $$Var\left( u_i \right)\; = E\left( u_i^{2} \right)\; =\; \sigma _{u}^{2}\; =\;Constant$$

3. $$u_i\; \sim \; N\left( 0, \;\sigma _{u}^{2} \right)$$

4. $$E\left( u_i,u_j \right)\; =\; 0$$

5. $$E\left( u_i,\; X_i \right)\; =\; X_i E\left(u_i \right)\; =\; 0$$

### Properties of BLUE (Best Linear Unbiased Estimators)

Lets take the regression line $$Y\; =\; \alpha \; +\; \beta X_{i}\; +\; u_{i}$$

1. Linearity i.e., $$\hat{\beta}$$

Lets take the regression line $$Y\; =\; \alpha \; +\; \beta X_i\; +\; u_{i}$$

As we know,

$$\hat{\beta} = \frac{\sum_{}^{}{x_{i}y_{i}}}{\sum_{}^{}{x_{i^{}}^{2}}}$$,

$$=\frac{\sum_{}^{}{\left( X- \bar{X} \right)\left( Y-\bar{Y} \right)}}{\sum_{}^{}{x_{i}^{2}}}$$,

$$= \frac{\sum_{}^{}{\left( X- \bar{X} \right)Y\; -\; \sum_{}^{}{\left( X-\bar{X} \right)\bar{Y}}}}{\sum_{}^{}{x_{i}^{2}}}$$,

$$=\; \frac{\sum_{}^{}{x_i\; Yi}}{\sum_{}^{}{x_{i}^{2}}}$$,

$$=\; \sum_{}^{}{k_{i}Y_{i}}$$         because     $$[\frac{x_{i}}{\sum_{}^{}{x_{i}^{2}}}=\; k_{i}]$$

2. Unbiasedness i.e., $$\hat\beta$$  is unbiased,

From Linearity,

$$\hat{\beta}=\; \sum_{}^{}{k_{i}Y_{i}}$$,

$$=\; \sum_{}^{}{k_{i}\; \left(\alpha \; +\; \beta X_{i}\; +\; u_{i} \right)}$$,

$$=\; \alpha \sum_{}^{}{k_{i}}+\; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}}$$,

$$=\; \; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}}$$, because $$[\sum_{}^{}{k_i}=\; 0]$$,

Taking expectations both sides,

$$E\left( \hat{\beta} \right)\; =\; \beta$$,         because        $$[\sum_{}^{}{k_ix_i}=\; 1 \;,\; E\left( u_i \right)\; =\; 0 ]$$,

3. Best i.e, $$\hat\beta$$ is best or $$\hat\beta$$ have minimum variance

As we know,

$$Var\left(\hat \beta \right)\; =\; E\left\{ \hat \beta \; -\beta \right\}^{2}$$,

$$=\;E\; \left\{ \beta \; +\; \sum_{}^{}{k_{i}u_{i}\; +\; \beta } \right\}^{2}$$,

$$=\;E\; \left\{ \sum_{}^{}{k_{i}u_{i}} \right\}^{2}$$,

$$=\; E\left\{ k_{1}^{2}u_{1}^{2}+k_{2}^{2}u_{2}^{2}+k_{3}^{2}u_{3}^{2}……….. \right\} + 2E\left\{ k_{1}k_{2}u_{1}u_{2}+ k_{2}k_{3}u_{2}u_{3} +…….. \right\}$$,

$$= E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\}\; +\; 2E\left\{ k_{i}k_{j}u_{i}u_{j} \right\}$$,

$$=\; E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\}$$,

$$=\; \sum_{}^{}{k_{i}^{2}\; E\left( u_{i}^{2} \right)}$$,  beacause       $$[E\left( k_{i}k_{j}u_{i}u_{j} \right)\; =\; 0]$$,

$$= \sigma _{ui}^{2} \;\sum_{}^{}{k_{i}^{2}\; }$$,

Now suppose there is another estimator,

$$var\left( \beta^\ast \right)\; =\; \sum_{}^{}{u_{i}^{2}}\sigma _{ui}^{2}$$,

Where $$k_{i}\; \neq \; w_{i}$$,

Let $$w_{i}\; =\; k_{i}\; +\; c_{i}$$,

So, $$Var\; \left( \beta^\ast \right)\; =\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)^{2}\; \sigma _{ui}^{2}}$$,

$$=\; \sigma _{ui}^{2}\; \sum_{}^{}{\left\{ k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right\}}$$,

$$=\; \sigma _{ui}^{2}\; \sum_{}^{}{k_{i}^{2}}+\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}}$$,  As $$[\sum k_{i}c_{i} = 0]$$,

$$=\; Var\left( \hat{\beta} \right)\; +\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}}$$,

So, $$Var\; \left( \hat\beta \right)\; <\; Var\left( \beta^\ast \right)$$,

Now for $$\alpha$$ ;

1. Linearity i.e.; $$\alpha$$ is linear,

As we know,

$$\hat\alpha \; =\; \bar{Y}\; -\; \hat\beta \bar{X}$$,

$$=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\hat\beta \bar{X}$$,

$$=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\; \sum_{}^{}{k_{i}Y_{i}\bar{X}}$$,

$$=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i$$,

So, $$\hat\alpha$$ is linear.

2. Unbiasedness i.e.; $$\hat\alpha$$  is unbiased,

From linearity,

$$\hat\alpha=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i$$,

$$=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\; \bar{X} \right\}\; \left( \alpha \; +\; \beta X_{i}\; +\; u_{i} \right)}$$,

$$=\; \sum_{}^{}{\left\{ \frac{\alpha }{n}\; -\; \alpha k_{i}\bar{X}\; +\; \frac{\beta X_{i}}{n}\; -\; \beta X_{i}k_{i}\bar{X}\; +\; \frac{u_{i}}{n}\; -\; u_{i}k_{i}\bar{X} \right\}}$$,

$$=\; \frac{n\alpha }{\alpha }\; -\; \alpha \bar{X}\sum_{}^{}{k_{i}\; +\; \beta \; \sum_{}^{}{\frac{X_{i}}{n}\; -\; \beta \bar{X}\sum_{}^{}{k_{i}X_{i}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}-\sum_{}^{}{k_{i}u_{i}\bar{X}}}}}\;$$,

$$=\; \alpha \; +\; \beta \bar{X}\; -\; \beta \bar{X}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}\; -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}$$,

Now taking expectations both sides,

$$E\left( \hat\alpha \right)\; =\; \alpha \; +\; E\left( ui \right)\; \frac{1}{n}$$,

$$E\left( \hat\alpha \right)\; =\; \alpha \;$$,

3. Best i.e.; $$\hat\alpha$$ is best or $$\hat\alpha$$ have minimum variance,

From unbiased,

$$\hat\alpha =\; \alpha \; +\;\; \frac{\sum{u_i}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}$$,

We know,

$$var\left( \alpha \right)\; =\; E\left\{ \hat\alpha -\alpha \right\}^{2}$$,

$$=\; E\left\{ \alpha +\frac{\sum_{}^{}{u_{i}}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}-\alpha \right\}^2$$,

$$=\; E\left\{ \sum_{}^{}{\left( \frac{1}{n}\; -\; k_{i}\bar{X} \right)^{2}}u_{i}^{2} \right\}$$,

$$=\; \sigma _{ui}^{2}\left\{ \sum_{}^{}{\left( \frac{1}{n^{2}}\; +\; k_{i}^{2}\bar{X}\; -2\; \frac{1}{n}ki\bar{X} \right)^{}} \right\}$$,

$$=\; \sigma _{ui}^{2}\left\{ \frac{n}{n^{2}}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}}-\; \frac{2}{n}\bar{X}\sum_{}^{}{ki} \right\}$$,

$$=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}} \right\}$$,     As $$[\sum_{}^{}{k_i}=\; 0 ]$$,

Lets take another estimator $$\alpha^\ast$$,

$$Var\left( \alpha^\ast \right)\; =\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{w_{i}^{2}} \right\}$$,

Where $$k_{i}\; \neq \; w_{i}$$ and let $$w_i=k_i\;+\;c_i$$,

Now,

$$Var\left( \beta \right)\; =\; \sigma _{ui}^{2}\; \left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)} \right\}$$,

$$=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right)} \right\}$$,

$$=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{k_{i}^{2}\; +\; \bar{X}\; \sum_{}^{}{c_{i}^{2}}} \right\}$$,           As $$[\sum k_{i}c_{i} = 0]$$,

$$=\;\sigma _{ui}^{2}\; \frac{1}{n}+\; \sigma _{ui}^{2}\bar{X}\sum_{}^{}{k_{i}^{2}\; +\; \sigma _{ui}^{2}\bar{X}\sum_{}^{}{c_{i}^{2}}}$$,

$$=\;\sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{c_{i}^{2}} \right\}\; +\; \sigma _{ui}^{2}\; \bar{X}\sum_{}^{}{c_{i}^{2}}$$,

So, $$var\left( \hat\alpha \right)\; <\; var \left( \alpha^\ast \right)$$