What is Var[b] in multiple regression?

trkalo84

trkalo84

Answered question

2022-09-27

What is V a r [ b ] in multiple regression?
Assume a linear regression model y = X β + ϵ with ϵ N ( 0 , σ 2 I ) and y ^ = X b where b = ( X X ) 1 X y. Besides H = X ( X X ) 1 X is the linear projection from the response space to the span of X, i.e., y ^ = H y
Now I want to calculate V a r [ b ] but what I get is an k × k matrix, not an n × n one. Here's my calculation:
V a r [ b ] = V a r [ ( X X ) 1 X y ] = ( X X ) 1 X V a r [ y ] = σ 2 I X ( X X ) 1 Here you can  see already this thing will be k  ×  k = σ 2 ( X X ) 1 X X I ( X X ) 1 = σ 2 ( X X ) 1 R k × k
What am I doing wrong?
Besides, are E [ b ] = β, E [ y ^ ] = H X β, V a r [ y ^ ] = σ 2 H, E [ y y ^ ] = ( I H ) X β, V a r [ y y ^ ] = ( I H ) σ 2 correct (this is just on a side note, my main question is the one above)?

Answer & Explanation

Jade Mejia

Jade Mejia

Beginner2022-09-28Added 8 answers

The covariance matrix for b (the estimator for β) should be k × k. If the X matrix is n × k then β has to be k × 1; otherwise the product X β wouldn't be n × 1
So if β is a constant vector of k parameters, then its estimator b is a random vector with k elements. Therefore the covariance matrix for b consists of covariances for all possible combinations of two members selected from the random vector, hence it must be a k × k matrix.
To answer your side notes, all your calculations are correct but some can be simplified further. Check that H X = X, so that E [ y ^ ] = X β, and E [ y y ^ ] = 0

Do you have a similar question?

Recalculate according to your conditions!

New Questions in Inferential Statistics

Ask your question.
Get an expert answer.

Let our experts help you. Answer in as fast as 15 minutes.

Didn't find what you were looking for?