I'm having problems to understand the definition of the level of significance α. I thought I knew wh

Cierra Castillo

Cierra Castillo

Answered question

2022-07-05

I'm having problems to understand the definition of the level of significance α. I thought I knew what α is but I realized I don't.When I stated to study statistics by myself I read this introductory book and everything was fine, the definition is very clear. He says on page 290:
You’re probably wondering, how small does a p-value have to be for us to achieve statistical significance? If we agree that a p-value of 0.0001 is clearly statistically significant and a p-value of 0.50 is not, there must be some point between 0.0001 and 0.50 where we cross the threshold between statistical significance and random chance. That point, measuring when something becomes rare enough to be called “unusual,” might vary a lot from person to person. We should agree in advance on a reasonable cutoff point.
Statisticians call this cutoff point the significance level of a test and usually denote it with the Greek letter α (alpha).
For example, if α=0.05 we say we are doing a 5% test and will call the results statistically significant if the p-value for the sample is smaller than 0.05. Often, short hand notation such as P<0.05 is used to indicate that the p-value is less than 0.05, which means the results are significant at a 5% level.
Now, I'm studying about statistical inference, a more advanced subject, and I realized there are some concepts that don't exactly have the same definition as I studied before. The level of significance is an example.
I'm reading this book and on page 352 he introduces the Neyman-Pearson lemma as a method to find the UMP test.
Example:
On the basis of a random sample of size 1 from the p.d.f. f ( x ; θ ) = θ x θ 1 ,   0 < x < 1   ( θ > 1 )
For θ 1 > θ 0 , the cutoff point is calculated by:
... C = ( 1 α ) 1 θ 0
For θ 1 < θ 0 , we have:
... C = α 1 θ 0
So in this second book, the cutoff point is not necessarily α, I'm confused.
MY ATTEMPT TO UNDERSTAND WITH THE HELP OF THE ANSWERS
The alpha is predetermined, but it doesn't mean I can't have a smaller rejection region. Then I end up having a smaller rejection region using NP lemma with the same level of significance alpha. Some introductory books let the cutoff point to be α by standard (why?), that's the reason of my confusion, I can shrink the rejection region keeping the value of α. Can someone say if I'm right?

Answer & Explanation

Brendan Bush

Brendan Bush

Beginner2022-07-06Added 14 answers

You are misunderstanding what "cutoff" means in the context of the Neyman-Pearson lemma. It is referring to a cutoff of the likelihood ratio in the test, not as a cutoff of what p-values are small enough to be significant.
A p-value (of some results) represents the probability under the null hypothesis of getting results at least as extreme/unusual as your current results. The idea is that seeing results that are unusual under the null hypothesis should be evidence to reject the null hypothesis. To decide how small a p-value is small enough to be significant, you must set a significance level, say α = 0.05; if your p-value is smaller than α, this indicates that your data is unusual under the null, so you reject the null.
Comparing p-values against α guarantees that your Type I error is α: if the null is true, the probability that your test is wrong is α. More generally, this is how you should be thinking about significance levels: as Type I error (probability that a test rejects the null if null is true). This is important for likelihood ratio tests, where there aren't p-values being computed anywhere.
"Cutoff" in Neyman-Pearson
In this context, you are forming a test based on a statistic called the likelihood ratio Λ ( x ) = L 1 ( θ x ) L 0 ( θ x ) . (I am following your textbook's convention of putting the null in the denominator.)
The test is of the form
reject Λ ( x ) < C
do not reject Λ ( x ) < C
for some cutoff C. (I'm ignoring the case where the likelihood ratio equals C for simplicity.) The intuition is that if the data x seem to support the alternative hypothesis, the numerator of the likelihood ratio would be large so we would lean toward rejecting; likewise if the data x seem to support the null, the denominator would be large and we would lean toward not rejecting. This is C is what the "cutoff" is in the Neyman-Pearson example. It is not a cutoff for what p-values are small enough to be significant.
How do we choose the cutoff? If C is large, then you reject less often (leading to small Type I error); if C is small you reject more often (leading to large Type I error). If you have set a significance level, then you must set C large enough to avoid a large Type I error. The example you are reading is solving for the smallest C that keeps the Type I error below α.

Do you have a similar question?

Recalculate according to your conditions!

New Questions in College Statistics

Ask your question.
Get an expert answer.

Let our experts help you. Answer in as fast as 15 minutes.

Didn't find what you were looking for?