[Corpora-List] Help in Applying Appropriate Statistical Test and Its Interpretation

[Rob Malouf]

The problem isn't the interpretation of p-values.  It's that corpus data often wildly violates the assumptions underlying the chi-square statistic, making the p-values meaningless. This is especially true when you're comparing really big counts with really small counts, which is I think what Adam's rule of thumb is meant to address.  Once you've decided that applying the chi-square test even makes sense, then questions like significance levels and Bonferroni corrections come into play.

In this particular case, even if the chi-square test is appropriate, it's probably not going to be very informative.  Something like a GLM is going to be a better bet for telling you what the patterns are (see Baayen's excellent recent textbook for background or the Bresnan et al. papers for an example of an application to similar data).

--
Rob Malouf <rmalouf at mail.sdsu.edu>
Department of Linguistics and Asian / Middle Eastern Languages
San Diego State University

On Jun 29, 2010, at 9:34 AM, Stefan Th. Gries wrote:

>> the null hypothesis-testing you discuss here doesn't work in corpus linguistics - for the argument see Language is never ever ever random.<http://kilgarriff.co.uk/Publications/2005-K-lineer.pdf> 2005 *Corpus Linguistics and Linguistic Theory* 1 (2): 263-276.
> and cf the following article in that journal for a rebuttal (which
> shows that once p-values are corrected for multiple testing as they
> should be anyway they sometimes do *exactly* what they're supposed to
> do).
> 
>> My rule of thumb is: it only counts if the ratio (of normalised frequencies) is greater than/less than a factor of two between two text types
> and how is that rule of thumb better than this other little rule of
> thumb that's been around in the sciences, the one that I think was
> something like "p<0.05"? ;-))) And what is the basis for the proposed
> rule of thumb?
> 
> (NB: I am not saying effect sizes are unimportant, just that bashing
> p-values for a shortcoming they exhibit when they are applied
> incorrectly is not exactly useful/wise ...)
> 
> STG
> --
> Stefan Th. Gries
> -----------------------------------------------
> University of California, Santa Barbara
> http://www.linguistics.ucsb.edu/faculty/stgries
> -----------------------------------------------
> 
> _______________________________________________
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
> 


_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora