[Corpora] [Corpora-List] Calculating statistical significant

[Grzegorz Chrupała]

It seems that the only reason to use McNemar's test these days is
tradition. The exact binomial test can be computed in a fraction of a
second with current hardware and software.
--

Grzegorz Chrupała
Communication and Information Sciences
Tilburg University
PO Box 90153
5000 LE Tilburg
The Netherlands

Web: grzegorz.chrupala.me
Phone: +31 13 466 3106
Email: g.chrupala at uvt.nl


On Mon, Nov 10, 2014 at 7:06 PM, Stefan Evert <stefanML at collocations.de> wrote:
>> If you have the outputs of both systems on each instance, you may try bootstrap resampling, as done here: http://genomebiology.com/2008/9/S2/S2
>
> Indeed, if you have the full system outputs and if you believe that your test data form a random sample from the population of interest, you can apply bootstrap resampling in order to obtain confidence intervals for non-trivial evaluation criteria such as P, R and F-score.
>
> If you just want to know whether there is a significant difference between the two systems, you can simply apply McNemar's test.  The bootstrap resampling – if implemented correctly – will give you the same answer at much greater computational cost.
>
> If you're satisfied with accuracy as an evaluation criterion, you can also compute (binomial) confidence intervals for the two systems directly without bootstrapping.  A confidence interval for the difference in accuracy can be derived from McNemar's test – I've implemented something along those lines for my PhD thesis long, long ago.
>
> Best,
> Stefan
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora