<div>
If you have the outputs of both systems on each instance, you may try bootstrap resampling, as done here: <a href="http://genomebiology.com/2008/9/S2/S2">http://genomebiology.com/2008/9/S2/S2</a>
</div>
<div><div><br></div><div><br></div><div>-- </div><div>Sérgio Matos</div><div><div>IEETA</div><div>Universidade de Aveiro</div></div><div><br></div></div>
<p style="color: #A0A0A8;">On Monday 10 November 2014 at 17:04, Jacob Eisenstein wrote:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div><div><blockquote type="cite"><div>That depends very much on what your task looks like. It might be easiest – and is often done in computational linguistics – to carry out a ten-fold cross-validation and apply a paired t-test to the quality measure of your choice (e.g. F-score). To be precise, sample A would be the F-scores achieved by Sys 1 across the ten folds, and sample B the F-scores achieved by Sys 2 on _exactly the same_ folds (and in _exactly the same order_).<br></div></blockquote><div><br></div><div>This seems overly conservative to me. Suppose there is a lot of variance across the folds, but system 1 does exactly 0.5% better than system 2 on every fold. It seems like what you want to do is a t-test on the difference in performance.</div><div><br></div><div>That said, there are definitely machine learning / stats papers that argue against computing variance across cross-validation folds. I can't find the exact reference I'm thinking of, but the related work section of Demsar (JMLR 2006) seems like a useful starting point.</div><div><a href="http://machinelearning.wustl.edu/mlpapers/paper_files/Demsar06.pdf">http://machinelearning.wustl.edu/mlpapers/paper_files/Demsar06.pdf</a><br></div><div><br></div><blockquote type="cite"><div>For a tagging task evaluated in terms of accuracy, you can apply McNemar's test to the output of the two systems. The samples correspond to all tokens in the test set, and the observed values are (i) whether Sys 1 is correct on this token and (ii) whether Sys 2 is correct on this token.<br></div></blockquote><div><br></div><div>One could also apply a sign test in this case, which I personally find easier to understand. The trouble is that you may not have access to Sys 2's outputs on each instance (suppose you only know its reported accuracy); in this case, you can't apply the sign test or McNemar's test.</div><div><br></div><div>-Jacob</div><div><br></div><div> </div><blockquote type="cite"><div>
<br>
Hope this helps,<br>
Stefan<br>
<div><div><br>
<br>
<br>
_______________________________________________<br>
UNSUBSCRIBE from this page: <a href="http://mailman.uib.no/options/corpora" target="_blank">http://mailman.uib.no/options/corpora</a><br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
</div></div></div></blockquote></div><br></div></div>
</div><div><div>_______________________________________________</div><div>UNSUBSCRIBE from this page: <a href="http://mailman.uib.no/options/corpora">http://mailman.uib.no/options/corpora</a></div><div>Corpora mailing list</div><div><a href="mailto:Corpora@uib.no">Corpora@uib.no</a></div><div><a href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>