[Corpora-List] Significance tests for n-fold cross validation

[Chris Fournier]

Having identified your desired metric for comparison, and having assumedly
calculated it as a micro or macro average, it would be beneficial to perform
an analysis ensure that the increase (or decrease) observed in the
comparison of your overall results is a result of a non-random effect.
 Which method to use, and how to apply it, could be determined by reading
the paper titled *Statistical comparisons of classifiers over multiple data
sets.*

Demšar J. Statistical comparisons of classifiers over multiple data sets.
The Journal of Machine Learning Research. 2006;7:1-30. Available at:
http://portal.acm.org/citation.cfm?id=1248547.1248548


On Sun, May 8, 2011 at 3:28 PM, Emad Mohamed <emohamed at umail.iu.edu> wrote:

> Dear Corpora Listers,
>
> I have run a number of POS tagging experiments in a 5 fold cross validation
> setting. Imagine the results of the 5 folds (in terms of accuracy) are these
> fictitious numbers:
>
> 93.4%    93.7%     92.9%     93%    93.1%
>
> I then added feature x to the feature set, which gave the following
> results:
>
> 93.7%  94%       92.7%       93.2%   93.4%
>
> My question is: Do I need to use any significance test to show  that adding
> feature x really helps? if no, why not? and if yes, what are the appropriate
> tests in such a case?
>
> Thank you in anticipation
>
>
> --
> Emad Soliman Ali Mohamed
> aka Emad Nawfal (*عماد نوفل*)
> Suez Canal University, Egypt
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20110508/78ef77b7/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora