Corpora: parser recommendation

[Miles Osborne]

yes, i agree: when making claims that approach X is better than approach
Y, people really ought to also consider training/testing on material other
than just wsj / bnc etc.  the key ideas here are bias and variance: if an
approach (eg neural nets, EM etc) has a high bias and variance (will give
different results if either parameters are varied or the training set
varies) then any results reported using a single distribution won't
necessarily hold in some other scenario.  or, just because your parser is
good at wsj doesn't necessarily mean that it will be good at susanne.

here's an example paper that, at least in my opinion, takes empirical
evaluation seriously:

http://robotics.stanford.edu/~ronnyk/vote.ps.gz

Bauer, Eric, Kohavi Ron, An Empirical Comparison of Voting Classification
Algorithms: Bagging, Boosting, and Variants. To appear in the journal
Machine Learning Vol 36, Nos. 1/2, July/August 1999, pages 105-139
compressed postscript (632K) updated 5/22 /99 or acrobat (PDF).

also check the "free lunch theorem" -can't find a link off-hand.

Miles Osborne