[Corpora-List] Precision and Recall

[Seth Grimes]

One thing to note and one thing to think about:

Note that information-retrieval accuracy is typically measured via the 
harmonic mean of weighted precision and recall scores where the weights 
are chosen to match the problem at hand.  That is, some problems can 
tolerate a higher proportion of false positives in order to catch every 
true positive you can, while others can afford to miss true positives 
in order to minimize the number of false negatives.  The hunt for a 
terrorist might be in the first category and application of the death 
penalty might be in the latter.

And an observation: What about all the information that's not part of your 
source space?  That is, you can have high recall and precision but still 
not "get it."  For instance, if you're looking at customer satisfaction, 
you're looking at an incomplete picture if you don't consider, say, 
postings on forums or review sites.

 					Seth


On Sat, 19 Apr 2008, Angus Grieve-Smith wrote:

> On Sat, 19 Apr 2008, Daniel Zeman wrote:
>
>> the false positives/negatives are absolute numbers. If you evaluate, say,
>> performance of a parser on two different data sets and you get fp=100 and
>> fn=100 for both, you still cannot say that both sets are equally hard for the
>> parser. It may well be that the sets were not the same size and that tp1=100
>> while tp2=1000.
>
> 	Okay, I see that you would want to know how many false negatives
> there are as a proportion - i.e. how many of the positives it found
> correctly - so I see the value in "recall," even if it doesn't make much
> sense as a name.  But it seems to me that the raw number of false
> negatives is also valuable.
>
> 	But false positives are false positives; why does it matter how
> many true positives there were?  Because it's a measure of how muddy the
> water is?  It seems like here, absolute numbers of false positives would
> be more valuable in many situations.  As Google found, it often doesn't
> matter how many false positives you have, as long as the most valuable
> true positives are close to the top of the list.
>
> 	Incidentally, this is not a purely academic line of questioning; I
> worked on an information retrieval project that failed in part because
> precision and recall did not accurately predict customer satisfaction.
>
> 					-Angus B. Grieve-Smith
> 					grvsmth at panix.com
>
> _______________________________________________
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>

--
Seth Grimes   Alta Plana Corp, analytical computing & data management
               Intelligent Enterprise magazine (CMP), Contributing Editor
grimes at altaplana.com       http://altaplana.com    301-270-0795

_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora