pos-errors.cut file

[Brian MacWhinney]

Dear Anna,
      Typically pos-errors.cut is empty.  It only shows errors if  
there is some major mismatch of categories between MOR and POST.  For  
example, if you train POST on one set of categories and then redo MOR  
without retraining using POSTTRAIN, then you could get errors.  But  
usually it is empty.  The file that is usually more full of errors is  
the errors file produced during POSTTRAIN.  Again, these errors can  
arise from changes to the MOR grammar over time without doing  
corresponding fixes to the %trn line in the training data files.
    One way of evaluating POST is to have a file with gold-standard  
codes on the %trn line and then run MOR and POST and then to use the  
TRNFIX program to detect disagreements between the %mor and %trn.
However, if both %trn and %mor are wrong in the same way, these  
errors will not get caught.  Also, sometimes %trn is wrong, although  
in theory it would never be wrong.   You can run FREQ on the files  
output from TRNFIX and count the deviations and then divide that by  
the number of words in the file to compute something close to  
accuracy, with the above exceptions being noted.

--Brian MacWhinney

On Oct 9, 2006, at 7:28 AM, Anna Gudmundson wrote:

>
> Dear info chibolts,
>
> I just wonder what kind of information is supposed to show up in  
> the pos-errors.cut file, i.e.
> the file that results from running the POST command. I ask the  
> question because every time I
> run POST, nothing is written to that file, i.e. it always turns out  
> to be empty even though I
> know there should be several errors. What kind of errors should be  
> reported in that file, and
> if there are no errors, does the program still output something  
> like “no errors found”?
>
> I also have a second question that concerns the evaluation of the  
> POST program. Is there a
> way to calculate some kind of accuracy rate, like for example  
> precision and recall. I have a
> vague memory of having seen a percentage value in some output file  
> (could it have been in
> the pos-error file maybe?)
>
> I would be so grateful if someone could help me.
>
> Thanks in advance
> Anna Gudmundson
>
>
>
>