[Corpora-List] Problems in using ACE evaluation scripts
Heng Ji
hengji at cs.nyu.edu
Wed Jun 23 15:57:11 UTC 2010
Hi Dingcheng,
You are using ACE05 scorer for ACE02 data. The xml formats are different
in the corpora from these two years.
The error msg indicates the entity type is missing:
<entity ID="ABC19980106.1830.0029-E1">
because ACE05 scorer is expecting something like:
<entity ID="CNN_CF_20030303.1900.00-E1" TYPE="PER" SUBTYPE="Individual"
CLASS="SPC">
I'd suggest you to make the scorer and data consistent (from the same
year).
Hope this helps,
Heng
On Wed, 23 Jun 2010, lixxx345 at umn.edu wrote:
> Dear All,
>
> I am using ACE corpus for relational discovery. I read ACE documents and
> tried to run a sample evaluation. But I get the following errors. I don't
> know what wrong here. Would anyone who is familiar with ACE evaluations give
> me helps?
>
> I tried two scripts and used the same ACE standard xml files as inputs, but
> got the same error message.
>
> The command line I used:
> ace05-eval-v14a.pl -R
> ace_phase2\data\ace2_devtest\bnews\ABC19981001.1830.1257.sgm.tmx.rdc.xml -T
> \ace_phase2\data\ace2_devtest\bnews\ABC19981001.1830.1257.sgm.tmx.rdc.xml
>
> The output:
> Entity scoring is LEVEL weighted
> Relation/Event mapping is allowed when ARGUMENTS criterion is satisfied
> Overlap of BOTH of Arg-* relation arguments is required for mapping
> Arguments contribute to scoring when arguments are OVERLAPPING
> Argument averaging is ARITHMETIC
> Scoring mode parameter set is "2005-TERN"
>
> min acceptable overlap of matching mention heads or names:
> 30.0 percent
> min acceptable run of matching characters in mention heads:
> 0.0 percent
> max acceptable extent difference for names and mentions to match:
> 4 chars for text sources
> 0.400 sec for audio sources
> 0.400 cm for image sources
>
> Entity mention values:
> 1.000 for type NAM
> 0.500 for type NOM
> 0.100 for type PRO
> Entity value weights for entity types:
> 1.000 for type FAC
> 1.000 for type GPE
> 1.000 for type LOC
> 1.000 for type ORG
> 1.000 for type PER
> 1.000 for type VEH
> 1.000 for type WEA
> Entity value weights for entity classes:
> 1.000 for class SPC
> 0.000 for class GEN
> 0.000 for class NEG
> 0.000 for class USP
> Entity value discounts for entity attribute recognition errors:
> 0.750 for CLASS errors
> 0.900 for SUBTYPE errors
> 0.500 for TYPE errors
> Entity mention value discounts for mention attribute recognition errors:
> 0.900 for ROLE errors
> 0.900 for STYLE errors
> 0.900 for TYPE errors
> Entity value (cost) weight for spurious (false alarm) entities: 0.750
> Entity mention value (cost) weight for spurious entity mentions: 0.750
> Entity mention value (cost) discount for incorrect coreference: 0.000
>
> Relation value weights for relation types:
> 1.000 for type ART
> 1.000 for type GEN-AFF
> 1.000 for type METONYMY
> 1.000 for type ORG-AFF
> 1.000 for type PART-WHOLE
> 1.000 for type PER-SOC
> 1.000 for type PHYS
> Relation value weights for relation modalities:
> 1.000 for modality Asserted
> 1.000 for modality Other
> 1.000 for modality Unspecified
> Relation value discounts for relation attribute recognition errors:
> 0.750 for MODALITY errors
> 0.700 for SUBTYPE errors
> 1.000 for TENSE errors
> 1.000 for TYPE errors
> Relation value (cost) weight for spurious (false alarm) relations: 0.750
> Relation value (cost) weight for spurious relation arguments: 0.000
> Relation argument value (cost) discount for argument role errors: 0.750
>
> Event value weights for event types:
> 1.000 for type Business
> 1.000 for type Conflict
> 1.000 for type Contact
> 1.000 for type Justice
> 1.000 for type Life
> 1.000 for type Movement
> 1.000 for type Personnel
> 1.000 for type Transaction
> Event value weights for event modalities:
> 1.000 for modality Asserted
> 1.000 for modality Other
> Event value discounts for event attribute recognition errors:
> 1.000 for GENERICITY errors
> 0.750 for MODALITY errors
> 1.000 for POLARITY errors
> 0.900 for SUBTYPE errors
> 1.000 for TENSE errors
> 0.500 for TYPE errors
> Event value (cost) weight for spurious (false alarm) events: 0.750
> Event value (cost) weight for spurious event arguments: 0.500
> Event argument value (cost) discount for argument role errors: 0.750
>
> Timex2 attribute value weights for timex2 attributes:
> 0.000 for type ANCHOR_DIR
> 0.000 for type ANCHOR_VAL
> 0.000 for type MOD
> 0.000 for type SET
> 0.500 for type TYPE
> 0.000 for type VAL
> Timex2 value (cost) weight for spurious (false alarm) timex2's: 0.750
> Timex2 mention value (cost) weight for spurious timex2 mentions: 0.750
> Timex2 mention value (cost) discount for incorrect coreference: 0.000
>
> Value value weights for value types:
> 1.000 for type Contact-Info
> 1.000 for type Crime
> 1.000 for type Illness
> 1.000 for type Job-Title
> 1.000 for type Numeric
> 1.000 for type Sentence
> Value value discounts for value attribute recognition errors:
> 0.900 for SUBTYPE errors
> 0.500 for TYPE errors
> Value value (cost) weight for spurious (false alarm) values: 0.750
> Value mention value (cost) weight for spurious value mentions: 0.750
> Value mention value (cost) discount for incorrect coreference: 0.000
>
> -------------------------------------------------------------------------------
>
> FATAL INPUT ERROR for entity 'ABC19981001.1830.1257-E1'
> in document 'ABC19981001.1830.1257'
> in file 'C:\Documents and
> Settings\lidi00000\SICCD_RD\ace_phase2\data\ace2_devtest\bnews\ABC19981001.1830.1257.sgm.tmx.rdc.xml'
> Missing entity TYPE attribute
>
> --------------------------------------------------------------------------------
>
> Above is the error message.
>
>
> Thanks,
>
> Dingcheng Li
>
>
> _______________________________________________
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora
More information about the Corpora
mailing list