Ergo's 1st ANNUAL PARSING CONTEST

[Philip A. Bralich, Ph.D.]

Ergo Linguistic Technologies would like to announce its first
annual parsing contest based on a fixed set of sentences and a
fixed set of tasks to be performed on that set of sentences.
The area of NLP to be explored is that of increased syntactic
analysis to provide: 1) improvements in navigation and control
technology through more complex grammar, 2) improvements in the
implementation of question/answer, statement/response dialogs
with computers and computer characters, and 3) improvements in
web and database searching using natural anguage queries.

The contest will be based on a comparison of results for parses of
a fixed set of sentences (included at end of this message) and
various tasks that can be performed as a result of those parses.
That is, the comparison will be based on the actual parse tree and
the ability to use that parsed output to generate theory independent
parse trees and output and to perform various NLP tasks.  The
judging will be based on the standards for evaluating NLP that have
been proposed previously on this list by myself and Derek Bickerton
and which are currently being developed into an ISO standard for the
Virtual Reality Modeling Language (VRML) as part of the VRML
Consortium's development efforts (http://www.vrml.org/WorkingGroups/
NLP-ANIM).  The standards proposed are theory and field independent
standards which allow both linguists and non-linguists to evaluate NLP
systems in the areas of navigation and control, question/answer
dialogues, and database and web searching.  I will also be at the
annual meeting of the Linguistic Society of America this week in Los
Angeles for those who would like to discuss this in more detail.

The sentences chosen for this contest are rather simple, but as we find
more and more parsers that can accomplish the tasks on this list, we
will add more complex sentences and tasks to the list.  Please, be aware
that systems that may be designed for large corpora of unrestricted text
actually cannot work in this domain.  Thus, while such systems may be
useful for certain searching tasks, they are not useful in the domain
explored in this contest — and this is evidenced by their inability to
perform on tests such as the one provide here.

The full contest instructions and an HTML document of Ergo's results in
this area can be found at http://www.ergo-ling.com.  The standards were
designed to allow the developers of a parsing system (statistical or
syntactic) to demonstrate the thoroughness and accuracy of the parses they
produce by using the parsed output to perform a number of straightforward,
traditional syntactic tasks such as changing a statement to a question or
an active to a passive as well as demonstrating an ability to create
standard trees (Using the Penn Treebank II guidelines) and standard
grammatical analyses.  All the standards chosen were chosen to be theory
independent measures of the accuracy of a parse through the use of standard
and ordinary grammatical and syntactic output.

The contest officially begins on January 15th and will be closed on March
31st.  This will allow developers 2.5 months to develop tools and to work
with trouble spots that they may have with the set of sentences offered in
this contest.  The contest will be offered in subsequent years from January
to March.  As time develops we hope the parsers, the contest rules, and the
test sentences will all grow in sophistication and scope.  However, as most
parsers have existed many more years than ours, it is reasonable to think
these tools exist already.

THE CONTEST RULES:
Anyone who joins must submit an HTML document and the parser (source code
only) that created it.  The parser can be in any format but it must require
a minimum of effort for the contest judges to set up and run. For example,
a WIN95 Interface that takes input files and produces the html output file
would be considered a minimum effort parser.  There will be tests to ensure
that the output is genuine parsed output rather than a synthesis such as a
series of print calls that merely present the correct output for a
particular string rather than generating it.

The HTML files of all contestants will be made available at the Ergo web
site (http://www.ergo- ling.com).  Those who wish to join even though their
parsing system is not robust or complete enough for all the tasks or all
the sentences in the contest are also welcome to join.  Reviewers will then
look at these documents as promising parsers for future contests.  Their
results will be posted on our web site as well.

Judging will be based on the percentage of sentences that parsed, the
percentage of the tasks that are completed and on the accuracy of the parses
that result and the success on the parsing tasks.  Currently, the judges will
be Derek Bickerton and myself, but we will welcome others to join in the task.
Because of the home court advantage of the judges, there will be printed
reports of the judging available on the Ergo web site for review by the
overall community of professionals in this area.  Complaints or criticisms
will also be posted.

Anyone who would like to review the judging and the comments on the judging
are welcome to do so.  Anyone who wishes to be a volunteer judge may also
contact us.  However, the criteria for all judging will be the accuracy of
the parser in creating a correct parse of all the sentences and completing
all the tasks set forth in the test materials.

We would like this contest to remain open not only to challengers but also
to those who would like to design and improve the contest itself through the
addition of more sentences or more tasks added to the parsing task.  There is
one condition, however, on being able to this, we will hold rigidly to the
rule that those who would improve on or add to the contest must first meet
the original challenge at a minimum level of 75% accuracy before being
allowed to contribute.   We are starting with a small set of relatively simple
sentences to make this as available as possible to as many people as possible.
In this manner researchers in industry, academia, and government will be able
to compare their results without exposing any proprietary or confidential
information.  We also do not want the contest to be unduly influenced by those
who would like to target some ideal of parsing that is not thoroughly grounded
in what is currently possible in these domains.

At a Virtual Reality and Multi-Media Conference in Japan (VSMM ‘98), Ergo was
awarded the "Best Technical Award" for its NLP technology.  I believe the
main reason that judges and others were able to notice this is because I was
able to point out that "THE ENTIRE FIELD OF VIRTUAL REALITY AN D MULTI-MEDIA
IS BEING HELD HOSTAGE BY GRAMMAR."  And then I went on to explain that the
main reason many VR and Multi-Meida sites and programs are not catching on
is because their users cannot ask even a simple question of the characters or
about the objects they encounter.  Thus, a UNESCO virtual world such as
reconstructed cathedral will receive many visitors but they will not stay
and explore because they cannot ask even the simplest questions like "How
many stairs in this Cathedral?"  "When was the Nave built?"   and so on. I
then pointed out that while speech and graphics were actually ready to work
with such projects, the fact that their grammatical abilities is so limited,
no one is using them with these products. The missing link between speech, VR
and multi- media and users actually talking to avatars and sites is GRAMMAR.
When I then demonstrated that this was so with the use of the Ergo tools, we
won the award.   The main reason I am sponsoring this contest is so that all
linguists and NLP researchers who would like to paticipate in this very large
future source of jobs can do so as soon as possible.  So in order to
stimulate research and interest this contest is proposed.

WE WOULD ESPECIALLY LIKE TO INVITE PROFESSORS, STUDENTS, AND STAFF AT CARNEGIE
MELON, STANFORD, XEROX PARC, MICROSOFT, IBM, DRAGON, LEARNOUT AND HAUSPIE,
PHILIPS, MIT, SUN MICROSYSTEMS (JAVASPEECH GROUP), NEW YORK UNIVERSITY, AND
GEORGETOWN UNIVERSITY TO SUBMIT ENTRIES TO THIS CONTEST.  WE WILL BE HAPPY TO
POST THEIR RESULTS AND WOULD ALSO BE HAPPY TO TELL THE WORLD IF THEY CAN
GENERATE A PARSE THAT IS BETTER THAN OURS ON THE STANDARDS PROVIDED HERE.
THIS IS A GREAT OPPORTUNITY FOR STUDENTS AND JUNIOR STAFF TO WORK WITH EXTANT
PARSERS TO COMBINE AND EXTEND TOOLS INTO THESE VERY USEFUL AND PRACTICAL
AREAS.

        THE SENTENCES
The full set of sentences for the contest is available at the
http://www.ergo-ling.com web site.  This list contains five from each of the
three sections: 1) theory independent parsing, 2) navigation and control, and
3) Question/answer, statement/response repartee.  The full list contains 105
sentences and will grow and be modified over the years as this annual contest
takes root.

Section 1:      Theory independent parsing.
        1.      there is a dog on the porch
        2.      John's house is bigger than mary's house
        3.      the tall thin man in the office is reading a technical report
        4.      the man who mary likes is reading the book that john gave her
        5.      learning how to cope with stress is of primary importance in the work world
Section 2:      Navigation and Control.
        1.      Erase all files that end in .doc
        2.      print the file called teach.doc
        3.      send an email to bob that says "meeting at eight"
        4.      send a fax to bob that says "there is a meeting at eight tonight"
        5.      go to yahoo and find information about golf courses in Georgia
Section 3:      Question and Answer/Statement Response Repartee.
        1.      bill's email is bill at server.com
                what is bill's email address
                what is bill's email
        2.      john has romantic books
                what kind of books does john have
        3.      My appointment with bob is at six o'clock
                what time is my appointment
                what time are my appointments
        4.      the tall thin man in the office is reading a technical report book
                what is the man reading
                what is the man doing
                is the man reading a report
                who is reading a report
        5.      John gave mary a book because it was her birthday
                who gave mary a book
                who did john give a book
                what did john give mary
                why did john give mary a book
                did john give mary a book
                did john give mary a book because it was her birthday
                did john give mary a pencil
                did john give mary a book because it was bob's birthday



Philip A. Bralich, President
Ergo Linguistic Technologies
2800 Woodlawn Drive, Suite 175
Honolulu, HI 96822
tel:(808)539-3920
fax:(880)539-3924