28.1288, FYI: Build It Break It: a New Shared Task for CL

[The LINGUIST List]

LINGUIST List: Vol-28-1288. Wed Mar 15 2017. ISSN: 1069 - 4875.

Subject: 28.1288, FYI: Build It Break It: a New Shared Task for CL

Moderators: linguist at linguistlist.org (Damir Cavar, Malgorzata E. Cavar)
Reviews: reviews at linguistlist.org (Helen Aristar-Dry, Robert Coté,
                                   Michael Czerniakowski)
Homepage: http://linguistlist.org

*****************    LINGUIST List Support    *****************
                       Fund Drive 2017
                   25 years of LINGUIST List!
Please support the LL editors and operation with a donation at:
           http://funddrive.linguistlist.org/donate/

Editor for this issue: Kenneth Steimel <ken at linguistlist.org>
================================================================


Date: Wed, 15 Mar 2017 19:00:15
From: Damir Cavar [damir at linguistlist.org]
Subject: Build It Break It: a New Shared Task for CL

 
Emily Benders pointed this out to us:

https://bibinlp.umiacs.umd.edu/sharedtask.html

Build It Break It is a new type of shared task for AI problems that pits AI
system ''builders'' against human ''breakers'' in an attempt to learn more
about the generalizability of current technology and the brittleness of
current NLP systems.

The goals are multi-fold: we want to...

- Build more reliable NLP technology
- Learn what linguistic phenomena our systems are capable of handling
- Encourage researchers to think about model assumptions
- Build an interesting test collection of examples
- Increase cross-talk between linguistics and NLPers

Our shared task will run in three rounds:

1. Building Round: We will release training data (which you can choose to use
or ignore as you like) and Builder Teams will build systems for solving that
task.

2. Breaking Round: Breakers must construct minimal pairs that they think will
fool the Builders' systems.

3. Judgment Round: All minimal pair sentences will be collected, shuffled, and
sent back to the builder teams. They must run their system as is and upload
the predictions on all the Breaker data.

We will run two tasks in parallel:

1. Sentence-level sentiment analysis, based on movie reviews originally
collected by Pang+Lee+Vaithyanathan.

2. Semantic role labelling as question answering: This task and data are
derived from the He+Lewis+Zettlemoyer's work on Question-Answer Driven
Semantic Role Labelling. The input is a sentence and a question related to one
of the predicates in the sentence, and the output is a span of the sentence
that answers the question.

The contest will run from late March through the end of May and you're welcome
to participate as a builder, a breaker, or both!
 



Linguistic Field(s): Computational Linguistics





 



------------------------------------------------------------------------------

*****************    LINGUIST List Support    *****************
                       Fund Drive 2017
Please support the LL editors and operation with a donation at:
            http://funddrive.linguistlist.org/donate/

This year the LINGUIST List hopes to raise $70,000. This money
will go to help keep the List running by supporting all of our 
Student Editors for the coming year.

Don't forget to check out the Fund Drive 2017 site!

http://funddrive.linguistlist.org/

We collect donations via the eLinguistics Foundation, a
registered 501(c) Non Profit organization with the federal tax
number 45-4211155. The donations can be offset against your
federal and sometimes your state tax return (U.S. tax payers
only). For more information visit the IRS Web-Site, or contact
your financial advisor.

Many companies also offer a gift matching program. Contact
your human resources department and send us the necessary form.

Thank you very much for your support of LINGUIST!
 


----------------------------------------------------------
LINGUIST List: Vol-28-1288	
----------------------------------------------------------