29.1081, FYI: Call for Participation: LoReHLT 2018 Evaluation

[The LINGUIST List]

LINGUIST List: Vol-29-1081. Thu Mar 08 2018. ISSN: 1069 - 4875.

Subject: 29.1081, FYI: Call for Participation: LoReHLT 2018 Evaluation

Moderators: linguist at linguistlist.org (Damir Cavar, Malgorzata E. Cavar)
Reviews: reviews at linguistlist.org (Helen Aristar-Dry, Robert Coté,
                                   Michael Czerniakowski)
Homepage: http://linguistlist.org

Please support the LL editors and operation with a donation at:
           http://funddrive.linguistlist.org/donate/

Editor for this issue: Kenneth Steimel <ken at linguistlist.org>
================================================================


Date: Thu, 08 Mar 2018 18:10:20
From: Shudong Huang [shudong.huang at nist.gov]
Subject: Call for Participation: LoReHLT 2018 Evaluation

 
National Institute of Standards and Technology (NIST) is pleased to announce
LoReHLT18 (Low Resource HLT 2018) - an open evaluation series first launched
in 2016 aimed to advance human language technologies that provide rapid and
effective response to emerging incidents where language resources are very
limited. The core tasks for 2018 will continue to be machine translation (MT),
entity discovery and linking (EDL), and situation frames (SF), but we are
introducing several changes to the program.

-Two surprise incident languages (IL), as in 2017, PLUS English for EDL and SF
for each IL
-Check points reduced to two (2) with a shorter evaluation window
-SF task redefined with updated performance scoring metrics
SF text and speech as a single task
-Previously evaluated languages, Oromo and Tigrinya from LoReHLT17, offered as
''Open Test on Additional Languages'' (OTAL), a separate and simplified
evaluation on both MT and EDL

The entire program is open to any participants. As before, open participants
may choose to participate in one or all tasks, in either one or all languages.

We expect the main LoReHLT evaluation to occur in July, but OTAL won't take
place until September. We'll be offering a dry run in Mandarin Chinese for new
participants to become familiar with data format and evaluation procedures
(but we will not provide any scoring). 

While we are hammering out the official evaluation plan, which we expect to
release to the public shortly, please visit our LoReHLT main page at
https://www.nist.gov/itl/iad/mig/lorehlt-evaluations to learn more about the
program overall and about past evaluations.

If you have any questions, please do not hesitate to contact us at
lorehlt_poc at nist.gov.
 



Linguistic Field(s): Computational Linguistics





 



------------------------------------------------------------------------------

*****************    LINGUIST List Support    *****************
Please support the LL editors and operation with a donation at:
            http://funddrive.linguistlist.org/donate/
 


----------------------------------------------------------
LINGUIST List: Vol-29-1081	
----------------------------------------------------------