<div dir="ltr"><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center" id="docs-internal-guid-d73703e2-3e49-fb56-53c8-b846d86c4f60"><span style="font-size:16px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Automatic and Manual Metrics for Operational Translation Evaluation</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><a href="http://mte2014.github.io/" style="text-decoration:none"><span style="font-size:16px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">http://mte2014.github.io/</span></a><span style="font-size:16px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><span style="font-size:16px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">26 May 2014</span></p>
<br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Workshop at Language Resources and Evaluation Conference (LREC) 2014</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><a href="http://lrec2014.lrec-conf.org" style="text-decoration:none"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">http://lrec2014.lrec-conf.org</span></a><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">In brief:</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">We invite you to join us for an interesting day of work (and play!) as we discuss metrics for machine translation quality assessment and participate in some hands-on task-based translation evaluation.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">This workshop on Automatic and Manual Metrics for Operational Translation Evaluation (MTE 2014) will be a full-day LREC workshop to be held on Monday, May 26, 2014 in Reykjavik, Iceland. The format of MTE 2014 will be interactive and energizing:  a half-day of short presentations and discussion of recent work on machine translation quality assessment, followed by a half-day of hands-on collaborative work with MT metrics that show promise for the prediction of task suitability of MT output. The afternoon hands-on work will follow from the morning’s presentations, with some of the hands-on exercises developed directly from the submissions to the workshop.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Details:</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">While a significant body of work has been done by the machine translation (MT) research community towards the development and meta-evaluation of automatic metrics to assess overall MT quality, less attention has been dedicated to more operational evaluation metrics aimed at testing whether translations are adequate within a specific context: purpose, end-user, task, etc., and why the MT system fails in some cases. Both of these can benefit from some form of manual analysis. Most work in this area is limited to productivity tests (e.g. contrasting time for human translation and MT post-editing). A few initiatives consider more detailed metrics for the problem, which can also be used to understand and diagnose errors in MT systems. These include the Multidimensional Quality Metrics (MQM) recently proposed by the EU F7 project QTLaunchPad, the TAUS Dynamic Quality Framework, and past projects such as FEMTI, EAGLES and ISLE. Some of these metrics are also applicable to human translation evaluation. A number of task-based metrics have also been proposed for applications such as topic ID / triage and reading comprehension. The purpose of this workshop is to bring together representatives from academia, industry and government institutions to discuss and assess metrics for manual and automatic quality evaluation, with an eye toward </span><span style="font-size:16px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">how they might be leveraged or further developed into task-based metrics for more objective “fitness for purpose” assessment. We will also consider comparisons to well-established metrics for automatic evaluation such as BLEU, METEOR and others, including reference-less metrics for quality prediction. </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">The workshop will benefit from datasets already collected and manually annotated for translation errors by the QTLaunchPad project</span><span style="font-size:16px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> (</span><a href="http://www.qt21.eu/launchpad/" style="text-decoration:none"><span style="font-size:16px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">http://www.qt21.eu/launchpad/</span></a><span style="font-size:16px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">) and will cover concepts from many the metrics proposed by participants in the half-day of hands-on tasks.</span></p>
<br><span style="font-size:16px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Up-to-the-minute information and (most importantly) Registration:</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Additional details and schedule will be posted at the workshop website</span><a href="http://mte2014.github.io/" style="text-decoration:none"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">http://mte2014.github.io/</span></a><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> as they become available. Register to attend via the LREC registration site at</span><a href="http://lrec2014.lrec-conf.org/en/registration/" style="text-decoration:none"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">http://lrec2014.lrec-conf.org/en/registration/</span></a><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">We look forward to seeing you there!</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">The MTE 2014 Organizing Committee</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline">Keith J. Miller (MITRE)</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline">Lucia Specia (University of Sheffield)</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline">Kim Harris (GALA and text & form)</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline">Stacey Bailey  (MITRE)</span></p>
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><br></div>