<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
</head>
<body bgcolor="#ffffff" text="#000000">
<p class="MsoNormal" style="text-align: center;" align="left"><b
style="">- </b><b><a href="#interspeech">LDC
Data Sheets Now
Available Online</a></b><b style=""> -</b><b style=""></b><b style=""></b>
</p>
<p class="MsoNormal" style="text-align: center;" align="center">- <b><a
href="#lre">2007
NIST Language Recognition
Evaluation Test Set</a></b> -
</p>
<p class="MsoNormal" style="text-align: center;" align="center">- <b> </b><b><a
href="#onto">OntoNotes
3.0</a></b> -<br>
</p>
<p class="MsoNormal" style="text-align: center;" align="center">- <b><a
href="#euro">Web
1T 5-gram,
10 European Languages Version 1</a></b> -<br>
</p>
<hr size="2" width="100%">
<p class="MsoNormal" style="text-align: center;" align="center"><b><a
name="datasheets">LDC Data Sheets Now Available Online</a></b></p>
<p class="MsoNormal" style="text-align: center;" align="center"><b
style=""><o:p> </o:p></b></p>
<p class="MsoNormal">In early 2009, LDC crafted <a
href="http://www.ldc.upenn.edu/DataSheets/">data sheets</a> to
describe in concise form current and past projects, daily operations
and our
technical capabilities. Print versions of these documents debuted at <a
href="http://www.interspeech2009.org/">Interspeech 2009</a> and have
received
positive feedback for both their content and design. <o:p></o:p></p>
<p class="MsoNormal">The data sheets were distributed on FSC certified
30%
recycled paper and were printed using environmentally-friendly toner.<span
style=""> </span>FSC certification means that the process that
developed the paper, from seed to final sheet, is in compliance with
international laws and treaties so that it employs fair labor standards
and
respects and conserves environmental resources.<o:p></o:p></p>
<p class="MsoNormal">LDC intends to expand the breadth of data sheet
categories
and the depth of information provided within each category. This will
help to
accurately represent our organization and highlight our staff’s
research and
development efforts.<br>
</p>
<p class="MsoNormal">[<a href="#top">top</a>]<br>
<br>
</p>
<div align="center"><b>New
Publications</b><o:p></o:p>
</div>
<p><b><a name="lre">(1)</a></b> <a
href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2009S04">2007
NIST Language Recognition Evaluation Test Set</a> consists of 66 hours
of
conversational telephone speech segments in the following languages and
dialects: Arabic, Bengali, Chinese (Cantonese), Mandarin Chinese
(Mainland,
Taiwan), Chinese (Min), English (American, Indian), Farsi, German,
Hindustani
(Hindi, Urdu), Korean, Russian, Spanish (Caribbean, non-Caribbean),
Tamil, Thai
and Vietnamese. <o:p></o:p></p>
<p>The goal of the <a href="http://www.itl.nist.gov/iad/">NIST
(National
Institute of Standards and Technology)</a> <a
href="http://www.itl.nist.gov/iad/mig/tests/lre/">Language Recognition
Evaluation (LRE)</a> is to establish the baseline of current
performance
capability for language recognition of conversational telephone speech
and to
lay the groundwork for further research efforts in the field. NIST
conducted
three previous language recognition evaluations, in <a
href="http://www.itl.nist.gov/iad/mig/tests/lre/1996/">1996</a>, <a
href="http://www.itl.nist.gov/iad/mig/tests/lre/2003/">2003</a> and <a
href="http://www.itl.nist.gov/iad/mig/tests/lre/2005/">2005</a>. The
most
significant differences between those evaluations and the 2007 task
were the
increased number of languages and dialects, the greater emphasis on a
basic
detection task for evaluation and the variety of evaluation conditions.
Thus,
in 2007, given a segment of speech and a language of interest to be
detected
(i.e., a target language), the task was to decide whether that target
language
was in fact spoken in the given telephone speech segment (yes or no),
based on
an automated analysis of the data contained in the segment. <o:p></o:p></p>
<p class="MsoNormal">Each speech file in the test data is one side of a
"4-wire" telephone conversation represented as 8-bit 8-kHz mu-law
format. There are 7530 speech files in SPHERE (.sph) format for a total
of 66
hours of speech. The speech data was compiled from LDC's CALLFRIEND,
Fisher
Spanish and Mixer 3 corpora and from data collected by Oregon Health
and
Science University, Beaverton, Oregon. The test segments contain three
nominal durations of speech: 3 seconds, 10 seconds and 30 seconds.
Actual
speech durations vary, but were constrained to be within the ranges of
2-4
seconds, 7-13 seconds and 23-35 seconds, respectively.<o:p></o:p></p>
<br>
<p>[<a href="#top">top</a>]<br>
<o:p></o:p></p>
<p style="margin-bottom: 12pt; text-align: center;" align="center">*<o:p></o:p></p>
<p><b><a name="onto">(2)</a></b> <a
href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2009T24">OntoNotes
3.0</a>. The OntoNotes project is a collaborative effort between BBN
Technologies, the University of Colorado, the University of
Pennsylvania, and
the University of Southern California's Information Sciences Institute.
The
goal of the project is to annotate a large corpus comprising various
genres of
text (news, conversational telephone speech, weblogs, use net,
broadcast, talk
shows) in three languages (English, Chinese, and Arabic) with
structural
information (syntax and predicate argument structure) and shallow
semantics
(word sense linked to an ontology and coreference)..<o:p></o:p></p>
<p><a
href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2007T21">OntoNotes
Release 1.0 (LDC2007T21)</a> contains 400k words of Chinese newswire
data and
300k words of English newswire data. <a
href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2008T04">OntoNotes
Release 2.0 (LDC2008T04)</a> added the following to the corpus: 274k
words of
Chinese broadcast news data; and 200k words of English broadcast news
data.
OntoNotes Release 3.0 incorporates the following new material: 250k
words of
English newswire data, 200k of English broadcast news data; 200k words
of
English broadcast conversation material; 250k words of Chinese newswire
data;
250k words of Chinese broadcast news material;150k words of Chinese
broadcast
conversation data; and 200k words of Arabic newswire material. <o:p></o:p></p>
<p>Natural language applications like machine translation, question
answering,
and summarization currently are forced to depend on impoverished text
models
like bags of words or n-grams, while the decisions that they are making
ought
to be based on the meanings of those words in context. That lack of
semantics
causes problems throughout the applications. Misinterpreting the
meaning of an
ambiguous word results in failing to extract data, incorrect alignments
for
translation, and ambiguous language models. Incorrect coreference
resolution
results in missed information (because a connection is not made) or
incorrectly
conflated information (due to false connections). OntoNotes builds on
two
time-tested resources, following the <a
href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC99T42">Penn
Treebank</a> for syntax and the <a
href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2004T14">Penn
PropBank</a> for predicate-argument structure. Its semantic
representation will
include word sense disambiguation for nouns and verbs, with each word
sense
connected to an ontology, and coreference. The current goals call for
annotation of over a million words each of English and Chinese, and
half a million
words of Arabic over five years.<o:p></o:p></p>
<br>
<p>[<a href="#top">top</a>]<br>
<o:p></o:p></p>
<p style="text-align: center;" align="center">*<o:p></o:p></p>
<p><b><a name="euro">(3)</a></b> <a
href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2009T25">Web
1T 5-gram, 10 European Languages Version 1</a> was created by Google,
Inc. It consists of word n-grams and their observed frequency counts
for
ten European languages: Czech, Dutch, French, German, Italian, Polish,
Portuguese, Romanian, Spanish and Swedish. The length of the n-grams
ranges
from unigrams (single words) to five-grams. The n-gram counts were
generated
from approximately one billion word tokens of text for each language,
or
approximately one trillion total tokens. <o:p></o:p></p>
<p>The n-grams were extracted from publicly-accessible web pages from
October
2008 to December 2008. This data set contains only n-grams that
appeared at least
40 times in the processed sentences. Less frequent n-grams were
discarded.
While the aim was to identify and collect pages from the specific
target
languages only, it is likely that some text from other languages may be
in the
final data. This dataset will be useful for statistical language
modeling,
including machine translation, speech recognition and other uses. The
input encoding of documents was automatically detected, and all text
was
converted to UTF8.<o:p></o:p></p>
[<a href="#top">top</a>]
<div align="center">
<hr size="2" width="100%"><font face="Courier New, Courier, monospace"><small><small><big><br>
<br>
Ilya
Ahtaridis<br>
Membership Coordinator</big><br>
<br>
</small>--------------------------------------------------------------------</small><small><br>
</small></font></div>
<div align="center">
<pre class="moz-signature" cols="72"><font
face="Courier New, Courier, monospace">Linguistic Data Consortium Phone: (215) 573-1275
University of Pennsylvania Fax: (215) 573-2175
3600 Market St., Suite 810 <a
class="moz-txt-link-abbreviated" href="mailto:ldc@ldc.upenn.edu">ldc@ldc.upenn.edu</a>
Philadelphia, PA 19104 USA <a
class="moz-txt-link-freetext" href="http://www.ldc.upenn.edu">http://www.ldc.upenn.edu</a></font></pre>
</div>
<pre class="moz-signature" cols="72">
</pre>
</body>
</html>