[Corpora-List] New from LDC

Fri Jun 22 19:40:23 UTC 2012

/New publications:/

LDC2012T09
*- Arabic-Dialect/English Parallel Text <ae> -*

LDC2012T08
*- Prague Czech-English Dependency Treebank 2.0 <ce> -*

------------------------------------------------------------------------

*New publications*

(1) Arab 
<http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2012T09>ic-Dialect/English 
Parallel Text 
<http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2012T09> 
was developed by Raytheon BBN Technologies <http://www.bbn.com/> (BBN), 
LDC and Sakhr Software <http://www.sakhr.com/>and contains approximately 
3.5 million tokens of Arabic dialect sentences and their English 
translations.

The data in this corpus consists of Arabic web text as follows:

1. Filtered automatically from large Arabic text corpora harvested from 
the web by LDC. The LDC corpora consisted largely of weblog and online 
user groups and amounted to around 350 million Arabic words. Documents 
that contained a large percentage of non-Arabic or Modern Standard 
Arabic (MSA) words were eliminated. A list of dialect words was manually 
selected by culling through the Levantine Fisher (LDC2005S07 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2005S07>, LDC2005T03 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2005T03>, LDC2007S02 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2007S02> 
and LDC2007T04 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2007T04>) and 
Egyptian CALLHOME speech corpora (LDC97S45 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC97S45>, 
LDC2002S37 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2002S37>, LDC97T19 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC97T19> 
and LDC2002T38 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2002T38>) distributed 
by LDC. That list was then used to retain documents that contained a 
certain number of matches. The resulting subset of the web corpora 
contained around four million words. Documents were automatically 
segmented into passages using formatting information from the raw data.

2. Manually harvested by Sakhr Software from Arabic dialect web sites.

Dialect classification and sentence segmentation, as needed, and 
translation into English were performed by BBN through Amazon's 
Mechanical Turk <https://www.mturk.com/mturk/welcome>. Arabic annotators 
from Mechanical Turk classified filtered passages as being either MSA or 
one of four regional dialects: Egyptian, Levantine, Gulf/Iraqi or 
Maghrebi. An additional "General" dialect option was allowed for 
ambiguous passages. The classification was applied to whole passages 
rather than individual sentences. Only the passages labeled Levantine 
and Egyptian were further processed. The segmented Levantine and 
Egyptian sentences were then translated. Annotators were instructed to 
translate completely and accurately and to transliterate Arabic names. 
They were also provided with examples. All segments of a passage were 
presented in the same translation task to provide context.

*

(2) P 
<http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2012T08>rague 
Czech-English Dependency Treebank (PCEDT) 2.0 
<http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2012T08> 
was developed by the Institute of Formal and Applied Linguistics 
<http://ufal.mff.cuni.cz/> at Charles University <http://www.cuni.cz/> 
in Prague, Czech Republic. It is a corpus of Czech-English parallel 
resources translated, aligned and manually annotated for dependency 
structure, semantic labeling, argument structure, ellipsis and anaphora 
resolution. This release updates Prague Czech-English Dependency 
Treebank 1.0 (LDC2004T25 
<http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2004T25>) by 
adding English newswire texts so that it now contains over two million 
words in close to 100,000 sentences.

The principal new material in PCEDT 2.0 is the inclusion of the entire 
Wall Street Journal data from Treebank-3 (LDC99T42 
<http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC99T42>). 
Not included from PCEDT 1.0 are the Reader's Digest material, the Czech 
monolingual corpus and the English-Czech dictionary.

Each section is enhanced with a comprehensive manual linguistic 
annotation in the Prague Dependency Treebank style (LDC2006T01 
<http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T01>), 
Prague Dependency Treebank 2.0). The main features of this annotation 
style are:

    -dependency structure of the content words and coordinating and
    similar structures (function words are attached as their attribute
    values)

    -semantic labeling of content words and types of coordinating structures

    -argument structure, including an argument structure ("valency")
    lexicon for both languages

    -ellipsis and anaphora resolution

This annotation style is called tectogrammatical annotation, and it 
constitutes the tectogrammatical layer in the corpus. Please consult the 
PCEDT website <http://ufal.mff.cuni.cz/pcedt2.0/> for more information 
and documentation.

------------------------------------------------------------------------

-- 
--

Ilya Ahtaridis
Membership Coordinator
--------------------------------------------------------------------
Linguistic Data Consortium                  Phone: 1 (215) 573-1275
University of Pennsylvania                    Fax: 1 (215) 573-2175
3600 Market St., Suite 810ldc at ldc.upenn.edu
Philadelphia, PA 19104 USAhttp://www.ldc.upenn.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20120622/01f61328/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora