[Corpora-List] SUMMARY: sentence boundary detectors

[Armin Schmidt]

Dear all,

thank you for all the helpful responses. I was preparing several
parallel corpora for a machine translation task between the languages
German, Russian, English, and Spanish. In order to achieve good results
from sentence alignment, I was looking for a sentence splitter that
would perform equally well on all the data sets and, if at all, make the
same or similar errors for all the languages. Also, I didn't have any
lists of abbreviations.

A particularly nice response I received from Jan Strunk who kindly
provided a preliminary implementation of his system 'Punkt'
(http://www.linguistics.ruhr-uni-bochum.de/~strunk/ks2005FINAL.pdf).
'Punkt' learns abbreviations and sentence boundaries in a
language-independent, unsupervised manner.

Links to similar tools for one or several languages were of great help,
too. They are:

Russian:
http://aot.ru/download/graphan.tar.gz (source in C++, dll is included in
http://aot.ru/download/shortrml.zip).

German, Russian, English:
http://www.cis.uni-muenchen.de/~wastl/misc/tokenizer.tgz
(fast, rule-based).

English:
http://text0.mib.man.ac.uk:8080/sentencebreaker/heuristic_tool
(rule-based, Java)

Language-independent:
Tools of the SRI LM toolkit: http://www.speech.sri.com/projects/srilm/

Needs to be provided with a set of abbreviations for a particular language:
http://www.pojkfilmsklubben.org/mickel/code/python/SentenceSplitter.py
(Python)

For Perl, there are several modules available on http://www.cpan.org/
which can be extended for other than the given languages. E.g. for
Russian, use EN::Sentence and add acronym list:
add_acronyms('тел','т','г','млрд','млн','тыс','др','р','кг','л','см','пп','им','ст','муж','жен','ул','пр','кв','ч','п','д','с','стр');

Thanks again & best regards,
Armin


Armin Schmidt schrieb:
> Dear list,
> 
> I was wondering if you could point me to good sentence splitters for the
> following languages: German, Russian, Spanish, English. It would be
> great if they were stand-alone programs or modules for Python (Perl
> would be ok, too ... although I'm already aware of the respective
> CPAN-modules for English and German).
> 
> Since I do have corpora in all the above mentioned languages I would
> also be very interested in available implementations (not papers) of any
> unsupervised learning methods for detecting sentence boundaries (or
> rather abbreviations).
> 
> Thanks,
> Armin
> 
> 
> 

-- 
http://diotavelli.net/people/armin/