[Corpora-List] Q: Hyphenation removal

[Jordi Porta Zamorano]

Dear Ronald,

Perhaps the data-driven solution described in
http://www.euppublishing.com/doi/abs/10.3366/cor.2011.0010 could help you.

Best,
--Jordi

On Thursday, 16 August 2012, Roland Schäfer wrote:

> Dear list members,
>
> are there any tools to remove hard-coded "hyphe- nation" from texts (or
> papers describing principled solutions to the problem). The
> tool/solution should ideally:
>
> * work even if the line break after the hyphen has been removed,
>
> * differentiate with near-perfect accuracy between actual hyphenation
> and other superficially identical graphematic constructions involving
> hyphens (like German truncated compound coordination as in "Bus- und
> Bahnticket" for "Busticket und Bahnticket"),
>
> * (consequently:) be trainable (ideally in an unsupervised way) on
> arbitrary languages which use some specifiable set of characters to
> indicate hyphenation,
>
> * possibly also detect cases of hyphenation which are not written with a
> space (as in "hyphe-nation") or additional spaces (as in "hyphe - nation"),
>
> * process UTF-8 or at least some Unicode encoding,
>
> * be open-source/patent-free and available in the form of a library or
> command line tool (e.g., not a GUI tool/part of some OCR product/web
> service).
>
> Of course, references which only partially match these requirements are
> also highly appreciated.
>
> Thanks a lot.
>
> Regards
> Roland
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no <javascript:;>
> http://mailman.uib.no/listinfo/corpora
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20120817/0faf07db/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora