<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2800.1476" name=GENERATOR></HEAD>
<BODY>
<DIV><FONT face=Arial size=2><SPAN class=662543516-22122004>We are facing the
task of having to find duplicate and near-duplicate documents in a collection of
about 1 million texts. Can anyone give us advice on how to approach this
challenge? </SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=662543516-22122004></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=662543516-22122004></SPAN></FONT><FONT
face=Arial size=2><SPAN class=662543516-22122004>The documents are in
various formats (html, PDF, MS-Word, plain text, ...) so that we intend
to first convert them to plain text. It is possible that the same text is
present in the document collection in different formats.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=662543516-22122004></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=662543516-22122004>For smaller
collections, we identify (near)-duplicates by applying hierarchical clustering
techniques, but with this approach, we are limited to a few thousand documents.
</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=662543516-22122004></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=662543516-22122004>Any pointers are
welcome. Thank you.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=662543516-22122004></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=662543516-22122004>Ralf
Steinberger</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=662543516-22122004>European Commission
- Joint Research Centre</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=662543516-22122004><A
href="http://www.jrc.it/langtech">http://www.jrc.it/langtech</A></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=662543516-22122004></SPAN></FONT> </DIV></BODY></HTML>