Cleaneval is a good place to find out the problems and many solutions. However, my experience is that it ultimately depends on your exact needs. And the methods can be broadly categorized in two classes: deterministic and learning based. Unless you want to work on data with completely arbitrary formats, learning doesn't seem to be a good idea.<br>
<br>There is some code for text extraction from HTML documents and one or two utilities in <a href="http://sanchay.co.in">Sanchay</a>, but there is no documentation and the it is not connected to the current public GUI. The available code will have to be slightly modified for specific formats: some simple code that uses the HTML parser library to effectively create a template for extraction of a specific format. For a single format, it is not very time consuming.<br>
<br><div class="gmail_quote">On Mon, Aug 2, 2010 at 4:54 PM, Siddhartha Jonnalagadda <span dir="ltr"><<a href="mailto:sid.kgp@gmail.com">sid.kgp@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<font face="verdana,sans-serif">Thanks all for your replies. I am trying BoilerPipe now; will also look into the other things mentioned.<br><br>thanks again,<br><font color="#888888">siddhartha<br></font></font><div><div>
</div><div class="h5"><br><div class="gmail_quote">On Mon, Aug 2, 2010 at 2:51 AM, Wouter Weerkamp <span dir="ltr"><<a href="mailto:w.weerkamp@uva.nl" target="_blank">w.weerkamp@uva.nl</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">In 2007 there was a workshop on content extraction from web pages. You could gave a look at the papers presented there:<br>
<a href="http://cleaneval.sigwac.org.uk/" target="_blank">http://cleaneval.sigwac.org.uk/</a><br>
<br>
If you intend to follow feeds, and need to extract content from these, you can use a learning approach. For each feed you collect a certain number of pages, and you learn which part of the page changes, and which parts don't. From that it shouldn't be hard to determine "real" content.<br>
<br>
You could also have a look at fivefilters, it works pretty good given the simple approach is uses:<br>
<a href="http://fivefilters.org/content-only/" target="_blank">http://fivefilters.org/content-only/</a><br>
(following a few links, you can get to the (php) code).<br>
<br>
Wouter<div><br>
<br>
<br>
On 8/1/10 8:08 PM, Beatrice Alex wrote:<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div>
You might want to check out Boilerpipe:<br>
<br>
<a href="http://code.google.com/p/boilerpipe/" target="_blank">http://code.google.com/p/boilerpipe/</a><br>
<br>
Best,<br>
<br>
Bea<br>
<br>
------------------<br>
Beatrice Alex<br>
Research Fellow and Project Manager at the School of Informatics, University of Edinburgh.<br>
<br>
<br>
On 1 Aug 2010, at 01:26, Siddhartha Jonnalagadda wrote:<br>
<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div>
Is it trivial to extract the title and relevant text (ignoring the ads and other irrelevant stuff)? For example, in the website: <a href="http://tvnz.co.nz/world-news/chelsea-clinton-marries-in-ny-3680168" target="_blank">http://tvnz.co.nz/world-news/chelsea-clinton-marries-in-ny-3680168</a><br>
<br>
I am only interested in extracting the tile: "Chelsea Clinton marries in NY"<br>
and the subject below. How easy is this?<br>
<br>
"Bill and Hillary Clinton's daughter married her long-time boyfriend in the picturesque New York village of Rhinebeck today in what has been dubbed America's royal wedding.<br>
Chelsea Clinton - the only child of the former US president and the US secretary of state - wed Marc Mezvinsky at Astor Courts, an historic 50-acre (20-hectare) estate on the Hudson River, about 160 km north of New York City.<br>
<br>
"Today, we watched with great pride and overwhelming emotion as Chelsea and Marc wed in a beautiful ceremony at Astor Courts, surrounded by family and their close friends," Bill and Hillary Clinton said in a statement.<br>
<br>
"We could not have asked for a more perfect day to celebrate the beginning of their life together, and we are so happy to welcome Marc into our family," the statement said.<br>
<br>
"On behalf of the newlyweds, we want to give special thanks to the people of Rhinebeck for welcoming us and to everyone for their well-wishes on this special day."<br>
<br>
The statement, sent just after 7:30 pm (12:30pm NZT today), did not indicate exactly when the nuptials took place.<br>
<br>
On Friday night, Bill and Hillary Clinton waved to crowds of onlookers as they arrived at the historic Beekman Arms Inn in the center of Rhinebeck for a late-night cocktail party for some of the wedding guests.<br>
<br>
<br>
<br></div><div>
Apart from the parents of the bride, the only other high profile guests seen in Rhinebeck have been Bill Clinton's former secretary of state, Madeleine Albright, actors Ted Danson and Mary Steenburgen and fashion designer Vera Wang.<br>
<br>
Also spotted was real estate scion and movie producer billionaire Steve Bing. Bing lent Bill Clinton his jet to fly to North Korea in August of last year to bring home American journalists Laura Ling and Euna Lee after they spent four months imprisoned in the reclusive communist state.<br>
<br>
Guests boarded buses in Rhinebeck to be taken to Astor Courts about 5 pm EDT (10am NZT)<br>
<br>
Chelsea Clinton, 30, and Mezvinsky, 32, have known each other since they were teenagers. He is an investment banker, whose parents Marjorie Margolies-Mezvinsky and Edward Mezvinsky were once Democratic US House of Representatives members.<br>
<br>
Chelsea Clinton, who worked at a New York hedge fund and has more recently studied health policy at Columbia University, has kept a low profile since her father left the White House in January 2001, although she campaigned for her mother during her failed run for the 2008 Democratic presidential nomination.<br>
<br>
Signs and pictures congratulating the newlyweds hang in many shop windows in Rhinebeck, which has been swarmed by media around the world for an event that experts estimate to have cost between $US3 million and $US5 million.<br>
<br>
Airspace above Rhinebeck has been closed for 12 hours from 3 pm EDT (7pm NZT) today for the wedding and media were kept well away from the entrance to Astor Courts. Security in the area was comparable to that surrounding state visits.<br>
<br>
The guest list was reported to be between 400 and 500, but did not include a very understanding President Barack Obama.<br>
<br>
"Hillary and Bill properly want to keep this as a thing for Chelsea and her soon-to-be husband," Obama said on The View talk show on Thursday. "It would be tough enough to have one president at a wedding. You don't want two presidents."<br>
<br>
"<br>
_______________________________________________<br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
</div></blockquote>
<br>
<br>
<br>
<br>
<br>
<br>
The University of Edinburgh is a charitable body, registered in<br>
Scotland, with registration number SC005336.<div><br>
<br>
<br>
<br>
_______________________________________________<br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
</div></blockquote><font color="#888888">
<br>
-- <br>
ISLA * University of Amsterdam * <a href="http://ilps.science.uva.nl" target="_blank">http://ilps.science.uva.nl</a></font><div><div></div><div><br>
<br>
_______________________________________________<br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
</div></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
<br></blockquote></div><br>