Given some random news article, I want to write a web crawler to find the largest body of text present, and extract it. The intention is to extract the physical news article on the page.

The original plan was to use a BeautifulSoup findAll(True) and to sort each tag by its .getText() value. EDIT: don't use this for html work, use the lxml library, it's python based and much faster than BeautifulSoup. command (which means extract all html tags)

But this won't work for most pages, like the one I listed as an example, because the large body of text is split into many smaller tags, like paragraph dividers for example.

Does anyone have any experience with this? Any help with something like this would be amazing.

At the moment I'm using BeautifulSoup along with python, but willing to explore other possibilities.

Here are some deadly helpful python libraries for the task in sorted order of how much it helped me:

#1 goose library Fast, powerful, consistent #2 readability library Content is passable, slower on average than goose but faster than boilerpipe #3 python-boilerpipe Slower & hard to install, no fault to the boilerpipe library (originally in java), but to the fact that this library is build on top of another library in java, which attributes to IO time & errors, etc.

I'll release benchmarks perhaps if there is interest.

Indirectly related libraries, you should probably install them and read their docs:

• NLTK text processing library This is too good not to install. They provide text analysis tools along with html tools (like cleanup, etc).
• lxml html/xml parser Mentioned above. This beats BeautifulSoup in every aspect but usability. It's a bit harder to learn but the results are worth it. HTML parsing takes much less time, it's very noticeable.
• python webscraper library I think the value of this code isn't the lib itself, but using the lib as a reference manual to build your own crawlers/extractors. It's very nicely coded / documented!

A lot of the value and power in using python, a rather slow language, comes from it's open source libraries. They are especially awesome when combined and used together, and everyone should take advantage of them to solve whatever problems they may have!

Goose library gets lots of solid maintenance, they just added Arabic support, it's great!

You might look at the python-readability package which does exactly this for you.