I am looking to scrape some content from some websites for research and I was hoping that using python and web scraping might speed up my process. I have used python and beautiful soup before for one small project to convert an xml from one format to another.

Depending on how redundant is the structure of the data you want to extract, you could use several tools.

• If you're looking for extracting data always stored in the same DOM structure, Scrapy could do the job.
• If the data is sparse and is stored in various places, maybe BeautfulSoup4 or lxml could help you.
• If the data is generated by some JS code, have a look at Selenium

Here are a couple of resources you might find useful:

• PyCon 2012 Tutorial about web-scraping: http://pyvideo.org/video/609/web-scraping-reliably-and-efficiently-pull-data/
• http://isbullsh.it/2012/04/Web-crawling-with-scrapy/ (full disclosure, I wrote that)
• http://www.packtpub.com/article/web-scraping-with-python
• http://wwwsearch.sourceforge.net/mechanize/