I'm trying to scrape this site: http://stats.swehockey.se/ScheduleAndResults/Schedule/3940

And I've gotten as far (thanks to alecxe) as retrieving the date and teams.

from scrapy.item import Item, Field
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelectorclass SchemaItem(Item):date = Field()teams = Field()class SchemaSpider(BaseSpider):name = "schema"allowed_domains = ["http://stats.swehockey.se/"]start_urls = ["http://stats.swehockey.se/ScheduleAndResults/Schedule/3940"]def parse(self, response):hxs = HtmlXPathSelector(response)rows = hxs.select('//table[@class="tblContent"]/tr')for row in rows:item = SchemaItem()item['date'] = row.select('.//td[2]/div/span/text()').extract()item['teams'] = row.select('.//td[3]/text()').extract()yield item

So, my next step is to filter out anything that ins't a home game of "AIK" or "Djurgårdens IF". After that I'll need to reformat to an .ics file which I can add to Google Calender.

EDIT: So I've solved a few things but still has a lot to do. My code now looks like this..

# -*- coding: UTF-8 -*-
from scrapy.item import Item, Field
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelectorclass SchemaItem(Item):date = Field()teams = Field()class SchemaSpider(BaseSpider):name = "schema"allowed_domains = ["http://stats.swehockey.se/"]start_urls = ["http://stats.swehockey.se/ScheduleAndResults/Schedule/3940"]def parse(self, response):hxs = HtmlXPathSelector(response)rows = hxs.select('//table[@class="tblContent"]/tr')for row in rows:item = SchemaItem()item['date'] = row.select('.//td[2]/div/span/text()').extract()item['teams'] = row.select('.//td[3]/text()').extract()for string in item['teams']:teams = string.split('-') #split ithome_team = teams[0]#.split(' ') #only the first name, e.g. just 'Djurgårdens' out of 'Djurgårdens IF'away_team = teams[1]#home_team[0] = home_team[0].replace(" ", "") #remove whitespace#home_team = home_team[0]if "AIK" in home_team:for string in item['date']:year = string[0:4]month = string[5:7]day = string[8:10]hour = string[11:13]minute = string[14:16]print year, month, day, hour, minute, home_team, away_team  elif u"Djurgårdens" in home_team:for string in item['date']:year = string[0:4]month = string[5:7]day = string[8:10]hour = string[11:13]minute = string[14:16]print year, month, day, hour, minute, home_team, away_team     

That code prints out the games of "AIK", "Djurgårdens IF" and "Skellefteå AIK". So my problem here is obviously how to filter out "Skellefteå AIK" games and if there is any easy way to make this program better. Thoughts on this?

Best regards!

I'm just guessing that home games are the ones with the team you're looking for first (before the dash).

You can do this in XPath or from python. If you want to do it in XPath, only select the rows which contain the home team name.

//table[@class="tblContent"]/tr[contains(substring-before(.//td[3]/text(), "-"), "AIK")orcontains(substring-before(.//td[3]/text(), "-"), "Djurgårdens IF")
]

You can savely remove all whitespace (including newlines), I just added them for readability.

For python you should be able to do much the same, maybe even more concise using some regular expressions.