Same thing asked 2.5 years ago in Downloading a web page and all of its resource files in Python but doesn't lead to an answer and the 'please see related topic' isn't really asking the same thing.

I want to download everything on a page to make it possible to view it just from the files.

The command

wget --page-requisites --domains=DOMAIN --no-parent --html-extension --convert-links --restrict-file-names=windows

does exactly that I need. However we want to be able to tie it in with other stuff that must be portable, so requires it to be in Python.

I've been looking at Beautiful Soup, scrapy, various spiders posted around the place, but these all seem to deal with getting data/links in clever but specific ways. Using these to do what I want seems like it will require a lot of work to deal with finding all of the resources, when I'm sure there must be an easy way.

thanks very much

You should be using an appropriate tool for the job at hand.

If you want to spider a site and save the pages to disk, Python probably isn't the best choice for that. Open source projects get features when someone needs that feature, and because wget does its job so well, nobody has bothered to try to write a python library to replace it.

Considering wget runs on pretty much any platform that has a Python interpreter, is there a reason you can't use wget?