I'm getting a "WindowsError: [Error 5] Access is denied" message when reading a website with urllib2.

from urllib2 import urlopen, Request
from bs4 import BeautifulSouphdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'}
req = Request('https://' + url, headers=hdr)
soup = BeautifulSoup( urlopen( req ).read() )

The full traceback is:

Traceback (most recent call last):File "<stdin>", line 1, in <module>File "C:\Python27\lib\urllib2.py", line 154, in urlopenreturn opener.open(url, data, timeout)File "C:\Python27\lib\urllib2.py", line 431, in openresponse = self._open(req, data)File "C:\Python27\lib\urllib2.py", line 449, in _open'_open', req)File "C:\Python27\lib\urllib2.py", line 409, in _call_chainresult = func(*args)File "C:\Python27\lib\urllib2.py", line 1240, in https_opencontext=self._context)File "C:\Python27\lib\urllib2.py", line 1166, in do_openh = http_class(host, timeout=req.timeout, **http_conn_args)File "C:\Python27\lib\httplib.py", line 1258, in __init__context = ssl._create_default_https_context()File "C:\Python27\lib\ssl.py", line 440, in create_default_contextcontext.load_default_certs(purpose)File "C:\Python27\lib\ssl.py", line 391, in load_default_certsself._load_windows_store_certs(storename, purpose)File "C:\Python27\lib\ssl.py", line 378, in _load_windows_store_certsfor cert, encoding, trust in enum_certificates(storename):
WindowsError: [Error 5] Access is denied

I've tried running the script from a command prompt with admin privileges, as suggested here, but it does not fix the problem.

Any suggestions on how to resolve this error?

It looks like this is a windows certificate store inconsistency. httplib - which is internally called by urllib2 - recently changed from no server certificate validation to enforce server certificate validation by default. Therefore you'll encounter this problem in any python script that is based on urllib, httplib and running within your user profile.

That said, something seems to be very wrong with your windows certificate store. httplib fails for you while trying to enumerate certificates for the named certificate stores CA certification authority (shows up as Intermediate Certification Authorities in certmgr.msc) but succeeds for ROOT which is the normal trusted root certificate store (see comments to question). I'd therefore suggest to check all the certificates in certmgr:intermediate certificate authorities for recently added certificates and/or the windows log for general errors. What is going on in your case is that urllib2 internally calls httplib which then tries to set up a default ssl context with certificate validation enforced and as part of this it enumerates the trusted certificate anchors of your system by calling ssl.enum_certificates. This function is implemented in C as _ssl_enum_certificates_impl and internally calls WINAPIs CertOpenSystemStore and CertEnumCertificatesInStore. For the certificate store location CA it just failes in one of the two winapi calls with an access denied.

If you want to further debug this you can also try to manually invoke the WINAPI:CertOpenSystemStore with LPTCSTR::'CA' as an argument and try to debug it from this side, try other windows certstore management tools and/or call microsoft support for asistance.

There are also indications that others had similar problems while interfacing that api call, see google:access denied CertOpenSystemStore

If you just want to make it work without fixing the root cause you could just try to use the following workaround that temporarily patches the _windows_cert_stores to not include the broken CA certstore or to completely disable the trust-anchor loading logic. (all other ssl.SSLContext invocations will be patched in the current process)

Note that this effectively disables server certificate verification.

ssl.SSLContext._windows_cert_stores = ("ROOT",)         # patch windows_cert_stores default to only include "ROOT" as "CA" is broken for you.
#ssl.SSLContext.load_default_certs = lambda s,x:None    # alternative, fully NOP load_default_certs to do nothing instead.
ctx = ssl.create_default_context()                      # create new sslcontext, not veryfing any certificates, hostnames.
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE                         hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'}
req = Request('https://' + url, headers=hdr)
x = urlopen( req , context=ctx).read() 
ssl.SSLContext._windows_cert_stores = ("ROOT","CA")   # UNDO PATCH

I hope this information will help you resolve the issue. good luck.