InsecureRequestWarning + MarkupResemblesLocatorWarning:

2024/9/22 18:24:10

I'd like to scrape a site for my office work. I am learning each day. I need your support guys.

Here is the Code:

url = https://www.eprocure.gov.bd/partner/ViewTenderPaymentDetails.jsp?payId=33767442&uId=12381&tenderId=860992&lotId=1332668&payTyp=ps)
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0","Accept-Encoding": "\*","Connection": "keep-alive"}
requests.get(url, verify=False, headers=headers)
soup = BeautifulSoup(url,'html.parser').textprint(soup)

Result:

InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.eprocure.gov.bd'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
warnings.warn(
1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.eprocure.gov.bd'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
warnings.warn(
57: MarkupResemblesLocatorWarning: The input looks more like a URL than markup. You may want to use an HTTP client like requests to get the document behind the URL, and feed that document to Beautiful Soup.
soup = BeautifulSoup(url,'html.parser').text

Please help me with scraping the site. I've just started coding.

Answer

InsecureRequestWarning is actually described in the warning you see in the output.
You have disabled the certificate verification (verify=False), hence made your request insecure.

You should be careful with such requests. If you want to disable this warning, see this article. Otherwise, follow the link from the warning message and read more details about the SSL verification.

Regarding BS part, you are passing the URL as a string to the constructor, instead, you should pass the content of the response.

The following code works for me (with InsecureRequestWarning):

import requests
from bs4 import BeautifulSoupurl = "https://www.eprocure.gov.bd/partner/ViewTenderPaymentDetails.jsp?payId=33767442&uId=12381&tenderId=860992&lotId=1332668&payTyp=ps"
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0","Accept-Encoding": "*","Connection": "keep-alive",
}response = requests.get(url, verify=False, headers=headers)
if response.status_code == 200:soup = BeautifulSoup(response.content, 'html.parser')# Continue with your parsing here# Continue with your parsing here# Continue with your parsing hereprint(soup.prettify())
else:print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
https://en.xdnf.cn/q/119101.html

Related Q&A

Fetching images from URL and saving on server and/or Table (ImageField)

Im not seeing much documentation on this. Im trying to get an image uploaded onto server from a URL. Ideally Id like to make things simple but Im in two minds as to whether using an ImageField is the b…

Comparing list with a list of lists

I have a list string_array = [1, 2, 3, 4, 5, 6] and a list of lists multi_list = [[1, 2], [2, 3], [2, 4], [4, 5], [5, 6]]The first element of each sub-list in multi_list will have an associated entry …

Cannot save data to database Python

I have a table called category TABLES["category"] = ("""CREATE TABLE category (category_id INTEGER NOT NULL AUTO_INCREMENT,category_name VARCHAR(120) NOT NULL,PRIMARY KEY (cate…

How to generate a permutation of list of lists in python

I have a list of lists say[[2, 4, 6], [2, 6, 10], [2, 12, 22], [4, 6, 8], [4, 8, 12], [6, 8, 10], [8, 10, 12], [8, 15, 22], [10, 11, 12]]How do I generate a combination of the lists for a given length?…

Issue sending file via Discord bot (Python)

if message.content.upper().startswith("!HEADPATS"):time.sleep(1)with open(tenor.gif, rb) as picture:await client.send_file(channel, picture)Ive got my discord bot up and running (everythings …

Matplotlib installation on Mavericks

Im having problem while installing matplotlib. Im using Mavericks and it complains about a deprecated NumPy API both installing via pip and installing from source (following the instructions here https…

Exact string search in XML files?

I need to search into some XML files (all of them have the same name, pom.xml) for the following text sequence exactly (also in subfolders), so in case somebody write some text or even a blank, I must …

Integrate a function by the trapezoidal rule- Python

Here is the homework assignment Im trying to solve:A further improvement of the approximate integration method from the last question is to divide the area under the f(x) curve into n equally-spaced tr…

Kivy module not found in vscode (Mac)

I have installed Kivy and when I used the IDLE app that came with Python I can import it and it runs perfectly. However, when I try to import it in vscode I get the error: ModuleNotFoundError: No modul…

How to get latest unique entries from sqlite db with the counter of entries via Django ORM

I have a SQLite db which looks like this:|ID|DateTime|Lang|Details| |1 |16 Oct | GB | GB1 | |2 |15 Oct | GB | GB2 | |3 |17 Oct | ES | ES1 | |4 |13 Oct | ES | ES2 | |5 |15 Oct | ES | ES3 …