How to find a source when a website uses javascript

2024/10/12 0:30:23

What I want to achieve

I am trying to scrape the website below using Beautiful-soup and when I load the page it does not give the table that shows various quotes. In my previous posts folks have helped me providing the website that actually fed the main website but I am not sure how did they find it.Once I have pulled the data I can do the rest.

Website

https://www.cmegroup.com/trading/energy/refined-products/methanol-t2-fob-rdam-icis.html

What has been tried.

I tried to use Selenium driver but getting different errors which might need more time and not comfortable using Selenium. Eventually I plan to create an exe that downloads the information to excel file.

Answer

Lots of pages in the web uses JS to change the page. These changes are not visible to Beautiful-soup because it doesn't execute JS. I can think of two options:

  • You could use tools like Selenium that actually runs a full fledged browser with JS.
  • You could open the website in Chrome or Firefox, open web inspector than refresh the page. Watch for XHR requests in network tab, you may find the request that brings the data you are looking for. If you found it you could directly load that page instead of the main page.
https://en.xdnf.cn/q/118258.html

Related Q&A

How to print a list of dicts as an aligned table?

So after going through multiple questions regarding the alignment using format specifiers I still cant figure out why the numerical data gets printed to stdout in a wavy fashion.def create_data(soup_ob…
READ MORE...

abstract classes in python: Enforcing type

My question is related to this question Is enforcing an abstract method implementation unpythonic? . I am using abstract classes in python but I realize that there is nothing that stops the user from …
READ MORE...

Convert image array to original svs format

Im trying to apply a foreground extraction to a SVS image (Whole Slide Image) usign OpenSlide library.First, I converted my image to an array to work on my foreground extraction:image = np.asarray(oslI…
READ MORE...

Printing bytestring via variable

I have the following Unicode text stored in variable:myvariable = Gen\xe8veWhat I want to do is to print myvariable and show this:GenveI tried this but failed:print myvariable.decode(utf-8)Whats the ri…
READ MORE...

Loop and arrays of strings in python

I have the following data set:column1HL111 PG3939HL11 HL339PG RC--HL--PGI am attempting to write a function that does the following:Loop through each row of column1 Pull only the alphabet and put into…
READ MORE...

2 Dendrograms + Heatmap from condensed correlationmatrix with scipy

I try to create something like this: plotting results of hierarchical clustering ontop of a matrix of data in pythonUnfortunatelly when I try to execute the code, I get the following warnings:Warning (…
READ MORE...

Iterator example from Dive Into Python 3

Im learning Python as my 1st language from http://www.diveintopython3.net/. On Chp 7, http://www.diveintopython3.net/iterators.html, there is an example of how to use an iterator.import redef build_mat…
READ MORE...

Getting a 500 Internal Server Error using render_template and Flask [duplicate]

This question already has answers here:How to debug a Flask app(13 answers)Comments not working in jinja2(2 answers)Closed 5 years ago.I am trying to use Flask to render an HTML template. I had it work…
READ MORE...

Bokeh use of Column Data Source and Box_Select

Im lost as to how to set up a Column Data Source so that I can select points from one graph and have the corresponding points highlighted in another graph. I am trying to learn more about how this work…
READ MORE...

How Does a Pyqtgraph Export for Three Subplots Look Like?

Using PyQtGraph, I would like to generate three sub plots in one chart and export this chart to a file.As I will repeat this a lot of times, it is quite performance sensitive. Therefore I do not need t…
READ MORE...
Latest Q&A

Copyright @ 2022~2023