selenium scraping data using children of elements

2024/10/7 10:16:28

Hi i'm trying to scrape some data from a live stocks website. I want to display the companies name and stock price, %change ect. The details of 25 companies are shown per page, and these details follow the same format, one row of data comes under the tag, have the same css selector and then the rest of the elements e.g. Name of the company share price, %change ect are children of that tag. Since all data follows the same format, my question is how do i filter through each tag displaying data for that row?

I thought about:

using a for loop to find the elements by class name (since they all have the same class) and then cycle through these but I don't how to select a child of that class:

Here' the html code: enter image description here

enter image description here

Here's the link to the site https://uk.finance.yahoo.com/most-active

Answer

Here ya go man! I had similar issues when trying to scrape data from a similar stock site

from bs4 import BeautifulSoup
from urllib.request import urlopen as uRequrl = "https://uk.finance.yahoo.com/most-active"
uClient = uReq(url)
html_page = uClient.read()
uClient.close()soup = BeautifulSoup(html_page, 'html.parser')
full_table = soup.find_all("tr", class_="simpTblRow")for tr in full_table:td = tr.find_all('td')row = [i.text for i in td]print(row)

You can make your own data structure to hold each of the printed values, it runs with this output (without labels, the output looks confusing but you can look at the site for reference)

['ICON.L', 'Iconic Labs Plc', '0.0132', '0.0000', '0.00%', '1.291B', '412.755M', 
'1.299M', 'N/A', '']
['SYME.L', 'Supply@ME Capital plc', '0.8020', '+0.2320', '+40.70%', '1.145B', 
'173.333M', '262.694M', 'N/A', '']
['UJO.L', 'Union Jack Oil plc', '0.2925', '-0.0075', '-2.50%', '210.666M', '154.613M', 
'45.165M', 'N/A', '']
['TOM.L', 'TomCo Energy Plc', '0.8750', '+0.0750', '+9.37%', '197.389M', '49.702M', 
'5.894M', 'N/A', '']
['PREM.L', 'Premier African Minerals Limited', '0.0680', '-0.0045', '-6.21%', 
'193.547M', '136.558M', '8.542M', 'N/A', '']
['COPL.L', 'Canadian Overseas Petroleum Limited', '0.4000', '-0.0200', '-4.76%', 
'139.094M', '211.933M', '9.549M', 'N/A', '']
['MSMN.L', 'Mosman Oil And Gas Limited', '0.1800', '+0.0200', '+12.50%', '136.04M', 
'51.481M', '2.854M', 'N/A', '']
['VAST.L', 'Vast Resources plc', '0.2100', '-0.0090', '-4.11%', '133.599M', 
'243.076M', '25.673M', 'N/A', '']
['UKOG.L', 'UK Oil & Gas Investments PLC', '0.2250', '+0.0150', '+7.14%', '114.909M', 
'275.501M', '24.974M', 'N/A', '']
['7DIG.L', '7digital Group plc', '2.8000', '-0.3000', '-9.68%', '111.529M', '23.568M', 
'68.752M', 'N/A', '']
['ALBA.L', 'Alba Mineral Resources PLC', '0.0925', '-0.0125', '-11.90%', '99.299M', 
'39.579M', '4.472M', 'N/A', '']
['LLOY.L', 'Lloyds Banking Group plc', '28.18', '-0.44', '-1.54%', '98.424M', 
'263.235M', '19.948B', '70.46', '']
['KOD.L', 'Kodal Minerals Plc', '0.0475', '0.0000', '0.00%', '111.583M', '75.563M', 
'5.278M', 'N/A', '']
['UFO.L', 'Alien Metals Limited', '0.3600', '+0.0500', '+16.13%', '84.677M', 
'52.319M', '8.516M', 'N/A', '']
['EDL.L', 'Edenville Energy Plc', '0.0500', '-0.0025', '-4.76%', '80.171M', '47.879M', 
'4.073M', 'N/A', '']
['EQT.L', 'EQTEC plc', '0.5300', '+0.0650', '+13.98%', '73.296M', '144.02M', 
'36.929M', 'N/A', '']
['BOIL.L', 'Baron Oil Plc', '0.1000', '-0.0025', '-2.44%', '73.45M', '71.853M', 
'4.426M', 'N/A', '']
['BREE.L', 'Breedon Group plc', '83.20', '+2.20', '+2.72%', '63.05M', '4.616M', 
'1.403B', '43.79', '']
['AMGO.L', 'Amigo Holdings PLC', '9.30', '+0.55', '+6.29%', '62.892M', '20.841M', 
'44.206M', 'N/A', '']
['INSP.L', 'Inspirit Energy Holdings Plc', '0.0800', '+0.0025', '+3.23%', '61.622M', 
'73.243M', '2.323M', 'N/A', '']
['TRAF.L', 'Trafalgar Property Group plc', '0.2000', '0.0000', '0.00%', '55.959M', 
'40.707M', '2.85M', 'N/A', '']
['RBD.L', 'Reabold Resources plc', '0.7300', '-0.0350', '-4.58%', '51.697M', 
'23.499M', '51.808M', 'N/A', '']
['CPI.L', 'Capita plc', '28.70', '-7.18', '-20.01%', '48.494M', '12.711M', '478.994M', 
'N/A', '']
['GGP.L', 'Greatland Gold plc', '12.20', '-0.80', '-6.15%', '44.829M', '33.176M', 
'461.41M', 'N/A', '']
['SRES.L', 'Sunrise Resources plc', '0.3000', '-0.0100', '-3.23%', '39.45M', 
'15.895M', '9.963M', 'N/A', '']
https://en.xdnf.cn/q/118835.html

Related Q&A

Python - ETFs Daily Data Web Scraping

Im trying to web scrape some daily info of differents ETFs. I found that https://www.marketwatch.com/ have a accurate info. The most relevant info is the open Price, outstanding shares, NAV, total asse…

How to create DataFrame with columns based on scraped data?

import requests, re from bs4 import BeautifulSoupdata = []soup = BeautifulSoup(requests.get(https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4ARfIAQzYAQHoAQH4AQKI…

How do i change the colour of a button border tkinter

How do i change the colour of a border in tkinterI have looked at other solutions which recommended using highlightcolor and highlightbackground, however these did not work. excercises_button = Button(…

module object has no attribute Gridspec despite calling help(gridspec) revealing the Gridspec class

If I run the python console and doimport matplotlib matplotlib.__version__ import matplotlib.gridspec as gsI see that the matplotlib version is 1.2.1.If I do help(gs) I see the Gridspec class.However t…

Python division doesnt work as expected for large numbers [duplicate]

This question already has answers here:What class to use for money representation?(6 answers)Closed 9 months ago.I have three variables a, b and c. I want to make sure that after doing this: c -= a*bc…

working out an average of the values in a dictionary

My dictionary as of now is like this:class_1 = {Bob:[9,5,4,3,3,4], John:[5,5,7,3,6], Andy:[7,5,6,4,5], Harris:[3,4,2,3,2,3,2]}What i am trying to make it look like is this:class_1_average ={Bob:[averag…

getting an error when trying to import a list into a mysql table

whenever i try to add a list into the mysql table I get an error : ProgrammingError: Not all parameters were used in the SQL statementive tried to look online but all i could found is that i need to us…

Getting a view does not return a valid response error message on my flask chatbot [duplicate]

This question already has answers here:Flask view return error "View function did not return a response"(3 answers)Closed 3 years ago.Trying to create a whatsapp bot on Twilio that limits the…

Django how to add data to Object from queryset

I would like show list of clients and show tags assigned to them but I have problem because I have my tags in other table and I dont know how to connect data together. Clients can have couple of tags o…

before_action ... only: how to do this in python flask? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.Want to improve this question? Add details and clarify the problem by editing this post.Closed 5 years ago.Improve…