Scraping Dynamic Information

2024/10/5 19:46:24

I recently started with coding, I use Python and Pycharm. I Installed and imported the needed "Add-ons" like Selenium.
For my first project I tried to get the "address" information from this website:

https://randomstreetview.com/#fullscreen

When I use the Chrome developer tools (F12) the information shows up after the div'address'. After lots of trial and error I managed to scrape the website in multiple ways but nothing ever showed up behind the div.
Due to that I looked it up and think it's a dynamic information, so I started trying to use Selenium and other code that I found online.
Nothing worked or I didn't manage to get it working.
The address changes everytime when you open the website but the place stays the same (afaik).
Can anyone help me with a working code or help me to get on the right way?
I could provide my different approaches but I won't think that would help.
Answer

If you want the element's address just get the element and print it's text.

driver.get("https://randomstreetview.com/")
wait = WebDriverWait(driver, 10)
elem = wait.until(EC.presence_of_element_located((By.ID, "address")))
print(elem.text)

Element

<div id="address">Nordre Ringvej 97, 2600 Glostrup, Dänemark</div>

Outputs

Nordre Ringvej 97, 2600 Glostrup, Dänemark

Imports

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
https://en.xdnf.cn/q/120105.html

Related Q&A

Combining three RGB images into a single RGB image

I have three RGB images, but each one has only 1 non-zero channel (ie. one has a red channel with 0s in the blue and green channels) and I want to combine them into a single RGB image with the correct …

Parse XML to Table in Python

Im trying to parse XML to table-like structure in Python. Imagine XML like this:<?xml version="1.0" encoding="UTF-8"?> <base><element1>element 1</element1>…

Use Artificial Intelligence to predict next number (n+1) in a sequence

The AI must predict the next number in a given sequence of incremental integers using Python, but so far I havent gotten the intended result. I tried changing the learning rate and iterations but so fa…

Calculating size folder python [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.Want to improve this question? Update the question so it focuses on one problem only by editing this post.Closed 8…

Answer Error, Only outputting Zero

I am coding in python and I cannot seem to figure out why when the amount of tickets sold is entered it does not calculate the full price of the tickets that were sold. Any help is appreciated, Thanks.…

finding a pattern match and concatenating the rest of lines in python

I have a small data set to clean. I have opened the text file in Pycharm. The data set is like this:Code-6667+ Name of xyz company+ Address + Number+ Contact person+ Code-6668+ Name of abc company, A…

Flat a list of containing only one dict

I have a dict which contain a list product which will contain only one dict: d = {"thickness": 90.0,"mass_surf": 37.8,"res_therm": 0.75,"codename": "codenam…

How to breakup a list of list in a given way in Python [closed]

Its difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying thi…

Incrementing a counter while assigning it in python

Is it possible to do the following in python somehow?nodename = node%s % (++num if ct[0] != J else num)Or do I need to first do the increment and the assign it on the next line:if ct[0] != J:num += 1n…

Append two arrays together into one? (Numpy/Python)

I currently have an array of strings and Im trying to join it together with another array of strings to form a complete word to do some web parsing. For example:`Var1 [A B C, .... ....] Var2 …