I want to download U.S. Department of Housing and Urban Development data using Python's Selenium. Here's my code.

import os
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Selectoptions = webdriver.ChromeOptions()
preferences= {"download.default_directory": os.getcwd(), "directory_upgrade": True}
options.add_experimental_option("prefs", preferences)
#options.headless = True
options.add_experimental_option('excludeSwitches', ['enable-logging'])url = "https://hudgis-hud.opendata.arcgis.com/datasets/deteriorated-paint-index-by-county/explore"# Path of my WebDriver
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)wait = WebDriverWait(driver, 60)# to maximize the browser window
driver.maximize_window()#get method to launch the URL
driver.get(url)paths = ["#ember97", "calcite-card > div > calcite-button"]for x in paths:wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, x))).click()

I can click the button to expand the side panel, where the CSV file button is located, but I cannot click the CSV file itself to download it. My first thought was to check for if the side panel existed within an IFRAME, so I did

seq = driver.find_elements_by_tag_name('iframe')
seq

And it returned nothing. The content is nested in a class called side-panel-ref. Is there a way to switch to this somehow so I can click that content, when iframes aren't there? What might I be missing?

Your button is inside a shadowroot.

You see this when you inspect in devtools:

Quickest and easiest way to handle this is with some JS .This is your script slightly refactored + the JS call:

url = "https://hudgis-hud.opendata.arcgis.com/datasets/deteriorated-paint-index-by-county/explore"
wait = WebDriverWait(driver, 60)
driver.maximize_window()
driver.get(url)wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#ember97'))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.dataset-download-card > hub-download-card")))driver.execute_script('document.querySelector("div.dataset-download-card > hub-download-card").shadowRoot.querySelector("calcite-card > div > calcite-button").click()')

It's a fairly lengthy JS call. It's reasonably self explanatory if you read it, there are 5 parts to it: document.querySelector(..).shadowRoot.querySelector(..).click() - but just ask if you need more support.

Please also be aware that selenium is bad at downloading files. There's no API that exposes the downloads progress. You'll need to ensure your browser remains open while you download the file.

It seems a pretty quick download so you might get away with a hard coded sleep.

Also worth a mention - If you're not a fan of the long JS, you can also break it down like so:

container = driver.find_element_by_css_selector("div.dataset-download-card > hub-download-card")
shadowRoot = driver.execute_script("return arguments[0].shadowRoot", container)
shadowRoot.find_element_by_css_selector("calcite-card > div > calcite-button").click()