I'm attempting to log into a website using Python Selenium using the following code:

import time
from contextlib import contextmanager
from selenium import webdriver
from selenium.webdriver.chrome.options import Options@contextmanager
def getBrowser(*options):chrome_options = Options()if options: [chrome_options.add_argument(option) for option in options]browser = webdriver.Chrome(chrome_options=chrome_options)try:yield browserfinally:browser.quit()with getBrowser() as browser:browser.get('https://www.vinted.com/members/notifications')time.sleep(20)browser.find_element_by_xpath('//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span').click()

It works perfectly, however when I add the --headless option to the browser, it raises a NoSuchElementException.

Error raising code:

with getBrowser('--headless') as browser:browser.get('https://www.vinted.com/members/notifications')time.sleep(20)browser.find_element_by_xpath('//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span').click()

Traceback:

Traceback (most recent call last):File "<ipython-input-4-fe0834deb137>", line 1, in <module>runfile('C:/Users/Alec/vinted test case.py', wdir='C:/Users/Alec')File "C:\Users\Alec\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfileexecfile(filename, namespace)File "C:\Users\Alec\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfileexec(compile(f.read(), filename, 'exec'), namespace)File "C:/Users/Alec/vinted test case.py", line 27, in <module>browser.find_element_by_xpath('//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span').click()File "C:\Users\Alec\selenium\webdriver\remote\webdriver.py", line 354, in find_element_by_xpathreturn self.find_element(by=By.XPATH, value=xpath)File "C:\Users\Alec\selenium\webdriver\remote\webdriver.py", line 832, in find_element'value': value})['value']File "C:\Users\Alec\selenium\webdriver\remote\webdriver.py", line 297, in executeself.error_handler.check_response(response)File "C:\Users\Alec\selenium\webdriver\remote\errorhandler.py", line 194, in check_responseraise exception_class(message, screen, stacktrace)NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span"}(Session info: headless chrome=65.0.3325.181)(Driver info: chromedriver=2.36.540470 (e522d04694c7ebea4ba8821272dbef4f9b818c91),platform=Windows NT 10.0.16299 x86_64)

This error only occurs when the browser is headless. What's causing this behavior? Can it be made to work in headless mode?

target HTML:

<div class="u-flex-grow"><a class="c-button--inverse c-button--normal c-button--amplified c-button " href="/member/general/login?ref_url=%2Fmembers%2Fnotifications"><span class="c-button__content">Log In</span></a></div>

I had the same issue. The solution was to add usr-agent:

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'
chrome_options.add_argument('user-agent={0}'.format(user_agent))

I found the reason by printing the page

browser.get(url)
print(browser.page_source)

then the return html says permission deny, so I google it. Follow the solution: How to access a site via a headless driver without being denied permission