Python dryscrape scrape page with cookies

2024/10/14 16:23:38

I wanna get some data from site, which requires loggin in.
I log in by requests

url = "http://example.com"
response = requests.get(url, {"email":"[email protected]", "password":"12345"})
cookies = response.cookies

Then I wanna get data from some JS page. Through requests it isn't possible, so I have to use dryscrape for this.

import dryscrape
url = "http://example.com/js-page"
sess = dryscrape.Session()
sess.visit(url)

Is it possible to pass cookies to visit() or I have to look for another solution?

Answer

Why not login by dryscrape?

session = dryscrape.Session()
session.visit('<url_where_is_login_form>')
name = session.at_xpath('//*[@name="username"]') # Where <input name="username">
name.set("<login>")
password = session.at_xpath('//*[@name="password"]') # Where <input name="password">
password.set("<password>")
# Push the button
name.form().submit()
session.visit("<url to visit with proper cookies>")
https://en.xdnf.cn/q/69394.html

Related Q&A

Python retry using the tenacity module

Im having having difficulty getting the tenacity library to work as expected. The retry in the following test doesnt trigger at all. I would expect a retry every 5 seconds and for the log file to refle…

How to write own logging methods for own logging levels

Hi I would like to extend my logger (taken by logging.getLogger("rrcheck")) with my own methods like: def warnpfx(...):How to do it best? My original wish is to have a root logger writing …

How to use pandas tz_convert to convert to multiple different time zones

I have some data as shown below with hour in UTC. I want to create a new column named local_hour based on time_zone. How can I do that? It seems like pandas tz_convert does not allow a column or panda…

virtualenv, python and subversion

Im trying to use the python subversion SWIG libraries in a virtualenv --no-site-packages environment. How can I make this work?

Float to Fraction conversion in Python

While doing exercise on the topic of float type to Fraction type conversion in Python 3.52, I found the difference between the two different ways of conversion.The first method is:>>> from fra…

How to update an SVM model with new data

I have two data set with different size.1) Data set 1 is with high dimensions 4500 samples (sketches).2) Data set 2 is with low dimension 1000 samples (real data). I suppose that "both data set ha…

Expanding NumPy array over extra dimension

What is the easiest way to expand a given NumPy array over an extra dimension?For example, suppose I have>>> np.arange(4) array([0, 1, 2, 3]) >>> _.shape (4,) >>> expand(np.…

Django-Haystack giving attribute error?

I am trying to use Haystack and Whoosh with my Django app. I followed the steps on Haystack docs, but i am getting this error when i do a searchAttributeError at /search/ module object has no attribute…

python calendar with holidays [duplicate]

This question already has answers here:Closed 12 years ago.Possible Duplicate:Holiday Calendars, File Formats, et al. Hi, Is there a calendar library in Python with which I can check for holidays, com…

How to choose your conda environment in Jupyter Notebook

I installed Anaconda 5.3 with Python 3.7 (root environment). After that I created a new environment (py36) using Python 3.6I activated the new environment with activate py36 conda env list shows that t…