How to give delay between each requests in scrapy?

2024/11/21 2:25:13

I don't want to crawl simultaneously and get blocked. I would like to send one request per second.

Answer

There is a setting for that:

DOWNLOAD_DELAY
Default: 0
The amount of time (in secs) that the downloader should wait beforedownloading consecutive pages from the same website. This can be usedto throttle the crawling speed to avoid hitting servers too hard.

DOWNLOAD_DELAY = 0.25    # 250 ms of delay

Read the docs: https://doc.scrapy.org/en/latest/index.html

https://en.xdnf.cn/q/26248.html

Related Q&A

preprocess_input() method in keras

I am trying out sample keras code from the below keras documentation page, https://keras.io/applications/What preprocess_input(x) function of keras module does in the below code? Why do we have to do …

How to calculate precision and recall in Keras

I am building a multi-class classifier with Keras 2.02 (with Tensorflow backend)，and I do not know how to calculate precision and recall in Keras. Please help me.

Django set range for integer model field as constraint

I have a django model,class MyModel(models.Model)qty = model.IntegerField()where I want to set constraint for qty something like this, >0 or <0,i.e the qty can be negative or positive but can no…

Increase resolution with word-cloud and remove empty border

I am using word cloud with some txt files. How do I change this example if I wanted to 1) increase resolution and 2) remove empty border. #!/usr/bin/env python2 """ Minimal Example =====…

How can I check if a list index exists?

Seems as thoughif not mylist[1]:return FalseDoesnt work.

Check if space is in a string

in word == TrueIm writing a program that checks whether the string is a single word. Why doesnt this work and is there any better way to check if a string has no spaces/is a single word..

Django F expressions joined field

So I am trying to update my model by running the following: FooBar.objects.filter(something=True).update(foobar=F(foo__bar))but I get the following error: FieldError: Joined field references are not pe…

How do I unit test PySpark programs?

My current Java/Spark Unit Test approach works (detailed here) by instantiating a SparkContext using "local" and running unit tests using JUnit.The code has to be organized to do I/O in one f…

Sorting by arbitrary lambda

How can I sort a list by a key described by an arbitrary function? For example, if I have:mylist = [["quux", 1, "a"], ["bar", 0, "b"]]Id like to sort "myl…

Add quotes to every list element

Im very new to python. I need a simple and clear script to add quotes to every list elements. Let me explain more. Here is the my code.parameters = [a, b, c] query = "SELECT * FROM foo WHERE bar I…

Latest Q&A