Scrapy get result in shell but not in script

2024/10/12 0:24:32

one topic again ^^ Based on recommendations here, I've implemented my bot the following and tested it all in shell :

    name_list = response.css("h2.label.title::text").extract()packaging_list = response.css("div.label.packaging::text").extract()ean = response.css("h1.page-title::text").extract_first()product_price = ''.join(response.css('.product-pricing__main-price  ::text').extract())company = "carrefour"for name, packaging, price in zip(name_list, packaging_list, product_price):item = ScrapybotItem()item['ean'] = eanitem['desc'] = name.replace("\n","").strip() + " " +  packagingitem['price'] = priceitem['company'] = companyyield item

Problem is with price field.

For price in shell, I have for instance :

In [2]: product_price
Out[2]: '\n                    5,65€\n\n  \n      '

Output from script for same product :

{'company': 'carrefour',
'desc': "Gel nettoyant anti-imperfections 5 en 1 L'Oréal Paris Men Expert le "'tube de 150ml','ean': '\n  1 résultat pour « 3600522418634 »\n','price': '\n'}

Do you know why don't I get result for prices with script ?

Answer

product_price is a string, given that you are joining the results of the selector in:

product_price = ''.join(response.css('.product-pricing__main-price  ::text').extract())

Then, when you use zip, you'll be splitting that string in parts, thus you'll have the \n for the first item, as it's probably the first character in product_price.

Check this example:

>>> for i, j, k in zip([1, 2, 3, 4], [5, 6, 7, 8], 'abcd'):print (i, j, k)

Output:

1 5 a
2 6 b
3 7 c
4 8 d
https://en.xdnf.cn/q/118259.html

Related Q&A

How to find a source when a website uses javascript

What I want to achieve I am trying to scrape the website below using Beautiful-soup and when I load the page it does not give the table that shows various quotes. In my previous posts folks have helped…

How to print a list of dicts as an aligned table?

So after going through multiple questions regarding the alignment using format specifiers I still cant figure out why the numerical data gets printed to stdout in a wavy fashion.def create_data(soup_ob…

abstract classes in python: Enforcing type

My question is related to this question Is enforcing an abstract method implementation unpythonic? . I am using abstract classes in python but I realize that there is nothing that stops the user from …

Convert image array to original svs format

Im trying to apply a foreground extraction to a SVS image (Whole Slide Image) usign OpenSlide library.First, I converted my image to an array to work on my foreground extraction:image = np.asarray(oslI…

Printing bytestring via variable

I have the following Unicode text stored in variable:myvariable = Gen\xe8veWhat I want to do is to print myvariable and show this:GenveI tried this but failed:print myvariable.decode(utf-8)Whats the ri…

Loop and arrays of strings in python

I have the following data set:column1HL111 PG3939HL11 HL339PG RC--HL--PGI am attempting to write a function that does the following:Loop through each row of column1 Pull only the alphabet and put into…

2 Dendrograms + Heatmap from condensed correlationmatrix with scipy

I try to create something like this: plotting results of hierarchical clustering ontop of a matrix of data in pythonUnfortunatelly when I try to execute the code, I get the following warnings:Warning (…

Iterator example from Dive Into Python 3

Im learning Python as my 1st language from http://www.diveintopython3.net/. On Chp 7, http://www.diveintopython3.net/iterators.html, there is an example of how to use an iterator.import redef build_mat…

Getting a 500 Internal Server Error using render_template and Flask [duplicate]

This question already has answers here:How to debug a Flask app(13 answers)Comments not working in jinja2(2 answers)Closed 5 years ago.I am trying to use Flask to render an HTML template. I had it work…

Bokeh use of Column Data Source and Box_Select

Im lost as to how to set up a Column Data Source so that I can select points from one graph and have the corresponding points highlighted in another graph. I am trying to learn more about how this work…