I am trying to get description text content of each block on this page

https://twitter.com/search?q=data%20mining&src=typd&vertical=default&f=users.

html for p tag looks like

<p class="ProfileCard-bio u-dir" dir="ltr" data-aria-label-part=""><a href="http://t.co/kwtDyFn6dC" rel="nofollow" dir="ltr" data-expanded-url="http://DataMiningBlog.com" class="twitter-timeline-link" target="_blank" title="http://DataMiningBlog.com"><span class="invisible">http://</span><span class="js-display-url">DataMiningBlog.com</span><span class="tco-ellipsis"><span class="invisible">&nbsp;</span></span></a> covers current challenges, interviews with leading actors and book reviews related to data mining, analytics and data science.</p>

my code:

productDivs = soup.findAll('div', attrs={'class' : 'ProfileCard-content'})
for div in productDivs:print div.find('p', attrs={'class' : 'ProfileCard-bio u-dir'}).text

anything wrong here? Getting exception here

Traceback (most recent call last):File "twitter_user_scrapper.py", line 91, in getImageListprint div.find('p', attrs={'class' : 'ProfileCard-bio u-dir'}).text
AttributeError: 'NoneType' object has no attribute 'text'

The issue might be that some div with class as ProfileCard-content may not have a child p element with class - ProfileCard-bio u-dir , when that happens , the following returns None -

div.find('p', attrs={'class' : ['ProfileCard-bio', 'u-dir']})

And that is the reason you are getting the AttributeError. You should get the return of above and save it in a variable , and check whether its None or not and take the text only if its not None.

Also, you should give class as a list of all the classes , not a single string, as -

attrs={'class' : ['ProfileCard-bio', 'u-dir']}

Example -

productDivs = soup.findAll('div', attrs={'class' : 'ProfileCard-content'})
for div in productDivs:elem = div.find('p', attrs={'class' : ['ProfileCard-bio', 'u-dir']})if elem:print elem.text