So basically: the goal is to click on each symbol for each sector on this website, that pops up a table with contact details, I want to copy all of that information and store it in a file. Right now everything else has been tested and is working correctly. I just can't get Selenium to nested loop through sector and then all the symbols correctly.
I've tried this code with hard-coded parameters and it works perfectly, I now just need to pass on the values by reference for each sector/symbol so that I can loop through the entire site.
Can see the format of the website and the 2 dialogue box things here. Am pasting the full workable code below but it's this bit thats causing me issues and doesnt loop through the site correctly:
sector = Select(driver.find_element_by_id("sector"))
symbol = Select(driver.find_element_by_id("Symbol"))for sec in sector.options[1:]:sec.click()for sym in symbol.options[1:]:table = sym.text
It behaves very very erratically, I tried running through it stepwise with sleep commands but even I can't make sense of how it's operating right now, the rest of the operations are working fine I've tested them out by inputting manual tickers. I just need to select a sector, loop through all of the symbols in it, then select the next sector and all of it's symbols then and so on and so forth.
Code block for reproducing:
chromedriver = "chromedriver.exe"
driver = webdriver.Chrome(chromedriver)
driver.get("https://www.psx.com.pk/psx/resources-and-tools/listings/listed-companies")
driver.maximize_window()sector = Select(driver.find_element_by_id("sector"))
symbol = Select(driver.find_element_by_id("Symbol"))for sec in sector.options[1:]:sec.click()for sym in symbol.options[1:]:table = sym.textwait = WebDriverWait(driver,30)findelement = wait.until(EC.presence_of_element_located((By.ID, table)))findelement.click()# we do this after thingy has been clicked uponwait = WebDriverWait(driver,30)extract = wait.until(EC.presence_of_element_located((By.ID, "addressbookdata")))#close the popupWebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@class='close'][@data-dismiss='modal']"))).click()with open("output.txt", "a") as myfile:myfile.write(extract.get_attribute('innerText'))driver.quit()
You have to have two-loop, especially nested ones.
One for Sector.
Second for Symbol.
Also, first symbol is default selected option which does not really have anything, so got to skip it.
PS: The application is a bit weird, for the first symbol, even manual address book does not show up.
Code:
driver.maximize_window()wait = WebDriverWait(driver, 30)
driver.get("https://www.psx.com.pk/psx/resources-and-tools/listings/listed-companies")sector = Select(wait.until(EC.visibility_of_element_located((By.ID, "sector"))))
symbol = Select(wait.until(EC.visibility_of_element_located((By.ID, "Symbol"))))allSectorOptions = sector.options
for sector_ in allSectorOptions:if sector_.get_attribute('value') != "0":print("Inside sector loop")sector_.click();time.sleep(5)print("Selected sector:", sector_.text)allSymbolOptions = symbol.optionsfor symbol_ in allSymbolOptions:if symbol_.get_attribute('value') != "0":print("Inside symbol loop")attempts = 0while attempts < 3:try:symbol_.click()breakexcept StaleElementReferenceException as stale:print(stale.msg)attempts = attempts + 1print("Selected symbol:", symbol_.text)try:address_data = wait.until(EC.visibility_of_element_located((By.ID, "addressbookdata")))print(address_data.get_attribute('innerText'))wait.until(EC.element_to_be_clickable((By.XPATH, "//button[@class='close'][@data-dismiss='modal']"))).click()except:print("addressbookdata could not found for", symbol_.text)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Output:
Inside sector loop
Selected sector: Automobile assembler
Inside symbol loop
Selected symbol: AGTL
addressbookdata could not found for AGTL
Inside symbol loop
Selected symbol: ATLH
Company Address Book
Atlas Honda LimitedRepresentative Mr. Saquib H. Shirazi
Designation Chief Executive
Company Atlas Honda Limited
Address 1-Mcleod Road , ,Lahore
Phone 042-37225015-17, 37233515-17
Phone 2
Fax 042-37233518-37351119
Date of Listing 1965-10-15
Email [email protected]
URL www.atlashonda.com.pk
Registrar Hameed Majeed Associates (Pvt.) Ltd
Inside symbol loop
Selected symbol: DFML
Company Address Book
Atlas Honda LimitedRepresentative Mr. Saquib H. Shirazi
Designation Chief Executive
Company Atlas Honda Limited
Address 1-Mcleod Road , ,Lahore
Phone 042-37225015-17, 37233515-17
Phone 2
Fax 042-37233518-37351119
Date of Listing 1965-10-15
Email [email protected]
URL www.atlashonda.com.pk
Registrar Hameed Majeed Associates (Pvt.) Ltd
Inside symbol loop
Selected symbol: GHNI
Company Address Book
Atlas Honda LimitedRepresentative Mr. Saquib H. Shirazi
Designation Chief Executive
Company Atlas Honda Limited
Address 1-Mcleod Road , ,Lahore
Phone 042-37225015-17, 37233515-17
Phone 2
Fax 042-37233518-37351119
Date of Listing 1965-10-15
Email [email protected]
URL www.atlashonda.com.pk
Registrar Hameed Majeed Associates (Pvt.) Ltd
Inside symbol loop
Selected symbol: GHNL
Company Address Book
Dewan Farooque Motors LimitedRepresentative Mr. Waseem-ul-Haque Ansari
Designation Chief Executive Officer
Company Dewan Farooque Motors Limited
Address Dewan Centre, 3-A, Lalazar , Beach Hotel Road,Karachi
Phone 111-364-111
Phone 2
Fax 021-35630860
Date of Listing 2000-07-31
Email [email protected]
URL www.yousufdewan.com
Registrar BMF Consultants Pakistan (Pvt.) Ltd.
Inside symbol loop
Selected symbol: HCAR
Company Address Book
Dewan Farooque Motors LimitedRepresentative Mr. Waseem-ul-Haque Ansari
Designation Chief Executive Officer
Company Dewan Farooque Motors Limited
Address Dewan Centre, 3-A, Lalazar , Beach Hotel Road,Karachi
Phone 111-364-111
Phone 2
Fax 021-35630860
Date of Listing 2000-07-31
Email [email protected]
URL www.yousufdewan.com
Registrar BMF Consultants Pakistan (Pvt.) Ltd.
Inside symbol loop
Selected symbol: HINO
Company Address Book
Ghandhara Industries LimitedRepresentative Mr. Ahmed Kuli Khan Khattak
Designation Chief Executive Officer
Company Ghandhara Industries Limited
Address F-3, Hub Chauki Road , S.I.T.E. Post Box No. 2706,Karachi
Phone 021-38709000
Phone 2 021-32560083-86
Fax 021-32560090-32564458
Date of Listing 1963-09-03
Email [email protected]
URL www.gil.com.pk
Registrar CDC Share Registrar Services Limited
Inside symbol loop
Selected symbol: INDU
Company Address Book
Hinopak Motors LimitedRepresentative Mr. Takayuki Kizawa
Designation President & Chief Executive Officer
Company Hinopak Motors Limited
Address D-2, S.I.T.E , Manghopir Road,Karachi
Phone 021-32563510-18, 111-252-525
Phone 2
Fax 021-32563028
Date of Listing 1988-05-25
Email [email protected]
URL www.hinopak.com
Registrar FAMCO Associates (Pvt) Limited
Inside symbol loop
Selected symbol: MTL
Company Address Book
Indus Motor Company LimitedRepresentative Mr. Ali Asghar Abbas Jamali
Designation Chief Executive Officer
Company Indus Motor Company Limited
Address Plot No.NWZ/1/P-1 , Port Qasim Authority, Bin Qasim,Karachi
Phone 021-34720041-48-34721100
Phone 2 021-111-869-682
Fax 021-34720056-34720054
Date of Listing 1992-06-29
Email [email protected]
URL www.toyota-indus.com
Registrar CDC Share Registrar Services Limited
Inside symbol loop
Selected symbol: PSMC
Company Address Book
Millat Tractors LimitedRepresentative Syed Muhammad Irfan Aqueel
Designation Chief Executive
Company Millat Tractors Limited
Address Sheikhupura Road , Distt: Shahdara,Lahore
Phone 042-111-200-786, 37911021-25
Phone 2
Fax 042-37925835-37924166
Date of Listing 1965-02-02
Email [email protected]
URL www.millat.com.pk
Registrar Hameed Majeed Associates (Pvt.) Ltd
addressbookdata could not found for PSMC
Inside symbol loop
Selected symbol: SAZEW
Company Address Book
Pak Suzuki Motor Company LimitedRepresentative Mr. Masafumi Harano
Designation Chief Executive Officer
Company Pak Suzuki Motor Company Limited
Address DSU-13, Pakistan Steel Industrial Estates , Bin Qasim,Karachi
Phone 021-34723552-58
Phone 2 021-34723532 (Direct)
Fax 021-34723521-23
Date of Listing 1985-09-10
Email [email protected]
URL www.paksuzuki.com.pk
Registrar CDC Share Registrar Services Limited
Inside symbol loop
Selected symbol: SAZEWR
Company Address Book
Pak Suzuki Motor Company LimitedRepresentative Mr. Masafumi Harano
Designation Chief Executive Officer
Company Pak Suzuki Motor Company Limited
Address DSU-13, Pakistan Steel Industrial Estates , Bin Qasim,Karachi
Phone 021-34723552-58
Phone 2 021-34723532 (Direct)
Fax 021-34723521-23
Date of Listing 1985-09-10
Email [email protected]
URL www.paksuzuki.com.pk
Registrar CDC Share Registrar Services Limited
Inside sector loop
Selected sector: Automobile parts & accessories
Inside symbol loop
Selected symbol: AGIL
addressbookdata could not found for AGIL
Inside symbol loop
Selected symbol: ATBA
addressbookdata could not found for ATBA
Inside symbol loop
Selected symbol: BELA
Company Address Book
Bela Automotives LimitedRepresentative Mr. Abdul Mateen Allahwala
Designation Chief Executive
Company Bela Automotives Limited
Address Plot No.1 & 3 Mouza Pathra , Hub Chowki,Baluchistan
Phone
Phone 2 Mob.0300-8271368
Fax
Date of Listing 1994-09-27
Email [email protected]
URL
Registrar M/s. JWAFFS Registrar Services (Pvt.) Limited
Inside symbol loop
Selected symbol: BWHL
Company Address Book
Bela Automotives LimitedRepresentative Mr. Abdul Mateen Allahwala
Designation Chief Executive
Company Bela Automotives Limited
Address Plot No.1 & 3 Mouza Pathra , Hub Chowki,Baluchistan
Phone
Phone 2 Mob.0300-8271368
Fax
Date of Listing 1994-09-27
Email [email protected]
URL
Registrar M/s. JWAFFS Registrar Services (Pvt.) Limited
addressbookdata could not found for BWHL
Inside symbol loop
Selected symbol: DWAE
Company Address Book
Baluchistan Wheels LimitedRepresentative Mr. Razzak H.M. Bengali
Designation Chief Executive
Company Baluchistan Wheels Limited
Address 1st Floor, State Life Building No.3 , Dr. Ziauddin Ahmed Road,Karachi
Phone 021-35689259-35687502
Phone 2
Fax 021-35684003
Date of Listing 1994-01-18
Email [email protected]
URL www.bwheels.com
Registrar THK Associates (Private) Limited
Inside symbol loop
Selected symbol: EXIDE
Company Address Book
Baluchistan Wheels LimitedRepresentative Mr. Razzak H.M. Bengali
Designation Chief Executive
Company Baluchistan Wheels Limited
Address 1st Floor, State Life Building No.3 , Dr. Ziauddin Ahmed Road,Karachi
Phone 021-35689259-35687502
Phone 2
Fax 021-35684003
Date of Listing 1994-01-18
Email [email protected]
URL www.bwheels.com
Registrar THK Associates (Private) Limited
Inside symbol loop
Selected symbol: GTYR
Company Address Book
Baluchistan Wheels LimitedRepresentative Mr. Razzak H.M. Bengali
Designation Chief Executive
Company Baluchistan Wheels Limited
Address 1st Floor, State Life Building No.3 , Dr. Ziauddin Ahmed Road,Karachi
Phone 021-35689259-35687502
Phone 2
Fax 021-35684003
Date of Listing 1994-01-18
Email [email protected]
URL www.bwheels.com
Registrar THK Associates (Private) Limited
Inside symbol loop
Selected symbol: LOADS
Company Address Book
Dewan Automotive Engineering LimitedRepresentative Mr. Waseem-ul-Haque Ansari
Designation Chief Executive Officer
Company Dewan Automotive Engineering Limited
Address Dewan Centre, 3-A, Lalazar , Beach Hotel Road,Karachi
Phone 111-364-111
Phone 2
Fax 021-35630860
Date of Listing 1986-09-02
Email [email protected]
URL www.yousufdewan.com
Registrar BMF Consultants Pakistan (Pvt.) Ltd.
Inside symbol loop
Selected symbol: PTL
Company Address Book
Ghandhara Tyre and Rubber Company Limited (Formerly: General Tyre)Representative Mr. Hussain Kuli Khan Khattak
Designation Chief Executive Officer
Company Ghandhara Tyre and Rubber Company Limited (Formerly: General Tyre)
Address H-23/2, Landhi Industrial Trading Estate , Landhi,Karachi
Phone 021-111-487-487
Phone 2 021-35080172-81
Fax 021-35080171
Date of Listing 1982-06-21
Email [email protected]
URL www.generaltyre.com.pk
Registrar CDC Share Registrar Services Limited
Inside symbol loop
Selected symbol: THALL
Company Address Book
Ghandhara Tyre and Rubber Company Limited (Formerly: General Tyre)Representative Mr. Hussain Kuli Khan Khattak
Designation Chief Executive Officer
Company Ghandhara Tyre and Rubber Company Limited (Formerly: General Tyre)
Address H-23/2, Landhi Industrial Trading Estate , Landhi,Karachi
Phone 021-111-487-487
Phone 2 021-35080172-81
Fax 021-35080171
Date of Listing 1982-06-21
Email [email protected]
URL www.generaltyre.com.pk
Registrar CDC Share Registrar Services Limited
addressbookdata could not found for THALL
Inside sector loop
Selected sector: Bonds
Inside symbol loop
Selected symbol: AGSILSC
addressbookdata could not found for AGSILSC
Inside symbol loop
Selected symbol: AKBLTFC6
addressbookdata could not found for AKBLTFC6
Inside symbol loop
Selected symbol: AKBLTFC7
addressbookdata could not found for AKBLTFC7
Inside symbol loop
Selected symbol: AZLCLTFC2
addressbookdata could not found for AZLCLTFC2
Inside symbol loop
Selected symbol: BAFLTFC6
addressbookdata could not found for BAFLTFC6
Inside symbol loop
Selected symbol: BAFLTFC7
addressbookdata could not found for BAFLTFC7
Inside symbol loop
Selected symbol: BIPLSC
addressbookdata could not found for BIPLSC
Inside symbol loop
Selected symbol: CNERGYSC
addressbookdata could not found for CNERGYSC
Inside symbol loop
Selected symbol: EPCLSC
addressbookdata could not found for EPCLSC
Inside symbol loop
Selected symbol: HBLTFC2
addressbookdata could not found for HBLTFC2
Inside symbol loop
Selected symbol: HUBCSC2
addressbookdata could not found for HUBCSC2
Inside symbol loop
Selected symbol: HUBCSC4
addressbookdata could not found for HUBCSC4
Inside symbol loop
Selected symbol: HUBPHLSC
addressbookdata could not found for HUBPHLSC
Inside symbol loop
Selected symbol: JSBLTFC1