Dynamic Web scraping using Selenium

Question

I am trying to scrape the tables from the below dynamic webpage. I am using the below code to find the data in tables (they are under tag name tr). But I am getting empty list as output. Is there anything that I am missing here?

https://www.taipower.com.tw/tc/page.aspx?mid=206&cid=406&cchk=b6134cc6-838c-4bb9-b77a-0b0094afd49d

from selenium import webdriver
chrome_path = r"C:\Users\upko\Downloads\My Projects\Ibrahim Projects\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://www.taipower.com.tw/tc/page.aspx?mid=206&cid=406&cchk=b6134cc6-838c-4bb9-b77a-0b0094afd49d")
driver.find_elements_by_tag_name('tr')

Answer 1

Website have iframes, you need switch into desired iframe to access data. Didnt tested code, but should work

iframe = driver.find_element_by_xpath("//iframe[@id='IframeId']")
driver.switch_to_frame(iframe)

#Now you can get data
trs = driver.find_elements_by_tag_name('tr')

Answer 2

The desired elements are within an <iframe> so you have to:

• Induce WebDriverWait for the desired frame to be available and switch to it.

• Induce WebDriverWait for the desired visibility_of_all_elements_located.

• You can use either of the following Locator Strategies:

• Using XPATH:

  driver.get("https://www.taipower.com.tw/tc/page.aspx?mid=206&cid=406&cchk=b6134cc6-838c-4bb9-b77a-0b0094afd49d")
  WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"//iframe[@id='IframeId']")))
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='container-fluid']//div[@class='span6']/strong")))])
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Console Output:

  ['??(Nuclear)', '??(Coal)', '????(Co-Gen)', '????-??(IPP-Coal)', '??(LNG)', '????-??(IPP-LNG)', ' ??(Oil)', '??(Diesel)', '??(Hydro)', '??(Wind)', '???(Solar)', '????(Pumping Gen)']
  

---

Reference

You can find a couple of relevant discussions in:

• Switch to an iframe through Selenium and python