'extract all items in Google map search first page
I'm interested in Google map scrapping through Python and selenium. I want to scrap all the leads for a specific search.
In the first page 20 leads are displayed, but my code extracts just 8 of them. When I try to zoom out to display all the items in first page, that could not load.
How can I modify that to extract all the items (so on will continued for next pages, but I can't start that coding yet because of this limitation)
here is the first part of my code which just loads the first 8 items
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path=r"D:\\Prohit\\chromedriver.exe")#,chrome_options=options)
# get url
url = "https://www.google.com/maps"
driver.get(url)
what_search_box = driver.find_element(By.ID,"searchboxinput")
what_search_box.send_keys("company eskişehir organize sanayi bölgesi, türkiye")
what_search_box.send_keys(Keys.ENTER)
driver.execute_script("document.body.style.zoom='10%'")
# or
# driver.execute_script("window.scrollTo(0,2800)")
# if manually find the xpath of the last item in first page and try to use that
a = WebDriverWait.until(By.XPATH, "/html/body/div[3]/div[9]/div[9]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]/div[41]")
try:
elem = WebDriverWait(driver, 30).until(
EC.presence_of_element_located((By.XPATH, "/html/body/div[3]/div[9]/div[9]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]/div[41]"))
)
finally:
driver.quit()
but that doesn't work with such an error:
File "C:\ProgramData\Anaconda31\lib\site-packages\selenium\webdriver\support\wait.py", line 89, in until
raise TimeoutException(message, screen, stacktrace)
how can I solve such a way that I can extract all of the items displayed in first page (20 items)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|