Scroll with Keys.PAGE_DOWN in Selenium Python

Question

Hello Every one can any one help me in scrolling https://www.grainger.com/category/black-pipe-fittings/pipe-fittings/pipe-tubing-and-fittings/plumbing/ecatalog/N-qu1?searchRedirect=products

i want to scroll this using

actions = ActionChains(browser)
actions.send_keys(Keys.PAGE_DOWN)
actions.perform()

till it reaches the bottom of the scroll where it will find an element "Load More"

loadMoreButton = browser.find_element_by_css_selector(
                        ".btn.list-view__load-more.list-view__load-more--js")
loadMoreButton.click()

and then ponce clicked the load more button it has to again perform the scroll action and then again the loadmore action until the load more button is not available.

I have to use this page down action as the element does not load until the page is scrolled till the element if anyone could suggest some solution will be of great help

Answer 1

This has worked for me with zero issues...

from selenium.webdriver.common.keys import Keys

driver.find_element_by_tag_name('body').send_keys(Keys.PAGE_DOWN)

Answer 2

To scroll the page https://www.grainger.com/category/black-pipe-fittings/pipe-fittings/pipe-tubing-and-fittings/plumbing/ecatalog/N-qu1?searchRedirect=products till it reaches the bottom of the page where it will find an element with text as View More and then click the element until the element is not available you can use the following solution:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.common.exceptions import StaleElementReferenceException
  from selenium.common.exceptions import TimeoutException
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_argument('disable-infobars')
  browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  browser.get("https://www.grainger.com/category/black-pipe-fittings/pipe-fittings/pipe-tubing-and-fittings/plumbing/ecatalog/N-qu1?searchRedirect=products")
  while True:
      try:
          browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,10).until(EC.visibility_of_element_located((By.XPATH, "//a[@class='btn list-view__load-more list-view__load-more--js' and normalize-space()='View More']"))))
          browser.execute_script("arguments[0].click();", WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='btn list-view__load-more list-view__load-more--js' and normalize-space()='View More']"))))
          print("View More button clicked")
      except (TimeoutException, StaleElementReferenceException) as e:
          print("No more View More buttons")
          break
  browser.quit()
  

• Console Output:

  View More button clicked
  View More button clicked
  No more View More buttons
  

Answer 3

@PedroLobito I am trying to retrieve the product links can you help me in this

---

No need for selenium in this case, just sniff the xhr requests via developer tools and go straight to the gold (json).

---

The url structure for products is as follows:

https://www.x.com/product/anything-Item#

Just add the Item # value in the json object at the end of the url, something like:

1. https://www.x.com/product/anything-5P540
2. https://www.x.com/product/anything-5P541

...

---

py3 example (for py2, just change the format syntax):

import json
import requests

main_cat = "WP7115916"
sub_cat = "4836"

x = requests.get(f"https://www.x.com/product/tableview/GRAINGER-APPROVED-Square-Head-Plugs-{main_cat}/_/N-qu1?searchRedirect=products&breadcrumbCatId={sub_cat}&s_pp=false").json()

for p in x['records']:
    for childs in p['children']:
        for item in json.loads(childs['collapseValues']):
            url = f"https://www.x.com/product/lol-{item['sku']}"
            print(url)

---

https://www.x.com/product/lol-5P540
https://www.x.com/product/lol-5P541
https://www.x.com/product/lol-5P542
https://www.x.com/product/lol-5P543
https://www.x.com/product/lol-5P544
https://www.x.com/product/lol-5P545
https://www.x.com/product/lol-5P546
https://www.x.com/product/lol-5P547
https://www.x.com/product/lol-5P548
...

Answer 4

One of the best method for smooth scrolling...

html = driver.find_element(By.XPATH,'//body')
total_scroled = 0
page_height = driver.execute_script("return document.body.scrollHeight")
while total_scroled < page_height:
    html.send_keys(Keys.PAGE_DOWN)
    total_scroled += 400
    time.sleep(.5)