Selenium Chrome Webdriver process Working Locally But Not On Heroku

Question

So I have the following Selenium Chrome Webdriver process that is working correctly locally. Basically, when the below code is run, a purchase is made through Best Buy, and an SMS message is sent to me from my Twilio phone number indicating whether or not the purchase was successful. The variables in all-caps are already defined. I also did not include the imports. Again, the below code is making the purchase SUCCESSFULLY when run locally. This means that the SMS message that I get from my Twilio phone number contains "Success!" every time the following code is run locally:

client = Client(ACCOUNT_SID, AUTH_TOKEN)
def runBestBuyBotLocal():
    driver = webdriver.Chrome()
    driver.get("https://www.bestbuy.com/site/spongebob-squarepants-mini-plush-styles-may-vary/6404213.p?skuId=6404213")
    wait = WebDriverWait(driver, 10)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body[@class='size-l']/div[@class='pl-page-content']//div[@class='container-v2']/div[@class='row v-m-bottom-g']/div[2]//div[@class='col-xs-12']/div[6]/div[@class='None']/div/div/div/button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='cartApp']/div[@class='page-spinner page-spinner--out']/div[@class='large-view size-l']//div[@class='fluid-large-view']//section[@class='fluid-large-view__sidebar']//button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-e']")))
    element.send_keys(EMAIL)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-p1']")))
    element.send_keys(PASSWORD)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='cia-app-container']/div[@class='cia-actual-full-page-wrapper lv']/section/main[@class='cia-wrapper container']//form/div[3]/button")))
    element.send_keys(Keys.RETURN)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']")))
    element.send_keys(CVV)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button")))
    element.send_keys(Keys.RETURN)
    
try:
    runBestBuyBotLocal()
    message = client.messages \
        .create(
            body='Success!',
            from_=TWILIONUMBER,
            to=MYNUMBER
        )
except:
    message = client.messages \
        .create(
            body='Fail! ' + traceback.format_exc(),
            from_=TWILIONUMBER,
            to=MYNUMBER
        )

The following code is the basically the same process but adapted slightly to function on Heroku remote dyno. It is a part of my Heroku web app. The below code is NOT making the purchase successfully when run on Heroku remote dyno. It is instead throwing a Timeout error on the line element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']"))) This is really weird because I have the same line in runBestBuyBotLocal, and when runBestBuyBotLocal gets is executed locally, no timeout exception gets thrown on that line, and the purchase is made successfully. So, because of the timeout exception being thrown, every time I run the following code on a Heroku remote dyno, I get an SMS message from my Twilio phone number containing "Fail!":

def runBestBuyBotRemote():
    chrome_options = webdriver.ChromeOptions()
    chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--no-sandbox")
    driver = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), chrome_options=chrome_options)
    driver.get("https://www.bestbuy.com/site/spongebob-squarepants-mini-plush-styles-may-vary/6404213.p?skuId=6404213")
    wait = WebDriverWait(driver, 10)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body[@class='size-l']/div[@class='pl-page-content']//div[@class='container-v2']/div[@class='row v-m-bottom-g']/div[2]//div[@class='col-xs-12']/div[6]/div[@class='None']/div/div/div/button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='cartApp']/div[@class='page-spinner page-spinner--out']/div[@class='large-view size-l']//div[@class='fluid-large-view']//section[@class='fluid-large-view__sidebar']//button[@type='button']")))
    element.send_keys(Keys.RETURN)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-e']")))
    element.send_keys(EMAIL)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='fld-p1']")))
    element.send_keys(PASSWORD)
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@class='cia-app-container']/div[@class='cia-actual-full-page-wrapper lv']/section/main[@class='cia-wrapper container']//form/div[3]/button")))
    element.send_keys(Keys.RETURN)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[@id='credit-card-cvv']")))
    element.send_keys(CVV)
    driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button"))))
    element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='checkoutApp']/div[@class='page-spinner page-spinner--out']/div[1]/div[1]//div[@class='checkout__container checkout__container-fast-track']/div[@class='checkout__col checkout__col--primary']//div[@class='button--place-order-fast-track']/button")))
    element.send_keys(Keys.RETURN)

client = Client(ACCOUNT_SID, AUTH_TOKEN)
try:
    runBestBuyBotRemote()
    message = client.messages \
        .create(
            body='Success!',
            from_=TWILIONUMBER,
            to=MYNUMBER
        )
except:
    message = client.messages \
        .create(
            body='Fail!' + traceback.format_exc(),
            from_=TWILIONUMBER,
            to=MYNUMBER
        )

So, the question is: does anyone have an idea as to why runBestBuyBotLocal is NOT throwing a timeout exception when run locally and why runBestBuyBotRemote IS throwing a timeout exception when run on my Heroku remote dyno? In other words, why am I getting a "Success!" SMS message when the process is run locally and a "Fail!" message when it is run on my Heroku remote dyno?

I'm confused why the process is succeeding when run locally and failing when run on Heroku remote dyno. The only difference between runBestBuyBoyLocal and runBestBuyBotRemote functions is how the webdriver is instantiated. So the reason for one failing and one succeeding could be due to this difference, but I'm not sure.

Let me know if I need to clear up anything!

Answer 1

import traceback

try:
    WebDriverWait(driver, 2).until(EC.element_to_be_clickable(
        (By.XPATH, '//b[@id = "country_england"]/preceding-sibling::input'))).click()
except:
    traceback.print_exc()

use traceback to print the full exception trace .

Answer 2

In my opinion, you should try:

1. Set a screen resolution: chrome_options.add_argument('window-size=1920x1080'); Websites can have different layout on different resolutions...
2. Not use full XPaths, for example:

"/html//div[@class='c-portal']/div[@role='dialog']/div[1]/div[@role='dialog']/div[@role='document']//a[@role='button']"

Instead use: "//a[@role='button']", if multiple elements exist in DOM with this property, try to expand condition i.e. "//a[@role='button' and contains(@id, 'foo')]" Look: How to Find XPath?

Answer 3

You need to consider a couple of things as follows:

• ChromeDriver initiated Headless Browsing Context i.e. google-chrome-headless opens with a reduced Viewport. So you need to set the window size maximized.

  • Example:

    options = webdriver.ChromeOptions()
    options.add_argument("--headless")
    options.add_argument("window-size=1400,900")
    

You can find a detailed discussion in Not able to maximize Chrome Window in headless mode.

• If your test step is to interact with the element instead of presence_of_element_located() you need to induce WebDriverWait for element_to_be_clickable().

  • Example:

    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body[@class='size-l']/div[@class='pl-page-content']//div[@class='container-v2']/div[@class='row v-m-bottom-g']/div[2]//div[@class='col-xs-12']/div[6]/div[@class='None']/div/div/div/button[@type='button']"))).send_keys(Keys.RETURN)
    

You can find a detailed discussion in How to insert a value in a field Text using Selenium?.

• Absolute xpath makes your tests brittle. You may like to replace them with relative xpath. As an example, replace:

  element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[@id='cartApp']/div[@class='page-spinner page-spinner--out']/div[@class='large-view size-l']//div[@class='fluid-large-view']//section[@class='fluid-large-view__sidebar']//button[@type='button']")))
  

  with:

  element = wait.until(EC.presence_of_element_located((By.XPATH, "//button[@type='button'][@attribute2='value'][@attribute3='value']")))
  

Answer 4

All the answers above helped me figure it out. It was all about the flags. Assuming you have the build packs installed, and config vars added (many other answers already mention these), then you may still need the correct flags added as options.

Even thought the Chrome Buildpack says it adds the flags automatically, for me, it did not. I also needed others. If you are scraping a page you need a window size, else you might see unknown error: session deleted because of page crash , and apparently there are sizes issues re: heroku v chrome (or something) here

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--disable-gpu")
## might not be needed
chrome_options.add_argument("--remote-debugging-port=9222")
chrome_options.add_argument('--window-size=1920x1480')