Why isn't Puppeteer page.click waiting (maybe Browserless?)

Question

Goal: I have a page that I need to get html from after first clicking something on the page.

Issue: The html that comes back is not waiting for that element click.

Here's one way that I've tried to do it.

await page.setViewport({width: 1400, height: 800});
await page.waitForSelector('.specs .accent', {visible: true, timeout: 80000});

const html = await page.evaluate(() => {
      const moreButton = document.querySelector('.specs .accent');
      moreButton.click();
      return document.body.innerHTML;
});

console.log(html);

Here's another way I've tried:

await page.setViewport({width: 1400, height: 800});
await page.waitForSelector('.specs .accent', {timeout: 80000});
await page.click('.specs .accent');
const html = await page.evaluate(() => document.body.innerHTML);
console.log(html);

The html that comes back is the html prior to the click happening. Leading me to believe that the click is broken, but not erroring out.

Notes:

• I've screenshot after the page.click. The element is there, but not clicked.
• There are no errors with page.click that I can find, but I might be debugging wrong.
• I recently implemented Browserless. The code above used to work, but sometime between then and when I implemented Browserless (https://www.browserless.io/docs/docker-quickstart), it stopped working. Maybe there's something wrong with their page.click or version of Puppeteer?
• There's also a proxy. So the flow is app -> browserless (puppeteer) -> proxy -> page.
• The thing I'm trying to click is an <a href="javascript:void(0)" class="accent">More</a> When clicked, it populates some with preloaded data.

Finally: I'm really not sure how to debug this further. Any clues?

Page for reference: https://www.article.com/product/16190/mod-blue-berry-armchair

Answer 1

The solution is that certain javascript requests being blocked can interrupt the ability for an element to be clicked by Puppeteer. If you're having trouble with this, make sure no requests are being blocked by Browerless, Puppeteer, or your proxy and then try the page.click again.

A simple debugging solution is to use page.screenshot to see if something is, in fact, clicked.

Answer 2

After testing with the URL that you sent to me through email to browserless support, I was able to see through the live debugger that the default viewport was a "mobile" and hence the structure of the page had changed... The hamburger button was present for the navbar and the Specifications section where the button you're trying to click is, was collapsed.

I added this line before navigating to the page:

await page.setViewport({
      width: 1920,
      height: 1080
  })

This made the viewport larger horizontally, and after that, I went to console and ran a document.querySelector('.specs .accent').click() and I was able to click and expand the information as desired.