puppeteer page.evaluate() returns empty object

Question

I am trying to scrape this web https://poe.ninja/challenge/builds?time-machine=day-6 using Puppeteer. I tried Puppeteer page.evaluate querySelectorAll return empty objects and saw lot of similar question here. but none of them solve my problem.

Here is my code:

const scrapeNinja = async () => {
    const browser = await puppeteer.launch({headless: false})

    const page = await browser.newPage()

    await page.goto(`https://poe.ninja/challenge/builds?time-machine=day-6`, {
        waitUntil: 'domcontentloaded',
    })

    const getArray = await page.evaluate(() => {
        return Array.from(document.querySelectorAll(
                '#openSidebar > div > section:nth-child(3) > div > div > div > ul li .css-1h2ruwl'
            )).map(e => e.textContent)
    })

    console.log(getArray)
}

I know the values returned from page.evaluate should be serializeable. isn't this Array.from(document.querySelectorAll('#openSidebar > div > section:nth-child(3) > div > div > div > ul li .css-1h2ruwl')).map(e => e.textContent) not a serializeable value? I tried use this on the dev tool section it return exacully what i want, but back to node.js, it only return empty array...

Am i do something wrong?

Answer 1

Looks like the problem is really with waiting, you are looking for elements even if full dom content isnt fully loaded.

  const scrapeNinja = async () => {
  const browser = await puppeteer.launch({headless: false})

  const page = await browser.newPage()

  await page.goto(`https://poe.ninja/challenge/builds?time-machine=day-6`, {
    waitUntil: 'networkidle2',
  })
  

  const getArray = await page.$$eval('#openSidebar > div > section:nth-child(3) > div > div > div > ul li .css-1h2ruwl',
    el => el.map(item => item.textContent))

  console.log(getArray)
}

scrapeNinja()

This code works perfectly for me, even you dont have to initialize array. In the future use networkidle2 in waitUntil option