Puppeteer - Facing Issue to scroll in-page content on linkedin sales navigator

Question

I am trying to scrape details of a few companies and their leads from the Linkedin Sales Navigator. The problem I am facing is that, the code only scrapes top 2-3 names. The reason behind this issue is that the leads are added dynamically when the page is scrolled to bottom. The page contains two in-page scrollers (highlighted in red) and I want to write code for one of them scroller to scroll till its bottom. The div section where I want to scroll has class="p4 _vertical-scroll-results_1igybl".enter image description here. Will be thankful If anyone could help me do this using puppeteer. Thanks in advance!!!

Answer 1

Here is a small example of how you can implement autoscroll:

let lastHeight = await page.evaluate("document.body.scrollHeight");

while (true) {
    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)");
    await page.waitForTimeout(2000);
    let newHeight = await page.evaluate("document.body.scrollHeight");
    if (newHeight === lastHeight) {
      break;
    }
    lastHeight = newHeight;
}

If the scroll is inside an element, then instead of document.body use document.querySelector("your_element").