Category "scrapy-splash"

Scroll the full js webpage using lua script to get the full source code

I want to scroll and get the full webpage source code using lua script. as example (http://note.com/ ) I want to scroll this full website to get the full source

Can this infomation be scraped from this site - if so, what I am not seeing

I am not new to Python, but new to Scrapy and Splash. Using Scrapy, I have successfully scraped static pages with tables, css and created .json files that were

How can I handle pagination with Scrapy and Splash, if the href of the button is javascript:void(0)

I am trying to scrape the names and links of universities from this website: https://www.topuniversities.com/university-rankings/world-university-rankings/2021,

How can I handle pagination with Scrapy and Splash, if the href of the button is javascript:void(0)

I am trying to scrape the names and links of universities from this website: https://www.topuniversities.com/university-rankings/world-university-rankings/2021,

Not able to create a docker image for splash scrapy app python?

I am using scrapy-splash to rendering javascript in web scrapping. So, splash run in a separate container. see here https://splash.readthedocs.io/en/latest/inst