'Sending two clicks with requests_html, but only the first one seems to be working
So, I have this webpage that I have to do some crawling. Here's what I have to do: I have to go on this page (https://www.digitalocean.com/pricing),
click on this div:
and then click on the view as table icon on the right corner of the page
so I can extract this table
Here's the Python code:
import requests_html
from jsscript import script
with requests_html.HTMLSession() as s:
url = "https://www.digitalocean.com/pricing"
r = s.get(url)
r.html.render(sleep=10, timeout=200, script=script, keep_page=False)
html_text = r.html.html
print(html_text)
and here's the JS script I wrote for clicking on both buttons:
script = """
() => {
function sleep (time) {
return new Promise((resolve) => setTimeout(resolve, time));
}
function clickButton () {
const item = document.querySelector("#__next > section > div.ContainerStyles__StyledContainer-sc-1vejnbq-0.fTRAxt.container > div.MasonryStyles__MasonryDiv-sc-n3swdp-0.cevqpZ > div:nth-child(1) > div:nth-child(1)");
if(item) {
item.click()
}
}
clickButton();
sleep(200).then(() => {
const item2 = document.querySelector('#__next > section > div.ContainerStyles__StyledContainer-sc-1vejnbq-0.fTRAxt.container > div.HeroPricing___StyledDiv-sc-13g3nko-39.kvnoDg > div > div.HeroPricingStyles__StyledToolsContainer-sc-h8uh2t-5.mBMrO.dropletsToolsContainer > div.HeroPricingStyles__StyledEndContainer-sc-h8uh2t-10.MZirY.dropletsEndContainer > button:nth-child(3) > img');
if(item2) {
item2.click()
}
})
}
"""
Problem is, once I print html_text
, all I get is the HTML after the first button was clicked, like the second wasn't clicked at all. I tried simulating this on the navigator console and it works fine, both buttons are clicked and I can get to the table page successfully.
I've already checked this problem (Sending a click with requests_html and pyppeteer python) and it has helped me a lot, but still doesn't solve my second click problem.
Is it possible at all to do multiple-clicks with requests-html?
Using selenium is not an option.
I've never used requests-html before, but even after checking documentation and some YouTube videos I still don't know what might be going wrong.
Thanks in advance.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|