'Puppeteer bet365 - certain matches not available
I am trying to scrape some table tennis betting odds using puppeteer. However, I am dealing with a problem trying to load Setka Cup table tennis matches.
This cup, with few other table tennis cups, aren't loading for me with a message (roughly translated): Sorry, this page is no longer available. Betting has ended or has been paused.
I've been able to load the odds of some other cups, other sports (while not using headless mode) and I reckon it's not a location based error since it loads via my regular Chrome browser and both browsers seem to send out same information (caught using the Network tab in Chrome Dev tools).
There are a lot of other tools/tips that I have found and tried but none of those resolved this.
Is there some additional scraping/bot prevention just for this sport/cup in particular? Hope I am not missing anything clear as I have just started with all of this. Thank you
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const randomUA = require('modern-random-ua');
const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);
const VIEWPORT = {width: 1200, height: 900};
const BET365 = 'https://www.bet365.com/#/AS/B92/';
function delay(time) {
return new Promise(function(resolve) {
setTimeout(resolve, time)
});
}
const escapeXpathString = str => {
const splitedQuotes = str.replace(/'/g, `', "'", '`);
return `concat('${splitedQuotes}', '')`;
};
const clickByText = async (page, text) => {
const escapedText = escapeXpathString(text);
const linkHandlers = await page.$x(`//span[contains(text(), ${escapedText})]`);
if (linkHandlers.length > 0) {
await linkHandlers[0].click();
} else {
throw new Error(`Link not found: ${text}`);
}
};
(async () => {
const browser = await puppeteer.launch({
headless: false,
args: [
"--disable-infobars",
"--no-sandbox",
"--disable-blink-features=AutomationControlled",
],
ignoreDefaultArgs: ["--enable-automation"],
});
const page = (await browser.pages())[0];
await page.evaluateOnNewDocument(() => {
delete navigator.__proto__.webdriver;
Object.defineProperty(navigator, 'maxTouchPoints', {
get() {
return 1;
},
});
navigator.permissions.query = i => ({then: f => f({state: "prompt", onchange: null})});
});
await page.viewport(VIEWPORT);
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36');
// await page.setUserAgent(randomUA.generate());
const client = await page.target().createCDPSession()
await client.send('Network.clearBrowserCookies')
await page.goto(BET365, { waitUntil: 'networkidle2' });
await page.waitForTimeout(5000);
await clickByText(page, `Setka Cup`);
await page.waitForTimeout(2230);
await page.screenshot({path: '1.png'});
console.log("screenshot 1");
await browser.close();
})()
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|