'How to bypass Cloudflare bot protection in selenium
I need to grab some information from a site just for education purpose, however i cannot send requests because of the protection. I get The typical Checking-your-browser page shows up first and then i'm being redirected repeatedly. how i can bypass this protection in python selenium?
Solution 1:[1]
I had this problem a long time ago and I was able to solve it. Use the code below and enjoy :)
options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(options=options, executable_path=r"webdriver\chromedriver.exe")
///////// EDIT //////////////// this way now is not working !
Solution 2:[2]
As March 2022 :
Hi, I had the same problem when using headless Selenium on a Docker Linux image.
I solved it by creating a virtualdisplay right before calling the webdriver:
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 800))
display.start()
Don't forget to install both pyvirtualdisplay and xvfb:
pip install pyvirtualdisplay
and sudo apt-get install xvfb
And you must remove the "headless" option in ChromeDriver, here is the complete code I use :
#Display in order to avoid CloudFare bot detection
display = Display(visible=0, size=(800, 800))
display.start()
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('start-maximized')
options.add_argument('enable-automation')
options.add_argument('--disable-infobars')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-browser-side-navigation')
options.add_argument("--remote-debugging-port=9222")
# options.add_argument("--headless")
options.add_argument('--disable-gpu')
options.add_argument("--log-level=3")
driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)
Since it was working nicely without headless on my local computer, I figured emulate a real display might do the work aswell. I do not really understand why, but from what I've understood, CloudFare tries to execute javascript code in order to confirm you're not a bot. Having a emulated webpage display helps to do so.
Solution 3:[3]
A bit of a late response and no wonder why developers still face this issue repetitively. I am using Java v17 and Gradle v7.4.2. My solution is related to the one explained above but the code is in Java.
@Before
public void setup() {
WebDriverManager.chromedriver().setup();
ChromeOptions options = new ChromeOptions();
// Bypass Cloudflare checks
options.setExperimentalOption("useAutomationExtension", false);
options.addArguments("--disable-blink-features=AutomationControlled");
driver = new ChromeDriver(options);
driver.manage().window().maximize();
}
Please refer to the Selenium ChromeOptions documentation for further details.
Happy coding.
Solution 4:[4]
SOLUTION JULY 2021
just add user agent argument in chrome options and set user agent to any value
ops = Options() ua='cat' ops.add_argument('--user-agent=%s' % ua) driver=uc.Chrome(executable_path=r"C:\chromedriver.exe",chrome_options=ops)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | |
Solution 2 | Jules Civel |
Solution 3 | Spicy strike |
Solution 4 | Haider Asad |