'How to pass security cloudflare server with php curl

I used curl for getting data from biorxiv site. This is not an illegal job.

$curl = curl_init();

    curl_setopt_array($curl, array(
        CURLOPT_URL => 'https://www.biorxiv.org/search/electron+microscopy+jcode%3Abiorxiv+limit_from%3A2021-11-08+limit_to%3A2021-11-10+numresults%3A75+sort%3Arelevance-rank+format_result%3Astandard,
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_ENCODING => '',
        CURLOPT_MAXREDIRS => 10,
        CURLOPT_TIMEOUT => 0,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
        CURLOPT_CUSTOMREQUEST => 'GET',         
    ));

I used this code, and this is working well in my postman, but not working on the server. I am seeing this wrong message now.

enter image description here

How can I fix this issue?

Thank you!

phpcodeignitercaptchacloudflarephp-curl


Solution 1:[1]

Well, what you are encountering is a bot detection system. Cloudflare uses this to prevent DDoS attacks by non-humans. In this case it means you will not be able to use curl this way since these checks specifically attempt to very a fully working browser. You might be able to bypass it temporarily by copying cookies from your browser, this will not last long since these get refreshed periodically.

Postman works btw because it runs on Chrome and thus is less likely to trigger bot detection.

Solution 2:[2]

Here is an example of URL behind CloudFlare: https://www.lendingtree.com/forms/mortgage/pecan/refi_hdl2b_ltv

Works fine from Chrome incognito window. When I copy the request as cURL, curl it from the same machine's command window, I get 403. curl request:

curl 'https://www.lendingtree.com/forms/mortgage/pecan/refi_hdl2b_ltv' \
  -H 'authority: www.lendingtree.com' \
  -H 'sec-ch-ua: "Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'dnt: 1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'sec-fetch-site: none' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-user: ?1' \
  -H 'sec-fetch-dest: document' \
  -H 'accept-language: en-US,en;q=0.9' \
  --compressed

What gives?

Solution 3:[3]

Cloudflare uses TLS fingerprinting to sift unwanted traffic. You can read more here.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Dylan Reimerink
Solution 2 savageGoat
Solution 3 Victor Velchev

Related Questions