You Look Like a Robot' Error , while using puppeteer in firebase cloud functions

Question

Im using the puppeteer package , in order to scrap a web page data that is fetched by clicking a button in this page

this are the presetting that I'm using:

const puppeteer = require('puppeteer-extra')

const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())

// Add adblocker plugin to block all ads and trackers (saves bandwidth)
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker')
puppeteer.use(AdblockerPlugin({ blockTrackers: true }))

those setting are made in order that I will not be detected as a robot.

here what I'm doing : (basically , creating a request by clicking a button , then this request return a json with a data that fill up some text info in a label , then I'm reading the data from that label

here's how im clicking the button :

const box = await btn.boundingBox();
const x = box.x + (box.width/2);
const y = box.y + (box.height/2);
console.log(x, y);
page.mouse.move(x,y,{step:1});
page.mouse.click(x,y)
await page.waitForTimeout(4000);

then afterwards ---> I'm getting the data from the data:

const [result] = await page.$x('//*[@id="content"]/div[1]/div[1]/div/div[2]/div');
//   const txt = await result.evaluate.toString
let value = await page.evaluate(el => el.textContent, result);
console.log(value);
console.log('done?');
await browser.close();
const dic = {};
dic['status'] = 200;
dic['data'] = {"message": value};
  
response.send(dic);

I'm also using the 'on' method in order to see if the im getting a response from the action of clicking the button , like so:

await page.on('response', async response =>{
      try {
       console.succ(await response.json());
      } catch (error) {
        //
        // console.error(error);
      }
  
  
  
  });

and it sure get one.

the problem is ---> that when I'm deploying it to the firebase cloud functions server,

firebase deploy --only functions

and then triggering the function --> I'm getting a json that look like this :

{ success: false, message: 'You look like a robot.' } 

But when deploying the same code to my local host like so

firebase serve --only functions

and then triggering the function --> I'm not detected as a robot and getting the json with a successful result --> and with that data that the clicking of a button supposed to fetch.

this is so weird , I'm trying to think that there's a connections between the firebase cloud functions and reCAPTCHA , because both are a google services but, its not seem's reasonable for it to be true .

that being said, what could be the reason for this? all that change is the environment that the code runs from. do you have any idea why this is happening ? and how to solve it of course .

Answer 1

Since your function runs properly locally, it's almost certainly not the function itself.

Sites take a variety of different approaches to detect bots, one of which is blocking traffic from known data centers like Google Cloud's. Using a residential IP proxy like those provided by BrightData will probably circumvent this.

Answer 2

I'm facing the same issue while using Puppeteer in Firebase Cloud Functions.

I'm using a residential IP proxy with the following set of packages puppeteer-extra, puppeteer-extra-plugin-stealth, puppeteer-extra-plugin-anonymize-ua', and user-agents`.

On localhost, all is working as expected while running Puppeteer in firebase Cloud Functions I'm getting a 404 response from the requested URL. So there must be some difference.