'Listing GitHub users with their repos count
I want to list GitHub users with their repos count using GitHub rest API.
I tried to send request (https://api.github.com/search/users?q=${q}&&per_page=${COUNTS_PER_PAGE}&&page=${page || 1}
) using axios in simple node app.
But there is no repo-related field in response except repos_url
.
So I tried to send request with repos_url
again per users but met rate limit exceeded.
Is there anybody help me with solving this issue? Any help will be helpful for me.
Solution 1:[1]
I ran into this same issue with scraping one of the Organizations I'm involved in. The way we got around it was to use python
and do a try/except
block to keep trying over and over.
while True:
params = {"cursor": endCursor}
try:
response = get_repos(params)
except Exception:
rand_sleep = randint(5, 30)
print(f"Gonna sleep for {rand_sleep} because of a 403 error...")
sleep(rand_sleep)
repos = response['organization']['repositories']['nodes']
all_repos = all_repos + repos
hasNextPage = response['organization']['repositories']['pageInfo']['hasNextPage']
endCursor = response['organization']['repositories']['pageInfo']['endCursor']
We leverage the https://api.github.com/graphql endpoint to get the data we need, and then just keep trying over and over to get the chunks of data.
Solution 2:[2]
The repo count can be retrieved using the superface sdk instead of calling the api using axios manually. the sdk handles the communication with the github API and returns the repo list , then you can get number from there. here are the steps
npm install @superfaceai/one-sdk
npx @superfaceai/cli install vcs/user-repos
const { SuperfaceClient } = require('@superfaceai/one-sdk');
const sdk = new SuperfaceClient();
async function run() {
// Load the installed profile
const profile = await sdk.getProfile('vcs/user-repos');
// Use the profile
const result = await profile
.getUseCase('UserRepos')
.perform({
user: 'superfaceai'
});
console.log(result.unwrap().repos.length);
}
run();
You can make the script to sleep in between requests to cope with the rate limiting
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | JJ Asghar |
Solution 2 |