List all the projects and all the groups

Question

What is the easiest method to list all the projects and groups in GitLab using my private token?

Answer 1

If only your private token is available, you can only use the API:

PROJECTS

Use the following command to request projects:

curl "https://<host>/api/v4/projects?private_token=<your private token>"

This will return you the first 20 entries. To get more you can add the paramater per_page

curl "https://<host>/api/v4/projects?private_token=<your private token>&per_page=100"

with this parameter you can request between 20and 100 entries. https://docs.gitlab.com/ce/api/README.html#pagination

If you now want all projects, you have to loop through the pages. To get to another page add the parameter page.

curl "https://<host>/api/v4/projects?private_token=<your private token>&per_page=100&page=<page_number>"

Now you may want to know how many pages there are. For that, add the curl parameter --head. This will not return the payload, but the header.

The result should look like this:

HTTP/1.1 200 OK
Server: nginx
Date: Thu, 13 Jul 2017 17:43:24 GMT
Content-Type: application/json
Content-Length: 29428
Cache-Control: no-cache
Link: <request link>
Vary: Origin
X-Frame-Options: SAMEORIGIN
X-Next-Page: 2
X-Page: 1
X-Per-Page: 20
X-Prev-Page:
X-Request-Id: 80ecc167-4f3f-4c99-b09d-261e240e7fe9
X-Runtime: 4.117558
X-Total: 312257
X-Total-Pages: 15613
Strict-Transport-Security: max-age=31536000

The two interesting parts here are X-Totaland X-Total-Pages. The first is the count of available entries and the second the count of total pages.

I suggest to use python or some other kind of script to handle the requests and concat the results at the end.

If you want to refine the search, consult this wiki page: https://docs.gitlab.com/ce/api/projects.html#projects-api

GROUPS

For groups simply replace projects with groups in the curls. https://docs.gitlab.com/ce/api/groups.html#list-groups

UPDATE:

Here is the official list of Gitlab API clients/wrappers: https://about.gitlab.com/partners/technology-partners/#api-clients
I highly recommend using one of these.

Answer 2

Using the python-gitlab module we can get all groups and projects inside it. But if you have more than 20 groups or 20 projects inside of any group use pagination.

import gitlab
gl = gitlab.Gitlab('https://gitlab.com', private_token='Token')
groups = gl.groups.list()
for each in groups:
    group = gl.groups.get(each, lazy=True)
    project_lst=group.projects.list(as_list=False)  #pagination
    for item in project_lst:
    project_id = gl.projects.get(item.attributes['id'])
        ......................................
        ......................................

Answer 3

Using python You can easily List all the groups and the Projects.Below is the code which you can try to list the project and groups:

import os
import gitlab

gl = gitlab.Gitlab('http://gitlab_hostname.com', 'your_private_token')
groups = gl.groups.list()
projects = gl.projects.list()
all_projects = gl.projects.list(all=True)
all_groups=gl.groups.list(all=True)
print("All groups are:",all_groups)
length=len(all_projects)
i=0
while i < length:
    project = gl.projects.get(all_projects[i].id)
    print(project)
    i=i+1

Answer 4

With python-gitlab you can do this with:

$ gitlab project list                                                                                                 [6:53:23]
id: 58
path: project-omega

id: 56
path: sigma

id: 54
path: theta

...

Answer 5

Use the graphql endpoint

It is very efficient to use the graphql endpoint of gitlab because you can get structured json result in a very concise and resource efficient way, here are two exemplary steps to achieve the task of "listing all projects and groups":

1. Use the graphql-explorer to find your query
2. Send requests to the graphql endpoint with your developed query (python example)

Use the graphql-explorer to find your query

1. Login to your gitlab instance (logging in is a required so that the queries also work on your private entities)
2. Go to the graphql explorer https://gitlab.com/-/graphql-explorer

After a bit of experimenting you would eventually come up with a query similar to this:

query {
    user(username:"YourUserName") {
    groups {
      nodes {
        id
        parent {
          id
        }
        name
        descendantGroups {
          nodes {
            id
            name
          }
        }
        projects {
          nodes {
            id
            name
          }
        }
      }
    }
  }
}

Send requests to the graphql endpoint with your developed query

Now you can e.g. via python post the above developed queries and work on the returned data:

import requests

query = """
query {
    user(username:"YourUserName") {
    groups {
      nodes {
        id
        parent {
          id
        }
        name
        descendantGroups {
          nodes {
            id
            name
          }
        }
        projects {
          nodes {
            id
            name
          }
        }
      }
    }
  }
}
"""
url = 'https://gitlab.com/api/graphql?private_token=your_private_token'
response = requests.post(url, json={'query': query})
data = response.json()
# work on data ...

Happy coding

Answer 6

Here's a small shell script to get all GitLab projects:

echo -n "Please enter private token: "
read -s PRIVATE_TOKEN
number_of_pages=$(curl -s --head "https://gitlab.example.org/api/v4/projects?private_token=$PRIVATE_TOKEN" | grep x-total-pages | awk '{print $2}' | tr -d '\r\n')
for page in $(seq 1 $number_of_pages); do
    curl -s "https://gitlab.example.org/api/v4/projects?private_token=$PRIVATE_TOKEN&page=$page"
done

Replace gitlab.example.org with the hostname of your GitLab server.

Adjust it as needed; for example, this uses jq to get just the SSH repo URL for each project:

for page in $(seq 1 $number_of_pages); do
    curl -s "https://gitlab.example.org/api/v4/projects?private_token=$PRIVATE_TOKEN&page=$page" | jq -r '.[] | .ssh_url_to_repo'
done

Answer 7

In bash for Gitlab API V4:

#!/bin/bash
GL_DOMAIN=""
GL_TOKEN=""
echo "" > gitlab_projects_urls.txt
for ((i=1; ; i+=1)); do
    contents=$(curl "$GL_DOMAIN/api/v4/projects?private_token=$GL_TOKEN&per_page=100&page=$i")
    if jq -e '. | length == 0' >/dev/null; then 
       break
    fi <<< "$contents"
    echo "$contents" | jq -r '.[].ssh_url_to_repo' >> gitlab_projects_urls.txt
done