Gitlab-Runner on Kubernetes/Openshift

Question

I'm trying to deploy Gitlab-runner(s) on Openshift/K8S, but can't succeed in having runners communicating with Gitlab (itself deployed in pods on OCP).

I followed couple of different instructions like these ones:

• https://www.youtube.com/watch?v=EwbhA53Jpp4
• https://docs.gitlab.com/runner/install/kubernetes.html
• https://github.com/oprudkyi/openshift-templates/blob/master/gitlab-runner/gitlab-runner.yaml

My gitlab-runner pod is starting properly, but it always receives HTTP 404 Not Found error messages.

Here is my toml config file:

# cat /etc/gitlab-runner/config.toml
concurrent = 6
check_interval = 0

[[runners]]
  name = "GitLab Runner"
  url = "http://gitlab-ce.MY_COMAIN.com/ci"
  token = "WHO_CARES?"
  executor = "kubernetes"
  [runners.kubernetes]
    namespace = "MINE"
    privileged = false
    host = ""
    cert_file = ""
    key_file = ""
    ca_file = ""
    image = ""
    cpus = ""
    memory = ""
    service_cpus = ""
    service_memory = ""
    helper_cpus = ""
    helper_memory = ""
    helper_image = ""
  [runners.cache]
    Type = "s3"
    ServerAddress = "minio-service:80"
    AccessKey = "GENERATED"
    SecretKey = "GENERATED"
    BucketName = "bkt-gitlab-runner"
    Insecure = true

And as soon as the pod starts, I have this in my logs:

Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
Running in system-mode.                            

Configuration loaded                                builds=0
Metrics server disabled                            
WARNING: Checking for jobs... failed                runner=WHO_CARES? status=404 Not Found
WARNING: Checking for jobs... failed                runner=WHO_CARES? status=404 Not Found
WARNING: Checking for jobs... failed                runner=WHO_CARES? status=404 Not Found

And in Gitlab, in the runners page (https://gitlab-ce.MY_COMAIN.com/group/project/settings/ci_cd) there is no "Runners activated for this project".

I can log in to my pod in its terminal and launch gitlab-runner register to register a new runner

/ # gitlab-runner register
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab-ce.MY_COMAIN.com
Please enter the gitlab-ci token for this runner:
WHO_CARES?
Please enter the gitlab-ci description for this runner:
[dc-gitlab-runner-service-1-ktw6v]: test
Please enter the gitlab-ci tags for this runner (comma separated):
test
Registering runner... succeeded                     runner=WHO_CARES?
Please enter the executor: docker+machine, kubernetes, ssh, docker-ssh, parallels, shell, virtualbox, docker-ssh+machine, docker:
kubernetes
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

But when I try to run it... I'm facing the exact same issue.

/ # gitlab-runner run
Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
Running in system-mode.

Configuration loaded                                builds=0
Metrics server disabled
WARNING: Checking for jobs... failed                runner=WHO_CARES? status=404 Not Found
WARNING: Checking for jobs... failed                runner=WHO_CARES? status=404 Not Found

Of course, I checked if I can access Gitlab from the Runner terminal, and it works

/ # ping
BusyBox v1.27.2 (2018-06-06 09:08:44 UTC) multi-call binary.

Usage: ping [OPTIONS] HOST

/ # ping gitlab-ce.MY_COMAIN.com
PING gitlab-ce.MY_COMAIN.com (1.2.3.4): 56 data bytes
64 bytes from 1.2.3.4: seq=0 ttl=63 time=0.268 ms
64 bytes from 1.2.3.4: seq=1 ttl=63 time=0.261 ms
64 bytes from 1.2.3.4: seq=2 ttl=63 time=0.288 ms
^C

Side note: I'm running OCP 3.9 / K8S 1.9

Do you see anything I'm doing wrong?

Cheers, Olivier

Answer 1

OK. Here is the solution.

The problem is coming from a weird behavior with the token. There are several token in Gitlab and one must choose carefuly the one to use.

This problem is also related here: https://gitlab.com/gitlab-org/gitlab-ce/issues/37807

You must first use the Runner token available in the admin page: https://gitlab-instance/admin/runners. This token is to be used in your config.toml, under the [runners.token] section.

Deploy and start the runner.

It appears in Gitlab, but is unavailable. Simply clik on its name in the admin area: https://gitlab-instance/admin/runners/38

Find here in the details the token associated to this Runner.

Copy it back to the config.toml file, still under the exact same [runners.token] section, as a replacement. Redeploy your runner.

Should work.

Answer 2

Apr. 2022: GitLab 14.10 does support that use case:

GitLab Runner Operator for Kubernetes

In GitLab 13.10, we released the GitLab Runner Operator for the Red Hat OpenShift container platform for Kubernetes.

That release provided OpenShift users with the automation and management capabilities of the Operator Framework and simplified the ongoing management of runners in an OpenShift Kubernetes cluster. Available starting in 14.10 is a GitLab Runner Operator v1.7.0 that you can use in non-OpenShift Kubernetes clusters. This GitLab Runner Operator is available on OperatorHub.io.

See Documentation and Issue.

---

Another option, which would avoid the token management, with GitLab 13.7 (December 2020)

GitLab Runner for Red Hat OpenShift

Available today is the GitLab Runner container image for the Red Hat OpenShift Container Platform.

To install the runner on OpenShift, you can use the new GitLab Runner Operator available from the beta channel in Red Hat’s Operator Hub - a web console for OpenShift cluster administrators to discover and select Operators to install on their cluster.

Operator Hub is deployed by default in the OpenShift Container Platform.
We plan to transition the GitLab Runner Operator to the stable channel, and by extension GA, in early 2021. Finally, we are also developing an operator for GitLab, so stay tuned to future release posts for those announcements.

See Documentation and Issue.

---

This is enhanced with GitLab 13.11 (April 2021)

Deploy GitLab on OpenShift and Kubernetes with the GitLab Operator (beta)

GitLab is working to offer full support for OpenShift. To accomplish this, we have released the MVP GitLab Operator. The operator aims to manage the full lifecycle of GitLab instances on Kubernetes and OpenShift container platforms.
Currently, this is a beta release and it is not recommended for production use.

The next steps will be to make the operator generally available (GA). In the future the operator will become the recommended installation method for Kubernetes and OpenShift, although the GitLab Helm chart will still be supported. We welcome you to try this operator and provide feedback on our issue tracker.

See Documentation and Issue.