Building Go apps with private gitlab modules in Docker

Question

I am trying to build my go apps on a docker file. Inside my go.mod there is private package that needs authentication/ssh. This question is similar to Building Go apps with private modules in Docker, but in my case is i have to pull package from gitlab not from github. Here is my dockerfile:

# builder image
FROM golang:1.14.11-alpine AS builder

# specific directory for build process
WORKDIR /usr/src/build

# copying the source code 
# to the current working directory
COPY . .
RUN apk add --no-cache openssh-client
RUN apk add --no-cache git

# create ssh directory
RUN mkdir ~/.ssh
RUN touch ~/.ssh/known_hosts
RUN ssh-keyscan -t rsa gitlab.com >> ~/.ssh/known_hosts

# allow private repo pull
RUN git config --global url."https://my-personal-access-token:[email protected]/".insteadOf "https://gitlab.com/"

ADD . /go/src/gitlab.com/my-repo/backends/backend-structs
CMD cd /go/src/gitlab.com/my-repo/backends/backend-structs; go get /go/src/gitlab.com/my-repo/backends/backend-structs && go build -o /go/bin/backend-structs


# executing build process
RUN GOOS=linux go build -ldflags="-s -w" -o app

# runtime image
FROM golang:1.14.11-alpine AS runtime

# create and use non-root user
# to increase container security 
# ref https://pythonspeed.com/articles/root-capabilities-docker-security/
RUN adduser myuser --disabled-password

USER myuser

WORKDIR /home/myuser

# copy the executable binary file from builder directory
# to the current working directory
COPY --from=builder /usr/src/build/app .

# exposing port
EXPOSE 8080

# run the application
CMD ["./app"]

i have tried to follow this tutorial https://divan.dev/posts/go_get_private/ , by changing github.com to gitlab.com still failed.

Here is the error details:

#17 5.830       remote: HTTP Basic: Access denied
#17 5.830       fatal: Authentication failed for 'https://gitlab.com/my-repo/backends.git/'
------
executor failed running [/bin/sh -c GOOS=linux go build -ldflags="-s -w" -o app]: exit code: 1

anyone here knows how to create dockerfile with golang private package(repo is hosted in gitlab.com) ?

Answer 1

In my experience, do NOT use git configs to solve this. Only use ~/.netrc. Here is a guide a made specifically for this: https://gist.github.com/MicahParks/1ba2b19c39d1e5fccc3e892837b10e21

I'll paste its contents below as well.

Problem

The go command line tool needs to be able to fetch dependencies from your private GitLab, but authenticaiton is required.

This assumes your private GitLab is hosted at privategitlab.company.com.

Environment variables

The following environment variables are recommended:

export GO111MODULE=on
export GOPRIVATE=privategitlab.company.com

The above lines might fit best in your shell startup, like a ~/.bashrc.

Explanation

GO111MODULE=on tells Golang command line tools you are using modules. I have not tested this with projects not using Golang modules on a private GitLab.

GOPRIVATE=privategitlab.company.com tells Golang command line tools to not use public internet resources for the hostnames listed (like the public module proxy).

Get a personal access token from your private GitLab

To future proof these instructions, please follow this guide from the GitLab docs. I know that the read_api scope is required for Golang command line tools to work, and I may suspect read_repository as well, but have not confirmed this.

Set up the ~/.netrc

In order for the Golang command line tools to authenticate to GitLab, a ~/.netrc file is best to use.

To create the file if it does not exist, run the following commands:

touch ~/.netrc
chmod 600 ~/.netrc

Now edit the contents of the file to match the following:

machine privategitlab.company.com login USERNAME_HERE password TOKEN_HERE

Where USERNAME_HERE is replaced with your GitLab username and TOKEN_HERE is replaced with the access token aquired in the previous section.

Common mistakes

Do not set up a global git configuration with something along the lines of this:

git config --global url."[email protected]:".insteadOf "https://privategitlab.company.com"

I beleive at the time of writing this, the SSH git is not fully supported by Golang command line tools and this may cause conflicts with the ~/.netrc.

Bonus: SSH config file

For regular use of the git tool, not the Golang command line tools, it's convient to have a ~/.ssh/config file set up. In order to do this, run the following commands:

mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/config
chmod 600 ~/.ssh/config

Please note the permissions on the files and directory above are essentail for SSH to work in it's default configuration on most Linux systems.

Then, edit the ~/.ssh/config file to match the following:

Host privategitlab.company.com
  Hostname privategitlab.company.com
  User USERNAME_HERE
  IdentityFile ~/.ssh/id_rsa

Please note the spacing in the above file matters and will invalidate the file if it is incorrect.

Where USERNAME_HERE is your GitLab username and ~/.ssh/id_rsa is the path to your SSH private key in your file system. You've already uploaded its public key to GitLab. Here are some instructions.

Answer 2

I ran in that problem with private github.com repositories. I used the ssh-agent as a solution, however, I had difficulties because there are all sorts of assumptions in other answers. In my case, I had an issue with the filename for the key and the other issue was AppArmor.

I talk about my setup and these issues in details below.

Experimental Docker Feature

First, in the Dockerfile, you want to allow newer options by adding a comment like so on the very first line:

# syntax=docker/dockerfile:experimental

This will allow us to use the --mount=type=ssh option.

Note: if the comment is not on the very first line, it will be ignored and the extensions such as the --mount=... fail.

Basic Dockerfile Setup

Like you, we use Go alpine. We also make sure to get openssh-client.

FROM golang:alpine AS build-env
RUN apk --no-cache add build-base git mercurial gcc curl openssh-client

Detail, we add our code (entire folder):

ADD . .

Create the known_hosts & .gitconfig Files

Now we have an issue because by default the key is not going to be recognized by SSH. To avoid the issue, we want to add the github.com (gitlab.com, in your case) key to our known hosts. We also want to make sure that github.com is used with SSH and not HTTPS.

RUN mkdir -p -m 0700 ~/.ssh && \
    ssh-keyscan github.com >> ~/.ssh/known_hosts && \
    echo -e "[url \"[email protected]:<company-name>\"]\n\tinsteadOf = https://github.com/<company-name>" >> ~/.gitconfig

Note: the ~ in a docker is always /root at this point.

Some people attach a volume to give Docker access to their known_hosts file. I don't think that's safe and it is likely to break if you run online tools such as CircleCI.

Build Go Application

This gives us all the necessary elements to finally build our application:

ENV GO111MODULE=on
ENV GOPRIVATE=github.com/<company-name>
RUN --mount=type=ssh cd cmd/app/ && go build -o app

Our Dockerfile has a few more lines, not presented here, to define additional files and the final ENTRYPOINT.

Load Key in ssh-agent

We're nearly ready to run docker build .... But we still need to define the key in our ssh-agent. This is very simple:

ssh-add id_rsa

SUPER VERY IMPORTANT: The name of the key MUST be one of the defaults that ssh expects. id_rsa is one of them. If you have a key name other than one of the defaults, it won't be picked up.

The following are the names checked in one of my tests. You see those when you run the ssh -A -v ... command (See below).

#18 0.828 debug1: identity file /root/.ssh/id_rsa type -1
#18 0.828 debug1: identity file /root/.ssh/id_rsa-cert type -1
#18 0.828 debug1: identity file /root/.ssh/id_dsa type -1
#18 0.828 debug1: identity file /root/.ssh/id_dsa-cert type -1
#18 0.829 debug1: identity file /root/.ssh/id_ecdsa type -1
#18 0.829 debug1: identity file /root/.ssh/id_ecdsa-cert type -1
#18 0.829 debug1: identity file /root/.ssh/id_ecdsa_sk type -1
#18 0.829 debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
#18 0.829 debug1: identity file /root/.ssh/id_ed25519 type -1
#18 0.829 debug1: identity file /root/.ssh/id_ed25519-cert type -1
#18 0.829 debug1: identity file /root/.ssh/id_ed25519_sk type -1
#18 0.829 debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
#18 0.829 debug1: identity file /root/.ssh/id_xmss type -1
#18 0.829 debug1: identity file /root/.ssh/id_xmss-cert type -1

You can check that the key was loaded using:

ssh-add -l

The name of each key should appear at the end of the line. It has to be one of the defaults as mentioned above (you may also be able to fiddle around with a Host entry in the docker .ssh/config file).

Build the Docker Image

To build the image, we now run docker like so:

DOCKER_BUILDKIT=1 docker build --progress=plain .

(you may, of course, use other options such as --build-arg GO_VERSION=... to force a version of golang)

The --progress=plain gives you the ability to better see what is going on. Somehow, the DOCKER_BUILDKIT=1 prevents Docker from saving the intermediate images and containers so you would not be able to debug much without the option.

The --ssh default Option

This command line option may still be required. I actually use it. However, in the latest versions of docker, it automatically gets turn on if a [email protected] (or similar?) is detected. I'm not so sure that it can detect such in all situations. If you are having problems, make sure to include that option on your ... docker build ... command line.

There is no need to specify any specifics in my experience. Just default is enough.

Debugging the SSH Connection

If you have issues with the connection (i.e. SSH telling you that the connection was refused), then you can add a RUN command before the RUN ... go build ... to debug just that part:

RUN ssh -A -v -l git github.com

The -A option tells SSH to use the ssh-agent to retrieve the private key.
The -v asks SSH to print out debug information.
The -l option defines the user name. For github.com, you are expected to use git as the user name. By default, ssh uses $USER which inside a Docker would be root. That won't work.

If the connection works, github.com tells you that you were authorized but there is no shell to connect to so you get kicked out immediately. If you do not see that friendly message, then the SSH is not yet correctly setup. In fact, you can test that connection in your console like so:

$ ssh -l git github.com
PTY allocation request failed on channel 0
Hi <your-name>! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

Issue 1: apparmor

All of that did not work for me. The fact is that the ssh-agent creates a socket which is hidden under /run/user/<uid>/keyring/ssh and that path is not allowed to docker ... tools & services by default. At least, if your kernel has apparmor like Ubuntu servers do, it's not going to work.

You can see that this happens by looking at your /var/log/syslog or such similar file (could be /var/log/auth.log). There will be a DENIED error like so:

Oct 28 10:42:13 ubuntu2004 kernel: [78018.511407] audit: type=1400 audit(1635442933.692:143): apparmor="DENIED" operation="connect" profile="snap.docker.docker" name="/run/user/1000/keyring/ssh" pid=36260 comm="docker" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000

We see the complete path to my keyring socket, the name of the apparmor profile that denied access, and the operation, which here is "connect". To fix the issue you first need to find the profile. This is under /var/lib/snapd:

/var/lib/snapd/apparmor/profiles/snap.docker.docker

Then run this command:

$ sudo apparmor_parser -r \
          /var/lib/snapd/apparmor/profiles/snap.docker.docker

to refresh the apparmor settings.

If you don't use the snap version of Docker, profiles are generally found under /etc/apparmor.d/..., but I could not see the file in newer versions of docker (2021)...

Edit that file, go to the end, and just before the closing } character, enter this line:

/run/user/1000/keyring/ssh rw,

This means docker will be able to read and write to this specific socket.

Obviously, the 1000 is a specific user. Use your user ID (id -u) or the user identifier that is to run docker build ... if not you.

You may also allow all the users on that computer, but this is not recommended:

/run/user/[0-9]*/keyring/ssh rw,

(It is still pretty safe since you only give that permission to docker, but you never know...)

Issue 2: Key Filename

I repeat the second issue here because that is very important. The key lookup from Docker is going to search for a key named id_rsa (and other similar default key names, see above). If you use a special name for your key, say github_rsa, then it won't be picked up by Docker.

You may be able to use the .ssh/config file for this purpose by adding:

Host github.com
  IdentityFile /root/.ssh/github_rsa

At some point, I fiddled with such but was not able to make it work. Probably because of Issue #1 (a.k.a. apparmor). If you are sharing your Dockerfile with many programmers, though, using a special name in this way needs to be well documented. Most programmers do not do such things and it could take them a while to find out why they can't create the Docker image on their system.

Do Not chmod Anything!

On many pages/answers, you see that permissions are often resolved using the chmod command. For example, someone who thinks that their key is not accessible to Docker because of the permissions being 700 (rwx-----) on their /run/user/1000 folder or 600 (rw-------) on their ~/.ssh/... files may think that changing those permissions will help. It won't. The entry in the ssh-agent is enough to share your private key as required.

Side Note About Using .netrc

As far as I can tell, when you use a .netrc, you include your credential in the Docker image. This means anyone who gets a copy of your image has your credentials. Probably not something you want. If your images are only used internally, it may be okay...