222 lines
7.3 KiB
Markdown
222 lines
7.3 KiB
Markdown
---
|
|
title: GitLab runners on Hetzner Cloud
|
|
date: 2019-09-10 18:30:00
|
|
---
|
|
|
|
Because my different projects use a lot of pipelines for tests and deployments,
|
|
I wanted to use the different possibilities of the cloud to reduce the load on
|
|
my GitLab server and instead run these short-lived containers on other
|
|
virtual machines.
|
|
|
|
In this post, I will explain the different steps needed to install a GitLab
|
|
Runner and the necessary adapter for Docker Machine in a container that
|
|
can be deployed on your server, configure the runner to launch new Hetzner Cloud
|
|
servers and deploy a shared cache to centralize the cache storage between
|
|
theses machines. The configuration presented here is based on the official
|
|
GitLab container configured and launched through Docker Compose.
|
|
|
|
First, on the Hetzner Cloud Console, create a new project and an API key
|
|
for this project. You should use this project only for the automatically
|
|
created runners. The key can be created under *Access* -> *API Tokens* -> *New*
|
|
in the project page on the Hetzner Cloud Console.
|
|
|
|
### Docker Compose
|
|
Then, on my `docker-compose.yml` that contains the GitLab service, I added
|
|
a `runner` service that will build a custom container with the Docker Machine
|
|
driver needed to talk to the Hetzner API.
|
|
|
|
```
|
|
version: '2'
|
|
|
|
services:
|
|
runner:
|
|
restart: on-failure
|
|
build: ./hetzner-gitlab-runner
|
|
volumes:
|
|
- /srv/gitlab-ce/runner:/etc/gitlab-runner
|
|
|
|
gitlab:
|
|
[...]
|
|
```
|
|
|
|
Change the volume storing the configuration with something that is matching
|
|
the volumes of your GitLab service container. Here, I will store the
|
|
configuration in the `/srv/gitlab-ce/runner/config.toml` configuration file.
|
|
|
|
We then need to add the instructions to build our custom hetzner-gitlab-runner
|
|
container image.
|
|
|
|
### GitLab Runner with the Hetzner Docker Machine driver
|
|
For this custom container, we will create a subfolder `hetzner-gitlab-runner`
|
|
containing the `Dockerfile` responsible for building the container image.
|
|
I took this route to facilitate matching the versions of your GitLab and
|
|
GitLab runner installation.
|
|
|
|
The `hetzner-gitlab-runner/Dockerfile` is as follow. It uses a builder to
|
|
download the 2.0.1 release of the Docker Machine driver and then copies
|
|
that into the official gitlab-runner container image. You can change the
|
|
gitlab-runner version here to match your deployed GitLab version.
|
|
|
|
```
|
|
FROM alpine:3.10 as builder
|
|
|
|
RUN apk add --no-cache --virtual .fetch-deps curl tar
|
|
|
|
WORKDIR /build
|
|
RUN curl -sLo hetzner.tar.gz https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/2.0.1/docker-machine-driver-hetzner_2.0.1_linux_amd64.tar.gz
|
|
RUN tar xf hetzner.tar.gz && chmod +x docker-machine-driver-hetzner
|
|
|
|
FROM gitlab/gitlab-runner:v12.3.0
|
|
COPY --from=builder /build/docker-machine-driver-hetzner /usr/bin
|
|
```
|
|
|
|
### GitLab runner configuration
|
|
|
|
In the `/srv/gitlab-ce/runner/config.toml` configuration, configure everything
|
|
needed for Docker Machine to create new runner virtual machines.
|
|
|
|
```
|
|
concurrent = 2
|
|
check_interval = 0
|
|
|
|
[[runners]]
|
|
name = "docker-runner"
|
|
url = "https://git.your.domain/"
|
|
token = "your-gitlab-token"
|
|
executor = "docker+machine"
|
|
[runners.docker]
|
|
tls_verify = false
|
|
image = "maven:3-jdk-8"
|
|
privileged = false
|
|
disable_cache = false
|
|
volumes = ["/cache"]
|
|
shm_size = 0
|
|
[runners.cache]
|
|
[runners.machine]
|
|
IdleCount = 0
|
|
IdleTime = 1800
|
|
MachineDriver = "hetzner"
|
|
MachineName = "gitlab-runner-%s"
|
|
MachineOptions = [
|
|
"hetzner-api-token=your-hetzner-api-key",
|
|
"hetzner-image=debian-10",
|
|
"hetzner-server-type=cx11",
|
|
"hetzner-server-location=fsn1",
|
|
]
|
|
```
|
|
|
|
In this file, you will have to insert the Hetzner Cloud API created in the
|
|
first step and the GitLab URL and registration token that can be found in
|
|
the Runner administration page on your GitLab instance.
|
|
|
|
Once these three changes are made, you can test these changes on GitLab,
|
|
you should see a new server being created on the Cloud Console, the runner
|
|
logs should show that a runner is executing the pipeline. For now, you
|
|
should get a warning in the logs that the cache will not be downloaded
|
|
from a shared cache server. This is normal as we have not yet configured
|
|
the cache.
|
|
|
|
### Minio service
|
|
|
|
To store the cache from the different runners, we will deploy a new Minio
|
|
service on our Docker Compose configuration. This will allow each cache
|
|
to be stored on the main server and be accessible to each runner machine.
|
|
|
|
On my server, I deployed it behind Apache to secure the access with HTTPS,
|
|
same as with the other containers running on it. You will need to adapt
|
|
that configuration to your system.
|
|
|
|
First, generate two random chains of characters that will act as access
|
|
and secret keys.
|
|
|
|
In the Docker Compose file, we will add a new Minio service with a volume
|
|
to persist the cache data and the environement variables defining our
|
|
keys :
|
|
|
|
```
|
|
version: '2'
|
|
|
|
services:
|
|
runner:
|
|
[...]
|
|
|
|
minio:
|
|
restart: on-failure
|
|
image: minio/minio:RELEASE.2019-09-26T19-42-35Z
|
|
ports:
|
|
- "19000:9000"
|
|
environment:
|
|
MINIO_ACCESS_KEY: 3eA4pp73u8rtsydZ
|
|
MINIO_SECRET_KEY: DRSweWE3GDmNygTa
|
|
command: server /data
|
|
volumes:
|
|
- /srv/gitlab-ce/minio-data:/data
|
|
|
|
gitlab:
|
|
[...]
|
|
```
|
|
|
|
As before, change the path to your volume so that the data storage matches
|
|
with what you used for your GitLab install. Adapt also the exported port.
|
|
Here I used 19000 because that's my convention, use whatever matches your
|
|
install.
|
|
|
|
For reference, the configuration for Apache is the following. You will need
|
|
to adapt for your domain, your SSL key and certificate and the port defined
|
|
in your Docker Compose file.
|
|
|
|
```
|
|
<VirtualHost *:443>
|
|
ServerName s3.your.domain
|
|
|
|
SSLEngine On
|
|
|
|
ProxyPreserveHost On
|
|
ProxyRequests Off
|
|
ProxyVia Block
|
|
|
|
ProxyPass / http://localhost:19000/
|
|
ProxyPassReverse / http://localhost:19000/
|
|
|
|
Include /etc/letsencrypt/options-ssl-apache.conf
|
|
SSLCertificateFile /etc/letsencrypt/live/your.domain/fullchain.pem
|
|
SSLCertificateKeyFile /etc/letsencrypt/live/your.domain/privkey.pem
|
|
</VirtualHost>
|
|
```
|
|
|
|
Once Apache is configured, you should be able to access s3.your.domain with
|
|
your browser and login using the access and secret keys defined before.
|
|
|
|
In the Minio web interface, create a new `runner` bucket by clicking on
|
|
the orange plus on the bottom right of the screen. This will be needed
|
|
in the next steps.
|
|
|
|
### Minio as a Runner cache
|
|
|
|
In the previous section about the GitLab Runner `config.toml` configuration,
|
|
we left the `[runners.cache]` section empty. We will now configure the
|
|
Runner to use the newly created Minio S3 storage as a shared cache.
|
|
|
|
```
|
|
[runners.cache]
|
|
Type = "s3"
|
|
Path = "cache"
|
|
Shared = false
|
|
[runners.cache.s3]
|
|
ServerAddress = "s3.your.domain"
|
|
AccessKey = "3eA4pp73u8rtsydZ"
|
|
SecretKey = "DRSweWE3GDmNygTa"
|
|
BucketName = "runner"
|
|
Insecure = false
|
|
```
|
|
|
|
Don't forget to update the AccessKey, the SecretKey and your Address to
|
|
match what was configured in the previous section.
|
|
|
|
Once the configuration is updated, you can stop the `gitlab_runner_1`
|
|
container and restart it using Docker Compose to force the GitLab Runner
|
|
to refresh its configuration.
|
|
|
|
Launching a new pipeline, you should now have a cache available. You can
|
|
verify that the runner uploads it after the run by checking the content
|
|
of the bucket after a successful execution of a job.
|