From ac737662d3f02bda2cee303cce2ce6fb16b70883 Mon Sep 17 00:00:00 2001 From: Thomas Schwery Date: Mon, 30 Sep 2019 20:16:19 +0200 Subject: [PATCH] First article about Hetzner Cloud and GitLab Runner with Docker Machine --- .../2019-09-24-hetzner-cloud-docker-ci.md | 222 ++++++++++++++++++ 1 file changed, 222 insertions(+) create mode 100644 articles/2019-09-24-hetzner-cloud-docker-ci.md diff --git a/articles/2019-09-24-hetzner-cloud-docker-ci.md b/articles/2019-09-24-hetzner-cloud-docker-ci.md new file mode 100644 index 0000000..538a071 --- /dev/null +++ b/articles/2019-09-24-hetzner-cloud-docker-ci.md @@ -0,0 +1,222 @@ +--- +title: GitLab runners on Hetzner Cloud +date: 2019-09-10 18:30:00 +--- + +Because my different projects use a lot of pipelines for tests and deployments, +I wanted to use the different possibilities of the cloud to reduce the load on +my GitLab server and instead run these short-lived containers on other +virtual machines. + +In this post, I will explain the different steps needed to install a GitLab +Runner and the necessary adapter for Docker Machine in a container that +can be deployed on your server, configure the runner to launch new Hetzner Cloud +servers and deploy a shared cache to centralize the cache storage between +theses machines. The configuration presented here is based on the official +GitLab container configured and launched through Docker Compose. + +First, on the Hetzner Cloud Console, create a new project and an API key +for this project. You should use this project only for the automatically +created runners. The key can be created under *Access* -> *API Tokens* -> *New* +in the project page on the Hetzner Cloud Console. + +### Docker Compose +Then, on my `docker-compose.yml` that contains the GitLab service, I added +a `runner` service that will build a custom container with the Docker Machine +driver needed to talk to the Hetzner API. + +``` +version: '2' + +services: + runner: + restart: on-failure + build: ./hetzner-gitlab-runner + volumes: + - /srv/gitlab-ce/runner:/etc/gitlab-runner + + gitlab: + [...] +``` + +Change the volume storing the configuration with something that is matching +the volumes of your GitLab service container. Here, I will store the +configuration in the `/srv/gitlab-ce/runner/config.toml` configuration file. + +We then need to add the instructions to build our custom hetzner-gitlab-runner +container image. + +### GitLab Runner with the Hetzner Docker Machine driver +For this custom container, we will create a subfolder `hetzner-gitlab-runner` +containing the `Dockerfile` responsible for building the container image. +I took this route to facilitate matching the versions of your GitLab and +GitLab runner installation. + +The `hetzner-gitlab-runner/Dockerfile` is as follow. It uses a builder to +download the 2.0.1 release of the Docker Machine driver and then copies +that into the official gitlab-runner container image. You can change the +gitlab-runner version here to match your deployed GitLab version. + +``` +FROM alpine:3.10 as builder + +RUN apk add --no-cache --virtual .fetch-deps curl tar + +WORKDIR /build +RUN curl -sLo hetzner.tar.gz https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/2.0.1/docker-machine-driver-hetzner_2.0.1_linux_amd64.tar.gz +RUN tar xf hetzner.tar.gz && chmod +x docker-machine-driver-hetzner + +FROM gitlab/gitlab-runner:v12.3.0 +COPY --from=builder /build/docker-machine-driver-hetzner /usr/bin +``` + +### GitLab runner configuration + +In the `/srv/gitlab-ce/runner/config.toml` configuration, configure everything +needed for Docker Machine to create new runner virtual machines. + +``` +concurrent = 2 +check_interval = 0 + +[[runners]] + name = "docker-runner" + url = "https://git.your.domain/" + token = "your-gitlab-token" + executor = "docker+machine" + [runners.docker] + tls_verify = false + image = "maven:3-jdk-8" + privileged = false + disable_cache = false + volumes = ["/cache"] + shm_size = 0 + [runners.cache] + [runners.machine] + IdleCount = 0 + IdleTime = 1800 + MachineDriver = "hetzner" + MachineName = "gitlab-runner-%s" + MachineOptions = [ + "hetzner-api-token=your-hetzner-api-key", + "hetzner-image=debian-10", + "hetzner-server-type=cx11", + "hetzner-server-location=fsn1", + ] +``` + +In this file, you will have to insert the Hetzner Cloud API created in the +first step and the GitLab URL and registration token that can be found in +the Runner administration page on your GitLab instance. + +Once these three changes are made, you can test these changes on GitLab, +you should see a new server being created on the Cloud Console, the runner +logs should show that a runner is executing the pipeline. For now, you +should get a warning in the logs that the cache will not be downloaded +from a shared cache server. This is normal as we have not yet configured +the cache. + +### Minio service + +To store the cache from the different runners, we will deploy a new Minio +service on our Docker Compose configuration. This will allow each cache +to be stored on the main server and be accessible to each runner machine. + +On my server, I deployed it behind Apache to secure the access with HTTPS, +same as with the other containers running on it. You will need to adapt +that configuration to your system. + +First, generate two random chains of characters that will act as access +and secret keys. + +In the Docker Compose file, we will add a new Minio service with a volume +to persist the cache data and the environement variables defining our +keys : + +``` +version: '2' + +services: + runner: + [...] + + minio: + restart: on-failure + image: minio/minio:RELEASE.2019-09-26T19-42-35Z + ports: + - "19000:9000" + environment: + MINIO_ACCESS_KEY: 3eA4pp73u8rtsydZ + MINIO_SECRET_KEY: DRSweWE3GDmNygTa + command: server /data + volumes: + - /srv/gitlab-ce/minio-data:/data + + gitlab: + [...] +``` + +As before, change the path to your volume so that the data storage matches +with what you used for your GitLab install. Adapt also the exported port. +Here I used 19000 because that's my convention, use whatever matches your +install. + +For reference, the configuration for Apache is the following. You will need +to adapt for your domain, your SSL key and certificate and the port defined +in your Docker Compose file. + +``` + + ServerName s3.your.domain + + SSLEngine On + + ProxyPreserveHost On + ProxyRequests Off + ProxyVia Block + + ProxyPass / http://localhost:19000/ + ProxyPassReverse / http://localhost:19000/ + + Include /etc/letsencrypt/options-ssl-apache.conf + SSLCertificateFile /etc/letsencrypt/live/your.domain/fullchain.pem + SSLCertificateKeyFile /etc/letsencrypt/live/your.domain/privkey.pem + +``` + +Once Apache is configured, you should be able to access s3.your.domain with +your browser and login using the access and secret keys defined before. + +In the Minio web interface, create a new `runner` bucket by clicking on +the orange plus on the bottom right of the screen. This will be needed +in the next steps. + +### Minio as a Runner cache + +In the previous section about the GitLab Runner `config.toml` configuration, +we left the `[runners.cache]` section empty. We will now configure the +Runner to use the newly created Minio S3 storage as a shared cache. + +``` + [runners.cache] + Type = "s3" + Path = "cache" + Shared = false + [runners.cache.s3] + ServerAddress = "s3.your.domain" + AccessKey = "3eA4pp73u8rtsydZ" + SecretKey = "DRSweWE3GDmNygTa" + BucketName = "runner" + Insecure = false +``` + +Don't forget to update the AccessKey, the SecretKey and your Address to +match what was configured in the previous section. + +Once the configuration is updated, you can stop the `gitlab_runner_1` +container and restart it using Docker Compose to force the GitLab Runner +to refresh its configuration. + +Launching a new pipeline, you should now have a cache available. You can +verify that the runner uploads it after the run by checking the content +of the bucket after a successful execution of a job.