Edenmal

Sysadmin Garden of Eden

GitLab + Kubernetes: Running CI Runners in Kubernetes

Table of Contents
  1. 1. Intro
    1. 1.1. Requirements
  2. 2. Step 1 - Verify Kubernetes cluster “connectivity”
  3. 3. Step 2 - Get GitLab CI Register Token from GitLab
  4. 4. Step 3 - Write Kubernetes manifests for GitLab CI Runners
    1. 4.1. Kubernetes API References
    2. 4.2. Namespace for the build tasks
    3. 4.3. ConfigMap for the Environment Variables
    4. 4.4. Secret for the Token Environment Variable
    5. 4.5. Statefulset for the actually running the Runners in Kubernetes
  5. 5. Step 4 - Create the manifests
  6. 6. Step 5 - Check the GitLab CI Runners
  7. 7. Troubleshooting
    1. 7.1. Check the logs of the GitLab CI runner Pods
  8. 8. Summary

Intro

In this post, I’ll be going over using GitLab CI to create your application’s container Continuous Delivery to Kubernetes.
This is the second post in the three post series about Kubernetes and GitLab. The first post can be found here: Edenmal - GitLab + Kubernetes: Perfect Match for Continuous Delivery with Container.

NOTE Please check the requirements before beginning.

Requirements

Step 1 - Verify Kubernetes cluster “connectivity”

To check if you have proper Kubernetes cluster connection, run the following command:

$ kubectl cluster-info
kubectl cluster-info
Kubernetes master is running at https://k8s.example.com:443
KubeDNS is running at https://k8s.example.com:443/api/v1/namespaces/kube-system/services/kube-dns/proxy
[...]
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$

The kubectl command cluster-info shows you if you can connect to the cluster and list the available cluster services (in the ouput: Kubernetes apiserver and KubeDNS service).

Step 2 - Get GitLab CI Register Token from GitLab

Go to your GitLab instance and go to the Admin area.

To go to your Admin area, click the wrench icon next to the search bar in the top right of any GitLab page:
GitLab Admin area icon position

Next click on the Runners tab in the navbar.
GitLab Admin area Runners tab

Et-voilà! There is your token (in the picture the token would be where the blanked out area is).
GitLab Admin area Runners Token

Now copy the token somewhere safe for usage in Step 3 - Write manifest for GitLab CI Runners.

NOTE Keep the token secure!

Step 3 - Write Kubernetes manifests for GitLab CI Runners

This step will show you the manifests for the GitLab CI runner manifests for Kubernetes.

Kubernetes API References

The Kubernetes API references can be found on the official Kubernetes.io page here: https://kubernetes.io/docs/reference/
All manifests should be compatible with Kubernetes 1.7.

Namespace for the build tasks

Replace YOUR_GITLAB_BUILD_NAMESPACE with a name for the namespace in which the GitLab CI builds are run. The separate namespace for the GitLab CI builds is useful for detecting stuck containers (for example when there was an issue with the runner not cleaning up).

apiVersion: v1
kind: Namespace
metadata:
name: YOUR_GITLAB_BUILD_NAMESPACE

ConfigMap for the Environment Variables

First we gonna start with the ConfigMap for the basic configuration using environment variables of the GitLab CI Runner image:
You have to point YOUR_GITLAB_CI_SERVER_URL to your GitLab instance’s URL with /ci appended to it (like this https://gitlab.example.com/ci).
Also you need to replace YOUR_GITLAB_BUILD_NAMESPACE with the name of the namespace from the step Namespace for the build tasks.

apiVersion: v1
data:
REGISTER_NON_INTERACTIVE: "true"
CI_SERVER_URL: "YOUR_GITLAB_CI_SERVER_URL"
METRICS_SERVER: "0.0.0.0:9100"
RUNNER_REQUEST_CONCURRENCY: "4"
RUNNER_EXECUTOR: "kubernetes"
KUBERNETES_NAMESPACE: "YOUR_GITLAB_BUILD_NAMESPACE"
KUBERNETES_PRIVILEGED: "true"
KUBERNETES_CPU_LIMIT: "1"
KUBERNETES_MEMORY_LIMIT: "1Gi"
KUBERNETES_SERVICE_CPU_LIMIT: "1"
KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
KUBERNETES_HELPER_CPU_LIMIT: "500m"
KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
KUBERNETES_PULL_POLICY: "if-not-present"
KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
KUBERNETES_POLL_INTERVAL: "5"
KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-cm

The above ConfigMap also adds resource limits to the build containers run. You can change them as long as they follow the Kubernetes resource limits units.

Secret for the Token Environment Variable

Then we’ll use the GitLab CI runner token to create a secret in Kubernetes for it to use it in the Statefulset.

To encode the token as base64, you can run something like this:

$ echo YOUR_GITLAB_CI_TOKEN | base64 -w0

(base64 should be available and already installed on most Linux distributions)

Replace YOUR_BASE64_ENCODED_TOKEN with the output from the above command.

apiVersion: v1
data:
token: YOUR_BASE64_ENCODED_TOKEN
kind: Secret
metadata:
name: gitlab-ci-token
labels:
app: gitlab-ci-runner

Statefulset for the actually running the Runners in Kubernetes

This container specification for the GitLab CI runner has a twist added to it. On start the runer tries to unregister any runner with the same name. This is especially useful when a node is lost (aka NodeLost event). It then tries to re-register itself and then begin to run. On a normal stop of the Pod, the GitLab CI runner will run the unregister command to try to unregister itself, so it won’t be used by GitLab anymore. This is done by using Kubernetes lifecycle “hooks”, documentation about them can be found here: Kubernetes.io - Container Lifecycle Hooks.

Another awesome thing is that using envFrom allows to specify Secrets and ConfigMaps to be used as environment variables (variables are only set when they match the specific regex for environment variables).

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: gitlab-ci-runner
labels:
app: gitlab-ci-runner
spec:
replicas: 2
serviceName: gitlab-ci-runner
template:
metadata:
labels:
app: gitlab-ci-runner
spec:
containers:
- image: gitlab/gitlab-runner:v9.5.0
command:
- /bin/bash
- -c
- "/usr/bin/gitlab-ci-multi-runner unregister -t $GITLAB_CI_TOKEN -n $RUNNER_NAME || true; /usr/bin/gitlab-ci-multi-runner register -r $GITLAB_CI_TOKEN && sed -i 's/^concurrent.*/concurrent = 3/' /etc/gitlab-runner/config.toml; exec /usr/bin/gitlab-ci-multi-runner run -u gitlab-runner"
envFrom:
- configMapRef:
name: gitlab-ci-runner-cm
- secretRef:
name: gitlab-ci-token
env:
- name: GITLAB_CI_TOKEN
valueFrom:
secretKeyRef:
name: gitlab-ci-token
key: token
- name: RUNNER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- containerPort: 9100
name: http-metrics
protocol: TCP
name: gitlab-ci-runner
lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- "/usr/bin/gitlab-ci-multi-runner unregister -t $GITLAB_CI_TOKEN -n $RUNNER_NAME"
restartPolicy: Always

Step 4 - Create the manifests

To interact with the Kubernetes cluster the kubectl programm is used. To create/“run” a manifest on the Kubernetes cluster the subcommand create is used.
An example, like this:

kubectl create -f FILE_NAME

The FILE_NAME would be the corresponding name of file you saved the manifest(s) in.
To specify multiple manifests in one go, just add -f FILE_NAME pepr file behind the create. Like this kubectl create -f FILE_NAME_1 -f FILE_NAME_2 -f FILE_NAME_3 ....

Now specify the manifests you created in the Step 3 - Write manifest for GitLab CI Runners.

Step 5 - Check the GitLab CI Runners

NOTE: For an example GitLab CI pipeline construct, please take a look at the repository on GitHub galexrt/presentation-gitlab-k8s.

Go to the GitLab Admin area and there to the “Runners” tab as shown above.
GitLab Runners List
In the table/list below where you got the GitLab runner token from, the runners should habe appeared.
If not check the Troubleshooting section below.

Troubleshooting

Check the logs of the GitLab CI runner Pods

To get logs from one of the GitLab CI runner pods, you can use the following command:

$ kubectl logs -f gitlab-ci-runner-0
[...]
GITLAB_CI_RUNNER_0_POD_LOGS_HERE
[...]

The -f option is causing the logs to be streamed, like with the tail command.
Check the logs for any errors.

Summary

Now you should have two (or more depending if you have changed the replicas count in the StatefulSet manifest) GitLab CI runner that use Kubernetes as an so called executor for CI tasks.

Have Fun!