2. JTI - K8s - production

Goals

In the last exercise we managed to get a Kubernetes (K8s) environment up for local development. Now we will look at how to create and deploy this to a self hosted kubernetes cluster using GitLab. Of course, you are free to choose what ever application you like, and you do not need to deploy to CSCloud, but this guide is heavily focused on doing just that.

Prerequisites

Make sure to have your application in a Gitlab project. This guide makes use of our example project and the files in this project. Pay extra attention to:

  1. ./k8s/taskit.yaml
  2. .gitlab-ci.yml

Exercise

The exercise will go through these steps:

  1. Create a Kubernetes cluster in CSCloud incl. a load balancer and a jump host
  2. Connect the cluster to GitLab (agent and container registry)

Build the infrastructure

We will start by creating a Kubernetes cluster in CSCloud. To make this step as easy as possible, we have prepared a Terraform script for doing this.

Please follow the steps in Example - Kubernetes Cluster Infrastructure.

K8s CSCloud

  • Jump Host Router - The Jump Host Router acts as the intermediary, directing SSH traffic from the public internet to the jump host within our network.

  • Jump Host - A pivotal node for secure access, the Jump Host provides a controlled pathway to the Kubernetes cluster.

  • K8s Control Plane 1 and 2 - The control planes nodes serve as the central coordination hubs for the Kubernetes cluster, facilitating Helm chart installations and overall cluster management.

  • K8S Worker Nodes 1, 2, and 3 - These worker nodes are the workforce of the cluster, responsible for running application containers and executing tasks.

  • K8s Load Balancer - Functioning as a traffic director, the load balancer ensures even distribution of network load across the cluster nodes, promoting service stability and efficiency.

Connect Cluster to GitLab and deploy

A nice feature in GitLab is that we can install a GitLab agent in our cluster to authorize GitLab to work with the cluster.

1. Connect to the cluster

Navigate to GitLab->Your Application Project->Operate->Kubernetes clusters and click "Connect a cluster".

In the searchbox, write "cscloud-agent" and press enter. Press "Register".

Now you will be presented with a Agent access token and installation suggestion for the cluster.

  1. Copy the instructions,
  2. open a ssh-connection to your jump host,
  3. from the jump host, connect to the control plane: shh k8s-control-plane-1
  4. paste the instructions into the server. (Helm is installed on k8s-control-plane-1)

Close the installation suggestion and you should now see the agent being connected. (reload page)

Agent connected

In the deploy-pipeline we will connect to the cluster using this agent. (go have a look)

You will see that to change K8s context we use the environment variable, K8S_CONTEXT. Create it (Settings->CI/CD->Variables) and set it to: <project-path>:cscloud-agent example: 2dv013/student/xx22yy/part2-architecture/jti-application:cscloud-agent

That should be it!

2. Deploy from Container Registry using a deploy token

Container registry

The build script that is in our example pipeline is building a new image of our Just Task It-application. This image is placed in our GitLab Container Registry.

If the pipeline have built the image, navigate to Deploy->Container Registry. Click the taskit URL and you should find the tag "latest". Copy the image path and edit ./k8s/taskit.yaml. Replace the path to the image under Deployment->taskit

Since this registry is private, the cluster will not be able to pull images from your registry. To allow this, we need to create a "Deploy Token" and place that as a secret in the cluster.

  1. Visit: Settings->Repository->Deploy tokens
  2. Create a token with the name gitlab-deploy-token and the Scope read_registry. (date and username can be left blank)

This token will automatically be copied to a cluster secret named regcred every time the GitLab CI pipeline is executed. Of course, we could copy the credentials in to a kubernetes secret manually instead. (In the cluster you can list secrets by kubectl get secrets, however, it will not show up until you have deployed at least once.)

Deploy!

Now, try to deploy by going to Build->Pipeline and manually start the deploy.

Deploy button

If everything works out, you should be able to go into your cluster, run kubectl get svc to list all services, incl. taskit. Example output:

‌‌‌‌‌

Try to curl the port on one or more of your nodes. Example:

‌‌‌‌‌

If you get the HTML of the JTI-application everything is working! But, we can not connect to the application from outside of CSCloud. Try to access it via your load balancer and you should get a "502 Bad Gateway" from the NGINX load balancer.

Set up an ingress controller

No the last piece of the puzzle, to create an ingress controller as the middle man between our load balancer and the taskit service.

  1. Add the NGINX Ingress Helm Repository.

    ‌‌‌‌‌

  2. Install the NGINX Ingress Controller.

    ‌‌‌‌‌

  3. Verify the installation

    ‌‌‌‌‌

    and make sure that the ingress will direct traffic to your taskit service:

    ‌‌‌‌‌

    Take note of the port mapped to 80, in the case above, 30411. We now need to configure our load balancer to connect on that port.

Configure the load balancer

Connect to the jump host and then to the load balancer:

‌‌‌‌‌

Edit the nginx-config:

‌‌‌‌‌

In the upstream worker-nodes block, add the port to the ingress controller:

Example:

‌‌‌‌‌

Save and restart NGINX:

‌‌‌‌‌

Test it!

Now, try to visit the load balancer IP in your browser. If everything works you should se Just Task It!

Environment URL (optional)

To get a nice way to reach your application from Gitlab Deployments->Environments. Set the environment variable TASKIT_URL to the IP you assigned to the load balancer.

Create the Session env as secrets (optional)

The session name and secret have so far been defined in the taskit manifest, but it is important that these secrets remain secret. We can manually create a secret in our cluster and reference that secret from our manifest file.

Log in to the control plane and run (please replace my examples with your own secrets):

‌‌‌‌‌

You can list secrets by:

‌‌‌‌‌

and see details about your secret by:

‌‌‌‌‌

Now, modify taskit.yaml by replacing the env. part of the sessions with:

‌‌‌‌‌

Try it out

Now every commit to GitLab will rebuild your image and make it available for deployment. Of course this is a bit extreme and you can tweak this a lot.