2. JTI - K8s - production

Goals

In the last exercise we managed to get a Kubernetes (K8s) environment up for local development. Now we will look at how to create and deploy this to a self hosted kubernetes cluster using GitLab. Of course, you are free to choose what ever application you like, and you do not need to deploy to CSCloud, but this guide is heavily focused on doing just that.

Prerequisites

Make sure to have your application in a Gitlab project. This guide makes use of our example project and the files in this project. Pay extra attention to:

1. ./k8s/taskit.yaml
2. .gitlab-ci.yml

Exercise

The exercise will go through these steps:

1. Create a Kubernetes cluster in CSCloud incl. a load balancer and a jump host
2. Connect the cluster to GitLab (agent and container registry)

Build the infrastructure

We will start by creating a Kubernetes cluster in CSCloud. To make this step as easy as possible, we have prepared a Terraform script for doing this.

Please follow the steps in Example - Kubernetes Cluster Infrastructure.

• Router: Connects the public network to the app-network, directing traffic as needed.

• Jump Host :

Acts as a secure gateway to the internal network. Routes administrative traffic, ensuring controlled entry and exit. Isolates internal resources from public threats. Host-name: jump-host

• Load Balancer:

Uses HAProxy to manage traffic load across nodes. Redirects HTTP to HTTPS and balances API traffic with least connections. Ensures backend node health with SSL checks. Routes requests to the cluster on port 6443 for Kubernetes API traffic. Enhances fault tolerance by distributing load to reduce downtime. Hostname: load-balancer

• Control Plane Nodes (control-plane-1, control-plane-2, control-plane-3):

Manage cluster operations and maintain high availability. Distribute roles like API servers and etcd management. Coordinate with the load balancer for request distribution.

• Worker Nodes (worker-1, worker-2, worker-3):

Host and run containerized applications. Manage workloads and enable scalable deployments.

Secure access to nodes is managed via the jump host, with all traffic routed through monitored channels to enhance security and management, ensuring an efficient and resilient environment crucial for scalable and robust application deployments.

Connect Cluster to GitLab and deploy

A useful feature in GitLab is the ability to install a GitLab agent in your Kubernetes cluster. This allows GitLab to interact seamlessly with the cluster. We have created an agent for CS Cloud that you can use. Here is how to add it to your repo.

Prerequisites

• You must be a Maintainer of the repository to perform the following steps.

Steps to Connect Your Cluster

1. Access GitLab Cluster Settings:

   • Navigate to GitLab -> Your Application Project -> Operate -> Kubernetes Clusters.
   • Click on "Connect a cluster".

2. Register the GitLab Agent:

   • In the search box, enter "cscloud-agent" and press enter.
   • Click on "Register" when it appears.

3. Install the Agent on Your Cluster:

   • You will receive an Agent access token along with installation instructions for your cluster.
   • Follow these steps to install the agent:
     1. Copy the installation instructions.
     2. Open an SSH connection to your jump host. ssh -i [PATH_TO_SSH_KEY_PEM_FILE] ubuntu@[YOUR_JUMP_HOST_PUBLIC_IP]
     3. Paste the copied instructions into the server terminal. (Note: Helm is already installed on jump-host).

4. Verify Agent Connection:

   • Once the installation is complete, close the installation dialog.
   • Refresh the page and you should see that the agent is successfully connected.

Configuration in the Deploy Pipeline

• The deployed pipeline connects to the cluster using the installed GitLab agent.
• To switch Kubernetes contexts easily, use the K8S_CONTEXT environment variable:
  • Navigate to Settings -> CI/CD -> Variables.
  • Create a new variable named K8S_CONTEXT and set its value to <project-path>:cscloud-agent.
    • Example: 2dv013/student/xx22yy/part2-architecture/jti-application:cscloud-agent.

This setup should enable smooth and secure interaction between GitLab and your Kubernetes cluster!

2. Deploy from the Container Registry Using a Deploy Token

Container Registry

When you run the example pipeline's build script, it creates a new image of our "Just Task It" application. This image is stored in your GitLab Container Registry.

• Locating Your Image:
  • After the pipeline builds the image, go to Deploy -> Container Registry.
  • Click on the "taskit" URL, where you should find an image tagged as "latest".
  • Copy the image path, then open and edit ./k8s/taskit.yaml. Update the image path under the section Deployment -> taskit to the copied path.

Accessing the Private Registry

Since our Container Registry is private, your Kubernetes cluster needs access credentials to pull images. We achieve this by creating a "Deploy Token" and storing it as a secret in the cluster.

1. Generate a Deploy Token:

   • Go to Settings -> Repository -> Deploy Tokens.
   • Create a new token named gitlab-deploy-token with the scope set to read_registry. (The expiration date and username fields can be left blank.)

2. Token Deployment:

   • The deploy token is automatically copied to a cluster secret named regcred every time the GitLab CI pipeline runs.
   • Alternatively, you could manually create a Kubernetes secret with these credentials. (Use kubectl get secrets to list secrets in the cluster. Note that the secret won't appear until you have executed a deployment at least once.)

By following these steps, you ensure that your cluster can securely access and pull images from your private registry.

Update Skaffold Configuration and Deploy

Update the Skaffold Configuration and CI/CD pipeline file.

Before deploying:

• Ensure that the image path in your skaffold.yaml file is updated to match your specific registry path. Ensure that the image path under the artifacts section matches the one configured in your k8s/taskit.yaml file. This synchronization is crucial for successful deployments.

• Update the same path in .gitlab-ci.yml:

  skaffold deploy --profile production --images ... command uses Skaffold to deploy the application with a specified profile (production), referencing the correct image path. It defaults to the latest commit tag if no specific tag is provided.

Deploy the Application

1. Initiate Deployment:
   • Go to Build -> Pipeline in GitLab.
   • Manually start the deployment process by clicking on the "Deploy" button.

2. Verify Deployment:

• Once the deployment is complete, you can verify it by accessing your cluster.
• Run the following command to list all services, including the "taskit" service:

Example output should display the details of all running services, confirming that "taskit" is correctly deployed.

Try to curl the port on one or more of your nodes. Example:

If you get the HTML of the JTI-application everything is working!

Try it out

Now every commit to GitLab will rebuild your image and make it available for deployment. Of course this is a bit extreme and you can tweak this a lot.

⚠️ WARNING At this point you still can not connect to the application from outside of CSCloud. Try to access it via your Load Balancer and you should get a "502 Bad Gateway" from the NGINX load balancer.

Set up an ingress controller

Now the last piece of the puzzle, to create an ingress controller as the middle man between our load balancer and the taskit service.

1. Add the NGINX Ingress Helm Repository.

   ‌‌‌‌‌

2. Install the NGINX Ingress Controller.

   ‌‌‌‌‌

3. Verify the installation

   ‌‌‌‌‌

   and make sure that the ingress will direct traffic to your taskit service:

   ‌‌‌‌‌

   ❗❗❗ Take note of the port mapped to 80, in the case above, 30411. We now need to configure our load balancer to connect on that port.

Configure the HAProxy Load Balancer

1. SSH into the Jump Host:

2. Access the HAProxy Load Balancer:

3. Backup and Edit Configuration Files:

Change to the HAProxy configuration directory and back up files.

4. Modify HAProxy Configuration:

   Edit the configuration to route traffic to the Ingress Controller.

Update the backend configuration:

5. Validate and Restart HAProxy:

Validate the configuration and restart HAProxy to apply changes.

6. Verify HAProxy:

Ensure HAProxy is running and serving requests correctly.

7. Test Access through HAProxy:

Use the public IP address of HAProxy to test access.

Look for a 200 OK response. Troubleshoot any issues using logs and configuration checks.

Test it!

Now, try to visit the load balancer IP in your browser. If everything works you should se Just Task It!

Environment URL (optional)

To get a nice way to reach your application from Gitlab Deployments->Environments. Set the environment variable TASKIT_URL to the IP you assigned to the load balancer.

Create the Session env as secrets (optional)

The session name and secret have so far been defined in the taskit manifest, but it is important that these secrets remain secret. We can manually create a secret in our cluster and reference that secret from our manifest file.

Log in to the control plane and run (please replace my examples with your own secrets):

You can list secrets by:

and see details about your secret by:

Now, modify taskit.yaml by replacing the env. part of the sessions with: