3. JTI - Production

Goals

In the previous exercies we have looked at how to develop our application locally. Now it is time to take the next step and push the code to production. This includes:

Setting up the infrastructure
Create production-versions of our Docker- and Compose-files.
Set up a pipeline for CI/CD that deploys the application on a staging and production server.

Prerequisites

You should by now have an application with Docker- and Docker Compose files in a Gitlab.lnu.se repository.

Watch/follow along in Demo #6.

1. Setting up the infrastructure

The application will be deployd to a server in Open Stack with the following installed:

Docker - runtime
Nginx - reversed proxy

We will use Terraform to set up the infrastructure and Ansible to provisioning the two servers, one server for staging and one for poduction. It is critical that the two servers are alike, so using Terraform and Ansible in this case is a good match.

The code for creating the infrastructure will be version controlled in a separete project called "infra" or "infrastructure".

1.1. Infrastructure using Terraform

Start by creating the Terraform configuration files for creating:

Router
Network
Security groups for
- SSH: 22
- HTTP: 80
- HTTPS: 443
Instances
- staging
- production

Note the IP-numbers for the next step.

Example project - JTI - Production - Infrastructure

1.2. Provisioning using Ansible

Host-file

Add the IP-numbers from 1.1 to a host.yml-file:

‌‌‌‌‌

(This manual step could probably be automated)

Playbook(s)

Create one or more playbook configuration files that fulfills:

Install Docker (incl. add the ubuntu-user to the docker group)
Start Docker
Install Nginx
Copy a nginx config file to /etc/nginx/sites-available/ and symlink it to /etc/nginx/sites-enabled/
(optional: Add Letsencrypt certs to nginx)
Start/restart Nginx

Apply the playbook:

‌‌‌‌‌

Example project - JTI - Production - Infrastructure

Confirm

Hopefully, if all is well, we should have the two provisioned servers up and running. Confirm by visting :80 (optional :443) in the browser. The should return a bad gateway since the applications is not started.

Troubleshooting

Check the status in CSCloud. Sometimes Terraform fails to build instances due to a bug in OpenStack.
SSH into the machines and look at logs.
- Is docker running? `docker ps``
- What is the nginx-status? systemctl status nginx.service

2. Docker and Docker Compose

We switch over to the application project and leave the infrastructure project.

We have already created Docker- and Docker Compose files, however, theese are optimized for local development and not production. Now we need to:

Create a Dockerfile for production
Create a DockerCompose structure for production and development

2.1 Dockerfile

We create a new Dockerfile named Dockerfile.production and make it production ready.

The main differences between Dockerfile and Dockerfile.production:

npm ci --only=production --omit=dev to not install dev dependencies.
We start by: CMD ["node", "src/server.js"]
Read more: Dockerizing a Node.js web app

2.2 Docker Compose

Docker Compose files are composable. We can provide many files and they will all merge together were the last file has the highest priority. Because of this we create two new files:

docker-compose-development.yaml
docker-compose-production.yaml

In the original docker-compose we have everything that is common in both development and production. This includes:

For the mongodb service
- The container name
- The image beeing useed
For the taskit service
- The container name
- build context

In the docker-compose-development.yaml we specify:

volumes for development

In the docker-compose-production.yaml we specify:

volume for mongodb in production. (no need for volumes for taskit)
environmentvariables that we can specify on this level for taskit.
That we want to use Dockerfile.production when building taskit.
Ports. Might differ between production and development.

Example project - JTI - Production - Application

Environment variables

We specify most of our environment variables in a .env-file and they will be accessible in our docker compose scripts.

For the development compose file:

‌‌‌‌‌

And in the production compose file:

‌‌‌‌‌

Confirm

We can test our production files by executing them locally but instruction docker to run them at our staging server.

Docker will look for the environment variable DOCKER_HOST. If found, docker will execute it´s commands against that host. We should be able to:

‌‌‌‌‌

...where you need to change the IP to the IP of your staging server. In this case, you also need to add your private ssh-key to the ssh-agent. ssh-add.

Do not proceed until you get everything to work locally. It is a slower process to troubleshoot using the Gitlab pipelines.

3. Deploy trough a pipeline

When everything works we want to be able to deploy our code when it is pushed to gitlab. There are many different branching strategies to choose from and we should probably have the main-branch protected so that you always merge it through a Merge Request that can be audited and approved before the code is pushed to production. We will however start of by just making sure that new commits to main will be automaticly deployed to the staging server, and manually deployed to the production server.

3.1 The .gitlab-ci.yml-file

To create a pipeline on gitlab, you simply create a file named '.gitlab-ci.yml' and place it in the root of your project. (Doing this in the web UI of gitlab is pretty neet.)

Create to stages named "integration-test" (staging) and "deploy" (production). We start with the staging stage first since the production stage is more or less a copy of staging.

Some tips on the way:

Setting the "environmet" to "staging" and providing an URL will make testing even easier.
We need to make sure that the Gitlab-runner executing the pipeline have access to our private SSH-key. Gitlab have good information on how to do this.
Pay attention to "Verifying the SSH host keys" in the documentation at gitlab. Our runner must trust our newly created servers. (see environment variables)
If you created the pipeline from a template, you can lower the sleeps that simulate testing and linting.

Example project - JTI - Production - Application.gitlab-ci.yml

Environment variables

We need to set some environment variables on Gitlab. (Settings->Ci/CD->Variables)

PRODUCTION_HOST

The IP-address of the production server

STAGING_HOST

The IP-address of the staging server

SSH_PRIVATE_KEY

The private key used to connection to the servers

SSH_KNOWN_HOSTS

The result of ssh-keyscan for the servers.

SESSION_NAME

Used by NODE and gets built into the container.

SESSION_SECRET

Used by NODE and gets built into the container.

DOCKER_PORT

On which port to start Docker.

BASE_URL

Are we deploying on a sub directory? If not, go with /

3.2 Confirm Staging

Time to confirm functionallity of the staging stage of the pipeline.

Start by validating the pipeline by entering the "CI/CD editor" (CI/CD->Editor) and clicking "validate". Here you can show a visualization of the pipeline and a merged version of your pipline if you are importing parts of the pipeline from other places.

To run the pipeline you can either:

Push a new commit to the project
Click "Run Pipeline" in CI/CD->Pipelines

You can always enter a stage in the pipeline and look at the commandline output to trouble shoot.

Make a notable change in the css or a view-file and wait for the pipeline to finnsh executing. Click Deployments->Environments to get the link to the application. Visit and confirm.

3.3 Deployment stage

If everything worked in the previous stage we can copy the code to the Deployment stage and change which IP is beeing used by changing the environtment variable used. We also need to add when: manual to make the deployment to production a manual step after we have confirmed and tested during the staging stage.

Wrapping up

We now have a CI/CD pipeline for our application. We can develop the application locally and new versions are easily deployable.