January 21, 2025 15 min to read

GitLab CI/CD Pipeline Configuration Guide

Understanding GitLab CI/CD workflow and .gitlab-ci.yml configuration

Overview

Let’s explore GitLab CI/CD workflow and learn how to write .gitlab-ci.yml files for automated pipelines. For GitLab and GitLab Runner installation, please refer to our previous posts:

What is GitLab CI/CD?

GitLab CI/CD (Continuous Integration/Continuous Deployment) is a built-in feature that automates the process of building, testing, and deploying applications.

It helps ensure code quality and stability by running tests and validations for every code change, minimizing human error and improving development efficiency.

How GitLab CI/CD Works

Core Components:

Configuration
Uses `.gitlab-ci.yml` file in repository root to define jobs, stages, and rules.
Pipeline
Triggered by commits or merge requests, follows stages defined in configuration, and can be manual or scheduled.
Jobs
Individual tasks within pipelines that run in containers or runners to build, test, or deploy actions.
Stages
Pipeline phases (e.g., build, test, deploy). Jobs in the same stage run in parallel, and stages run sequentially.
Runners
Agents that execute jobs. They can run on various platforms and be shared or specific to projects.
Artifacts & Caching
Store build outputs, share between jobs, and cache dependencies for efficient execution.

GitLab Predefined Variables

Common predefined variables used in pipelines:

CI_COMMIT_REF_NAME    # Branch name being built
CI_COMMIT_SHORT_SHA   # First 8 characters of commit SHA
CI_REGISTRY           # GitLab Container Registry address
CI_PROJECT_DIR        # Full path where repository is cloned
CI_REGISTRY_USER      # GitLab registry user
CI_REGISTRY_PASSWORD  # GitLab registry password

Pipeline Examples

1. Basic Pipeline

stages:
  - build
  - test
  - deploy

image: alpine

build_a:
  stage: build
  script:
    - echo "This job builds something."

test_a:
  stage: test
  script:
    - echo "This job tests something."

deploy_a:
  stage: deploy
  script:
    - echo "This job deploys something."
  environment: production

2. DAG (Directed Acyclic Graph) Pipeline

stages:
  - build
  - test
  - deploy

build_a:
  stage: build
  script:
    - echo "Building component A"

test_a:
  stage: test
  needs: [build_a]
  script:
    - echo "Testing component A"

deploy_a:
  stage: deploy
  needs: [test_a]
  script:
    - echo "Deploying component A"

3. Parent-Child Pipeline

# Root .gitlab-ci.yml
stages:
  - triggers

trigger_a:
  stage: triggers
  trigger:
    include: a/.gitlab-ci.yml
  rules:
    - changes:
        - a/*

# a/.gitlab-ci.yml
stages:
  - build
  - deploy

build_a:
  stage: build
  script:
    - echo "Building service A"

deploy_a:
  stage: deploy
  script:
    - echo "Deploying service A"

Practical Example

Here’s a real-world example with build and deploy stages:

stages:
  - build
  - deploy

variables:
  BRANCH:
    value: "dev1"
    description: "Select branch: dev1 or dev2"
  NAMESPACE:
    value: "dev1"
    description: "Select namespace: dev1 or dev2"
  SERVICE:
    value: "game"
    description: "Select service type"
  CI_REGISTRY_IMAGE: $CI_REGISTRY/$SERVICE
  BUILD_TAG: $BRANCH-$CI_COMMIT_SHORT_SHA
  IMAGE_URL: '${CI_REGISTRY_IMAGE}:${BUILD_TAG}'

.build_image: &build_image
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - /kaniko/executor 
      --context $CI_PROJECT_DIR 
      --dockerfile $CI_PROJECT_DIR/$SERVICE.Dockerfile 
      --destination $IMAGE_URL 
      --build-arg NODE_ENV=$BRANCH

.deploy_job: &deploy_job
  stage: deploy
  image: dtzar/helm-kubectl
  script:
    - kubectl set image deployment $SERVICE-$NAMESPACE 
      app=$IMAGE_URL -n $NAMESPACE

build_image:
  <<: *build_image
  rules:
    - if: '($CI_PIPELINE_SOURCE == "web")'
    - if: '($CI_PIPELINE_SOURCE == "trigger")'
  tags:
    - build-runner

deploy_job:
  <<: *deploy_job
  rules:
    - if: '($CI_PIPELINE_SOURCE == "web")'
    - if: '($CI_PIPELINE_SOURCE == "trigger")'
  tags:
    - deploy-runner

Advanced Pipeline Features

1. Change-Based Triggers

.change_files: &change_files
  changes:
    - apps/**/*
    - config/*
    - libs/**/*
    - *.Dockerfile
    - .gitlab-ci.yml

build_image:
  rules:
    - if: '($CI_PIPELINE_SOURCE == "push")'
      <<: *change_files

2. Cross-Project Triggers

build_job:
  after_script:
    - curl -X POST 
      -F token=${TRIGGER_TOKEN} 
      -F ref=${CI_COMMIT_REF_NAME} 
      -F variables[project_id]=${CI_PROJECT_ID} 
      http://gitlab.example.com/api/v4/projects/100/trigger/pipeline

Multi-Environment Deployment with Auto DevOps

Here’s an example pipeline that automatically deploys to different environments based on branch names:

stages:
  - build
  - test
  - review
  - staging
  - production

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""
  ROLLOUT_RESOURCE_TYPE: deployment

include:
  - template: Auto-DevOps.gitlab-ci.yml

build:
  stage: build
  image: docker:20.10.16
  services:
    - docker:20.10.16-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

test:
  stage: test
  image: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  script:
    - npm ci
    - npm test

review:
  stage: review
  script:
    - kubectl apply -f k8s/review/
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: https://review-$CI_COMMIT_REF_SLUG.$KUBE_INGRESS_BASE_DOMAIN
    on_stop: stop_review
  rules:
    - if: $CI_COMMIT_BRANCH != "main" && $CI_COMMIT_BRANCH != "production"

stop_review:
  stage: review
  script:
    - kubectl delete -f k8s/review/
  environment:
    name: review/$CI_COMMIT_REF_NAME
    action: stop
  when: manual
  rules:
    - if: $CI_COMMIT_BRANCH != "main" && $CI_COMMIT_BRANCH != "production"

staging:
  stage: staging
  script:
    - kubectl apply -f k8s/staging/
  environment:
    name: staging
    url: https://staging.$KUBE_INGRESS_BASE_DOMAIN
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

production:
  stage: production
  script:
    - kubectl apply -f k8s/production/
  environment:
    name: production
    url: https://$KUBE_INGRESS_BASE_DOMAIN
  rules:
    - if: $CI_COMMIT_BRANCH == "production"
  when: manual

Caching and Artifacts

Efficient usage of caching and artifacts can dramatically speed up pipelines:

stages:
  - build
  - test
  - deploy

# Global cache definition
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .yarn
    - .npm

variables:
  npm_config_cache: .npm
  YARN_CACHE_FOLDER: .yarn

build_app:
  stage: build
  image: node:16-alpine
  script:
    - yarn install
    - yarn build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

test_app:
  stage: test
  image: node:16-alpine
  script:
    - yarn install
    - yarn test
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

# Using cache and artifacts together
deploy_app:
  stage: deploy
  image: alpine:latest
  script:
    - apk add --no-cache curl
    - cd dist
    - tar -czf application.tar.gz *
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file application.tar.gz "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my-package/1.0.0/application.tar.gz"'
  dependencies:
    - build_app

GitLab CI/CD with Docker Compose

This example shows how to use Docker Compose for integration testing:

stages:
  - build
  - test
  - deploy

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_TLS_CERTDIR: ""

services:
  - docker:20.10.16-dind

build:
  stage: build
  image: docker:20.10.16
  script:
    - docker-compose build
  artifacts:
    paths:
      - docker-compose.yml

integration_test:
  stage: test
  image: docker:20.10.16
  script:
    - docker-compose up -d
    - sleep 10  # Wait for services to start
    - docker-compose exec -T app npm run integration-tests
    - docker-compose down
  dependencies:
    - build

Parallel Testing for Faster Pipelines

Breaking tests into parallel jobs can significantly speed up pipelines:

stages:
  - build
  - test
  - deploy

build_app:
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
      - node_modules/

.test_template: &test_template
  stage: test
  needs:
    - build_app
  artifacts:
    when: always
    reports:
      junit: junit-*.xml

unit_tests:
  <<: *test_template
  script:
    - npm run test:unit

integration_tests:
  <<: *test_template
  script:
    - npm run test:integration

e2e_tests_group1:
  <<: *test_template
  script:
    - npm run test:e2e -- --group=1

e2e_tests_group2:
  <<: *test_template
  script:
    - npm run test:e2e -- --group=2

deploy_app:
  stage: deploy
  script:
    - npm run deploy
  environment: production
  only:
    - main
  when: manual

Advanced Configuration Techniques

Dynamic Environment Configuration

This example demonstrates dynamic environment configuration based on Git branches:

.env_config: &env_config
  before_script:
    - |
      if [[ "$CI_COMMIT_BRANCH" == "main" ]]; then
        export ENVIRONMENT="staging"
        export API_URL="https://api.staging.example.com"
      elif [[ "$CI_COMMIT_BRANCH" == "production" ]]; then
        export ENVIRONMENT="production"
        export API_URL="https://api.example.com"
      else
        export ENVIRONMENT="review"
        export API_URL="https://api.review.example.com"
      fi
    - echo "Configuring for $ENVIRONMENT environment"

build_config:
  <<: *env_config
  script:
    - echo "Building for $ENVIRONMENT using $API_URL"
    - cat > .env << EOF
      API_URL=$API_URL
      ENVIRONMENT=$ENVIRONMENT
      BUILD_TIME=$(date)
      COMMIT_SHA=$CI_COMMIT_SHORT_SHA
      EOF
    - npm run build

Using GitLab CI with Monorepos

For monorepo setups, you can use different pipelines for different components:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH == "develop"

stages:
  - detect_changes
  - build
  - test
  - deploy

detect_changes:
  stage: detect_changes
  image: alpine:latest
  script:
    - |
      echo "Detecting changes..."
      git diff --name-only $CI_COMMIT_BEFORE_SHA $CI_COMMIT_SHA > changes.txt
      
      if grep -q "^frontend/" changes.txt; then
        echo "FRONTEND_CHANGES=true" >> variables.env
      else
        echo "FRONTEND_CHANGES=false" >> variables.env
      fi
      
      if grep -q "^backend/" changes.txt; then
        echo "BACKEND_CHANGES=true" >> variables.env
      else
        echo "BACKEND_CHANGES=false" >> variables.env
      fi
  artifacts:
    reports:
      dotenv: variables.env

build_frontend:
  stage: build
  script:
    - echo "Building frontend..."
    - cd frontend && npm ci && npm run build
  needs:
    - detect_changes
  rules:
    - if: $FRONTEND_CHANGES == "true"

build_backend:
  stage: build
  script:
    - echo "Building backend..."
    - cd backend && npm ci && npm run build
  needs:
    - detect_changes
  rules:
    - if: $BACKEND_CHANGES == "true"

Troubleshooting GitLab CI Pipelines

Common Pipeline Issues and Solutions:

1. Pipeline Won't Start: - Check runner availability and tags
- Verify `.gitlab-ci.yml` syntax with the CI Lint tool
- Ensure project has correct permissions
- Check if you've reached concurrent job limits

2. Job Failures: - Check job logs for specific error messages
- Verify environment variables are set correctly
- Check if Docker image is available and correct
- Ensure runner has sufficient resources

3. Pipeline Performance Issues: - Use caching for dependencies
- Implement parallel jobs for tests
- Only run necessary jobs with conditional rules
- Use more specific Docker images

4. Cache Problems: - Verify cache paths are correct
- Use different cache keys for different branches
- Check if cache is too large
- Ensure runner has sufficient storage

Pipeline Debugging Tips

debug_job:
  stage: test
  script:
    # List environment variables
    - env | sort
    
    # Show GitLab CI variables
    - echo "Project path: $CI_PROJECT_PATH"
    - echo "Job name: $CI_JOB_NAME"
    - echo "Commit SHA: $CI_COMMIT_SHA"
    
    # Show file system information
    - df -h
    - ls -la
    
    # Show network information
    - ip addr
    - cat /etc/hosts
    
    # Show Docker info (if available)
    - docker info || echo "Docker not available"
  when: manual

Best Practices

GitLab CI/CD Best Practices:

1. Pipeline Structure: - Keep pipelines simple and focused
- Use stages to organize jobs logically
- Group related jobs using templates
- Use parent-child pipelines for complex workflows
- Implement DAG (Directed Acyclic Graph) dependencies

2. Performance Optimization: - Use caching effectively to speed up builds
- Parallelize tests when possible
- Use specific, lightweight Docker images
- Only run necessary jobs with rules
- Keep build artifacts small and focused

3. Security Considerations: - Use secret variables for sensitive information
- Implement branch protection rules
- Run security scanners in your pipeline
- Limit access to protected environments
- Review Docker images for vulnerabilities

4. Maintainability: - Document pipeline behavior in README
- Use includes to share common configurations
- Keep pipeline configuration in version control
- Follow naming conventions for jobs and stages
- Use anchors and templates to reduce duplication

somaz v3.1.2

GitLab CI/CD Pipeline Configuration Guide

Overview

What is GitLab CI/CD?

How GitLab CI/CD Works

Core Components:

GitLab Predefined Variables

Pipeline Examples

1. Basic Pipeline

2. DAG (Directed Acyclic Graph) Pipeline

3. Parent-Child Pipeline

Practical Example

Advanced Pipeline Features

1. Change-Based Triggers

2. Cross-Project Triggers

Multi-Environment Deployment with Auto DevOps

Caching and Artifacts

GitLab CI/CD with Docker Compose

Parallel Testing for Faster Pipelines

Advanced Configuration Techniques

Dynamic Environment Configuration

Using GitLab CI with Monorepos

Troubleshooting GitLab CI Pipelines

Pipeline Debugging Tips

Best Practices

Reference

GitLab CI/CD Template Guide

Somaz

Comments

GitLab CI/CD Pipeline Configuration Guide

Overview

What is GitLab CI/CD?

How GitLab CI/CD Works

Core Components:

GitLab Predefined Variables

Pipeline Examples

1. Basic Pipeline

2. DAG (Directed Acyclic Graph) Pipeline

3. Parent-Child Pipeline

Practical Example

Advanced Pipeline Features

1. Change-Based Triggers

2. Cross-Project Triggers

Multi-Environment Deployment with Auto DevOps

Caching and Artifacts

GitLab CI/CD with Docker Compose

Parallel Testing for Faster Pipelines

Advanced Configuration Techniques

Dynamic Environment Configuration

Using GitLab CI with Monorepos

Troubleshooting GitLab CI Pipelines

Pipeline Debugging Tips

Best Practices

Reference

GitLab CI/CD Template Guide

Share

Somaz

Comments