66 min to read

Docker Compose - Complete Guide to Multi-Container Application Orchestration

Master containerized application configuration and deployment with Docker Compose

Featured image


Overview

Docker Compose is a powerful tool for defining and running multi-container Docker applications. Using a simple YAML file, you can configure your application’s services, networks, and volumes, then create and start all services with a single command. This approach transforms complex multi-container deployments into manageable, reproducible configurations.

Docker Compose excels in development, testing, and staging environments, while also being suitable for production use cases. It simplifies the orchestration of related services, making it an essential tool for modern containerized application development.


Evolution from V1 to V2

Important Migration Notice
  • July 2023: Docker Compose V1 reached end of life and no longer receives updates
  • V2 is now the official standard: Integrated as a Docker CLI subcommand rather than standalone binary
  • Command syntax change: docker-compose → docker compose (note the space)
  • Enhanced features: Better performance, improved error handling, and new capabilities


Core Concepts and Architecture

Understanding Docker Compose’s architecture helps you design better multi-container applications.


Docker Compose Components

Component Description
Services Define containers and their configuration (image, ports, volumes, environment)
Networks Enable communication between containers with automatic DNS resolution
Volumes Persist data and share files between containers and host system
Configs Manage configuration files separately from container images
Secrets Securely manage sensitive data like passwords and API keys


Compose Workflow Architecture

graph TD A[docker-compose.yml] --> B[Docker Compose CLI] B --> C[Docker Engine] C --> D[Container 1] C --> E[Container 2] C --> F[Container N] G[Custom Networks] --> D G --> E G --> F H[Persistent Volumes] --> D H --> E H --> F style A fill:#f9f,stroke:#333,stroke-width:2px style B fill:#bbf,stroke:#333,stroke-width:2px style C fill:#bfb,stroke:#333,stroke-width:2px


Installation and Setup

Install Docker Compose V2 for modern container orchestration capabilities.


Official Installation Methods

# Docker Desktop includes Compose V2 by default
# Download from: https://www.docker.com/products/docker-desktop

# Verify installation
docker compose version

Method 2: Manual Installation (Linux)


Method 3: Package Manager Installation

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install docker-compose-plugin

# CentOS/RHEL/Fedora
sudo yum install docker-compose-plugin
# or
sudo dnf install docker-compose-plugin

# macOS with Homebrew
brew install docker-compose


Post-Installation Verification

# Check Docker Compose version
docker compose version

# Verify Docker Engine compatibility
docker --version

# Test basic functionality
echo "version: '3.8'
services:
  hello:
    image: hello-world" > test-compose.yml

docker compose -f test-compose.yml up
docker compose -f test-compose.yml down

# Clean up test files
rm test-compose.yml


Essential Commands Reference

Master the core Docker Compose commands for efficient container management.


Lifecycle Management Commands

Service Startup and Shutdown

# Start all services defined in docker-compose.yml
docker compose up

# Start services in detached mode (background)
docker compose up -d

# Start specific services only
docker compose up service1 service2

# Build images before starting (if Dockerfile changes)
docker compose up --build

# Recreate containers even if configuration hasn't changed
docker compose up --force-recreate

# Stop all services
docker compose stop

# Stop specific services
docker compose stop service1 service2

# Stop and remove containers, networks, and volumes
docker compose down

# Remove everything including volumes (destructive!)
docker compose down -v

# Remove everything including images
docker compose down --rmi all

Build and Rebuild Operations

# Build or rebuild services
docker compose build

# Build specific service
docker compose build service-name

# Build with no cache (clean build)
docker compose build --no-cache

# Build with progress output
docker compose build --progress=plain

# Pull latest images before building
docker compose build --pull


Monitoring and Debugging Commands

Container Status and Logs

# List running containers
docker compose ps

# List all containers (including stopped)
docker compose ps -a

# Show container resource usage
docker compose top

# View logs from all services
docker compose logs

# Follow logs in real-time (like tail -f)
docker compose logs -f

# View logs for specific service
docker compose logs -f service-name

# View last N lines of logs
docker compose logs --tail=100 service-name

# Show timestamps in logs
docker compose logs -t service-name

Container Interaction

# Execute command in running container
docker compose exec service-name command

# Start interactive bash session
docker compose exec service-name bash

# Execute command as specific user
docker compose exec --user root service-name bash

# Run one-off command in new container
docker compose run service-name command

# Run command without dependencies
docker compose run --no-deps service-name command

# Run command and remove container after execution
docker compose run --rm service-name command


Advanced Management Commands

Configuration and Validation

# Validate docker-compose.yml syntax
docker compose config

# View the actual configuration that will be used
docker compose config --services

# Show only service names
docker compose config --services

# Resolve environment variables and show final config
docker compose config --resolve-env-vars

# Check for configuration errors
docker compose config --quiet

Service Scaling and Updates

# Scale service to specific number of replicas
docker compose up --scale service-name=3

# Update service configuration without downtime
docker compose up -d --no-deps service-name

# Restart specific services
docker compose restart service-name

# Restart all services
docker compose restart

# Pause/unpause services
docker compose pause service-name
docker compose unpause service-name


YAML Configuration Deep Dive

Master the docker-compose.yml file structure for complex application architectures.


Basic Compose File Structure

Standard Web Application Stack

version: '3.8'

services:
  # Database service
  database:
    image: postgres:15-alpine
    container_name: app_database
    environment:
      POSTGRES_DB: ${DB_NAME:-myapp}
      POSTGRES_USER: ${DB_USER:-postgres}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-secret}
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d
    ports:
      - "${DB_PORT:-5432}:5432"
    networks:
      - backend
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres}"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

  # Redis cache service
  cache:
    image: redis:7-alpine
    container_name: app_cache
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD:-cache_secret}
    volumes:
      - redis_data:/data
      - ./redis.conf:/usr/local/etc/redis/redis.conf
    ports:
      - "${REDIS_PORT:-6379}:6379"
    networks:
      - backend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "--raw", "incr", "ping"]
      interval: 30s
      timeout: 3s
      retries: 5

  # Application service
  app:
    build:
      context: ./app
      dockerfile: Dockerfile
      args:
        BUILD_ENV: ${NODE_ENV:-development}
        APP_VERSION: ${APP_VERSION:-latest}
    container_name: app_backend
    environment:
      NODE_ENV: ${NODE_ENV:-development}
      DATABASE_URL: postgres://${DB_USER:-postgres}:${DB_PASSWORD:-secret}@database:5432/${DB_NAME:-myapp}
      REDIS_URL: redis://cache:6379
      JWT_SECRET: ${JWT_SECRET:-your-jwt-secret}
      API_PORT: 3000
    volumes:
      - ./app:/usr/src/app
      - /usr/src/app/node_modules
      - app_uploads:/usr/src/app/uploads
    ports:
      - "${APP_PORT:-3000}:3000"
    depends_on:
      database:
        condition: service_healthy
      cache:
        condition: service_healthy
    networks:
      - frontend
      - backend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s

  # Nginx reverse proxy
  nginx:
    image: nginx:alpine
    container_name: app_nginx
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/sites-enabled:/etc/nginx/sites-enabled:ro
      - ./static:/usr/share/nginx/html:ro
      - nginx_logs:/var/log/nginx
    ports:
      - "${HTTP_PORT:-80}:80"
      - "${HTTPS_PORT:-443}:443"
    depends_on:
      - app
    networks:
      - frontend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3

# Network definitions
networks:
  frontend:
    driver: bridge
    name: app_frontend
  backend:
    driver: bridge
    name: app_backend
    internal: true  # Backend network is isolated

# Volume definitions
volumes:
  postgres_data:
    driver: local
    name: app_postgres_data
  redis_data:
    driver: local
    name: app_redis_data
  app_uploads:
    driver: local
    name: app_uploads
  nginx_logs:
    driver: local
    name: app_nginx_logs


Advanced Configuration Patterns

Microservices Architecture Example

version: '3.8'

services:
  # API Gateway
  api-gateway:
    build:
      context: ./api-gateway
      dockerfile: Dockerfile
    environment:
      - SERVICES_USER=user-service:3001
      - SERVICES_ORDER=order-service:3002
      - SERVICES_PAYMENT=payment-service:3003
    ports:
      - "8080:8080"
    depends_on:
      - user-service
      - order-service
      - payment-service
    networks:
      - api-network

  # User microservice
  user-service:
    build:
      context: ./services/user
      dockerfile: Dockerfile
    environment:
      - DB_HOST=user-db
      - DB_PORT=5432
      - DB_NAME=users
      - JWT_SECRET=${JWT_SECRET}
    depends_on:
      user-db:
        condition: service_healthy
    networks:
      - api-network
      - user-network
    deploy:
      replicas: 2
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M

  user-db:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: users
      POSTGRES_USER: ${USER_DB_USER}
      POSTGRES_PASSWORD: ${USER_DB_PASSWORD}
    volumes:
      - user_db_data:/var/lib/postgresql/data
    networks:
      - user-network
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${USER_DB_USER}"]
      interval: 30s
      timeout: 10s
      retries: 3

  # Order microservice
  order-service:
    build:
      context: ./services/order
      dockerfile: Dockerfile
    environment:
      - DB_HOST=order-db
      - MESSAGE_QUEUE_URL=amqp://rabbitmq:5672
    depends_on:
      - order-db
      - rabbitmq
    networks:
      - api-network
      - order-network
      - message-network

  order-db:
    image: mongodb/mongodb-community-server:latest
    environment:
      MONGODB_INITDB_ROOT_USERNAME: ${MONGO_USER}
      MONGODB_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}
    volumes:
      - order_db_data:/data/db
    networks:
      - order-network

  # Payment microservice
  payment-service:
    build:
      context: ./services/payment
      dockerfile: Dockerfile
    environment:
      - PAYMENT_PROVIDER_API_KEY=${PAYMENT_API_KEY}
      - MESSAGE_QUEUE_URL=amqp://rabbitmq:5672
    depends_on:
      - rabbitmq
    networks:
      - api-network
      - message-network
    secrets:
      - payment_api_key

  # Message queue
  rabbitmq:
    image: rabbitmq:3-management-alpine
    environment:
      RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq
    ports:
      - "15672:15672"  # Management interface
    networks:
      - message-network

  # Monitoring and logging
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - monitoring-network

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
      - ./monitoring/dashboards:/etc/grafana/provisioning/dashboards
    ports:
      - "3001:3000"
    depends_on:
      - prometheus
    networks:
      - monitoring-network

networks:
  api-network:
    driver: bridge
  user-network:
    driver: bridge
    internal: true
  order-network:
    driver: bridge
    internal: true
  message-network:
    driver: bridge
    internal: true
  monitoring-network:
    driver: bridge

volumes:
  user_db_data:
  order_db_data:
  rabbitmq_data:
  prometheus_data:
  grafana_data:

secrets:
  payment_api_key:
    file: ./secrets/payment_api_key.txt


Development-Optimized Configuration

Hot Reload and Development Tools

version: '3.8'

services:
  # Development API with hot reload
  api-dev:
    build:
      context: ./api
      dockerfile: Dockerfile.dev
      target: development
    environment:
      - NODE_ENV=development
      - DEBUG=app:*
      - DATABASE_URL=postgres://dev:dev@postgres:5432/dev_db
    volumes:
      - ./api:/app
      - /app/node_modules
      - api_logs:/app/logs
    ports:
      - "3000:3000"
      - "9229:9229"  # Node.js debugger port
    depends_on:
      - postgres
      - redis
    networks:
      - dev-network
    develop:
      watch:
        - action: sync
          path: ./api/src
          target: /app/src
        - action: rebuild
          path: ./api/package.json
          target: /app/package.json

  # Frontend development server
  frontend-dev:
    build:
      context: ./frontend
      dockerfile: Dockerfile.dev
    environment:
      - VITE_API_URL=http://localhost:3000
      - VITE_HOT_RELOAD=true
    volumes:
      - ./frontend:/app
      - /app/node_modules
    ports:
      - "5173:5173"  # Vite dev server
    networks:
      - dev-network
    develop:
      watch:
        - action: sync+restart
          path: ./frontend/src
          target: /app/src

  # Development database with initialization
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: dev_db
      POSTGRES_USER: dev
      POSTGRES_PASSWORD: dev
    volumes:
      - postgres_dev_data:/var/lib/postgresql/data
      - ./database/init:/docker-entrypoint-initdb.d
      - ./database/dev-data:/dev-data
    ports:
      - "5432:5432"
    networks:
      - dev-network

  # Redis for development
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    networks:
      - dev-network

  # Development tools
  mailhog:
    image: mailhog/mailhog:latest
    ports:
      - "1025:1025"  # SMTP server
      - "8025:8025"  # Web interface
    networks:
      - dev-network

  # Database administration tool
  pgadmin:
    image: dpage/pgadmin4:latest
    environment:
      PGADMIN_DEFAULT_EMAIL: admin@dev.local
      PGADMIN_DEFAULT_PASSWORD: admin
    ports:
      - "8080:80"
    depends_on:
      - postgres
    networks:
      - dev-network

networks:
  dev-network:
    driver: bridge
    name: dev_network

volumes:
  postgres_dev_data:
  api_logs:


Environment Management and Configuration

Implement flexible environment-specific configurations for different deployment scenarios.


Environment Variables and .env Files

Comprehensive .env Configuration

# .env file - Default development configuration

# Application Settings
APP_NAME=MyApplication
APP_VERSION=1.0.0
NODE_ENV=development
APP_DEBUG=true
APP_PORT=3000

# Database Configuration
DB_CONNECTION=postgres
DB_HOST=database
DB_PORT=5432
DB_NAME=myapp_dev
DB_USER=postgres
DB_PASSWORD=dev_secret_123
DATABASE_URL=postgres://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}

# Redis Configuration
REDIS_HOST=cache
REDIS_PORT=6379
REDIS_PASSWORD=redis_secret_456
REDIS_URL=redis://:${REDIS_PASSWORD}@${REDIS_HOST}:${REDIS_PORT}

# Security Settings
JWT_SECRET=your-super-secret-jwt-key-change-in-production
JWT_EXPIRES_IN=7d
SESSION_SECRET=session-secret-key
ENCRYPTION_KEY=32-character-encryption-key-here

# External Services
STRIPE_PUBLIC_KEY=pk_test_xxxxxxxxxxxxx
STRIPE_SECRET_KEY=sk_test_xxxxxxxxxxxxx
SENDGRID_API_KEY=SG.xxxxxxxxxxxxx
AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AWS_REGION=us-east-1
AWS_S3_BUCKET=my-app-uploads

# Feature Flags
FEATURE_NEW_UI=true
FEATURE_ANALYTICS=false
FEATURE_BETA_FEATURES=true

# Monitoring and Logging
LOG_LEVEL=debug
SENTRY_DSN=https://xxxxxxxxxxxxx@sentry.io/xxxxxxx
NEW_RELIC_LICENSE_KEY=xxxxxxxxxxxxx

# Development Tools
HOT_RELOAD=true
DEBUG_MODE=true
MOCK_EXTERNAL_SERVICES=true

Environment-Specific Override Files

Production Environment (.env.production)

# .env.production - Production overrides

NODE_ENV=production
APP_DEBUG=false
LOG_LEVEL=info

# Secure production database
DB_HOST=prod-db.company.com
DB_NAME=myapp_production
DB_USER=prod_user
DB_PASSWORD=${PROD_DB_PASSWORD}  # Set via CI/CD secrets

# Production Redis cluster
REDIS_HOST=prod-redis-cluster.company.com
REDIS_PASSWORD=${PROD_REDIS_PASSWORD}

# Production security
JWT_SECRET=${PROD_JWT_SECRET}
SESSION_SECRET=${PROD_SESSION_SECRET}

# Production services
STRIPE_PUBLIC_KEY=${PROD_STRIPE_PUBLIC_KEY}
STRIPE_SECRET_KEY=${PROD_STRIPE_SECRET_KEY}

# Feature flags for production
FEATURE_BETA_FEATURES=false
MOCK_EXTERNAL_SERVICES=false

# Production monitoring
SENTRY_DSN=${PROD_SENTRY_DSN}
NEW_RELIC_LICENSE_KEY=${PROD_NEW_RELIC_KEY}

Testing Environment (.env.test)

# .env.test - Testing configuration

NODE_ENV=test
APP_DEBUG=true
LOG_LEVEL=warn

# Test database (typically in-memory or isolated)
DB_NAME=myapp_test
DB_USER=test_user
DB_PASSWORD=test_pass

# Mock external services for testing
MOCK_EXTERNAL_SERVICES=true
STRIPE_SECRET_KEY=sk_test_mock
SENDGRID_API_KEY=test_key

# Fast test execution
FEATURE_ANALYTICS=false
FEATURE_NEW_UI=true

# Test-specific settings
TEST_TIMEOUT=30000
PARALLEL_TEST_WORKERS=4


Advanced Environment Variable Usage

Dynamic Configuration in Compose

version: '3.8'

services:
  app:
    image: myapp:${APP_VERSION:-latest}
    environment:
      # Direct environment variable assignment
      - NODE_ENV=${NODE_ENV:-development}
      - APP_PORT=${APP_PORT:-3000}
      
      # Computed environment variables
      - DATABASE_URL=postgres://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}
      - REDIS_URL=redis://:${REDIS_PASSWORD}@${REDIS_HOST}:${REDIS_PORT}
      
      # Environment-specific feature flags
      - FEATURE_DEBUG=${NODE_ENV:-development == 'development' ? 'true' : 'false'}
      
      # External file references
      - AWS_CREDENTIALS_FILE=/run/secrets/aws_credentials
    
    # Environment file precedence (later files override earlier ones)
    env_file:
      - .env                    # Base configuration
      - .env.${NODE_ENV:-dev}   # Environment-specific overrides
      - .env.local              # Local developer overrides (git-ignored)
    
    # Secret management
    secrets:
      - source: aws_credentials
        target: /run/secrets/aws_credentials
        mode: 0400

secrets:
  aws_credentials:
    file: ./secrets/aws_credentials.json

Environment Variable Validation

# docker-compose.yml with validation
version: '3.8'

services:
  app:
    build: .
    environment:
      - NODE_ENV=${NODE_ENV:?NODE_ENV is required}
      - DATABASE_URL=${DATABASE_URL:?DATABASE_URL is required}
      - JWT_SECRET=${JWT_SECRET:?JWT_SECRET must be set for security}
      - API_KEY=${API_KEY:?API_KEY is required for external service integration}
    
    # Health check that validates configuration
    healthcheck:
      test: ["CMD", "node", "scripts/health-check.js"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s


Override Files and Multi-Environment Support

Manage different environments efficiently using Docker Compose override mechanisms.


Override File Hierarchy

Base Configuration (docker-compose.yml)

# docker-compose.yml - Base configuration for all environments
version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    environment:
      - NODE_ENV=${NODE_ENV:-development}
    volumes:
      - ./src:/app/src
    networks:
      - app-network

  database:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=${DB_NAME}
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
    volumes:
      - db_data:/var/lib/postgresql/data
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

volumes:
  db_data:

Development Override (docker-compose.override.yml)

# docker-compose.override.yml - Automatically applied in development
version: '3.8'

services:
  app:
    ports:
      - "3000:3000"
      - "9229:9229"  # Debug port
    environment:
      - DEBUG=app:*
      - HOT_RELOAD=true
    volumes:
      - ./src:/app/src:cached      # Optimized for development
      - /app/node_modules          # Prevent overwriting
    command: npm run dev
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src
        - action: rebuild
          path: package.json

  database:
    ports:
      - "5432:5432"  # Expose database for development tools
    environment:
      - POSTGRES_DB=myapp_dev
      - POSTGRES_USER=dev
      - POSTGRES_PASSWORD=dev

  # Development-only services
  pgadmin:
    image: dpage/pgadmin4:latest
    environment:
      - PGADMIN_DEFAULT_EMAIL=admin@dev.local
      - PGADMIN_DEFAULT_PASSWORD=admin
    ports:
      - "8080:80"
    depends_on:
      - database
    networks:
      - app-network

  mailhog:
    image: mailhog/mailhog:latest
    ports:
      - "1025:1025"  # SMTP
      - "8025:8025"  # Web interface
    networks:
      - app-network

Production Override (docker-compose.prod.yml)

# docker-compose.prod.yml - Production-specific configuration
version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.prod
      target: production
    restart: unless-stopped
    deploy:
      replicas: 3
      resources:
        limits:
          memory: 1G
          cpus: '0.5'
        reservations:
          memory: 512M
          cpus: '0.25'
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first
    environment:
      - NODE_ENV=production
      - LOG_LEVEL=info
    # Remove development volumes
    volumes: []
    # Production command
    command: npm start
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

  database:
    # Production database configuration
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '1.0'
    # Remove development port exposure
    ports: []
    environment:
      - POSTGRES_DB=${PROD_DB_NAME}
      - POSTGRES_USER=${PROD_DB_USER}
      - POSTGRES_PASSWORD=${PROD_DB_PASSWORD}
    
    # Production volume with backup configuration
    volumes:
      - prod_db_data:/var/lib/postgresql/data
      - ./backups:/backups

  # Production-specific services
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/prod.conf:/etc/nginx/nginx.conf:ro
      - ./ssl:/etc/nginx/ssl:ro
      - ./static:/usr/share/nginx/html:ro
    depends_on:
      - app
    restart: unless-stopped
    networks:
      - app-network

  # Log aggregation
  fluentd:
    image: fluent/fluentd:latest
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - ./logs:/fluentd/log
    networks:
      - app-network

volumes:
  prod_db_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /opt/app/data/postgres

Testing Override (docker-compose.test.yml)

# docker-compose.test.yml - Testing environment configuration
version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.test
      target: test
    environment:
      - NODE_ENV=test
      - CI=true
      - COVERAGE=true
    command: npm run test:ci
    volumes:
      - ./coverage:/app/coverage
      - ./test-results:/app/test-results

  database:
    environment:
      - POSTGRES_DB=myapp_test
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
    tmpfs:
      - /var/lib/postgresql/data  # In-memory database for faster tests

  # Test dependencies
  test-db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=test_db
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
    tmpfs:
      - /var/lib/postgresql/data

  # Mock external services
  mock-api:
    image: wiremock/wiremock:latest
    ports:
      - "8081:8080"
    volumes:
      - ./test/mocks:/home/wiremock
    networks:
      - app-network


Environment-Specific Usage Commands

# Development (uses docker-compose.override.yml automatically)
docker compose up -d

# Production deployment
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Testing environment
docker compose -f docker-compose.yml -f docker-compose.test.yml up --abort-on-container-exit

# Staging environment
docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d

# Local development with custom overrides
docker compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.local.yml up -d

# Multiple environment files (later files override earlier ones)
docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.monitoring.yml up -d


Advanced Features and Patterns

Explore sophisticated Docker Compose capabilities for complex applications.


Health Checks and Dependency Management

Comprehensive Health Check Configuration

version: '3.8'

services:
  # Database with health check
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s
    volumes:
      - postgres_data:/var/lib/postgresql/data

  # Redis with health check
  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 30s
      timeout: 3s
      retries: 5
      start_period: 30s
    volumes:
      - redis_data:/data

  # Application with sophisticated health check
  app:
    build: .
    environment:
      - DATABASE_URL=postgres://user:password@postgres:5432/myapp
      - REDIS_URL=redis://redis:6379
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s
    ports:
      - "3000:3000"

  # Web server with dependency on app health
  nginx:
    image: nginx:alpine
    depends_on:
      app:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf

volumes:
  postgres_data:
  redis_data:


Watch Mode and Live Development

Advanced Watch Configuration

version: '3.8'

services:
  # Frontend with hot module replacement
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile.dev
    volumes:
      - ./frontend:/app
      - /app/node_modules
    ports:
      - "3000:3000"
    environment:
      - FAST_REFRESH=true
      - CHOKIDAR_USEPOLLING=true
    develop:
      watch:
        # Sync JavaScript/TypeScript changes
        - action: sync
          path: ./frontend/src
          target: /app/src
          ignore:
            - "**/*.test.js"
            - "**/*.spec.ts"
            
        # Rebuild on package.json changes
        - action: rebuild
          path: ./frontend/package.json
          
        # Sync configuration files
        - action: sync+restart
          path: ./frontend/vite.config.js
          target: /app/vite.config.js

  # Backend API with nodemon
  api:
    build:
      context: ./api
      dockerfile: Dockerfile.dev
    volumes:
      - ./api:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
    develop:
      watch:
        # Sync source code changes
        - action: sync+restart
          path: ./api/src
          target: /app/src
          
        # Rebuild on dependency changes
        - action: rebuild
          path: ./api/package.json
          
        # Sync environment configuration
        - action: sync
          path: ./api/.env.development
          target: /app/.env

  # Database migrations on schema changes
  migrator:
    build:
      context: ./api
      dockerfile: Dockerfile.dev
    command: npm run migrate:watch
    volumes:
      - ./api/migrations:/app/migrations
    depends_on:
      - postgres
    develop:
      watch:
        # Run migrations on schema changes
        - action: exec
          path: ./api/migrations
          command: npm run migrate:latest

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: development
      POSTGRES_USER: dev
      POSTGRES_PASSWORD: dev
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Usage with watch mode:

# Start services with file watching
docker compose watch

# Start specific services with watching
docker compose watch frontend api

# Watch with verbose output
docker compose watch --verbose


Network Configuration and Security

Advanced Network Topologies

version: '3.8'

services:
  # Public-facing load balancer
  traefik:
    image: traefik:latest
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"  # Traefik dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - public
      - internal

  # Frontend application
  frontend:
    build: ./frontend
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.frontend.rule=Host(`app.localhost`)"
      - "traefik.http.routers.frontend.entrypoints=web"
      - "traefik.http.services.frontend.loadbalancer.server.port=3000"
    networks:
      - internal

  # API backend
  api:
    build: ./api
    environment:
      - DATABASE_URL=postgres://user:pass@postgres:5432/app
      - REDIS_URL=redis://redis:6379
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`api.localhost`)"
      - "traefik.http.routers.api.entrypoints=web"
    depends_on:
      - postgres
      - redis
    networks:
      - internal
      - database

  # Database (isolated network)
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - database

  # Cache (isolated network)
  redis:
    image: redis:7-alpine
    networks:
      - database

  # Background workers (database access only)
  worker:
    build: ./worker
    environment:
      - DATABASE_URL=postgres://user:pass@postgres:5432/app
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis
    networks:
      - database

  # Monitoring (separate network)
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    networks:
      - monitoring
      - internal

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3001:3000"
    networks:
      - monitoring

networks:
  # Public network for external access
  public:
    driver: bridge
    
  # Internal network for inter-service communication
  internal:
    driver: bridge
    internal: false
    
  # Database network (isolated)
  database:
    driver: bridge
    internal: true
    
  # Monitoring network
  monitoring:
    driver: bridge

volumes:
  postgres_data:
  prometheus_data:
  grafana_data:


Resource Management and Scaling

Resource Limits and Constraints

version: '3.8'

services:
  # Resource-constrained web application
  app:
    build: .
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
          pids: 100
        reservations:
          cpus: '0.25'
          memory: 256M
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      update_config:
        parallelism: 2
        delay: 10s
        order: start-first
        failure_action: rollback
      rollback_config:
        parallelism: 1
        delay: 5s
        order: stop-first
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  # Database with resource management
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
        reservations:
          cpus: '0.5'
          memory: 512M
      placement:
        constraints:
          - node.role == manager
    volumes:
      - postgres_data:/var/lib/postgresql/data
    command: >
      postgres
      -c max_connections=200
      -c shared_buffers=256MB
      -c effective_cache_size=1GB
      -c work_mem=4MB

  # Scalable worker service
  worker:
    build: ./worker
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.1'
          memory: 128M
      restart_policy:
        condition: on-failure
        max_attempts: 5
    environment:
      - WORKER_CONCURRENCY=2
      - QUEUE_URL=redis://redis:6379

  # Redis with memory limit
  redis:
    image: redis:7-alpine
    deploy:
      resources:
        limits:
          memory: 256M
        reservations:
          memory: 64M
    command: redis-server --maxmemory 200mb --maxmemory-policy allkeys-lru

volumes:
  postgres_data:


Best Practices and Anti-Patterns

Learn to avoid common mistakes and implement production-ready configurations.


Security Best Practices

Secure Compose Configuration

version: '3.8'

services:
  # Application with security hardening
  app:
    build:
      context: .
      dockerfile: Dockerfile
    user: "1001:1001"  # Non-root user
    read_only: true    # Read-only filesystem
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:noexec,nosuid,size=100m
      - /var/tmp:noexec,nosuid,size=50m
    volumes:
      - type: bind
        source: ./app-data
        target: /app/data
        read_only: false
        bind:
          propagation: rprivate
    environment:
      - NODE_ENV=production
    secrets:
      - source: app_secret_key
        target: /run/secrets/secret_key
        mode: 0400
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  # Database with security hardening
  postgres:
    image: postgres:15-alpine
    user: postgres
    read_only: true
    tmpfs:
      - /tmp:noexec,nosuid,size=100m
      - /var/run/postgresql:noexec,nosuid,size=100m
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - type: tmpfs
        target: /dev/shm
        tmpfs:
          size: 256M
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER_FILE: /run/secrets/postgres_user
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
    secrets:
      - postgres_user
      - postgres_password
    command: >
      postgres
      -c ssl=on
      -c ssl_cert_file=/etc/ssl/certs/server.crt
      -c ssl_key_file=/etc/ssl/private/server.key
      -c log_statement=mod
      -c log_connections=on
      -c log_disconnections=on
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 30s
      timeout: 10s
      retries: 5

secrets:
  app_secret_key:
    file: ./secrets/app_secret_key.txt
  postgres_user:
    file: ./secrets/postgres_user.txt
  postgres_password:
    file: ./secrets/postgres_password.txt

volumes:
  postgres_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /secure/postgres/data

networks:
  default:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.enable_icc: "false"
      com.docker.network.bridge.enable_ip_masquerade: "true"


Common Anti-Patterns to Avoid

❌ Poor Practices

# DON'T DO THIS - Anti-patterns
version: '3.8'

services:
  bad-app:
    image: app:latest              # ❌ Always use specific versions
    privileged: true               # ❌ Avoid privileged mode
    user: root                     # ❌ Don't run as root
    network_mode: host             # ❌ Host network breaks isolation
    volumes:
      - /:/host                    # ❌ Don't mount entire host filesystem
    environment:
      - PASSWORD=hardcoded123      # ❌ Never hardcode secrets
      - DATABASE_HOST=localhost    # ❌ Use service names, not localhost
    restart: always                # ❌ 'unless-stopped' is usually better
    
  bad-db:
    image: mysql                   # ❌ No version specified
    environment:
      - MYSQL_ROOT_PASSWORD=       # ❌ Empty password
    volumes:
      - ./data:/var/lib/mysql      # ❌ Host directory instead of volume
    # ❌ No health check
    # ❌ No resource limits

✅ Best Practices Implementation

# DO THIS - Best practices
version: '3.8'

services:
  good-app:
    image: myapp:1.2.3            # ✅ Specific version tag
    user: "1001:1001"             # ✅ Non-root user
    read_only: true               # ✅ Read-only filesystem
    cap_drop:
      - ALL                       # ✅ Drop all capabilities
    volumes:
      - app_data:/app/data        # ✅ Named volume
    environment:
      - NODE_ENV=production
      - DATABASE_HOST=database    # ✅ Service name
    env_file:
      - .env                      # ✅ Environment file for non-secrets
    secrets:
      - jwt_secret                # ✅ Use secrets for sensitive data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    restart: unless-stopped       # ✅ Better restart policy
    deploy:
      resources:
        limits:
          memory: 1G              # ✅ Resource limits
          cpus: '0.5'
    networks:
      - app-network               # ✅ Custom network

  good-db:
    image: postgres:15.4-alpine   # ✅ Specific version
    user: postgres                # ✅ Non-root user
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER_FILE: /run/secrets/db_user
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_user                   # ✅ Secrets for credentials
      - db_password
    volumes:
      - postgres_data:/var/lib/postgresql/data  # ✅ Named volume
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 30s
      timeout: 10s
      retries: 5
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 2G
    networks:
      - app-network

secrets:
  jwt_secret:
    file: ./secrets/jwt_secret.txt
  db_user:
    file: ./secrets/db_user.txt
  db_password:
    file: ./secrets/db_password.txt

volumes:
  app_data:
  postgres_data:

networks:
  app-network:
    driver: bridge


Performance Optimization Guidelines

Docker Compose Performance Tuning

version: '3.8'

services:
  # Optimized application configuration
  app:
    build:
      context: .
      dockerfile: Dockerfile.optimized
      cache_from:
        - myapp:cache               # Use build cache
    image: myapp:${VERSION:-latest}
    
    # Memory and CPU optimization
    deploy:
      resources:
        limits:
          memory: 1G
          cpus: '1.0'
        reservations:
          memory: 256M
          cpus: '0.25'
    
    # Optimized volume mounts
    volumes:
      - type: volume
        source: app_cache
        target: /app/cache
        volume:
          nocopy: true              # Don't copy initial data
      - type: tmpfs
        target: /tmp
        tmpfs:
          size: 100M                # Limit tmpfs size
    
    # Environment-specific optimizations
    environment:
      - NODE_ENV=production
      - UV_THREADPOOL_SIZE=16       # Optimize Node.js thread pool
      - NODE_OPTIONS=--max-old-space-size=768  # Optimize V8 heap
    
    # Networking optimization
    networks:
      backend:
        aliases:
          - api.internal            # Custom DNS alias
    
    # Health check optimization
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s                   # Shorter timeout
      retries: 2                    # Fewer retries
      start_period: 30s             # Reduced start period

  # Optimized database configuration
  postgres:
    image: postgres:15.4-alpine
    
    # PostgreSQL performance tuning
    command: >
      postgres
      -c max_connections=100
      -c shared_buffers=256MB
      -c effective_cache_size=1GB
      -c maintenance_work_mem=64MB
      -c checkpoint_completion_target=0.9
      -c wal_buffers=16MB
      -c default_statistics_target=100
      -c random_page_cost=1.1
      -c effective_io_concurrency=200
      -c work_mem=4MB
      -c min_wal_size=1GB
      -c max_wal_size=4GB
    
    # Volume optimization
    volumes:
      - type: volume
        source: postgres_data
        target: /var/lib/postgresql/data
        volume:
          driver_opts:
            type: none
            o: bind
            device: /opt/app/postgres  # SSD mount point
    
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '2.0'
        reservations:
          memory: 512M
          cpus: '0.5'

  # Redis optimization
  redis:
    image: redis:7.2-alpine
    command: >
      redis-server
      --maxmemory 256mb
      --maxmemory-policy allkeys-lru
      --tcp-keepalive 60
      --tcp-backlog 511
      --timeout 0
      --save 900 1 300 10 60 10000  # Optimized persistence
    
    volumes:
      - type: volume
        source: redis_data
        target: /data

volumes:
  app_cache:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs
      o: size=500M,uid=1001,gid=1001

  postgres_data:
    driver: local

  redis_data:
    driver: local

networks:
  backend:
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: 1450


Monitoring, Logging, and Debugging

Implement comprehensive observability for your Compose applications.


Comprehensive Monitoring Stack

Full Observability Configuration

version: '3.8'

services:
  # Main application with monitoring
  app:
    build: .
    environment:
      - NODE_ENV=production
      - METRICS_ENABLED=true
      - JAEGER_ENDPOINT=http://jaeger:14268/api/traces
    labels:
      - "prometheus.io/scrape=true"
      - "prometheus.io/port=3000"
      - "prometheus.io/path=/metrics"
    networks:
      - app-network
      - monitoring
    depends_on:
      - postgres
      - redis
    logging:
      driver: "fluentd"
      options:
        fluentd-address: "localhost:24224"
        tag: "app.logs"

  # Database
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - app-network
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"

  # Redis
  redis:
    image: redis:7-alpine
    networks:
      - app-network
    command: redis-server --appendonly yes

  # Prometheus monitoring
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./monitoring/prometheus/rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    networks:
      - monitoring
    restart: unless-stopped

  # Grafana dashboards
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./monitoring/grafana/provisioning:/etc/grafana/provisioning
      - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards
    networks:
      - monitoring
    restart: unless-stopped
    depends_on:
      - prometheus

  # Node Exporter for system metrics
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    networks:
      - monitoring
    restart: unless-stopped

  # cAdvisor for container metrics
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    privileged: true
    devices:
      - /dev/kmsg
    networks:
      - monitoring
    restart: unless-stopped

  # Jaeger for distributed tracing
  jaeger:
    image: jaegertracing/all-in-one:latest
    container_name: jaeger
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
      - "16686:16686"
      - "14268:14268"
      - "14250:14250"
      - "9411:9411"
    environment:
      - COLLECTOR_ZIPKIN_HOST_PORT=:9411
    networks:
      - monitoring
    restart: unless-stopped

  # FluentD for log aggregation
  fluentd:
    build: ./monitoring/fluentd
    container_name: fluentd
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    volumes:
      - ./monitoring/fluentd/conf:/fluentd/etc
      - ./logs:/fluentd/log
    networks:
      - monitoring
    restart: unless-stopped

  # Alertmanager for alerts
  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./monitoring/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
      - alertmanager_data:/alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--web.external-url=http://localhost:9093'
    networks:
      - monitoring
    restart: unless-stopped

  # Elasticsearch for log storage
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - xpack.security.enabled=false
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - monitoring
    restart: unless-stopped

  # Kibana for log visualization
  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - monitoring
    restart: unless-stopped

networks:
  app-network:
    driver: bridge
  monitoring:
    driver: bridge

volumes:
  postgres_data:
  prometheus_data:
  grafana_data:
  alertmanager_data:
  elasticsearch_data:


Debugging and Troubleshooting Tools

Debug-Enabled Development Configuration

version: '3.8'

services:
  # Application with debugging enabled
  app-debug:
    build:
      context: .
      dockerfile: Dockerfile.debug
      target: debug
    environment:
      - NODE_ENV=development
      - DEBUG=app:*,express:*,sequelize:*
      - LOG_LEVEL=debug
      - ENABLE_DEBUG_ENDPOINTS=true
    volumes:
      - ./src:/app/src
      - ./logs:/app/logs
      - debug_data:/debug
    ports:
      - "3000:3000"
      - "9229:9229"    # Node.js debugger
      - "9464:9464"    # Debug metrics
    command: >
      sh -c "npm install &&
             npm run debug"
    stdin_open: true
    tty: true
    networks:
      - debug-network
    labels:
      - "debug.enabled=true"

  # Database with query logging
  postgres-debug:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: myapp_debug
      POSTGRES_USER: debug
      POSTGRES_PASSWORD: debug
    volumes:
      - postgres_debug_data:/var/lib/postgresql/data
      - ./database/debug-scripts:/docker-entrypoint-initdb.d
    ports:
      - "5432:5432"
    command: >
      postgres
      -c log_statement=all
      -c log_duration=on
      -c log_line_prefix='%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '
      -c log_checkpoints=on
      -c log_connections=on
      -c log_disconnections=on
      -c log_lock_waits=on
      -c shared_preload_libraries=pg_stat_statements
    networks:
      - debug-network

  # Redis with debugging
  redis-debug:
    image: redis:7-alpine
    command: >
      redis-server
      --loglevel debug
      --logfile /redis-logs/redis.log
    volumes:
      - redis_debug_data:/data
      - redis_logs:/redis-logs
    ports:
      - "6379:6379"
    networks:
      - debug-network

  # Debug tools container
  debug-tools:
    image: nicolaka/netshoot:latest
    container_name: debug-tools
    command: sleep infinity
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - debug_tools:/debug
    networks:
      - debug-network
    privileged: true
    stdin_open: true
    tty: true

  # Log aggregation for debugging
  debug-fluentd:
    build: ./debug/fluentd
    volumes:
      - ./debug/fluentd/conf:/fluentd/etc
      - debug_logs:/fluentd/log
    ports:
      - "24224:24224"
    networks:
      - debug-network

  # Web-based log viewer
  dozzle:
    image: amir20/dozzle:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "8888:8080"
    environment:
      - DOZZLE_LEVEL=debug
      - DOZZLE_TAILSIZE=1000

networks:
  debug-network:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: debug-bridge

volumes:
  postgres_debug_data:
  redis_debug_data:
  redis_logs:
  debug_data:
  debug_tools:
  debug_logs:

Useful Debugging Commands

# Container debugging commands
docker compose exec app bash                    # Interactive shell
docker compose exec --user root app bash       # Root shell for debugging
docker compose logs -f --tail=100 app          # Follow recent logs
docker compose logs --since="1h" app           # Logs from last hour
docker compose top                              # Process information
docker compose exec app ps aux                 # Processes inside container

# Network debugging
docker compose exec debug-tools ping app       # Test connectivity
docker compose exec debug-tools nslookup app   # DNS resolution
docker compose exec debug-tools netstat -tlnp  # Port listening
docker compose exec debug-tools ss -tulpn      # Socket statistics

# Performance debugging
docker compose exec app htop                   # Process monitoring
docker compose exec app iotop                  # I/O monitoring
docker compose stats                           # Resource usage
docker compose exec app strace -p 1            # System call tracing

# Database debugging
docker compose exec postgres-debug psql -U debug -d myapp_debug
docker compose exec postgres-debug pg_stat_activity
docker compose exec redis-debug redis-cli monitor

# Log analysis
docker compose logs app | grep ERROR           # Filter error logs
docker compose logs --since="2h" --until="1h" # Time range
docker compose logs -t app                     # Timestamps


Production Deployment Strategies

Implement enterprise-ready deployment patterns with Docker Compose.


Blue-Green Deployment

Blue-Green Deployment Configuration

# docker-compose.blue-green.yml
version: '3.8'

services:
  # Load balancer / Proxy
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/blue-green.conf:/etc/nginx/nginx.conf
      - ./ssl:/etc/nginx/ssl
    depends_on:
      - app-blue
      - app-green
    networks:
      - frontend
    environment:
      - ACTIVE_ENVIRONMENT=${ACTIVE_ENV:-blue}
    restart: unless-stopped

  # Blue environment
  app-blue:
    image: myapp:${BLUE_VERSION:-latest}
    environment:
      - NODE_ENV=production
      - APP_ENV=blue
      - DATABASE_URL=${BLUE_DATABASE_URL}
      - REDIS_URL=${BLUE_REDIS_URL}
    networks:
      - frontend
      - blue-backend
    deploy:
      replicas: ${BLUE_REPLICAS:-3}
      resources:
        limits:
          memory: 1G
          cpus: '0.5'
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  # Green environment
  app-green:
    image: myapp:${GREEN_VERSION:-latest}
    environment:
      - NODE_ENV=production
      - APP_ENV=green
      - DATABASE_URL=${GREEN_DATABASE_URL}
      - REDIS_URL=${GREEN_REDIS_URL}
    networks:
      - frontend
      - green-backend
    deploy:
      replicas: ${GREEN_REPLICAS:-0}
      resources:
        limits:
          memory: 1G
          cpus: '0.5'
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  # Shared database
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - blue-backend
      - green-backend
    restart: unless-stopped

  # Shared cache
  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    networks:
      - blue-backend
      - green-backend
    restart: unless-stopped

networks:
  frontend:
    driver: bridge
  blue-backend:
    driver: bridge
  green-backend:
    driver: bridge

volumes:
  postgres_data:
  redis_data:

Blue-Green Deployment Scripts

#!/bin/bash
# blue-green-deploy.sh

set -euo pipefail

COMPOSE_FILE="docker-compose.blue-green.yml"
NEW_VERSION="${1:-latest}"
CURRENT_ENV="${ACTIVE_ENV:-blue}"
TARGET_ENV=$([ "$CURRENT_ENV" = "blue" ] && echo "green" || echo "blue")

echo "Current environment: $CURRENT_ENV"
echo "Target environment: $TARGET_ENV"
echo "Deploying version: $NEW_VERSION"

# Deploy to target environment
echo "Deploying to $TARGET_ENV environment..."
if [ "$TARGET_ENV" = "green" ]; then
    export GREEN_VERSION="$NEW_VERSION"
    export GREEN_REPLICAS=3
    export GREEN_DATABASE_URL="$DATABASE_URL"
    export GREEN_REDIS_URL="$REDIS_URL"
else
    export BLUE_VERSION="$NEW_VERSION"
    export BLUE_REPLICAS=3
    export BLUE_DATABASE_URL="$DATABASE_URL"
    export BLUE_REDIS_URL="$REDIS_URL"
fi

# Start target environment
docker compose -f "$COMPOSE_FILE" up -d "app-$TARGET_ENV"

# Wait for health checks
echo "Waiting for $TARGET_ENV environment to be healthy..."
timeout 300 bash -c "
    while ! docker compose -f '$COMPOSE_FILE' exec app-$TARGET_ENV curl -f http://localhost:3000/health; do
        echo 'Waiting for health check...'
        sleep 10
    done
"

# Run smoke tests
echo "Running smoke tests against $TARGET_ENV environment..."
./scripts/smoke-test.sh "$TARGET_ENV"

# Switch traffic
echo "Switching traffic to $TARGET_ENV environment..."
export ACTIVE_ENV="$TARGET_ENV"
docker compose -f "$COMPOSE_FILE" up -d nginx

# Scale down old environment
echo "Scaling down $CURRENT_ENV environment..."
if [ "$CURRENT_ENV" = "green" ]; then
    export GREEN_REPLICAS=0
else
    export BLUE_REPLICAS=0
fi

docker compose -f "$COMPOSE_FILE" up -d "app-$CURRENT_ENV"

echo "Blue-green deployment completed successfully!"
echo "Active environment: $TARGET_ENV"


Rolling Updates and Zero-Downtime Deployment

Rolling Update Configuration

# docker-compose.rolling.yml
version: '3.8'

services:
  # Load balancer with health checks
  haproxy:
    image: haproxy:latest
    ports:
      - "80:80"
      - "443:443"
      - "8404:8404"  # Stats page
    volumes:
      - ./haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    networks:
      - frontend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8404/stats"]
      interval: 30s
      timeout: 10s
      retries: 3

  # Application with rolling update support
  app:
    image: myapp:${APP_VERSION:-latest}
    environment:
      - NODE_ENV=production
      - GRACEFUL_SHUTDOWN_TIMEOUT=30
    networks:
      - frontend
      - backend
    deploy:
      replicas: ${APP_REPLICAS:-5}
      update_config:
        parallelism: 1           # Update one container at a time
        delay: 30s               # Wait between updates
        order: start-first       # Start new before stopping old
        failure_action: rollback # Rollback on failure
        monitor: 60s             # Monitor period
        max_failure_ratio: 0.2   # Max 20% failure rate
      rollback_config:
        parallelism: 1
        delay: 10s
        order: stop-first
        monitor: 30s
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      resources:
        limits:
          memory: 1G
          cpus: '0.5'
        reservations:
          memory: 256M
          cpus: '0.1'
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    stop_grace_period: 45s       # Graceful shutdown time

  # Database with backup during updates
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./backups:/backups
    networks:
      - backend
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '1.0'
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $DB_USER"]
      interval: 30s
      timeout: 10s
      retries: 5

  # Migration service
  migrate:
    image: myapp:${APP_VERSION:-latest}
    command: npm run migrate
    environment:
      - NODE_ENV=production
      - DATABASE_URL=${DATABASE_URL}
    networks:
      - backend
    depends_on:
      postgres:
        condition: service_healthy
    restart: "no"

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge

volumes:
  postgres_data:

Rolling Update Script

#!/bin/bash
# rolling-update.sh

set -euo pipefail

COMPOSE_FILE="docker-compose.rolling.yml"
NEW_VERSION="${1:-latest}"
HEALTH_CHECK_URL="${2:-http://localhost/health}"

echo "Starting rolling update to version: $NEW_VERSION"

# Pre-deployment checks
echo "Running pre-deployment checks..."
docker image pull "myapp:$NEW_VERSION"

# Create database backup
echo "Creating database backup..."
docker compose -f "$COMPOSE_FILE" exec postgres pg_dump -U "$DB_USER" "$DB_NAME" > "backup-$(date +%Y%m%d_%H%M%S).sql"

# Run database migrations
echo "Running database migrations..."
export APP_VERSION="$NEW_VERSION"
docker compose -f "$COMPOSE_FILE" run --rm migrate

# Update application with rolling strategy
echo "Starting rolling update of application..."
docker compose -f "$COMPOSE_FILE" up -d app

# Monitor deployment
echo "Monitoring deployment progress..."
for i in {1..30}; do
    if curl -f "$HEALTH_CHECK_URL" >/dev/null 2>&1; then
        echo "Health check passed (attempt $i/30)"
        HEALTHY_COUNT=$((HEALTHY_COUNT + 1))
        if [ $HEALTHY_COUNT -ge 3 ]; then
            echo "Rolling update completed successfully!"
            break
        fi
    else
        echo "Health check failed (attempt $i/30)"
        HEALTHY_COUNT=0
    fi
    
    if [ $i -eq 30 ]; then
        echo "Rolling update failed - triggering rollback"
        docker compose -f "$COMPOSE_FILE" rollback app
        exit 1
    fi
    
    sleep 10
done

# Cleanup old images
echo "Cleaning up old images..."
docker image prune -f

echo "Rolling update completed successfully!"


Disaster Recovery and Backup Strategies

Comprehensive Backup Configuration


Backup and Recovery Scripts

#!/bin/bash
# backup-restore.sh

set -euo pipefail

COMPOSE_FILE="docker-compose.backup.yml"
BACKUP_TYPE="${1:-full}"
RESTORE_POINT="${2:-}"

case "$BACKUP_TYPE" in
    "full")
        echo "Creating full backup..."
        docker compose -f "$COMPOSE_FILE" exec backup /scripts/full-backup.sh
        ;;
    
    "incremental")
        echo "Creating incremental backup..."
        docker compose -f "$COMPOSE_FILE" exec backup /scripts/incremental-backup.sh
        ;;
    
    "restore")
        if [ -z "$RESTORE_POINT" ]; then
            echo "Error: Restore point required"
            echo "Usage: $0 restore <backup-file-or-timestamp>"
            exit 1
        fi
        
        echo "Restoring from: $RESTORE_POINT"
        
        # Stop application
        docker compose -f "$COMPOSE_FILE" stop app
        
        # Restore database
        docker compose -f "$COMPOSE_FILE" exec postgres-primary /scripts/restore.sh "$RESTORE_POINT"
        
        # Start application
        docker compose -f "$COMPOSE_FILE" start app
        ;;
    
    "pitr")
        if [ -z "$RESTORE_POINT" ]; then
            echo "Error: Point-in-time restore requires timestamp"
            echo "Usage: $0 pitr 'YYYY-MM-DD HH:MM:SS'"
            exit 1
        fi
        
        echo "Point-in-time restore to: $RESTORE_POINT"
        docker compose -f "$COMPOSE_FILE" exec pitr /scripts/pitr-restore.sh "$RESTORE_POINT"
        ;;
    
    "verify")
        echo "Verifying backup integrity..."
        docker compose -f "$COMPOSE_FILE" exec backup /scripts/verify-backup.sh
        ;;
    
    *)
        echo "Usage: $0 {full|incremental|restore|pitr|verify} [restore-point]"
        exit 1
        ;;
esac


Key Points

Docker Compose Mastery Summary
  • Modern Compose Usage
    - Use Compose V2 with docker compose command (space, not hyphen)
    - Leverage new features like watch mode and improved networking
    - Implement proper health checks and dependency management
    - Follow security best practices with non-root users and read-only filesystems
  • Production Readiness
    - Implement comprehensive monitoring and logging solutions
    - Use resource limits and restart policies appropriately
    - Design for high availability with multiple replicas and health checks
    - Plan disaster recovery with backup and restore procedures
  • Development Excellence
    - Utilize environment-specific override files effectively
    - Implement hot reload and debugging capabilities
    - Maintain clean separation between development and production configs
    - Use secrets management for sensitive configuration data


Conclusion

Docker Compose has evolved into an indispensable tool for modern application development and deployment. The transition from V1 to V2 brought significant improvements in performance, usability, and integration with the broader Docker ecosystem.

From simple development environments to complex production deployments, Docker Compose provides the flexibility and power needed to manage multi-container applications effectively. The YAML-based configuration approach makes complex infrastructures readable, maintainable, and version-controllable.


Strategic Implementation Guidelines

  1. Start with Development: Use Compose to standardize development environments across teams
  2. Progress to Staging: Implement override files for environment-specific configurations
  3. Scale to Production: Add monitoring, logging, and deployment automation
  4. Plan for Growth: Design with scalability and disaster recovery in mind
  5. Maintain Security: Implement proper secrets management and security hardening


Future Considerations

As container orchestration continues to evolve, Docker Compose remains relevant for single-host deployments, development environments, and smaller production workloads. For larger, distributed systems, consider Kubernetes or Docker Swarm, but Docker Compose provides an excellent foundation and learning platform for container orchestration concepts.

The combination of simplicity, power, and extensive ecosystem support ensures that Docker Compose will continue to be a cornerstone tool for developers and DevOps professionals working with containerized applications.


References