Infrastructure as Code: Docker and Total Reproducibility of My Projects

A data project that only runs on the machine where it was built is not a project. It is a local script with aspirations. The test of a real project is whether someone with no prior knowledge of the environment can clone the repository, run one or two commands, and have a working system. Infrastructure as code is what makes that possible: the environment is declared in version-controlled files, not configured manually on a specific machine and then hoped to be replicated correctly by whoever comes next.

The problem infrastructure as code solves

The "works on my machine" failure mode is well-known. Less often articulated is why it happens: the environment on the developer's machine accumulated state over time through manual installation steps, version choices, configuration changes, and dependencies installed for other projects that happen to satisfy this project's requirements. None of that state is recorded anywhere. When the project moves to another machine, CI environment, or production server, the state is different, and the project fails in ways that are difficult to diagnose because the failure is environmental rather than code-level.

Docker solves this by making the environment a build artifact: a Dockerfile that declares the base image, installs dependencies from a pinned requirements file, copies source code, and sets a deterministic entry point. The same Dockerfile produces the same image on any machine with Docker installed. The image is the environment, and the environment is version-controlled.

A Dockerfile for a data pipeline project

# Dockerfile
FROM python:3.11.9-slim

# Non-root user for security
RUN useradd --create-home --shell /bin/bash appuser

WORKDIR /app

# Install system dependencies first (changes infrequently, cached)
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Install Python dependencies (changes sometimes, cached until requirements change)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code last (changes frequently, not cached)
COPY src/ ./src/
COPY dbt_project/ ./dbt_project/
COPY tests/ ./tests/

USER appuser

# Default command: run the test suite
CMD ["pytest", "tests/", "-v", "--tb=short"]

The layer ordering in the Dockerfile is deliberate. Docker builds images layer by layer and caches each layer until its inputs change. Putting the rarely-changing layers (system dependencies, Python packages) before the frequently-changing layers (source code) means a code change triggers only the final copy step, not a full reinstall of all packages. A build that reinstalls 50 packages on every code change is a build that takes 5 minutes instead of 30 seconds.

docker-compose for the full local stack

A data project typically depends on more than one service: a database, a message broker, an orchestrator, a warehouse emulator. Docker Compose declares all of these services and their relationships in a single file, so the entire local development stack starts with one command.

# docker-compose.yml
version: "3.9"

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER:     ${POSTGRES_USER:-data}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-data}
      POSTGRES_DB:       ${POSTGRES_DB:-warehouse}
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./sql/init.sql:/docker-entrypoint-initdb.d/init.sql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U data -d warehouse"]
      interval: 10s
      timeout: 5s
      retries: 5

  pipeline:
    build: .
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      DATABASE_URL: postgresql://data:data@postgres:5432/warehouse
      PYTHONPATH: /app/src
    volumes:
      - ./src:/app/src          # mount source for hot reload during dev
      - ./data:/app/data
    command: python -m pipeline.main --date ${RUN_DATE:-today}

  dbt:
    build: .
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      DATABASE_URL: postgresql://data:data@postgres:5432/warehouse
    volumes:
      - ./dbt_project:/app/dbt_project
    command: dbt run --project-dir /app/dbt_project --profiles-dir /app/dbt_project

volumes:
  postgres_data:

The healthcheck on the Postgres service means the pipeline service waits until the database is actually ready to accept connections before starting, not just until the container has started. This eliminates the race condition where the pipeline starts before the database is ready and fails with a connection error that looks like a code bug.

The clone-build-run pattern

The end state of this infrastructure is that the README for any project I publish contains exactly these three lines:

git clone https://github.com/andrematiello/[project].git
cd [project]
docker compose up

Anyone who runs these commands on any machine with Docker installed gets a working instance of the project. No Python version management, no virtual environment setup, no manual database creation, no configuration file editing. The project works, and the output is deterministic.

Environment variables and secrets

Credentials and environment-specific configuration should never be in the repository, even in a Docker context. The pattern is a .env.example file committed to the repository (showing the names of all required variables with placeholder values) and a .env file in .gitignore (holding the real values on each machine).

# .env.example (committed to repository)
POSTGRES_USER=data
POSTGRES_PASSWORD=change_me_in_env
POSTGRES_DB=warehouse
COINGECKO_API_KEY=your_key_here
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=us-east-1

# .gitignore (prevents .env from ever being committed)
.env
*.env
!.env.example

Docker Compose automatically reads from a .env file in the same directory. The developer copies .env.example to .env, fills in the real values, and runs docker compose up. The secrets never enter the repository, the environment is still reproducible, and the onboarding instructions remain three lines.