FinOps for Data Engineering: Estimating and Controlling Costs in AWS

A pipeline that processes data efficiently but ignores the cost of doing so is not a well-engineered pipeline. In cloud environments, every read, write, compute minute, and byte stored has a price. That price is predictable from the design of the pipeline, which means the cost is an engineering decision made before the first deployment, not a surprise discovered at the end of the month when the bill arrives.

FinOps for data engineering is not about penny-pinching. It is about accountability: understanding what resources a system consumes, why, and whether the cost is proportionate to the value the system provides. An engineer who cannot estimate the monthly cost of a pipeline they designed is an engineer who has not fully understood that pipeline.

Storage cost: tiers and lifecycle policies

S3 storage pricing varies by access tier. Data that is accessed frequently belongs in Standard. Data that is accessed occasionally belongs in Standard-IA (Infrequent Access). Data that is accessed rarely but must be retrievable within minutes belongs in Glacier Instant Retrieval. Data that is never accessed except for compliance belongs in Glacier Deep Archive.

# S3 lifecycle policy: automate tier transitions (CloudFormation / boto3)
import boto3

s3 = boto3.client("s3")

lifecycle_policy = {
    "Rules": [{
        "ID": "data-lake-tiering",
        "Status": "Enabled",
        "Filter": {"Prefix": "raw/"},
        "Transitions": [
            {
                "Days": 30,
                "StorageClass": "STANDARD_IA"  # ~$0.0125/GB vs $0.023/GB Standard
            },
            {
                "Days": 90,
                "StorageClass": "GLACIER_IR"   # ~$0.004/GB
            },
            {
                "Days": 365,
                "StorageClass": "DEEP_ARCHIVE"  # ~$0.00099/GB
            },
        ],
        "Expiration": {
            "Days": 2555  # 7-year retention, then delete
        }
    }]
}

s3.put_bucket_lifecycle_configuration(
    Bucket="data-lake-raw",
    LifecycleConfiguration=lifecycle_policy
)

A raw data lake at 10 TB in Standard costs $230/month in storage. The same 10 TB with a 30-day Standard to Standard-IA transition (assuming 80% of data is older than 30 days) costs approximately $78/month. The lifecycle policy is a one-time configuration that reduces the storage bill by 66% on an ongoing basis.

Query cost: scan-based pricing

Athena charges $5 per TB of data scanned. A query that scans 1 TB costs $5. The same query, rewritten to use partition pruning and columnar format compression, might scan 20 GB and cost $0.10. The difference is not luck: it is partitioning and format choice made at data loading time.

# Cost estimator for Athena queries
def estimate_athena_cost(
    table_size_gb: float,
    partition_filter_fraction: float,  # fraction of partitions that match the query
    compression_ratio: float = 0.25,   # Parquet is ~75% smaller than equivalent CSV
    price_per_tb: float = 5.0
) -> dict:
    scanned_gb = table_size_gb * partition_filter_fraction * compression_ratio
    cost = (scanned_gb / 1024) * price_per_tb

    return {
        "table_size_gb":    table_size_gb,
        "scanned_gb":       round(scanned_gb, 2),
        "estimated_cost":   round(cost, 4),
        "cost_per_1000_runs": round(cost * 1000, 2),
    }

# Example: 500 GB CSV table, querying 1 week out of 52 (1/52 partitions)
csv_result = estimate_athena_cost(500, 1/52, compression_ratio=1.0)
# {'scanned_gb': 9.62, 'estimated_cost': 0.0469, 'cost_per_1000_runs': 46.88}

# Same data in Parquet with date partitioning
pq_result = estimate_athena_cost(500, 1/52, compression_ratio=0.25)
# {'scanned_gb': 2.40, 'estimated_cost': 0.0117, 'cost_per_1000_runs': 11.72}

The Parquet + partitioning combination reduces the per-query scan cost by 75%. At 1,000 queries per month, that difference is $35/month from a format and partitioning decision made once at pipeline design time.

Compute right-sizing

The most common compute waste pattern in data engineering is over-provisioning: running a pipeline on a cluster sized for peak load when most runs process a fraction of the peak volume. Right-sizing is the practice of matching the compute resource to the actual workload, using metrics from real runs rather than guesses.

# AWS Cost Explorer: identify over-provisioned EMR clusters via boto3
import boto3
from datetime import datetime, timedelta

ce = boto3.client("cost-explorer", region_name="us-east-1")

response = ce.get_cost_and_usage(
    TimePeriod={
        "Start": (datetime.today() - timedelta(days=30)).strftime("%Y-%m-%d"),
        "End":   datetime.today().strftime("%Y-%m-%d"),
    },
    Granularity="DAILY",
    Filter={
        "Tags": {
            "Key": "Project",
            "Values": ["orders-pipeline"]
        }
    },
    Metrics=["UnblendedCost"],
    GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
)

for day in response["ResultsByTime"]:
    for group in day["Groups"]:
        service = group["Keys"][0]
        cost    = group["Metrics"]["UnblendedCost"]["Amount"]
        if float(cost) > 1.0:
            print(f"{day['TimePeriod']['Start']}  {service}: ${float(cost):.2f}")

Tagging and attribution

A cloud bill without tagging is a list of service charges with no connection to the engineering decisions that produced them. Tagging connects the cost to the team, project, and pipeline responsible for it, which is what makes cost reduction actionable: you cannot reduce what you cannot attribute.

import boto3

def tag_pipeline_resources(resource_arn: str, pipeline_name: str, team: str, env: str) -> None:
    client = boto3.client("resourcegroupstaggingapi")
    client.tag_resources(
        ResourceARNList=[resource_arn],
        Tags={
            "Project":     pipeline_name,
            "Team":        team,
            "Environment": env,          # dev / staging / prod
            "CostCenter":  "data-eng",
            "ManagedBy":   "terraform",  # or cloudformation
        }
    )

With consistent tagging, Cost Explorer reports can show the monthly cost of a specific pipeline, team, or environment. A pipeline whose monthly cost increased 40% from one month to the next is visible as a tagged cost anomaly, not buried in an aggregate line for EC2 or S3. That visibility is what closes the loop between engineering decisions and their financial consequences.

Cost as a first-class engineering concern

The practice I follow is to include a cost estimate in every architecture decision record and every significant pipeline design before implementation begins. The estimate uses the patterns above: expected data volume, query frequency, compute hours, and storage tier. It does not need to be exact. An order-of-magnitude estimate at design time is worth more than a precise measurement of a cost that has already been incurred for six months.

An engineer who designs a pipeline without estimating its cost is an engineer who has made a financial decision without realizing it. Cloud resources are not free at design time and expensive at billing time: they are expensive at design time, and the design is where the cost is determined. FinOps is the discipline of making that explicit before the first deploy.