S3 Log Archival

Overview

Loki supports S3 as a storage backend for long-term log retention. In production, this replaces the default filesystem storage with durable, cost-effective cloud storage.

For local development, LocalStack (already in docker-compose.yml under profiles: ["localstack"]) provides S3-compatible storage.

Loki S3 Configuration

To enable S3 storage, update docker/loki/loki-config.yml:

Replace the common.storage block:

common:
  storage:
    s3:
      bucketnames: ${LOKI_S3_BUCKET:-crm-logs}
      region: ${AWS_REGION:-us-east-1}
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}
      # For LocalStack development:
      # endpoint: http://localstack:4566
      # insecure: true
      # s3forcepathstyle: true

Retention Tiers

S3 Lifecycle Rules

Log type	Loki (hot, searchable)	S3 Standard (warm)	S3 Glacier Instant (cold)	Delete
Application logs	30 days	30-90 days	-	90 days
Security logs	30 days	30-365 days	365 days - 1 year	1 year
Audit logs	30 days	30-365 days	365 days - 1 year	1 year

Implementation

S3 lifecycle rules should be configured via AWS CLI or Terraform:

aws s3api put-bucket-lifecycle-configuration \
  --bucket crm-logs \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "app-logs-retention",
        "Filter": { "Prefix": "app/" },
        "Status": "Enabled",
        "Transitions": [
          { "Days": 90, "StorageClass": "GLACIER_IR" }
        ],
        "Expiration": { "Days": 90 }
      },
      {
        "ID": "security-audit-logs-retention",
        "Filter": { "Prefix": "security/" },
        "Status": "Enabled",
        "Transitions": [
          { "Days": 365, "StorageClass": "GLACIER_IR" }
        ],
        "Expiration": { "Days": 365 }
      }
    ]
  }'

Estimated AWS Cost

At ~10 GB/month compressed log volume:

Component	Monthly cost
S3 Standard (90 GB)	~$2.07
S3 Glacier Instant (90 GB)	~$0.36
PUT requests	~$0.50
Total	~$3/month

Environment Variables

Add to .env for production S3 storage:

LOKI_S3_BUCKET=crm-logs
# AWS credentials are shared with the existing AWS_* vars in .env

For local development with LocalStack:

LOKI_S3_ENDPOINT=http://localstack:4566