S3 Output

Writes Snappy-compressed Parquet files to Amazon S3. This is a lightweight alternative to the Athena output when you only need S3 storage without Glue catalog or Athena integration.

How it works

1. Serializes compacted WAL segments into Parquet with Snappy compression.
2. Derives the S3 key from the namespace, time partitioning, and a content-based hash.
3. Uploads the Parquet file to S3 using PutObject.

Configuration

DATA_OUTPUT_PLUGIN_NAME=S3
DATA_OUTPUT_S3_BUCKET=my-output-bucket
DATA_OUTPUT_S3_PREFIX=warehouse/events

Or via YAML pipeline config:

data_sinks:
  sink:
    S3:
      s3_bucket: "my-output-bucket"
      s3_prefix: "warehouse/events"

Configuration variables

Variable	Default	Description
s3_bucket / DATA_OUTPUT_S3_BUCKET	(required)	S3 bucket for Parquet output
s3_prefix / DATA_OUTPUT_S3_PREFIX		Key prefix for output objects

S3 layout

s3://{s3_bucket}/{s3_prefix}/{namespace}/
  {partition_path}/
    {time_partition}/
      {content_hash}.parquet

Time partitioning is applied automatically based on pipeline configuration.

AWS credentials

S3 access uses the standard AWS credential chain (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, instance profile, etc.).

AWS permissions required

The IAM identity running Skipprd needs:

• s3:PutObject on the output bucket/prefix