S3 Input

Reads data from an Amazon S3 bucket.

Supported formats

yaml

data_sources:
  sample:
    S3:
      s3_bucket: skippr-public-sample-data
      s3_prefix: bike-hire

JSON (including gzipped), CSV, and Parquet are supported.

bash

DATA_SOURCE_PLUGIN_NAME=s3
DATA_SOURCE_S3_BUCKET=my-source-bucket
DATA_SOURCE_S3_PREFIX=events/
AWS_DEFAULT_REGION=us-east-1

Variable	Default	Description
`DATA_SOURCE_S3_BUCKET`	(required)	S3 bucket name
`DATA_SOURCE_S3_PREFIX`		Key prefix to filter objects
`DATA_SOURCE_S3_DELIMITER`	`/`	S3 listing delimiter
`DATA_SOURCE_S3_PREFIX_ORDERED_DEPTH`	`0`	Depth for ordered prefix scanning
`DATA_SOURCE_BATCH_SIZE_BYTES`	`1024000`	Bytes per read batch
`DATA_SOURCE_BATCH_SIZE_SECONDS`	`600`	Max seconds per batch

Uses the standard AWS credential chain: environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), instance profile, or IAM role.

Variable	Default	Description
`S3_DOWNLOAD_CONCURRENCY`	auto	Concurrent S3 download streams
`S3_DOWNLOAD_MEMORY_MB`	auto	Memory budget for downloads
`S3_INFLATE_RATIO`	`4`	Expected compression ratio for gzipped data
`INGEST_PENDING_MAX_BYTES`	auto	Max bytes in the pending ingest queue

The S3 input tracks progress by object key and last-modified timestamp. On restart, already-ingested objects are skipped.

S3 access uses standard AWS credentials.

Variable	Description
`AWS_ACCESS_KEY_ID`	AWS access key
`AWS_SECRET_ACCESS_KEY`	AWS secret key
`AWS_DEFAULT_REGION`	AWS region (if not inferred from bucket location)

bash

export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-east-1"

Symptom	Fix
`Access Denied`	Check `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set, and the IAM policy allows `s3:GetObject` and `s3:ListBucket` on the bucket
`NoSuchBucket`	Verify the bucket name and region
`No files found`	Check `s3_prefix` matches the actual key prefix in the bucket