Configuration Overview

Skipprd is configured primarily with skippr.yml. Environment variables are still supported for secrets, deployment overrides, and backwards-compatible engine configuration.

Start with:

• skippr.yml for the canonical project shape
• Input Source for data_sources
• Output Destination for ingest data_sinks and schema_sinks

skipprd discover / skipprd schema / skipprd sync run the engine against this file. There is no separate warehouses: dialect.

Environment overrides

Variable	Default	Section	Description
Pipeline identity
PIPELINE_NAME	default	Pipeline	Pipeline name
WORKSPACE_NAME	default	Pipeline	Workspace/domain name
TENANT	default	Pipeline	Tenant identifier
Input source
DATA_SOURCE_PLUGIN_NAME	(required)	Input	Source plugin: s3, file
DATA_SOURCE_S3_BUCKET		Input	S3 source bucket
DATA_SOURCE_S3_PREFIX		Input	S3 source key prefix
DATA_SOURCE_PATH		Input	Local file source path
DATA_SOURCE_BATCH_SIZE_BYTES	plugin-defined	Input	Batch size in bytes
DATA_SOURCE_BATCH_SIZE_SECONDS	plugin-defined	Input	Batch size in seconds
Output destination
DATA_OUTPUT_S3_BUCKET		Output	Destination S3 bucket for Parquet
DATA_OUTPUT_S3_PREFIX		Output	Destination S3 key prefix
SCHEMA_OUTPUT_GLUE_DATABASE_NAME		Output	Glue catalog database name
DATA_OUTPUT_ATHENA_WORKGROUP_NAME		Output	Athena workgroup
DATA_OUTPUT_ATHENA_RESULTS_S3_BUCKET		Output	S3 bucket for Athena query results
DATA_OUTPUT_MAX_ASYNC_UPLOADS	16	Output	Max concurrent Parquet uploads
Transforms
TRANSFORM_NAMESPACE_FIELDS		Transforms	Fields that define event type / namespace
TRANSFORM_BATCH_PARTITION_FIELDS		Transforms	Fields for Hive partitioning
TRANSFORM_BATCH_TIME_FIELDS		Transforms	Timestamp field(s) for time partitioning
TRANSFORM_BATCH_TIME_UNIT		Transforms	Time granularity: year, month, day, hour, minute
TRANSFORM_FLATTEN_EVENTS	no	Transforms	Flatten nested structures
TRANSFORM_BATCH_ORDER_FIELDS		Transforms	Sort rows within Parquet files for predicate pruning
Buffering & WAL
BUFFER_THRESHOLD_BYTES	10485760	Buffering	Buffer flush threshold (bytes)
BUFFER_THRESHOLD_SECONDS	60	Buffering	Buffer flush threshold (seconds)
WAL_STORAGE	disk	Buffering	WAL backend: disk, s3, or clustered
WAL_BYTES_PER_FILE	auto	Buffering	Optional WAL segment size override
WAL_MAX_DELAY_SECONDS	60	Buffering	Coarse max WAL segment age before flush
Skipprd state
SKIPPR_S3_BUCKET		Advanced	S3 bucket for metadata, offsets, WAL (when S3), deadletters
SKIPPRD_EL_STORAGE_MODE	s3	Advanced	Internal Skipprd EL metadata and stats persistence: s3 (default) or local
DATA_DIR	./data	Advanced	Local directory for WAL segments and offsets DB
Operational
SKIPPR_CHAOS_MODE	no	Advanced	Enable chaos mode (random SIGKILL for testing)
SKIPPR_ENV	prod	Advanced	Environment label
SCHEMA_AUTO_APPROVE	true	Advanced	Auto-approve schema changes
RESET_OFFSETS	false	Advanced	Reset offsets on startup
RESET_METADATA	false	Advanced	Reset metadata on startup
SYNC_FREQUENCY		Advanced	Sync frequency (seconds)