Skip to content

0059 all pipeline parameters must be stored in postgresql and edi

ADR-59 — All Pipeline Parameters Must Be Stored in PostgreSQL and Editable via Console

Status: Decided (2026-04-17)

Context: FTF baseline parameters (BASE_ENV) were hardcoded in ablation_matrix.py. Every parameter change required a code commit, CI build, image push, and pod restart — a 15-minute cycle for changing a single value like CVN_ATR_PERIOD. This created friction during the tuning protocol, encouraged batch changes (multiple parameters changed at once, breaking ceteris paribus), and made it impossible to iterate quickly between FTF runs. Two incidents resulted from stale parameters surviving across builds because the code change was merged but the image wasn't redeployed.

Decision: All pipeline parameters that affect FTF runs MUST be stored in PostgreSQL (ftf_config table) and editable via the Console UI (console.cvntrade.eu). The JSON file (config/ftf_baseline.json) serves only as a fallback for local development and tests. Code MUST NOT contain hardcoded parameter values for anything that is FTF-tunable.

Invariants:

  • Single source of truth: The ftf_config table (single row, base_env JSONB) is the authoritative source for all pipeline parameters at runtime. ablation_matrix.py reads from PostgreSQL first, JSON file fallback only when DB is unavailable (local dev, CI tests).
  • Console is the only editor: Parameter changes MUST go through the Console UI (Streamlit). No direct SQL updates, no kubectl exec, no code changes for parameter tuning. The Console enforces change reason and operator identity.
  • Audit trail: Every change is logged in ftf_config_history with timestamp, operator, and change reason. The history is immutable — no deletes, no updates.
  • Run snapshot: When an FTF run starts, the current base_env is snapshot into finetune_runs.base_env (JSONB). This is the immutable proof of which parameters were active during that run, regardless of subsequent edits.
  • No silent drift: If a parameter exists in the code (env var read) but not in ftf_config.base_env, the pipeline MUST fail-fast with a clear error — not silently use a default. New parameters must be added to ftf_config via the Console before they can be used.
  • All values are strings: Consistent with env var semantics. Type conversion happens at the consumer, not at the config layer.

What goes in ftf_config: - All CVN_* environment variables that affect training, inference, filtering, or execution - Cost model parameters (fee, slippage, funding rate) - HPO parameters (objective, trials, early stopping) - Filter toggles and thresholds

What does NOT go in ftf_config: - Infrastructure config (CPU, memory, node selectors) — stays in Helm values - Secrets (API keys, DB passwords) — stays in K8s Secrets - Airflow DAG config (schedule, retries) — stays in DAG code - Code-level constants (outlier caps, log format) — stays in code

Alternatives rejected: - Helm ConfigMap for parameters: Requires helm upgrade + pod restart. Same friction as code changes. - Environment variables via Airflow UI: Scattered across multiple DAGs, no audit trail, no single view. - Git-only config file: Requires commit + build + deploy. The 15-minute cycle is the problem this ADR solves. - Direct database editing: No audit trail, no validation, error-prone. The Console provides guardrails.

Files: scripts/ftf_config_ui.py, src/commun/finetune/ablation_matrix.py, infra/migrations/007_ftf_config.sql, infra/k8s/console.yaml