Architecture — Training Pipeline¶

Version: 3.3 Date: 2026-05-13 (V3.3 — restructured for committee review : flat structure, harness-first, S18 isolated, legacy demoted) Status: Canonical harness path in production ; model promotion blocked pending S18-S19 regression diagnosis

Status decomposition (committee gate)¶

1. Executive Summary¶

The training pipeline is now canonically routed through the Hamilton-based training harness for XGBoost, LightGBM and CatBoost. Since PR #904, all numeric hyperparameter defaults and HPO ranges are resolved at runtime from PG ftf_config.base_env via ADR-90 ; source-code hyperparameter values are forbidden and enforced by CI gate G5.

However, the post-harness training path is currently under diagnostic. The S17 canary FTF run exposed shallow-training behavior : LightGBM stops after 1-2 iterations across all trials, XGBoost shows median best_iteration around 17, and model quality is materially below the pre-#891 baseline. Current post-S17 f1_buy results on defi_top5 5m are : CatBoost 0.39, LightGBM 0.37 and XGBoost 0.09, versus a pre-#891 baseline around 0.42.

As a result, the harness architecture is production-canonical, but harness-trained model promotion is frozen until CVN-N001-EE-S18 and S19 are closed. The current leaderboard is informational only and must not be used for live trading promotion.

Pipeline flow (canonical, post-#898)¶

OHLCV → Enrichment → Labeling → Feature Engineering
    → Temporal Split → Harness DAG
    → { XGB | LGB | CB } via `harness.train_one(model_type, datasets, hpo_params)`
    → Evaluation → MLflow

CUSUM is applied at inference (per ADR-46 + committee reco) ; it is NOT a training filter. The legacy "Stage 7 HPO + XGBoost Training" name is retained in §8 Legacy Pipeline Analysis for historical context but the current canonical path is described in §4-§5.

Consolidated performance snapshot¶

Single source of truth — referenced by every other section :

Promotion-freeze decision is anchored on this table : no harness-trained model goes live until f1_buy on defi_top5 5m ATR0.5_1.5_H4 returns to ≥ 0.40 on ≥ 4/5 cryptos for all 3 model types (mlops_readiness.md §4 canary criterion).

2. Current Status & Promotion Freeze¶

Until both CVN-N001-EE-S18 (diagnostic) and the follow-up CVN-N001-EE-S19 (remediation) close with a verified canary, the operator MUST NOT promote any harness-trained model to live trading. Backtests and shadow runs are permitted ; auto-promotion is OFF (ADR-2).

3. Current Canonical Architecture¶

The pipeline below is the version in production as of 2026-05-13. See §8 Legacy Pipeline Analysis for the pre-#891 path (kept for historical bottleneck analysis ; do NOT reference for current implementation).

Binance OHLCV
   ↓
Enrichment (technical indicators)
   ↓
Labeling (triple-barrier ATR-dynamic)
   ↓
Feature Engineering (Hamilton DAG, fitted at train, frozen at inference)
   ↓
Temporal split (walk-forward, 5 folds × {train, val, test})
   ↓
Harness DAG (Hamilton, plugin registry)
   ↓
   ┌─ XGB ──┐
   ├─ LGB ──┤ → resolver `resolve(MODEL, TF, PARAM)` reads from PG ftf_config.base_env
   └─ CB ──┘   (455 keys, Console-editable, ADR-90)
   ↓
Evaluation (per-fold f1_buy / sortino / expectancy / drawdown, with bootstrap CI)
   ↓
MLflow registry (versioned model + feature contract pinned, ADR-23 / ADR-24)
   ↓
{Promotion gate} ← currently FROZEN pending S18-S19 — see §2

Each upstream stage (OHLCV / Enrichment / Labeling / FE / Split) is unchanged from the legacy ; only the training step (Stage 7 in legacy numbering) was replaced. The harness is therefore a drop-in replacement at the training boundary.

4. Harness Path¶

The legacy per-model autonomous trainer approach (Stages 7-9 of the legacy pipeline, see §8 Legacy Pipeline Analysis) has been replaced by a Hamilton-based training harness with plugin registries (ADR-89). This is the canonical path for every single-model and ensemble training in production since 2026-05-10 (PR #891 + #898). Promotion of harness-trained models is currently FROZEN — see §2 Current Status & Promotion Freeze.

High-level data flow¶

flowchart LR
    subgraph FTF[FTF runner / orchestrator]
        OR["CVNTrade_AutonomousOrchestrator
._create_autonomous_trainer(model_type)"]
    end

    subgraph Harness[training.harness]
        REG{{"MODEL_REGISTRY
ENSEMBLE_REGISTRY"}}
        AMT[CVNTrade_AutonomousModelTrainer
generic, parameterised by model_type]
        AET[CVNTrade_AutonomousEnsembleTrainer
generic, parameterised by ensemble_name]
        T1["train_one(model_type, datasets, hpo)
Hamilton Driver"]
        TE["train_ensemble(name, datasets, hpo)
Hamilton Driver"]
    end

    subgraph DAGs[harness.dags]
        XGD["xgboost_dag.py"]
        LGD["lightgbm_dag.py"]
        CBD["catboost_dag.py"]
        ENS1["blend_avg_dag.py"]
        ENS2["stack_logreg_dag.py"]
        ENS3["stack_xgb_meta_dag.py"]
    end

    subgraph Adapters[harness.adapters]
        XA[XGBAdapter]
        LA[LGBAdapter]
        CA[CBAdapter]
        EA[EnsembleAdapter]
    end

    OR --> REG
    REG -- single model --> AMT
    REG -- ensemble --> AET
    AMT --> T1
    AET --> TE
    T1 --> XGD & LGD & CBD
    TE --> ENS1 & ENS2 & ENS3
    XGD --> XA
    LGD --> LA
    CBD --> CA
    ENS1 & ENS2 & ENS3 --> EA
    EA -. composes .-> XA & LA & CA

Adding a new model¶

1. Drop a new file src/training/harness/dags/models/<name>_dag.py declaring the Hamilton DAG nodes (<name>_native, <name>_adapter, <name>_metrics_val, <name>_metrics_test, <name>_artifact) + the per-model HPO search space. Each numeric default MUST be read via commun.finetune.hyperparams.resolve(_MODEL, current_timeframe(), "<PARAM>") (per ADR-90 + CI grep gate G5) ; HPO range MIN/MAX/SCALE via resolve_hpo_range(_MODEL, current_timeframe(), "<PARAM>").
2. Append a register_model(...) call at the bottom of the file (or use the convention scan in harness/registry.py).
3. Seed the Console with the model's HP defaults + HPO ranges for the 5 timeframes (1M / 5M / 15M / 30M / 1H). Add the values to scripts/seed_hyperparams_console.py::LEGACY_DEFAULTS + LEGACY_HPO_RANGES, update commun.finetune.hyperparams.expected_keys() to enumerate the new (model, param) pairs, then run : python scripts/seed_hyperparams_console.py --apply against prod PG (via kubectl exec on a scheduler pod, see documentation/runbooks/break-glass-hyperparams.md for the Console-bypass pattern when needed). The parity tests tests/unit/training_harness/test_hyperparams_seeding_parity.py MUST pass before merge.
4. Done. The orchestrator picks the new model up automatically — no edits to any caller. Synthetic-pickup test : tests/unit/training_harness/test_phase4_lgb_cutover.py::TestSyntheticFourthModelPickup.

Adding a new ensemble¶

Same pattern under dags/ensembles/. The EnsembleAdapter composes N base TrainedArtifacts ; Hamilton resolves the base model dependencies and caches them within a driver call. Ensembles do NOT own hyperparameters themselves today — they reuse the per-base-model seeded values from ftf_config.base_env. If an aggregator gains tunable HPs (e.g., LogRegShrinkAggregator.l2 floor), externalize via ADR-90 with a CVN_HPO_ENS_<NAME>_<PARAM> namespace extension (currently unused — file an ADR addendum if needed).

File map (current)¶

Backtest contract¶

Every harness DAG returns a TrainedArtifact whose .adapter exposes the canonical predict_proba(X) -> (n, 2) Protocol. The backtest engine's create_model_adapter factory recognises harness adapters via duck-typing and wraps them through _HarnessAdapterShim to satisfy the legacy CVNTrade_ModelAdapter ABC. No type-sniffing in consumer code (regime trainer, backtest engine, OOS predictor).

5. Hyperparameter Resolution (ADR-90)¶

¶

Since PR #904 (CVN-N001-EE-S17, 2026-05-12) :

The resolver contract :

• Env key present → parse to typed value (per _PARAM_TYPES in commun/finetune/hyperparams.py)
• Env key missing + fallback=None → RuntimeError with canonical message pointing to ADR-90 (operator MUST seed the key)
• Env key missing + fallback=<value> → emit WARN event=hpo_fallback_applied then return the (type-coerced) fallback (transitional warning window per ADR-90 Clause 2)
• NaN / ±Inf rejected at parse time (_parse fail-fasts)
• HPO range MIN >= MAX rejected at resolve_hpo_range exit
• Fallback type-coerced through the same _parse path as env-sourced values (string "150" → int 150 for EARLY_STOPPING_ROUNDS)

Operator workflow : - Tune a value : Console UI → Config → ftf_config → edit key → save (writes audit row to ftf_config_history). No deploy needed (rule 1 of ADR-90). - Inspect coverage : Grafana dashboard cvntrade-hp-coverage (5 panels : 24h fallback count, 6h fallback %, 7d timeseries, top missing keys, training stack health). Target = 0 fallback events. - Rollback bad write : Console UI restore-from-history button NOT IMPLEMENTED today (Epic CCP wp#149 scope) ; until then, use the break-glass PG path with the gating ritual in documentation/runbooks/break-glass-hyperparams.md. - Seed a new namespace : python scripts/seed_hyperparams_console.py --apply after extending LEGACY_DEFAULTS / LEGACY_HPO_RANGES / expected_keys(). Idempotent (re-running is a no-op on already-seeded keys).

6. Known Regression — S18¶

Post-S17 canary FTF ftf_20260512_184337_fdee27_ATR0.5_1.5_H4 (2026-05-12) exposed that harness training stops shallow for all 3 model types :

Root cause is upstream of S17. Hyperparameter externalization (PR #904) is verified clean : event=hpo_fallback_applied count = 0, Optuna picks are within seeded ranges, parity tests test_hyperparams_seeding_parity.py confirm byte-for-byte legacy values. The regression is in the #891 harness migration itself — diagnostic Story scope under CVN-N001-EE-S18 plan dossier (plan_review committee session 4298520f PASSED OK strong consensus, 0 blockers, OP wp#154, OP Meeting #133).

Diagnostic plan (S18) — 6 steps, 5.25-day budget¶

Verdict tree (per ADR-79, adapted for diagnostic Story) :

• LOCK : root cause + fix < 30 LOC → open CVN-N001-EE-S19 1-week plan
• KEEP_AVAILABLE : root cause + fix non-trivial → S19 with 2-3 sprint plan + design dossier
• ABANDON : 30d no convergence OR fix > 3× revert surface → S19 reverts #891 + #896 + #899, accept ADR-89 loss, redo migration

Until S18 + S19 close, the harness FTF leaderboard is informational only ; no live trading promotion of harness-trained models is permitted (see §2).

7. Observability & Operator Workflow¶

Observability contract — Loki event schema¶

Every harness DAG emits the same event= schema, traceable in Loki via the {namespace="cvntrade"} stream :

See also¶

• ADR-89 — Training harness as plugin registry
• ADR-90 — Training hyperparameters in PG Console only (V3.2 addition)
• ADR-67 — Pluggable feature-selection registry (sister registry pattern)
• Plan dossier (harness original) — documentation/reviews/2026-05-09-cvn-n001-ee-s16-training-harness-unification-plan.md
• Plan dossier (hyperparameters externalization) — documentation/reviews/2026-05-11-cvn-n001-ee-s17-hyperparams-externalization-plan.md
• Plan dossier (shallow-training regression diagnostic, in flight) — documentation/reviews/2026-05-13-cvn-n001-ee-s18-harness-shallow-training-diagnostic-plan.md
• Break-glass runbook (ADR-90 transition window) — documentation/runbooks/break-glass-hyperparams.md
• Structurizr view — TrainingHarness (Components of Airflow → harness module)

8. Legacy Pipeline Analysis¶

Historical reference only. The current canonical implementation is described in §3-§7 above. Stages 1-6 below remain technically accurate as inputs to the harness (Stage 7 is the part replaced). Stages 7-9 are the pre-#891 trainers — deprecated since 2026-05-10. This section is retained for the bottleneck analysis it informs (§9) and for the FIX A-H historical action plan (Appendix A, §12) that motivated the harness refactor.

8.1 Pipeline Stages 1-6 (still current as harness inputs)¶

Stage 1: Data Ingestion (OHLCV)¶

Checkpoint: Raw OHLCV dataframe, no NaN, timezone-aware index.

Stage 2: Enrichment (Technical Indicators)¶

Checkpoint: Enriched dataframe with ~300 columns, ~23,500 rows (after warmup).

Stage 3: Labeling (Triple Barrier)¶

Class distribution (typical for ATR1.5_3.0_H5):

BUY (+1):   ~15-25%  ← minority class
HOLD (0):   ~40-50%  ← majority class (timeout)
SELL (-1):  ~25-35%

Checkpoint: Labeled dataframe. ~20,000 rows after NaN drop. Imbalanced toward HOLD.

Known issue: High HOLD ratio means model defaults to HOLD → low BUY recall.

Stage 4: CUSUM Pre-Filter¶

Checkpoint: Filtered dataframe. ~1,000-1,500 rows from 20,000. 95% data loss.

CRITICAL BOTTLENECK #1

Input:  20,000 labeled candles
CUSUM:  ~1,000 survive (h=3.0σ)
Output: 1,000 rows → split 70/15/15 → Train: 700, Val: 150, Test: 150

With more aggressive h (e.g. h=4.0σ from env):
Output: ~300 rows → Train: 210, Val: 45, Test: 45

Stage 5: Feature Engineering¶

Checkpoint: Feature matrix X (n_samples × n_features), label vector y.

BOTTLENECK #2 — Aggressive dimensionality reduction

300+ raw indicators → 30-50 selected features
Potential signal loss: MACD variants, stochastic slopes, volume patterns removed
Model capacity limited by feature count

Stage 6: Train/Val/Test Split¶

Checkpoint: Three datasets (X_train, y_train), (X_val, y_val), (X_test, y_test).

BOTTLENECK #3 — Tiny datasets after CUSUM

With 1,000 post-CUSUM rows:
  Train: 700 samples
  Val: 150 samples
  Test: 150 samples

With 300 post-CUSUM rows (aggressive h):
  Train: 210 samples
  Val: 45 samples
  Test: 45 samples

XGBoost on 210 training samples with 30 features = severe overfitting risk

8.2 Pipeline Stages 7-9 — Deprecated, kept for historical analysis¶

Stage 7: HPO + XGBoost Training (legacy — superseded by harness in PR #891 + #898)¶

Hyperparameter search space (from config, timeframe-dependent):

max_depth:        [3, 8]
learning_rate:    [0.01, 0.3]
n_estimators:     [100, 500]
subsample:        [0.6, 1.0]
colsample_bytree: [0.6, 1.0]
min_child_weight: [1, 10]
gamma:            [0, 5]
reg_alpha:        [0.01, 5.0]  (timeframe-dependent)
reg_lambda:       [0.5, 7.0]   (timeframe-dependent)
threshold_buy:    [0.30, 0.50]
threshold_sell:   [0.30, 0.50]

Objective function (CVN_HPO_OBJECTIVE):

Guard conditions (kill trial if):

if action_rate < 0.08 or action_rate > 0.60:  → score = 0.0
if recall_buy < 0.15:                          → score = 0.0

BOTTLENECK #4 — HPO guards constrain the solution space

action_rate [8%, 60%] forces model to predict BUY on 8-60% of samples
recall_buy > 15% forces minimum BUY detection
Combined: model must predict BUY often enough but not too often
Result: model converges to "safe middle" with f1_buy ~0.45

Class balancing (XGBoost trainer lines 900-911):

if _is_class_balancing_enabled():  # CVN_CLASS_BALANCING=1
    class_weights = compute_class_weight("balanced", classes, y_train)
    sample_weights = [class_weights[int(y)] for y in y_train]
else:
    sample_weights = ones(len(y_train))  # NO BALANCING (default)

BOTTLENECK #5 — Class balancing disabled by default

With 70% HOLD, 15% BUY, 15% SELL:
  Model learns to predict HOLD for most samples (safe bet)
  BUY precision high but recall low → f1_buy ~0.45
  Enabling balancing: 0.05-0.10 improvement expected

Early stopping (trainer line 957):

early_stopping_rounds = config.early_stopping_rounds  # default 150

Calibration (trainer lines 961-963):

if config.calibration != "none":
    self._apply_calibration(X_train, y_train, config.calibration)
# Options: "none" (default), "isotonic", "platt"

Stage 8: Model Evaluation (legacy — superseded by harness, kept for context)¶

Typical metrics (ATR1.5_3.0_H5, defi_top5):

f1_buy:        0.43-0.49     ← weak
precision_buy: 0.50-0.65     ← decent
recall_buy:    0.20-0.35     ← very low (model too conservative)
auc_buy:       0.55-0.65     ← barely above random (0.50)
action_rate:   0.10-0.30     ← low (few BUY predictions)

Stage 9: MLflow Storage (legacy — superseded by harness, kept for context)¶

9. Historical Bottlenecks¶

Historical reference only. The bottleneck analysis below was the V1 / V2 committee assessment that motivated the harness refactor (now done) and the FIX A-H action plan (Appendix A, §12). It describes the pre-#891 pipeline ; the harness path has different bottlenecks, currently under diagnosis in CVN-N001-EE-S18 (see §6).

Bottleneck Map¶

                     Raw Data (24,000 candles)
                              │
                    ┌─────────┴─────────┐
                    │  ENRICHMENT        │ → 300+ features
                    └─────────┬─────────┘   No data loss
                              │
                    ┌─────────┴─────────┐
                    │  LABELING          │ → ~20,000 labeled
                    └─────────┬─────────┘   10-20% NaN drop
                              │                              ← BOTTLENECK #6: label dropout
                    ┌─────────┴─────────┐
                    │  CUSUM FILTER      │ → ~1,000 rows     ← BOTTLENECK #1: 95% data loss
                    └─────────┬─────────┘   (h=3.0σ)
                              │
                    ┌─────────┴─────────┐
                    │  FEATURE ENGIN.    │ → 30-50 features   ← BOTTLENECK #2: signal loss
                    └─────────┬─────────┘   (adaptive cap)
                              │
                    ┌─────────┴─────────┐
                    │  SPLIT 70/15/15   │ → Train: 700
                    └─────────┬─────────┘   Val: 150          ← BOTTLENECK #3: tiny datasets
                              │             Test: 150
                    ┌─────────┴─────────┐
                    │  HPO (30 trials)  │ → f1_buy ~0.45      ← BOTTLENECK #4: guards limit
                    └─────────┬─────────┘                     ← BOTTLENECK #5: no class balance
                              │
                    ┌─────────┴─────────┐
                    │  EVALUATION        │ → 1-10 trades/fold
                    └─────────┬─────────┘   (tiny test set)
                              │
                    ┌─────────┴─────────┐
                    │  MLFLOW STORAGE    │
                    └───────────────────┘

Bottleneck Severity Ranking¶

10. Roadmap / Open Stories¶

Current in-flight + immediate-next Stories tied to this pipeline :

Promotion-gate criteria (S19 closure)¶

Recover the pre-#891 baseline AND demonstrate the harness path matches or exceeds it :

Only when ALL four gates pass on the canary FTF does the operator unfreeze harness-trained model promotion (§2). If S18 returns ABANDON, the promotion-freeze persists indefinitely and the operator pivots to revert.

11. ADR Compliance¶

12. Appendix A — Historical Action Plan (FIX A-H, V1+V2 Committee Recos)¶

Historical action plan that motivated the harness refactor. The FIX A-H items below were the V1 + V2 committee deliverables that drove the V3 plan. Some are now closed (e.g. FIX B — class balancing — integrated into the harness class_balance node ; FIX A — CUSUM decoupling — applied at inference per ADR-46). Others (FIX D trading-centric HPO, FIX F MLOps controls) remain partially open. Do NOT use this as the current action plan — the current open work is in §10. This appendix is kept for the rationale and traceability it provides.

4. Committee Verdict & Action Plan¶

V1 Committee Result: REJECTED (STARVATION, 4.0/10)¶

8 recommendations received. ALL addressed below with concrete fixes.

Action Plan — 3 Structural Fixes + 5 Improvements¶

PRIORITY 1 — CRITICAL (blocks everything)
┌─────────────────────────────────────────────────────────────────┐
│  FIX A: Decouple CUSUM from training                            │
│  FIX B: Enable class balancing by default                       │
│  FIX C: Integrate realistic cost model into HPO                 │
└─────────────────────────────────────────────────────────────────┘

PRIORITY 2 — HIGH (multiplier on Fix A/B/C)
┌─────────────────────────────────────────────────────────────────┐
│  FIX D: Trading-centric HPO objective (Sortino-based)           │
│  FIX E: Increase feature cap + regularization                   │
│  FIX F: MLOps operational controls                              │
└─────────────────────────────────────────────────────────────────┘

PRIORITY 3 — MEDIUM (validation & exploration)
┌─────────────────────────────────────────────────────────────────┐
│  FIX G: Test CUSUM/ML paradigm conflict                         │
│  FIX H: Binary classification (#517)                            │
└─────────────────────────────────────────────────────────────────┘

5. FIX A: Decouple CUSUM from Training (committee reco #1, #7)¶

Problem¶

CUSUM filters 95% of training data at h=3.0σ. Result: 20,000 candles → 1,000 → split → Train: 700 samples. XGBoost on 700 samples with 30 features = severe overfitting, unreliable metrics.

Solution: Train on ALL data, CUSUM only at inference¶

CURRENT (broken):
OHLCV → Enrichment → Label → CUSUM FILTER → FE → Split → Train
                              ▲
                              └── 95% data lost HERE

TARGET (fixed):
OHLCV → Enrichment → Label → FE → Split → Train  (CUSUM REMOVED from training)
                                                    │
                                                    ▼ at inference only:
                                            CUSUM → Inference → Filters → Trade

Implementation¶

File: src/commun/cache/components/cvntrade_autonomous_fe.py

Current (~line 125):

# Apply CUSUM before split (issue #295)
filtered_df = self._apply_cusum_before_split(enriched_df)
X_train, X_val, X_test = self._temporal_split(filtered_df)

Target:

# Train on ALL data — CUSUM only at inference (committee reco #1)
cusum_training_mode = os.environ.get("CVN_CUSUM_TRAINING_MODE", "disabled")
if cusum_training_mode in ("enabled", "relaxed_1_5", "legacy_3_0"):
    filtered_df = self._apply_cusum_before_split(enriched_df)
else:
    filtered_df = enriched_df  # NO CUSUM filtering for training
X_train, X_val, X_test = self._temporal_split(filtered_df)

New env var: CVN_CUSUM_TRAINING_MODE — canonical variants: {disabled, relaxed_1_5, legacy_3_0} (default: disabled)

FTF factor: cusum_training_mode with variants {disabled, relaxed_1_5, legacy_3_0} to A/B test the impact.

Expected Impact¶

CUSUM remains at inference¶

CUSUM is still applied at inference time (backtest candle loop, line 869). It gates which candles are processed by the ML model. The model is trained on ALL data but only makes predictions on CUSUM-validated candles.

Rationale: The model learns patterns from the full distribution but only acts during regime transitions. This resolves Hypothesis G (CUSUM/ML paradigm conflict) — the model is no longer trained on a biased subset.

Paradigm Test (committee reco #7)¶

To validate this architectural change, run 3 variants via FTF:

Compare Sortino, f1_buy, trades/fold. If cusum_disabled wins → confirm architectural fix.

6. FIX B: Enable Class Balancing by Default (committee reco #2)¶

Problem¶

CVN_CLASS_BALANCING=0 (disabled). With 70% HOLD, 15% BUY, 15% SELL, the model learns to predict HOLD for safety. BUY recall = 0.25 (misses 75% of opportunities). ADR-46 non-compliant.

Solution¶

Change default to CVN_CLASS_BALANCING=1 in BASE_ENV (ablation_matrix.py).

File: src/commun/finetune/ablation_matrix.py

BASE_ENV = {
    ...
    "CVN_CLASS_BALANCING": "1",  # Changed from "0" — ADR-46 compliance
    ...
}

Effect: compute_class_weight("balanced") → BUY weight ~3.3×, SELL weight ~1.5×, HOLD weight ~0.7×.

Expected Impact¶

Trade-off: More BUY predictions → more trades → better statistical power BUT potentially more false positives. The filters (trend, meta-label, regime) catch false positives.

7. FIX C: Realistic Cost Model in HPO (committee reco #3)¶

Problem¶

HPO optimizes precision_recall_auc which ignores transaction costs entirely. A model with f1_buy=0.50 and 30 trades at 15bps cost may have negative expectancy. HPO doesn't know.

Solution: Cost-aware HPO objective¶

New objective: sortino_net — backtest Sortino after costs.

# In hyperoptimizer, after model training:
# 1. Quick backtest on validation set with cost model
# 2. Compute net Sortino
# 3. Use as HPO score

def _compute_sortino_net(model, X_val, y_val, ohlcv_val, cost_bps=15):
    """Quick backtest for HPO scoring — net of costs."""
    predictions = model.predict(X_val)
    trades = _simulate_trades(predictions, ohlcv_val, cost_bps)
    return _sortino_ratio(trades)

Cost components: - Base fee: CVN_TRADE_FEE_BPS (default 15) - Slippage: base_bps + impact × √(size/volume) (from cost_model.py) - Funding rate: CVN_FUNDING_RATE_BPS × expected hold duration (from Binance API)

FTF factor: hpo_objective already has variants {fbeta_buy, precision_recall_auc, f1_macro}. Add sortino_net.

Implementation¶

File: src/training/XGBoost/cvntrade_XGBoost_hyperoptimizer.py

Add new objective handler:

elif objective == "sortino_net":
    from commun.config.cost_model import compute_trade_cost
    # Quick backtest on validation fold
    net_sortino = _quick_backtest_sortino(model, datasets, cost_bps=15)
    if net_sortino is None or np.isnan(net_sortino):
        return 0.0
    return max(0.0, net_sortino / 10.0)  # normalize to [0, 1] range

8. FIX D: Trading-Centric HPO & Relaxed Guards (committee reco #4)¶

Problem¶

HPO guards kill promising trials: - action_rate < 0.08 → score = 0 (too few BUY predictions) - action_rate > 0.60 → score = 0 (too many BUY predictions) - recall_buy < 0.15 → score = 0 (not enough BUY detection)

These guards constrain the solution space. With class balancing (Fix B), the model will naturally have higher action_rate.

Solution¶

Relax guards and shift to trading-centric scoring:

Current guards:

if action_rate < 0.08 or action_rate > 0.60: return 0.0
if recall_buy < 0.15: return 0.0

Target guards (relaxed):

if action_rate < 0.03 or action_rate > 0.80: return 0.0  # Much wider
if recall_buy < 0.05: return 0.0  # Minimal floor

Objective shift: From classification-centric to trading-centric:

9. FIX E: Feature Selection Strategy (committee reco #5)¶

Problem¶

Adaptive cap: min(80, max(10, n_features / 20)) → typically 30-50 features from 300+. Too aggressive — removes signal.

Solution¶

1. Increase cap: min(150, max(30, n_features / 10)) → typically 80-150 features
2. Rely on XGBoost regularization instead of pre-selection:
3. reg_alpha (L1) already in HPO range [0.01, 5.0]
4. reg_lambda (L2) already in HPO range [0.5, 7.0]
5. max_depth [3, 8] limits tree complexity
6. XGBoost handles feature selection internally via feature_importance
7. OOS feature selection (strict): Feature importance computed ONLY on training fold, applied to val/test

Implementation: - src/commun/cache/components/cvntrade_autonomous_fe.py: Change cap formula - BASE_ENV: CVN_MAX_FEATURES=0 (0 = auto) → keep, but change auto formula - FTF factor n_features already tests {top_30, top_50, top_100, full}

Expected Impact¶

More features → more signal → BUT only if regularization prevents overfitting. With Fix A (20× more training data), overfitting risk is dramatically reduced. 700 samples + 150 features = overfit. 14,000 samples + 150 features = healthy ratio (93:1).

10. FIX F: MLOps Operational Controls (committee reco #6)¶

Kill-Switch¶

Live Observability¶

Drift Detection¶

Staged Rollout¶

11. FIX G: CUSUM/ML Paradigm Test (committee reco #7)¶

Experimental Design¶

3-arm study via FTF cusum_training_mode factor:

Success Criteria¶

Expected Outcome¶

Arm A should dominate: - 20× more training data → better model generalization - CUSUM at inference → same signal quality (only regime-change trades) - f1_buy improvement: +0.10-0.20 (from 0.45 to 0.55-0.65)

If Arm A does NOT dominate → fundamental signal-to-noise issue (Hypothesis F). Requires alternative approaches (different features, different model architecture, different timeframe).

12. FIX H: Binary Classification (committee reco #8)¶

Already implemented as FTF factor classification_mode (issue #517):

Rationale: 3-class wastes model capacity learning SELL patterns we don't act on (LdP pipeline only opens long positions). Binary focuses 100% of capacity on the BUY decision.

13. Appendix B — V2 Stricter Recos (historical)¶

13. Walk-Forward Leakage Prevention (V2 reco #5)¶

Every component in the pipeline must be strictly in-sample for each walk-forward fold:

Fold k timeline:
├── [                   Train window                   ]──[Purge]──[  Val  ]──[Emb]──[  Test  ]
│    FE fitted here only                                   10 bars   15%        5 bars   15%
│    CUSUM sigma calibrated here (lagged 500 bars)
│    Feature selection on train only
│    Class weights computed on train only
│    Thresholds optimized on val only
│    Test: NEVER seen during training or tuning

Per-Component Leakage Audit¶

Verification Protocol¶

Before each sprint: 1. Code audit: grep -n "X_test\|y_test" src/training/ — no test data in training 2. FE pipeline: verify pipeline.fit(X_train) not pipeline.fit(X_full) 3. Feature selection: verify importance computed on (X_train, y_train) only 4. CUSUM sigma: verify sigma_window_end < train_start - purge_bars

14. Uncertainty Quantification (V2 reco #8)¶

All key metrics must be reported with confidence intervals to quantify statistical reliability.

Methods¶

Reporting Standard¶

Every metric in FTF reports and Grafana dashboards includes:

Sortino: 1.75 [1.43, 2.10] (95% CI, n=51 runs)

If CI includes 0 → metric is not statistically distinguishable from zero. Flag in report.

If CI width > mean → metric is unreliable. Flag as "WIDE CI — insufficient data".

BH Correction for Multiple Comparisons¶

Already implemented: ablation_stats.py:benjamini_hochberg() applied to all pairwise variant comparisons. Controls false discovery rate at 5%.

15. Minimum Trade Count Gates (V2 reco #7)¶

Problem¶

With 1-10 trades per fold, ALL metrics are unreliable. A single trade can swing Sortino from 0 to 100.

Trade count thresholds (committed)¶

Implementation¶

1. FTF report: Already flags underpowered variants. Strengthen: exclude < 30 trades from variant ranking.
2. Grafana: n_trades >= 3 filter (outlier protection). Add panel showing trade count per variant.
3. Promotion gate: Model cannot be promoted to Production in MLflow if any evaluation fold has < 30 trades.

Expected post-Fix-A trade counts¶

With CUSUM removed from training (Fix A): - Training: 14,000 samples (20× current) - Model sees more examples → better action_rate calibration - Expected: 30-100 trades per fold (vs 1-10 current)

16. Operational Controls & Runbooks (V2 reco #6)¶

Kill-Switch Hierarchy¶

Configuration Audit Trail¶

Every config change is traceable: - Git: Helm values changes → PR → CodeRabbit → merge (audit via git log) - MLflow: Model promotion → registry (version, timestamp, user) - Committee: Design decisions → committee/sessions/*.json (ADR-52) - Airflow: DAG runs → execution logs (who triggered, when, params)

Rollback Procedures¶

Runbooks (to create in documentation/RUNBOOKS/)¶

14. Appendix C — Market Hypothesis & Edge (V2 reco #9)¶

What edge are we exploiting?¶

The system targets short-term mean-reversion at regime transition points in DeFi altcoin markets.

Thesis: When a DeFi altcoin's volatility regime shifts (detected by CUSUM), the initial price reaction overshoots. The ML model identifies overshooting candles where the probability of a profitable mean-reversion trade (TP hit before SL) exceeds the breakeven probability after costs.

Why this edge should exist¶

1. DeFi altcoin microstructure: Lower liquidity than BTC/ETH → larger overreactions to regime shifts → mean-reversion opportunity.
2. Retail-dominated order flow: DeFi tokens have higher retail participation → predictable behavioral patterns (panic selling on vol spikes, FOMO buying on breakouts).
3. CUSUM as regime detector: CUSUM identifies structural breaks in volatility — these are exactly the moments when market participants overreact and create temporary mispricings.
4. Triple barrier as target: The ATR-based SL/TP captures the mean-reversion: TP is set at the "fair" reversion level, SL limits the cost of being wrong.

Why it hasn't worked yet¶

The edge exists in principle but the pipeline has structural issues that prevent the model from capturing it: - CUSUM during training eliminates 95% of examples → model can't learn the full distribution of regime transitions - Class imbalance → model defaults to HOLD instead of predicting BUY at transition points - HPO objective optimizes classification accuracy, not trading profit → model maximizes F1 but generates few trades - Feature cap removes market microstructure features (volume patterns, order flow proxies) that capture the edge

How we validate the edge¶

1. Statistical test: Sortino of our model > Sortino of random entry at same CUSUM-filtered candles (ADR-29 baseline)
2. Economic test: Net expectancy per trade > 0 after costs at 30 bps
3. Robustness test: Edge persists across 5 cryptos, 5 folds, 3 cost scenarios
4. Decay test: Edge doesn't degrade significantly over recent folds (concept drift monitoring)

Success criteria for edge validation¶

15. Appendix D — Pre-Harness Implementation Roadmap (historical)¶

Sprint 1: Critical Fixes — DEADLINE: 1 week¶

Sprint 2: HPO & Features — DEADLINE: 1 week¶

Sprint 3: Validation & Operations — DEADLINE: 1 week¶

Success Gate — Hard Criteria¶

After Sprint 3, ALL must be met to proceed to production:

If gate not met: Escalate to architectural review. Options: 1. Alternative models (LightGBM, transformer, LSTM) 2. Alternative features (order book, funding rate dynamics, cross-asset) 3. Alternative timeframes (1h, 4h) 4. Alternative strategy (momentum instead of mean-reversion)

16. Files Reference¶