ATFT Training Skill

Mission

• Launch production-grade training for the Graph Attention Network forecaster with correct dataset/version parity.
• Tune hyper-parameters (LR, batch size, horizons, latent dims) exploiting 80GB GPU headroom.
• Safely resume, stop, or monitor long-running jobs and record experiment metadata.

Engagement Triggers

• Requests to “train”, “fine-tune”, “HP optimize”, “resume training”, or “monitor training logs”.
• Need to validate new dataset compatibility with model code.
• Investigations into training stalls, divergence, or GPU under-utilization.

Preflight Safety Checks

1. Dataset freshness:

   ls -lh output/ml_dataset_latest_full.parquet

   then

   python scripts/utils/dataset_guard.py --assert-recency 72

   .
2. Environment health:

   tools/project-health-check.sh --section training

   .
3. GPU allocation:

   nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv

   (target >60% util, <76GB used baseline).
4. Git hygiene:

   git status --short

   ensure working tree state is understood (avoid accidental overrides during long runs).

Training Playbooks

1. Production Optimized Training (default 120 epochs)

1. make train-optimized DATASET=output/ml_dataset_latest_full.parquet

   — compiles TorchInductor + FlashAttention2.

2. make train-monitor

   — tails

   _logs/training/train-optimized.log

   .

3. make train-status

   — polls background process; ensure ETA < 7h.
4. Post-run validation:

   • python scripts/eval/aggregate_metrics.py runs/latest

     — compute Sharpe, RankIC, hit ratios.
   • Update

     results/latest_training_summary.md

     .

2. Quick Validation / Smoke

1. make train-quick EPOCHS=3

   — run in foreground.

2. python scripts/smoke_test.py --max-epochs 1 --subset 512

   for additional regression guard.

3. pytest tests/integration/test_training_loop.py::test_forward_backward

   if suspicious gradients.

3. Safe Mode / Debug

1. make train-safe

   — disables compile, single-worker dataloading.

2. make train-stop

   if hung jobs detected (consult

   _logs/training/pids/

   ).

3. python scripts/integrated_ml_training_pipeline.py --profile --epochs 2 --no-compile

   — capture flamegraph to

   benchmark_output/

   .

4. Hyper-Parameter Exploration

1. Ensure

   mlflow

   backend running if required (

   make mlflow-up

   ).

2. make hpo-run HPO_TRIALS=24 HPO_STUDY=atft_prod_lr_sched

   — uses Optuna integration.

3. make hpo-status

   — track trial completions.
4. Promote winning config →

   configs/training/atft_prod.yaml

   and document in

   EXPERIMENT_STATUS.md

   .

Monitoring & Telemetry

• Training logs:

  _logs/training/*.log

  (includes gradient norms, learning rate schedule, GPU temp).
• Metrics JSONL:

  runs/<timestamp>/metrics.jsonl

  .
• Checkpoint artifacts:

  models/checkpoints/<timestamp>/epoch_###.pt

  .
• GPU telemetry:

  watch -n 30 nvidia-smi

  or

  python tools/gpu_monitor.py --pid $(cat _logs/training/pids/train.pid)

  .

Failure Handling

• NaN loss → run

  make train-safe

  with

  FP32=1

  , inspect

  runs/<ts>/nan_batches.json

  .
• Slow dataloading → regenerate dataset with

  make dataset-gpu GRAPH_WINDOW=90

  or enable PyTorch compile caching.
• OOM → set

  GRADIENT_ACCUMULATION_STEPS=2

  or reduce

  BATCH_SIZE

  ; confirm memory fragments via

  python tools/gpu_memory_report.py

  .
• Divergent metrics → verify

  configs/training/schedule.yaml

  ; run

  pytest tests/unit/test_loss_functions.py

  .

Codex Collaboration

• Invoke

  ./tools/codex.sh --max "Design a new learning rate policy for ATFT-GAT-FAN"

  when novel optimizer or architecture strategy is required.
• Use

  codex exec --model gpt-5-codex "Analyze runs/<timestamp>/metrics.jsonl and suggest fixes"

  for automated postmortems.
• Share Codex-discovered tuning insights in

  results/training_runs/

  and update config files/documents accordingly.

Post-Training Handoff

• Persist summary in

  results/training_runs/<timestamp>.md

  noting dataset hash and commit SHA.
• Push model weights to

  models/artifacts/

  with naming

  gatfan_<date>_Sharpe<score>.pt

  .
• Notify research team via

  docs/research/changelog.md

  .