Claude-skill-registry-data markov-regime-features

Debugging constant Markov regime features in RL observations - when HMM probabilities show uniform values instead of dynamic regime estimates

install

source · Clone the upstream repo

git clone https://github.com/majiayu000/claude-skill-registry-data

Claude Code · Install into ~/.claude/skills/

T=$(mktemp -d) && git clone --depth=1 https://github.com/majiayu000/claude-skill-registry-data "$T" && mkdir -p ~/.claude/skills && cp -r "$T/data/markov-regime-features" ~/.claude/skills/majiayu000-claude-skill-registry-data-markov-regime-features && rm -rf "$T"

manifest: data/markov-regime-features/SKILL.md

source content

markov-regime-features - Research Notes

Experiment Overview

Item	Details
Date	2025-12-13
Goal	Debug why Markov features (indices 19-24) appeared constant in observation heatmaps while other features showed high variability
Environment	Python 3.10, PyTorch 2.0+, alpaca_trading package
Status	Success

Context

When visualizing RL observation features as a heatmap, the Markov regime features (volatility and trend probabilities) appeared as constant horizontal bands with values ~0.33, while all other features showed expected variability over time. This made the Markov features useless for regime detection.

Root Cause

The standalone

build_inference_observation()

function defaults to uniform priors when

markov_vol_probs

and

markov_trend_probs

are not provided:

# From inference_obs_builder.py:391-401
if include_markov:
    if markov_vol_probs is not None:
        obs[:, feat_idx:feat_idx+3] = markov_vol_probs
    else:
        obs[:, feat_idx:feat_idx+3] = [0.33, 0.34, 0.33]  # Uniform prior

The 6 Markov features are:

Index	Feature	Description
19	vol_prob_low	P(low volatility regime)
20	vol_prob_medium	P(medium volatility regime)
21	vol_prob_high	P(high volatility regime)
22	trend_prob_down	P(downtrend regime)
23	trend_prob_neutral	P(neutral regime)
24	trend_prob_up	P(uptrend regime)

Verified Workflow

Solution: Use InferenceObservationBuilder class

The

InferenceObservationBuilder

class maintains Markov state and updates probabilities based on price history:

from alpaca_trading.gpu.inference_obs_builder import InferenceObservationBuilder

# Create stateful builder with GPU Markov system
obs_builder = InferenceObservationBuilder(window=100, use_gpu_markov=True)

# Build observation - Markov states are updated from price history
obs = obs_builder.build(
    prices=prices,
    high=high,
    low=low,
)

# Access current regime estimates
print(f"Volatility: {obs_builder.vol_probs}")  # e.g., [0.15, 0.60, 0.25]
print(f"Trend: {obs_builder.trend_probs}")      # e.g., [0.20, 0.30, 0.50]

Visualizing Regime Evolution

To see how regimes change over time, build observations at multiple points:

eval_points = list(range(150, n_bars, 10))
vol_history = {'low': [], 'med': [], 'high': []}

for end_idx in eval_points:
    obs_builder.build(prices=prices[:end_idx], high=high[:end_idx], low=low[:end_idx])
    vol_history['low'].append(obs_builder.vol_probs[0])
    vol_history['med'].append(obs_builder.vol_probs[1])
    vol_history['high'].append(obs_builder.vol_probs[2])

plt.plot(eval_points, vol_history['high'], 'r-', label='P(High Vol)')
plt.plot(eval_points, vol_history['low'], 'g-', label='P(Low Vol)')

Failed Attempts (Critical)

Attempt Why it Failed Lesson Learned

Using

build_inference_observation()

directly

No Markov state passed, defaults to uniform [0.33, 0.34, 0.33]

Use

InferenceObservationBuilder

class for stateful Markov tracking

Setting

include_markov=False

Removes features entirely, breaks model compatibility

Keep Markov features, just ensure proper initialization

Manual Markov probability calculation

Complex HMM implementation needed

Use

GPUMarkovSystem

via

use_gpu_markov=True

Final Parameters

# Correct usage for dynamic Markov features
obs_builder = InferenceObservationBuilder(
    window=100,           # Must match training window
    use_gpu_markov=True   # Enable GPU-accelerated HMM
)

# For live trading - maintain single builder instance across bars
# For backtesting - create new builder per symbol, maintain across timesteps

Key Insights

The standalone function is for single-shot inference where Markov state is passed externally
The builder class is for stateful inference where Markov state evolves with price data
GPU Markov system uses last 20 returns to estimate regime probabilities
Uniform priors [0.33, 0.34, 0.33] indicate Markov system not receiving updates
In heatmaps, constant horizontal bands at indices 19-24 are a red flag for this issue

References

alpaca_trading/gpu/inference_obs_builder.py

- Feature building implementation

```
alpaca_trading/gpu/harmonized_gpu.py
```
- GPUMarkovSystem class
```
notebooks/develop_branch_testing.ipynb
```
- Visualization examples