id: 4e07160539b247b9a94010f27bf913d2
parent_id: 4928523c8cb84045af1d2f2dde00a7f0
item_type: 1
item_id: c7a39e16770542a08004fd73e971ca1f
item_updated_time: 1785522860458
title_diff: "[{\"diffs\":[[0,\" — R\"],[-1,\"einforcement Learning\"],[1,\"angeNet\"],[0,\" (St\"]],\"start1\":5,\"start2\":5,\"length1\":29,\"length2\":15}]"
body_diff: "[{\"diffs\":[[0,\" — R\"],[-1,\"einforcement Learning\\\n\\\n> **Current state,\"],[1,\"angeNet (State of the Art)\\\n\\\n> **Declared\"],[0,\" 202\"]],\"start1\":7,\"start2\":7,\"length1\":49,\"length2\":48},{\"diffs\":[[0,\"-07-\"],[-1,\"22\"],[1,\"31\"],[0,\".** \"],[-1,\"Counterfactual rollou\"],[1,\"Gen 5 = G48 formula + RangeNet (neural opponen\"],[0,\"t \"],[-1,\"t\"],[0,\"ra\"],[-1,\"ining. CF data collection v2 in progress with fixed opponent fold model.\\\n\\\n## Architecture (2026-07-22)\\\n\\\n```\\\nRangeNet v8 → opponent_range (169-dim)\\\n                ↓\\\n    Action Equity NN ← hero_features (459-dim = 290 hero + 169 range)\\\n                ↓\\\n    Dueling Q-values: Q(s,a) = V(s) + A(s,a) - mean(A over leg\"],[1,\"nge prediction). Pure NN (alpha=1.0), no heuristic blending.\\\n\\\n## What Changed from Gen 4\\\n\\\nGen 4 (G48) uses **heuristic ranges** built from observer stats (VPIP/PFR/AF → hand range narrowing). Gen 5 replaces that with a **neural network** (RangeNet) that predicts the opponent's hand distribution directly from observable features.\\\n\\\nThe formula (G47 preflop + G48 postflop) and MC equity engine are unchanged. Only the range source changes:\\\n\\\n```\\\nGen 4:  observer stats → heuristic range builder → MC equity → formula decision\\\nGen 5:  observable features → RangeNet NN → 169-dim softmax → MC equity → formula decision\\\n```\\\n\\\n## RangeNet Architecture\\\n\\\n```\\\nInput(81) → Linear(512) → ReLU → LayerNorm\\\n         → Linear(512) → ReLU → LayerNorm + residual\\\n         → Linear(512) → ReLU → LayerNorm + residu\"],[0,\"al\"],[-1,\")\"],[0,\"\\\n   \"]],\"start1\":56,\"start2\":56,\"length1\":358,\"length2\":863},{\"diffs\":[[0,\"    \"],[-1,\"  \"],[1,\"→ Linear(256) → ReLU\\\n\"],[0,\"     \"],[-1,\"↓\\\n\"],[0,\"    \"],[-1,\"Pick argmax Q (with AllIn/Bet2x masked at >25 BB)\\\n```\"],[1,\"→ Linear(169) → softmax\\\n```\\\n\\\nOutput: 169-dim softmax over hand types (13 pocket pairs, 78 suited, 78 offsuit).\\\nEach type probability is divided among its combos (2-pass: count available combos per type after dead-card exclusion, then assign `type_prob / count` per combo).\"],[0,\"\\\n\\\n##\"]],\"start1\":921,\"start2\":921,\"length1\":74,\"length2\":310},{\"diffs\":[[0,\"Vector (\"],[-1,\"459\"],[1,\"81\"],[0,\" dimensi\"]],\"start1\":1240,\"start2\":1240,\"length1\":19,\"length2\":18},{\"diffs\":[[0,\"s = \"],[-1,\"290 hero + 169 range\"],[1,\"49 base + 32 action history)\\\n\\\n### Base Features (49\"],[0,\")\\\n\"],[-1,\"\\\n\"],[0,\"| Ca\"]],\"start1\":1260,\"start2\":1260,\"length1\":31,\"length2\":61},{\"diffs\":[[0,\"|\\\n| \"],[-1,\"Card/Equity | 20 | hs, ppot, npot, nutpot, rpot + hand category (9 one-hot) + draws (6) |\\\n| Range-weighted equity | 5 | exact enumeration vs tightest opponent, weighted by HandRange |\\\n| Multiway MC equity | 5 | 1000-sample MC vs all opponents\"],[1,\"Player EMA stats | 11 | vpip, pfr, af, fold_to_bet, bet/raise/call/cbet/threebet/wtsd/call_raise_freq |\\\n| Confidence | 1 | hands_observed / 50 |\\\n| Table stats | 2 | table_vpip, table_pfr |\\\n| Position | 3 | normalized pos, is_sb, is_bb |\\\n| Game context | 5 | ln(pot_bb), ln(stack_bb), num_active, num_seats, street\"],[0,\" |\\\n|\"]],\"start1\":1358,\"start2\":1358,\"length1\":250,\"length2\":321},{\"diffs\":[[0,\"e | \"],[-1,\"10 | board_count, wetness, suit_count, connectivity, high_card |\\\n| Betting/Situation | 25 | pot_bb, to_call_bb, spr, stack_bb + position (9) + street (4) + counts |\\\n| Hero st\"],[1,\"4 | high_rank, paired, monotone, connected |\\\n| Board cards | 10 | 5 ranks + 5 suits |\\\n| Max sizing per street | 4 | pot-fraction capped at 3× |\\\n| In-hand action aggreg\"],[0,\"ate\"],[1,\"s\"],[0,\" | \"],[-1,\"5 | wagered_bb, is_aggressor, is_pfr_aggressor, facing_bet |\\\n| Opponent models (×3) | 60 | 20 fields each from Gen 4 PlayerStats |\\\n|\"],[1,\"9 | aggressive/passive counts per street + total_wagered_bb |\\\n\\\n### Rich\"],[0,\" Action \"],[-1,\"h\"],[1,\"H\"],[0,\"istory (\"],[-1,\"×8) | 160 | Per-action: player_offset, street, action_type, amount_bb, pot_after_bb |\\\n| RangeNet prediction | 169 | Opponent range probabilities (169 hand classes) |\\\n| **Total** | **459** | |\\\n\\\n## CF Training Pipeline\\\n\\\n1. Collect G48 teacher transitions (also records CF rollout EVs)\\\n2. CF rollout: for each postflop decision, sample 10 opponent hands from RangeNet, run 5 rollouts per legal action\\\n3. Train dueling Q-network on per-action EV vectors (weighted MSE over legal actions)\\\n4. At inference: argmax Q-value over legal actions (AllIn/Bet2x masked at >25 BB)\\\n\\\n### Key Bug Fixes (2026-07-22)\\\n- **CF collection not producing data**: binary was outdated, rebuilding fixed it\\\n- **CF collection script missing cf_path**: sed substitution didn't include cf_data.jsonl path\\\n- **CF training exits early**: `load_jsonl` ran on CF data first (different JSON schema), got 0 samples, exited. Restruc\"],[1,\"32 = 8 per street × 4 streets)\\\n| Feature | Description |\\\n|---|---|\\\n| num_raises | 0-3 |\\\n| num_calls | 0-3 |\\\n| total_wagered_bb | log normalized |\\\n| max_bet_frac | largest bet as pot fraction |\\\n| faced_bet | did player face aggression before acting? |\\\n| first_action | action code |\\\n| last_action | action code |\\\n| aggression_ratio | raises / (raises + calls) |\\\n\\\nShared function `compute_rich_action_history()` in `range_recorder.rs` — used by both training and inference to guarantee consistency.\\\n\\\n## Pure NN Decision (alpha=1.0)\\\n\\\nAlpha blending: `final_range = alpha × NN_range + (1-alpha) × heuristic_range`\\\n\\\n- **Gen 5 (production)**: `alpha = 1.0` (pure NN, heuristic output discarded)\\\n- Configurable via TOML: `range_alpha_min`, `range_alpha_max` in `Gen48Config`\\\n- `blend_alpha()` in `range_predictor.rs` interpolates by hands observed (min at 10 hands, max at 50)\\\n\\\n### Why Pure NN?\\\n- Sweep: pure NN (+138 BB/100 seed 42) > blend 0.3-0.7 (+80.5) > heuristic only (-0.5)\\\n- NN is better even at cold start (0 hands): default stats + game context + action history is richer input than heuristic VPIP/PFR defaults\\\n- Combo-count fix was critical: `probs_to_hand_range()` was inflating offsuit 3× vs suited. Fixed → \"],[0,\"tur\"],[1,\"n\"],[0,\"ed \"],[-1,\"`main()` to branch CF mode before loading transitions.\\\n- **CF features not normalized**: normalization computed from transition data, never applied to CF samples. Fixed: CF mode computes its own normalization.\\\n- **Q-value model inference used softmax**: `predict()` applied softmax to dueling advantages. Added `predict_q()` that computes proper Q-values via `V + A - mean(A)`.\\\n- **CF opponent fold model was fixed/unconditional**: AllIn got 65% fold regardless of hand strength → AllIn was argmax 43.9% of time. Fixed: fold probability now depends on showdown hand comparison (behind = high fold, ahead = low fold).\\\n\\\n### CF v1 Results (before fold model fix)\\\n- 447K CF samples from 500K hands (8 tables, 3-6 handed)\\\n- Training: loss 733 → 41.7 over 30 epochs\\\n- Evaluation: **-849 BB/100** (catastrophic — AllIn every hand due to biased EVs)\\\n\\\n### CF v2 (with fixed fold model) — IN PROGRESS\\\n- Collecting 500K hands with hand-strength-aware fold probabilities\\\n- Expected: fold prob 50-75% when opponent behind, 10-15% when ahead\\\n\\\n## Key Files\\\n\\\n| File | Content |\\\n|---|---|\\\n| `holdem_bots/src/gen5/counterfactual.rs` | CF rollout engine + opponent fold model |\\\n| `holdem_bots/src/gen5/rl_strategy.rs` | RL strategy + CF collection + Q-value inference |\\\n| `holdem_bots/src/gen5/network.rs` | Dueling DQN + `predict_q()` + CF trainer |\\\n| `holdem_bots/src/bin/train_gen5.rs` | Unified trainer (imitation / Q-reg / CF modes) |\\\n| `configs/bots/gen5_play.toml` | Play config with CF model (`q_value_model = true`) |\\\n| `configs/bots/gen5_collect_g48.toml` | Collection config with `cf_mode = true` |\\\n| `scripts/gen5_collect_g48.sh` | Collection script (transitions + range + CF data) |\\\n\\\n## Design Decisions\\\n\\\n- **Dueling DQN**: Q(s,a) = V(s) + A(s,a) - mean(A over legal). Stabilizes training.\\\n- **CF rollouts over Q-reg**: Q-reg only gets gradient for taken action (correlation ceiling ~0.595). CF gives per-action EV targets for ALL legal actions.\\\n- **AllIn/Bet2x masked at deep stacks**: filtered from training and masked at inference when stack > 25 BB.\\\n- **459-dim input**: 290 hero features + 169 RangeNet range prediction (+5.3% correlation vs 290-only).\\\n- **RangeNet v8**: 57-dim, 23.4M samples, top-10=21.5% (deployed in G49 live).\"],[1,\"blend from -16.4 to +15.1 BB/100\\\n\\\n## Range Model History\\\n\\\n| Version | Dims | Samples | Top-1 | Top-10 | Notes |\\\n|---------|------|---------|-------|--------|-------|\\\n| v10-v13 | 57 | 2-4M | ~3% | ~19% | Hit ceiling with coarse 8-dim action history |\\\n| v14_b1 | 81 | 1M | 2.86% | 19.0% | Accuracy unchanged, but combo-count fix improved downstream equity |\\\n\\\n**Accuracy ceiling**: ~19% top-10 across all versions. Bottleneck is inherent poker unpredictability (opponent hands are hard to predict from betting patterns), not features or data quantity.\\\n\\\n## Training Pipeline\\\n\\\n1. **Collection**: `equity_v3/` — 8 tables, formula-guided play, range-only recording (no transitions). ~6K samples/min.\\\n2. **Rolling training**: `range_rolling_train.sh` auto-trains at each 1M-sample milestone.\\\n3. **Trainer CLI**: `train_gen5_range range_merged.jsonl 30 512 1e-4 models/range_v14_bX.safetensors`\\\n\\\n### Current State\\\n- `models/range_v14_b1.safetensors`: 81-dim, 1M samples\\\n- `equity_v3/`: ~1.7M / 4M target samples\\\n- Batch 2 trains at 2M, batch 3 at 3M, final at 4M\\\n\\\n## Validation Status (2026-07-31)\\\n\\\n| Gate | Result |\\\n|---|---|\\\n| Harrington | 17/18 pass (Harr 4-5: raises turn with 99, book says check — judgment call) |\\\n| Self-play audit | ✓ passed |\\\n| Gen2 | ✓ +0.5 BB/100 |\\\n| Gen3 | ✗ -0.1 BB/100 (marginal) |\\\n| G48 | running |\\\n| TAG/Nit | pending |\\\n\\\n## Configs\\\n\\\n| Config | Purpose |\\\n|---|---|\\\n| `cash_nl_g49.toml` | Gen 5 sim config (alpha=1.0, 10K MC budget) |\\\n| `cash_nl_g49_live.toml` | Gen 5 live config (40K MC budget, profiles) |\\\n| `gen5_equity_collect.toml` | Range data collection (formula-only teacher) |\\\n\\\n## Key Files\\\n\\\n| File | Content |\\\n|---|---|\\\n| `holdem_bots/src/gen5/range_net.rs` | RangeNet network + `RangeTrainer` |\\\n| `holdem_bots/src/gen5/range_recorder.rs` | Feature extraction (81 dims), `compute_rich_action_history()` |\\\n| `holdem_bots/src/gen4/range_predictor.rs` | `RangePredictor`, `probs_to_hand_range()` (combo-count fix), `blend_alpha()` |\\\n| `holdem_bots/src/gen4/formula_postflop.rs` | G48 strategy + `Gen48Config` (alpha params, model path) |\\\n| `holdem_bots/src/gen4/mod.rs` | `build_ranges_and_equity()` — blends NN + heuristic (alpha=1.0) |\\\n| `holdem_bots/src/gen5/rl_strategy.rs` | RL strategy (teacher mode, range recording) |\\\n| `holdem_bots/src/bin/train_gen5_range.rs` | Range model training CLI |\\\n| `scripts/gen5_equity_collect.sh` | 8-table range collection |\\\n| `scripts/range_rolling_train.sh` | Rolling training at 1M milestones |\\\n\\\n## Dropped Gen 5 Approaches (historical)\\\n\\\nThese were attempted and abandoned before settling on RangeNet:\\\n\\\n- **Direct-action DQN** (imitation/Q-reg on 290-dim features): 71.5% accuracy, catastrophic mistakes (All-In with weak draws). Action selection without opponent range info is fundamentally blind.\\\n- **Counterfactual rollout training**: CF rollouts gave per-action EV targets for dueling DQN. Fold model was biased → AllIn 43.9% of time → -849 BB/100. Even after fold-model fix, the approach was superseded by formula + range prediction.\\\n- **Equity NN** (`model_infer_equity`): Pure value betting works in 6-max (+121 BB/100) but fails in 9-max (-148 BB/100). Redundant with G48's MC computation.\\\n- **Combined inference** (`model_infer_combined`): Single-street EV formula too aggressive. -157 to -2140 BB/100.\\\n- **FoldEquityNet**: Overfit, insufficient data.\\\n\\\n## Where This Goes Next\\\n\\\n- **Gen 5 tuning**: faster NN inference (sim bottleneck), better training data, formula parameter sweep with NN ranges\\\n- **Gen 6 (future)**: RL self-play for continuous improvement — use Gen 5 formula as teacher, RL fine-tunes against self-play data\"]],\"start1\":1692,\"start2\":1692,\"length1\":3463,\"length2\":5094}]"
metadata_diff: {"new":{},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-07-31T18:39:43.056Z
created_time: 2026-07-31T18:39:43.056Z
is_locked: 0
type_: 13