id: 4928523c8cb84045af1d2f2dde00a7f0
parent_id: 64972b502b8c482a9e953bdbd8a00f3d
item_type: 1
item_id: c7a39e16770542a08004fd73e971ca1f
item_updated_time: 1784717433257
title_diff: "[]"
body_diff: "[{\"diffs\":[[0,\"-07-\"],[-1,\"17\"],[1,\"22\"],[0,\".** \"],[-1,\"Imitation learning with G48 teacher. Data collection in progress.\"],[1,\"Counterfactual rollout training. CF data collection v2 in progress with fixed opponent fold model.\\\n\\\n## Architecture (2026-07-22)\\\n\\\n```\\\nRangeNet v8 → opponent_range (169-dim)\\\n                ↓\\\n    Action Equity NN ← hero_features (459-dim = 290 hero + 169 range)\\\n                ↓\\\n    Dueling Q-values: Q(s,a) = V(s) + A(s,a) - mean(A over legal)\\\n                ↓\\\n    Pick argmax Q (with AllIn/Bet2x masked at >25 BB)\\\n```\"],[0,\"\\\n\\\n##\"]],\"start1\":57,\"start2\":57,\"length1\":79,\"length2\":434},{\"diffs\":[[0,\"Vector (\"],[-1,\"290\"],[1,\"459\"],[0,\" dimensi\"]],\"start1\":500,\"start2\":500,\"length1\":19,\"length2\":19},{\"diffs\":[[0,\"mensions\"],[1,\" = 290 hero + 169 range\"],[0,\")\\\n\\\n| Cat\"]],\"start1\":514,\"start2\":514,\"length1\":16,\"length2\":39},{\"diffs\":[[0,\" 5 |\"],[-1,\" hs_rw, ppot_rw, npot_rw, nutpot_rw, win_prob_rw —\"],[0,\" exa\"]],\"start1\":708,\"start2\":708,\"length1\":58,\"length2\":8},{\"diffs\":[[0,\" 5 |\"],[-1,\" hs_mc, ppot_mc, npot_mc, nutpot_mc, win_prob_mc —\"],[0,\" 100\"]],\"start1\":799,\"start2\":799,\"length1\":58,\"length2\":8},{\"diffs\":[[0,\"|\\\n| \"],[-1,\"**Total** | **290** | |\\\n\\\n## Action Space (11 discrete → 3+5 multi-head)\\\n\\\nUnderlying: `Fold, Check, Call, BetThird(0.33), BetHalf(0.50), BetTwoThird(0.67), BetPot(1.0), Bet2x(2.0), Raise25x(2.5×), RaisePot(pot), AllIn`\\\n\\\nMulti-head mapping: action_type (passive/call/aggressive) + sizing_class (third/half/2third/pot+/allin).\"],[1,\"RangeNet prediction | 169 | Opponent range probabilities (169 hand classes) |\\\n| **Total** | **459** | |\"],[0,\"\\\n\\\n##\"],[1,\" CF\"],[0,\" Tra\"]],\"start1\":1269,\"start2\":1269,\"length1\":335,\"length2\":118},{\"diffs\":[[0,\"ne\\\n\\\n\"],[-1,\"### Phase 1: Imitation Learning (GPU, CUDA)\\\n1. Collect transitions from teacher bot in simula\"],[1,\"1. Collect G48 teacher transitions (also records CF rollout EVs)\\\n2. CF rollout: for each postflop decision, sample 10 opponent hands from RangeNet, run 5 rollouts per legal ac\"],[0,\"tion\\\n\"],[-1,\"2\"],[1,\"3\"],[0,\". Train \"],[-1,\"multi-head MLP with class-weighted cross-entropy + sizing-weighted MSE\\\n3. Feature normalization (per-feature standardization saved to `.norm.json`)\\\n\\\n**Latest model**: `postflop_v13` — trained on G48 teacher data.\\\n\\\n### Data Collection Statu\"],[1,\"dueling Q-network on per-action EV vectors (weighted MSE over legal actions)\\\n4. At inference: argmax Q-value over legal actions (AllIn/Bet2x masked at >25 BB)\\\n\\\n### Key Bug Fixe\"],[0,\"s (2\"]],\"start1\":1399,\"start2\":1399,\"length1\":354,\"length2\":373},{\"diffs\":[[0,\"-07-\"],[-1,\"17\"],[1,\"22\"],[0,\")\\\n- **\"],[-1,\"Short-handed (3-6 handed)**: ✅ COMPLETE — 24.1M transitions from 10M hands (39GB)\\\n- **Full-ring (7-8 handed)**: 🔄 RUNNING — ~3.5M of ~20M transitions (7 opponent types: chump, flock, gen1, gen2, gen3, loose, tight)\\\n- **Teacher**: G48 (`teacher_postflop = \\\"formula\\\"`, 10K MC samples)\\\n- **Config**: `configs/bots/gen5_collect_g48.toml`\\\n\\\n### Phase 2: RL Fine-Tuning (REINFORCE) — PLANNED\\\n- Frozen backbone + RL-adapted heads only\\\n- Mixed table: 1 RL + mixed opponents\\\n- Quick eval every 50 iterations, deep eval every 200\\\n\\\n### Phase 3: Live Fine-Tuning (future)\\\n- Collect Torn session data, fine-tune on real opponent distributions\\\n\\\n## Next Steps\\\n1. Wait for full-ring collection → merge all transitions\\\n2. Train `postflop_v14` on combined ~44M transitions\\\n3. Evaluate v14 vs G48 in simulation\\\n4. PPO fine-tuning if imitation plateaus\\\n5. Deploy via `gen5_rl.toml`\\\n\\\n## Key Technical Details\\\n\\\n### Candle LayerNorm Bug (FIXED)\\\nCandle-nn 0.8.4's `ops::layer_norm()` has no backward pass. Replaced with manual layer_norm in `multi_head.rs`, `network.rs`, `range_net.rs`.\\\n\\\n### Equity Feature Sources\\\n- **Raw equity** (features 0-4): `HandPotential::compute()` — exact enumeration, uniform opponent\\\n- **Range-weighted equity** (features 20-24): `compute_vs_range()` — exact enumeration weighted by tightest opponent's HandRange\\\n- **Multiway MC equity** (features 25-29): `compute_vs_range_multiway_shared()` — 1000 samples\\\n\\\n## Key Files\\\n\\\n| File | Content |\\\n|---|---|\\\n| `holdem_bots/src/gen5/features.rs` | 290-dim feature extraction + range-weighted + multiway equity |\\\n| `holdem_bots/src/gen5/multi_head.rs` | Multi-head model + trainer (LayerNorm fix, REINFORCE) |\\\n| `holdem_bots/src/gen5/rl_strategy.rs` | RL strategy + `compute_range_equity()` + `teacher_postflop` config |\\\n| `holdem_bots/src/gen5/network.rs` | Candle MLP policy + `best_device()` GPU selection |\\\n| `holdem_bots/src/gen5/recorder.rs` | JSONL transition recorder |\\\n| `holdem_bots/src/gen5/actions.rs` | Action space + masking + `action_to_heads()` |\\\n| `holdem_bots/src/bin/train_gen5_multi.rs` | Imitation trainer (GPU/CUDA) |\\\n| `holdem_bots/src/bin/train_gen5_rl.rs` | RL trainer (REINFORCE) |\\\n| `scripts/gen5_collect_g48.sh` | Short-handed collection (3-6 handed) |\\\n| `scripts/gen5_collect_g48_fullring.sh` | Full-ring collection (7-8 handed) |\\\n| `models/postflop_v13.safetensors` | Current imitation model |\\\n| `models/rl_current.safetensors` | RL model (latest) |\\\n\\\n## Design Decisions\\\n\\\n- **NN takes equity + opponent stats as input** (not range prediction NN) — pragmatic, builds on Gen 4\\\n- **Unified network** for preflop + postflop (preflop zeroes board features)\\\n- **Opponent features as input** — cold start = all-zeros, network adapts as stats accumulate\\\n- **Hybrid training**: offline imitation → offline RL → live fine-tuning (live play too slow for weight updates)\\\n- **Curriculum plan**: Beat Gen 1 → 2 → 3 → 4 → self-play convergence (Nash approximation)\\\n\"],[1,\"CF collection not producing data**: binary was outdated, rebuilding fixed it\\\n- **CF collection script missing cf_path**: sed substitution didn't include cf_data.jsonl path\\\n- **CF training exits early**: `load_jsonl` ran on CF data first (different JSON schema), got 0 samples, exited. Restructured `main()` to branch CF mode before loading transitions.\\\n- **CF features not normalized**: normalization computed from transition data, never applied to CF samples. Fixed: CF mode computes its own normalization.\\\n- **Q-value model inference used softmax**: `predict()` applied softmax to dueling advantages. Added `predict_q()` that computes proper Q-values via `V + A - mean(A)`.\\\n- **CF opponent fold model was fixed/unconditional**: AllIn got 65% fold regardless of hand strength → AllIn was argmax 43.9% of time. Fixed: fold probability now depends on showdown hand comparison (behind = high fold, ahead = low fold).\\\n\\\n### CF v1 Results (before fold model fix)\\\n- 447K CF samples from 500K hands (8 tables, 3-6 handed)\\\n- Training: loss 733 → 41.7 over 30 epochs\\\n- Evaluation: **-849 BB/100** (catastrophic — AllIn every hand due to biased EVs)\\\n\\\n### CF v2 (with fixed fold model) — IN PROGRESS\\\n- Collecting 500K hands with hand-strength-aware fold probabilities\\\n- Expected: fold prob 50-75% when opponent behind, 10-15% when ahead\\\n\\\n## Key Files\\\n\\\n| File | Content |\\\n|---|---|\\\n| `holdem_bots/src/gen5/counterfactual.rs` | CF rollout engine + opponent fold model |\\\n| `holdem_bots/src/gen5/rl_strategy.rs` | RL strategy + CF collection + Q-value inference |\\\n| `holdem_bots/src/gen5/network.rs` | Dueling DQN + `predict_q()` + CF trainer |\\\n| `holdem_bots/src/bin/train_gen5.rs` | Unified trainer (imitation / Q-reg / CF modes) |\\\n| `configs/bots/gen5_play.toml` | Play config with CF model (`q_value_model = true`) |\\\n| `configs/bots/gen5_collect_g48.toml` | Collection config with `cf_mode = true` |\\\n| `scripts/gen5_collect_g48.sh` | Collection script (transitions + range + CF data) |\\\n\\\n## Design Decisions\\\n\\\n- **Dueling DQN**: Q(s,a) = V(s) + A(s,a) - mean(A over legal). Stabilizes training.\\\n- **CF rollouts over Q-reg**: Q-reg only gets gradient for taken action (correlation ceiling ~0.595). CF gives per-action EV targets for ALL legal actions.\\\n- **AllIn/Bet2x masked at deep stacks**: filtered from training and masked at inference when stack > 25 BB.\\\n- **459-dim input**: 290 hero features + 169 RangeNet range prediction (+5.3% correlation vs 290-only).\\\n- **RangeNet v8**: 57-dim, 23.4M samples, top-10=21.5% (deployed in G49 live).\"]],\"start1\":1775,\"start2\":1775,\"length1\":2940,\"length2\":2540}]"
metadata_diff: {"new":{},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-07-22T10:57:35.742Z
created_time: 2026-07-22T10:57:35.742Z
is_locked: 0
type_: 13