Gen5 CF Rollout Fixes + Self-Play Iteration 1

# Gen5 CF Rollout Fixes + Self-Play Iteration 1

## Date: 2026-07-22

## Critical Bug Fixes

### 1. CF Rollout Fold EV Bug (CATASTROPHIC)
- **Location**: `counterfactual.rs:248` `simulate_rollout()`
- **Bug**: When opponent folds, returned `hero_invests + pot_bb` instead of `pot_bb`
- **Impact**: For a 75 BB all-in with 5 BB pot, EV was 80 instead of 5
- **Result**: Model learned all-ins were insanely profitable (AllIn "best" action 46.7% of time, avg EV 9.24)
- **Fix**: `return pot_bb;`

### 2. Opponent Raise Pot Calculation
- **Bug**: Missing hero's extra call in pot after opponent raises
- **Fix**: `current_pot += raise_total + extra;`

### 3. Multi-Street Call Pot
- **Bug**: Missing hero's call in pot during multi-street betting
- **Fix**: Added `current_pot += opp_bet;` after hero calls

### 4. predict_q() Legal Mask Bug
- **Location**: `network.rs:166`
- **Bug**: Argmax over ALL 11 actions, not just legal ones
- **Fix**: Filter by `legal_mask` before argmax

## CF Rollout Improvements

### Diverse Opponent Archetypes
5 opponent profiles randomly selected per rollout:
| Archetype | fold_base | raise_freq | bet_freq |
|-----------|-----------|------------|----------|
| nit       | 0.65      | 0.08       | 0.30     |
| tag       | 0.45      | 0.18       | 0.50     |
| lag       | 0.28      | 0.35       | 0.65     |
| station   | 0.10      | 0.04       | 0.35     |
| maniac    | 0.15      | 0.50       | 0.75     |

### Anti-Cheating
- Fold decisions based on opponent's CURRENT-BOARD hand strength (not showdown)
- `hand_strength()` normalizes HandValue::as_u32() to [0, 1]
- Multi-street continuation: opponent bets for value (hs>0.45) or bluffs (hs<0.25)

## Model Results

| Model | Loss/Corr | Harrington | vs Gen3 (BB/100) |
|-------|-----------|------------|-------------------|
| CF v6 (buggy EVs) | 0.07 | 49/53 | -321 |
| CF v7 (fixed EVs, unmasked) | 1.25 | 49/53 | -209 |
| CF v7 (aggressive masking) | 1.25 | 49/53 | **-20.5** |
| Q-reg v1 (real outcomes, masked) | corr=0.605 | - | -47.2 |

### Aggressive Action Masking
At stack > 25 BB, mask: BetPot(6), Bet2x(7), Raise25x(8), RaisePot(9), AllIn(10)
Only allow: Fold, Check, Call, Bet1/3, Bet1/2, Bet2/3
This prevents catastrophic over-betting from CF model biases.

## Remaining CF Bias
Even with fixes, CF data still overvalues aggressive actions:
- AllIn avg EV: 9.24 (best 46.7% of time) — fold prob too high for big bets
- Root cause: simplified opponent model doesn't condition calling ranges on bet size
- Real opponents who call big bets have strong hands; model doesn't capture this

## Self-Play Iteration 1 (Running)
- **Config**: `gen5_selfplay.toml` — CF v7 model plays postflop with temp=0.3
- **Opponents**: Gen3, Gen2, chump, flock, nit, tag, lag, station
- **Data**: 500K hands, recording transitions with actual hand outcomes
- **Goal**: Generate unbiased data from model's own play for next training iteration

## Key Insight
CF rollouts have fundamental limitations for direct play:
1. Opponent model too simple (doesn't condition on bet sizing)
2. Single-opponent rollouts (no multiway modeling)
3. No proper calling range analysis

Self-play with real outcomes is the path forward. Starting from -20 BB/100 (with masking), each iteration should improve as the model learns from actual mistakes.

## Files Modified
- `counterfactual.rs`: Opponent profiles, hand_strength(), fold_probability(), simulate_rollout() rewrite, 3 EV bugs fixed
- `network.rs`: predict_q() legal mask fix
- `rl_strategy.rs`: Aggressive action masking at deep stacks
- `gen5_play.toml`: Points at CF v7 model
- `gen5_selfplay.toml`: New self-play config with exploration
- `scripts/gen5_selfplay_collect.sh`: Self-play data collection script
- `scripts/gen5_collect_g48.sh`: Added virtual opponents to field

## Models Kept
- `gen5_cf_v7.safetensors` — CF ranking model (fixed EVs, best for play)
- `gen5_qreg_v1.safetensors` — Q-reg model (real outcomes)
- `gen5_cf_v6.safetensors` — CF model (buggy EVs, for reference)


id: 7791a2daf5e54e8a8318404903663281
parent_id: 5a06903f
created_time: 2026-07-22T15:00:49.134Z
updated_time: 2026-07-22T15:00:49.134Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 1784732449134
user_created_time: 2026-07-22T15:00:49.134Z
user_updated_time: 2026-07-22T15:00:49.134Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
share_id: 
conflict_original_id: 
master_key_id: 
user_data: 
deleted_time: 0
is_locked: 0
extracted_resource_ids: 
type_: 1