Gen 6 — RL Self-Play (State of the Art)

# Gen 6 — RL Self-Play (Future)

> **Research phase, not yet implemented.** Gen 6 = Gen 5 + RL for continuous improvement through self-play.

## Goal

Use reinforcement learning to improve the strategy beyond what the G48 formula + RangeNet can achieve alone. The formula provides a strong teacher baseline; RL fine-tunes to find exploitative adjustments that the formula's fixed thresholds cannot discover.

## Foundation

Gen 5 (G48 formula + RangeNet pure NN) provides:
- A strong, stable baseline strategy
- Neural opponent range prediction (169-dim softmax)
- MC equity computation
- Observer-based opponent modelling (EMA stats)

Gen 6 will add a learning loop on top of this.

## Potential Approaches

### 1. Policy Gradient on Self-Play
- Use Gen 5 formula as initial policy
- Generate self-play transitions (Gen 5 vs Gen 5, or Gen 5 vs mixed opponents)
- Policy gradient (REINFORCE / PPO) to adjust action selection
- Risk: may diverge from stable formula behavior

### 2. Q-Regression on Self-Play Outcomes
- Collect state-action-reward transitions from self-play
- Train Q-network to predict expected value of each action
- Formula-guided exploration: formula selects most actions, Q-network occasionally overrides
- Previous attempt (Gen 5 DQN) failed due to action selection without range info — Gen 6 would use range features

### 3. Opponent Counter-Exploitation
- Self-play generates opponent models
- Detect exploitable patterns (over-folds, over-calls, tilt)
- Adjust strategy parameters dynamically
- Closest to what the formula already does via observer EMA, but with learned adjustments

## Lessons from Dropped Approaches

Gen 6 must avoid the pitfalls that killed earlier RL attempts:
- **Direct-action NN without range input** → catastrophic mistakes (All-In with weak draws)
- **Single-street EV formulas** → too aggressive, don't account for future streets
- **Biased fold models in CF rollouts** → exploit inflated, AllIn every hand
- **Action selection is fundamentally blind without opponent range** → any RL approach MUST use range features (Gen 5's RangeNet output)

## Prerequisites

1. **Gen 5 must prove stable in live play** — sanity check passing, positive BB/100 over meaningful sample
2. **Faster NN inference** — current sim bottleneck makes large-scale self-play slow
3. **Sufficient compute** — self-play + training needs many hands

## Status

Not started. Gen 5 tuning and live validation come first.

id: 78ad1a9e50b840559ff15f38d1a4d324
parent_id: 42f3b0b681634f9ea2487bf4a561388a
created_time: 2026-07-31T18:34:44.136Z
updated_time: 2026-07-31T18:34:44.136Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 1785522884136
user_created_time: 2026-07-31T18:34:44.136Z
user_updated_time: 2026-07-31T18:34:44.136Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
share_id: 
conflict_original_id: 
master_key_id: 
user_data: 
deleted_time: 0
is_locked: 0
extracted_resource_ids: 
type_: 1