Development Environment & Server Topology # Development Environment & Server Topology > Canonical reference for **all** development projects. Defines which machine does what, the toolchain, the repo/workspace layout, and the standard dev → test → prod flow. Every project should consult this before deciding where to build, test, and deploy. > Created 2026-08-02. Last updated 2026-08-02. --- ## 1. Machine Inventory | Name | Hardware | OS | Primary Role | Verified facts | |---|---|---|---|---| | **workhorse** (basement desk) | AMD Ryzen 9 5950X (16C / 32T), 128 GiB RAM, RTX 5060 Ti 16 GB; `/` 457 GB SSD, `/home` 3.6 TB (2.2 TB free) | Ubuntu 26.04 LTS (Studio) | **Development** | cargo 1.95.0, Node v22.22.1, PostgreSQL installed | | **laptop** | Windows 11, RTX 3060 (laptop) | Windows 11 | Work / secondary dev | Kilo + JetBrains installable on demand | | **basement server** | Dual Xeon; ample RAM & disk | FreeBSD | **Jenkins CI + staging VM + file server** | LAN-only; Jenkins CI; can host **bhyve** VMs | | **Hetzner box** | Decent (not a powerhouse) | Debian Linux | **Git host (Forgejo) + Production + Codeberg mirror** | Public-facing; Forgejo (public + private repos); Caddy (TLS); planned self-hosted **Codeberg** mirror | ### Virtualization available - **bhyve** on the FreeBSD basement server → Debian staging VM (mirrors prod OS). - **VirtualBox** on workhorse / laptop → throwaway guests if needed. --- ## 2. Role Assignment (the model) ``` workhorse (Ubuntu) basement (FreeBSD) Hetzner (Debian) ================= ================ ================ CODE + LOCAL RUN → Jenkins CI → PRODUCTION + GIT RustRover / WebStorm bhyve: Debian staging Forgejo (primary) local PG (native) artifact + file server bare systemd services Kilo + JetBrains + Caddy (TLS) ``` - **Develop** on **workhorse** — coding (Kilo + JetBrains IDEs), local PostgreSQL (native Ubuntu package), fast incremental builds (32 threads). - **Test / stage** on the **basement server** — Jenkins drives CI; a **bhyve Debian VM** mirrors prod; file server + artifact store. - **Produce / host** on the **Hetzner box** — **Forgejo** (primary git host, public + private repos), production game services (bare systemd + Caddy TLS), planned **Codeberg** mirror for redundancy. - **laptop** — work machine and occasional mobile/secondary dev. --- ## 3. Toolchain | Concern | Tool | Where | What it does | |---|---|---|---| | AI coding | **Kilo** (Kilocode) — z.ai coding plan, primarily **GLM 5.1**, sometimes **GLM 5.2** | Any machine; primary: workhorse | AI-assisted coding in the terminal | | IDEs | **JetBrains All Products** (RustRover, WebStorm, IntelliJ, …) | workhorse (primary); laptop on demand | Code editing, debugging, refactoring | | Source control (primary) | **Forgejo** (self-hosted) | Hetzner (Debian, public) | Self-hosted git platform: repo hosting over SSH/HTTPS, web UI, issues, PRs, wiki. Supports **public and private repos**. | | Source control (mirror) | **Codeberg** (self-hosted instance, planned) | Hetzner (Debian, public) | Backup mirror of all Forgejo repos; off-site redundancy. | | CI / CD | **Jenkins** | basement server (FreeBSD, LAN-only) | Build, test, cross-compile, deploy to staging/prod | | Reverse proxy / TLS | **Caddy** | Hetzner (prod) | Public HTTPS, auto Let's Encrypt certs | | Deployment | **Bare systemd services** | Hetzner (prod); staging VM | Each service (PG, Redis, app) as a native systemd unit | | VM hypervisor | **bhyve** (basement) | basement server | Debian staging VM to mirror prod OS | | Rust toolchain | cargo / rustup (1.95.0) | workhorse | Build backend | | Node toolchain | Node v22.22.1 | workhorse | Build frontend | > **No Docker.** Services run as native systemd units (Linux) or plain services (FreeBSD). No containers. --- ## 4. Services Glossary ### Forgejo Self-hosted git platform — the same open-source software that [Codeberg.org](https://codeberg.org) runs. Provides: - **Git hosting** — push/pull repos over SSH (`git@hetzner:repo.git`) or HTTPS - **Web UI** — browse code, diffs, commits in a browser (publicly accessible) - **Issues & Pull Requests** — issue tracking, code review, merge workflows - **Wiki** — per-repo documentation - **Actions** — CI runner (Forgejo Actions, Gitea-compatible; an alternative to Jenkins if desired) - **Public & private repos** — visibility set per repo. Private repos require authentication; public repos are browseable by anyone. ### Codeberg (self-hosted mirror) A second instance of the Codeberg/Forgejo software, running alongside Forgejo on the same Hetzner box or as a backup target. Mirrors all repos from Forgejo for redundancy. Planned, not yet standing. ### Redis In-memory data store used as **cache + pub/sub**. Session cache, leaderboards (sorted sets), WebSocket pub/sub. **Deferrable for MVP** — the Rust process can hold session/leaderboard data in memory at low player counts. PostgreSQL handles all persistent data regardless. ### PostgreSQL Relational database. All persistent game state. Runs natively on workhorse for local dev; runs on Hetzner for production. ### Jenkins CI/CD server on the basement box (LAN-only). Watches Forgejo for pushes (deploy key + webhook), runs lint/test/build pipelines, cross-compiles the Rust binary for Linux x86_64, and deploys to staging and prod. --- ## 5. Workspace & Repository Layout (workhorse) ### Convention: `~/Development///` ``` ~/Development/ # canonical workspace root ├── skara-brae/ # multi-repo project: parent is a plain folder, NOT a git repo │ ├── skara-brae-server/ # open in RustRover │ ├── skara-brae-client/ # open in WebStorm │ └── skara-brae-common/ # shared; 3rd window or "Attach" ├── poker/ # (existing) │ ├── old_bot/ │ └── references/ ├── super-marvin/ # single-repo projects are fine too └── racecraft/ ``` **Why `~/Development//`:** multi-repo projects need siblings under one parent for path deps. Per-IDE default folders break this. One root = one mental model. JetBrains opens any folder. The `~/Development/` root was chosen over `~/dev/` because it already holds `poker/` and matches the Jenkins note path convention. ### Current legacy state (being migrated) ``` ~/RustroverProjects/ keyfob-station, rusty-marvin, rusty-telemetry, rusty_emu, super_marvin, top_drives ~/WebstormProjects/ joplin-plugin-mcp, racecraft, super-marvin-userscripts ~/IdeaProjects/ ac-telemetry-plugin ``` ### Symlinks (for IDE discoverability) ```bash git clone git@hetzner:skara-brae/skara-brae-server.git ~/Development/skara-brae/skara-brae-server ln -s ~/Development/skara-brae/skara-brae-server ~/RustroverProjects/skara-brae-server ``` Open the **real path** (`~/Development/...`) in the IDE so path deps resolve. Symlinks are for IDE "recent projects" only. ### Skara Brae — workspace status (2026-08-02) All three repos initialized, committed, and symlinks created: | Repo | Real path | IDE symlink | Git status | |---|---|---|---| | skara-brae-server | `~/Development/skara-brae/skara-brae-server/` | `~/RustroverProjects/skara-brae-server` | Committed; `cargo check` passes; health endpoint at `127.0.0.1:3000` | | skara-brae-client | `~/Development/skara-brae/skara-brae-client/` | `~/WebstormProjects/skara-brae-client` | Committed; Vue 3 + Vite + Pinia scaffolded | | skara-brae-common | `~/Development/skara-brae/skara-brae-common/` | — | Committed; `data/items/weapons.json`, `rust/` crate `sb-common`, `ts/` package `@skara-brae/common` | **Known issue — cross-repo Cargo path:** Cargo workspaces cannot resolve `path = "../skara-brae-common/rust"` (above workspace root). Workaround: absolute path `/home/jan/Development/skara-brae/skara-brae-common/rust` in each crate's `Cargo.toml`. Do **not** declare `sb-common` in `[workspace.dependencies]`. Long-term fix: git submodules. See [Tech Stack & Architecture § Cross-language dependency path workaround](joplin://980c3eb587294e4383474b94988f2f88). **No remote configured yet.** Next: create bare repos on the basement server (192.168.1.2) or push to Codeberg as temporary remote. When Hetzner Forgejo is stood up, migrate there. --- ## 6. Standard Dev → Test → Prod Flow ### Develop — workhorse - Local PostgreSQL (native Ubuntu package). Backend via `cargo run`, frontend via Vite. - 32 threads → fast `cargo` builds; 128 GiB RAM → room for IDEs + Kilo + PG. - Commit → push to **Forgejo** on Hetzner (internet SSH). ### Test / Stage — basement server (FreeBSD) - **Jenkins** watches Forgejo (webhook). Pipeline: lint → test → cross-compile Linux x86_64 → transfer binary. - Staging target: a **bhyve Debian VM** running systemd services (PG, app binary, Caddy). Accessible on LAN. - Basement box is also the file server and artifact store. ### Produce — Hetzner box (Debian) - Public entry point. **Caddy** (TLS) fronts Forgejo (git), the game API, and the static frontend. - Bare **systemd** services: PostgreSQL, Redis (when needed), Skara Brae binary. - Jenkins deploys via SSH: copy binary → `systemctl restart skara-brae`. - **Sizing caveat:** watch combined RAM (Forgejo + Codeberg + PG + app). Be ready to upsize or split. --- ## 7. Git Hosting ``` Hetzner (Forgejo, public) ├── public repos (browseable by anyone — open-source projects) └── private repos (auth required — work repos, game server, etc.) │ └── mirror → Codeberg instance (same box, backup) ``` - **Primary git host:** Forgejo on **Hetzner**. - **Public repos:** published projects (open-source tools, game client). - **Private repos:** work-in-progress, proprietary code, game server internals. - **Codeberg mirror** (planned): repo redundancy on the same box. - **Jenkins** on the basement box connects via deploy key + webhook. - The old [GIT / Jenkins Workflow](joplin://b7728728858f4107a4de7971e2b1ddd9) note is superseded by Forgejo. --- ## 8. Networking, TLS, Backups - **DNS:** `git.` (Forgejo), `api.`, `play.` → Hetzner A/AAAA. Caddy auto-issues Let's Encrypt. - **Git transport:** `git@hetzner` (Forgejo, internet SSH, primary). HTTPS also available for public repos. - **Basement server:** LAN-only, not publicly reachable. Jenkins connects outbound to Forgejo. - **Backups:** - Forgejo → nightly dump → Hetzner Storage Box or basement file server (pull, not push). - Codeberg mirror → Hetzner Storage Box. - PostgreSQL (prod on Hetzner) → `pg_dump` → off-site. - workhorse dev: repos recoverable from Forgejo; local DB is ephemeral. - **Restore test:** exercise one restore of each before relying on it. --- ## 9. Project-Specific Application - **Skara Brae** — first project to adopt this topology. See [Project Plan](joplin://088737f31c514f1da21cc42c0ab6acc1) and [Tech Stack & Architecture](joplin://980c3eb587294e4383474b94988f2f88). - Future projects inherit: code on workhorse, CI/staging on basement, git+prod on Hetzner. --- ## 10. Open Decisions (cross-project) - [ ] Migrate legacy IDE-folder repos into `~/Development/` now, or symlink-on-touch. - [ ] Stand up a **bhyve Debian staging VM** on the basement box. - [ ] Confirm Hetzner box headroom for Forgejo + Codeberg mirror + game workload; set a RAM threshold for splitting. - [ ] Backups: pick off-site targets per service and run a restore test. - [ ] Codeberg mirror: same Hetzner box or a different host? Needed at all if Forgejo backups are solid? id: bbc49a3bbada4fcdb8fedc66abf5014e parent_id: cd0501fe1cc24c93bc6bc3e78fb324fd created_time: 2026-08-02T08:06:50.225Z updated_time: 2026-08-02T11:32:15.313Z is_conflict: 0 latitude: 0.00000000 longitude: 0.00000000 altitude: 0.0000 author: source_url: is_todo: 0 todo_due: 0 todo_completed: 0 source: joplin-desktop source_application: net.cozic.joplin-desktop application_data: order: 1785658010225 user_created_time: 2026-08-02T08:06:50.225Z user_updated_time: 2026-08-02T11:32:15.313Z encryption_cipher_text: encryption_applied: 0 markup_language: 1 is_shared: 0 share_id: conflict_original_id: master_key_id: user_data: deleted_time: 0 is_locked: 0 extracted_resource_ids: type_: 1