# CHEAT_SHEET — Principal SWE, Roblox Database

## 1. Troubleshooting method (Mechanism–Measure–Move)

1. **Mechanism** — state the internals path first (B-tree descent, MVCC visibility check, WAL replay, planner cost choice, compaction level). One sentence, no guessing.
2. **Measure** — pull the first data before theorizing (see §5). Quote the number, not the vibe.
3. **Move** — commit to pick + trap ruled out + stay-fixed (runbook, automation, upstream issue). Say which depth tag you are in (Debug / Recognize / Do-not-claim).

Say aloud: "Mechanism, Measure, Move — path first, number second, fix that stays fixed."

## 2. PG internals ladder (shallow → deep, say where you stop)

1. **SQL/plan surface (Debug):** EXPLAIN ANALYZE, buffers/timing, plan shapes. Trap: reading cost numbers as time.
2. **Access methods (Debug):** heap + B-tree descent, index-only vs bitmap, HOT updates. Trap: assuming an index always wins (write amplification, bloat).
3. **MVCC (Debug):** snapshots, xmin/xmax/cmax, visibility map,_hint bits, vacuum/AUTOVACUUM, wraparound. Trap: confusing bloat (space) with wraparound (correctness emergency).
4. **WAL + recovery (Debug):** LSN chain, full-page writes, checkpoints, archive/PITR. Trap: claiming zero RPO on async replicas.
5. **Replication (Debug):** streaming/walsender, sync vs quorum vs async, pg_stat_replication lag, timelines, switchover vs failover. Trap: failover without fencing = split-brain.
6. **Planner/optimizer (Recognize):** pg_statistic, selectivity, join costing, parallel paths, partition pruning. Trap: `enable_*` hacks instead of ANALYZE/schema fix. Depth: narrate method, do not claim executor C++ work.
7. **Buffer/lock/concurrency (Recognize):** shared_buffers, clock sweep, LWLock waits, lock queues. Trap: raising buffers without measuring hit ratio + eviction pressure.
8. **Storage engine C++ (Do-not-claim):** executor nodes, AM implementations, WAL-insert paths. Say: "I read the structure and describe the path; I have not implemented at this layer — closing with targeted reps."

## 3. Raft/Paxos quorum lines (memorize)

- Quorum = 2f+1 nodes tolerate f failures: 3 nodes → quorum 2, tolerate 1; 5 nodes → quorum 3, tolerate 2.
- Leader election: candidate needs majority votes; terms prevent stale leaders.
- Log replication: entry committed once on majority; leader applies + notifies followers via commit index.
- Safety lines: "A minority partition cannot commit writes." / "Two leaders in one term is impossible with correct terms + majority."
- Liveness: minority loss keeps serving; majority loss stops writes until recovery.
- Split-brain trap: failover without fencing/STONITH risks two writers. Always name the fence.
- Roblox mapping: multi-region replication needs the same quorum + fencing story as RAC voting disk and Data Guard FSFO observer — say that bridge aloud.

## 4. K8s-for-databases method (control-plane answer shape)

1. **State:** StatefulSet + PVC per Pod, storage class for IOPS/latency SLO, Pod identity sticks to storage.
2. **Safety:** PodDisruptionBudgets, readiness gates (lag-bounded before serving), liveness separate from readiness.
3. **Moves:** drain → fence old writer → promote → verify lag/RPO → cut traffic → rollback ready. Zero-downtime schema/tenant moves = online change + dual-read/verify + cutover.
4. **Ops:** operator reconciles desired vs actual (scaling, healing, certs, backups); autoscaling on real signals (lag, p99, disk), not CPU alone.
5. **Observability:** per-tenant QPS/latency/error/lag dashboards; alerts on replication lag, disk, p99, failed moves.
- Trap lines: "Stateless rollout habits kill stateful data — never treat a DB Pod like a web Pod." / "Nomad container-GC history is why fencing + durable PVCs get their own paragraph."
- Depth: Debug the runbook/method; Recognize operator internals; Do-not-claim operator/framework C++ or Go implementation.

## 5. Tail-latency drill (first data + narration)

- Read p50/p99/p999 + histogram, never the mean. `pgbench -c N -j M -T 60 --latency-limit` + latency histogram.
- First data: `EXPLAIN (ANALYZE, BUFFERS, TIMING)` for query tail; `pg_stat_activity` + wait events for lock/IO; OS: iostat/vmstat + `perf top` / eBPF read narration (Recognize: describe what the trace would prove, do not claim kernel authorship).
- Non-query p99 causes: checkpoints/full-page writes, autovacuum, compaction (LSM), GC pauses, noisy neighbor/CPU steal, network retransmits, replica lag on read-your-write.
- Mitigations: spread checkpoints, tune autovacuum/vacuum cost, connection pooling, admission control, QoS/cgroups, isolate tenants, cache hot path, async where RPO allows.
- Say: "p99 went from Xms to Yms as measured by Z, by doing W" — never quote a fix without the before/after numbers.

## 6. Decision quick-matrix — Need / Pick / Why + Trap

| Need | Pick | Why + Trap |
|------|------|------------|
| OLTP correctness, MVCC readers | Postgres heap + B-tree | Snapshot isolation without read locks. TRAP: bloat + vacuum debt |
| Covering/partial lookups | Composite/covering/partial index | Index-only scans cut heap fetches. TRAP: write amplification + unused indexes |
| Zero-RPO regional failover | Sync streaming replication | Every commit on 2 nodes. TRAP: latency cost + failover still needs fencing |
| Read scale, RPO>0 OK | Async replicas | Cheap scale. TRAP: lag reads stale; no zero-RPO claim |
| Write-heavy KV ingest | RocksDB/LSM | Sequential-write friendly. TRAP: compaction debt → read tail |
| Multi-region consensus | Raft (2f+1) | Majority commit = safety. TRAP: minority writes rejected by design |
| Elastic multi-tenant PG | Sharded PG + control plane | Shard key + isolation + moves. TRAP: cross-shard joins/transactions |

## 7. STAR story index (Situation 2–3 sent / Task "I owned" / Action longest / Result X-as-measured-by-Y-by-doing-Z + stay-fixed)

| Principle | Story | Metric to quote |
|----|-------|-----------------|
| Ownership | Sev/outage or perf escalation owned end to end | MTTR + runbook cut next MTTR |
| Deep internals | Root-cause others missed via pg_stat/plan/WAL evidence | Diagnostic time down (e.g. hours→minutes) via data |
| Bias for Action | 24h mitigation, full fix next cycle | Time-to-mitigate + ship date |
| Customer obsession | Angry-customer/stakeholder turnaround | CSAT / renewal / explicit kudos |
| Highest standards | Review/tool/best-practice others adopted | Repeat incidents down, defects stay fixed |
| Influence/mentorship | Unblocked team, design review, mentoring | Delivery unblocked across teams |

Mechanics: internals depth in EVERY technical round, not just deep-dive. "I" not "we". Data + recency. 2 stories per principle minimum. Map each to Debug/Recognize honestly.

## 8. Intro spine — Jesus, 23-yr Oracle/TSE (90 sec, Roblox San Mateo)

> "I'm Jesus — 23 years solving migration and performance problems on production Oracle: RAC, Data Guard, GoldenGate, AWR-driven tuning, and partnering with Engineering on product fixes.
> I do the same job this posting describes: build and harden a multi-tenant database platform at elastic scale — engine features, control-plane moves with zero downtime, and tail-latency work backed by measurement.
> My method is Mechanism-Measure-Move: name the internals path, pull the first number, and leave a fix that stays fixed through runbooks and automation.
> I'm targeting Principal Software Engineer, Roblox Database in San Mateo because correctness-plus-scale for millions of players is exactly the problem I've spent my career on."

Gap line (honest): "My gap is C++ engine implementation versus Oracle/DBA internals — I read engine code for method and narrate the path precisely, and where the role needs implementation depth I'll say Recognize versus Debug out loud. Closing it with the labs in this 12-day plan."
Standing fact: Exceeds-Expectations-level performer since 2018 (mirror from prior kit — update title/year to current record before quoting).