Roblox-Postgres-Study

1 · Bloat is back after last night's VACUUM

A 40M-row marketplace table shows seq-scan growth a day after VACUUM FULL. Dead-tuple count climbs again by morning. What mechanism explains the regrowth, what do you measure first, and what stay-fixed change stops the cycle?

Answer: repeated HOT-ineligible updates (indexed columns churned) create dead versions faster than autovacuum reclaims. First data: pg_stat_user_tables n_dead_tup + last_autovacuum vs last_autoanalyze, plus pgstattuple bloat ratio. Stay-fixed: cut indexed-column churn or add covering design + tune autovacuum scale factors for this table. Trap: confusing bloat (space/scan cost) with wraparound (correctness emergency).

2 · Replica ten seconds behind at peak

Read-your-write complaints spike; pg_stat_replication shows replay_lag 10s but flush_lag near zero. Is this network, disk, or apply load — and what proves it?

Answer: replay (apply) lag, not network or flush — the standby receives fine but cannot apply fast enough under peak write load. Prove with sent/flush/replay LSN gaps; fix = spread batch load, check long transactions holding replay, consider sync only where RPO demands it. Trap: claiming zero RPO on an async replica.

3 · Plan flipped overnight, nothing changed

Same query, same schema, nested loop today where hash join won yesterday. Stats job was skipped during a deploy freeze. State the mechanism, the one command, and what you re-verify.

Answer: stale pg_statistic misestimate picks the wrong join shape. Command: ANALYZE the tables (never enable_* hacks in production). Re-verify: EXPLAIN ANALYZE row-estimate vs actual + p95 for that query hash. Trap: pinning the plan instead of fixing the statistics.

Daily say-aloud reps

• "Mechanism, Measure, Move — path first, number second, fix that stays fixed."
• "Durability comes from WAL; checkpoints bound recovery time."
• "Writers don't block readers; dead tuples cost until vacuumed."
• "Sync replication trades latency for zero data loss."
• "A minority partition stops accepting writes — that is safety."
• "Averages lie; fix the tail by measuring the tail."

Drill 1 · EXPLAIN read (Debug)

Given EXPLAIN (ANALYZE, BUFFERS) showing Seq Scan on 40M rows, Buffers shared hit=12 read=380k, and a nested loop over it — narrate: top line first, the one fix with biggest payoff, and what you re-measure.

Read: full-table scan feeds the loop, ~380k buffer reads = the cost. Fix: composite/covering index on filter+join columns (equality first, range last). Re-measure: execution time + buffers hit ratio for that query hash. Trap: reading cost numbers as time; and ANALYZE first before DDL in case stats are stale. Depth: Debug — you ran the plan yourself.

Drill 2 · C++ reading comprehension — buffer eviction (Recognize)

// Reading only: describe the path, do not claim authorship.
for (page : clock_sweep(shared_buffers)) {
  if (page.usage_count > 0) { page.usage_count--; continue; }
  if (page.pin_count == 0 && !page.dirty) return evict(page);
}
// Q: what liveness bug appears if usage_count never decays on a hot scan?

Name the mechanism, the measurement that would prove the failure mode, and the one-line fix direction — as a reader, not the author.

Read: clock-sweep gives hot pages a second chance via usage_count; without decay every scanned page stays pinned-eligible and eviction stalls on dirty/pinned pages. Proof: buffer hit ratio collapse + eviction stalls in wait events. Fix direction: restore decay on each sweep pass. Depth: Recognize — "I read the structure and describe the path; I have not implemented at this layer." Never claim C++ engine implementation.

Drill 3 · C++ reading comprehension — WAL insert path (Do-not-claim)

// Reading only: describe the path, do not claim authorship.
xid = assign_xid();
lsn = wal_reserve(len);
memcpy(wal_buf(lsn), record, len);
wal_publish(lsn);          // what breaks if publish is reordered before reserve?
// Q: which interleaving loses durability on crash?

State which interleaving loses durability, the measurement that catches it, and the honesty line.

Read: publishing an LSN before the reservation is visible lets a concurrent flush believe the record is complete — a crash then replays a torn record and durability is lost. Catch: WAL-replay / crash-recovery test with concurrent inserters + checksum failures. Honesty line: Do-not-claim — method narration is the claim, engine implementation is not. Redirect to learning plan.