A careless ALTER can pause writes across your app. Prefer expand/contract over big-bang schema changes.
A careless ALTER can pause writes across your app.
Expand schema first, dual-write or backfill, then contract old columns.
Schema changes are production changes. Treating them like local experiments creates multi-hour outages.
Why this shows up in real IT work
Information technology fails in boring, repeated ways: unclear ownership, untested recovery, and tools used without a question. Expand schema first, dual-write or backfill, then contract old columns. That sentence is the filter for everything below.
Most teams already own the software that could help. What they lack is a sequence they can run under pressure. A careless ALTER can pause writes across your app. The gap between knowing the tool exists and using it deliberately is where outages and slow weeks live.
You do not need a new platform to start. You need a first check small enough to survive a noisy on-call, then a second check that confirms the first one. That is how operational skill compounds without requiring a conference talk every morning.
The mechanism in plain language
Schema changes are production changes. Treating them like local experiments creates multi-hour outages. When you treat the problem as bad luck, you get folklore. When you treat it as a diagnosable system, you get a checklist.
Notice how often teams wait for a perfect rewrite before they fix the path in front of them. Delivery follows verification more often than the reverse. Expand schema first, dual-write or backfill, then contract old columns. Prove the symptom, then change one variable.
Also notice the stories people tell mid-incident: that this stack is uniquely cursed, that only one hero can fix it, that documentation is for other people. Those stories are loud and mostly useless. Replace them with a sequence you can run with half a dashboard and a half-honest status page.
A practice you can run this week
- Add new columns nullable (or with safe defaults) without dropping old ones.
- Deploy code that reads/writes both shapes if needed.
- Backfill in batches with throttling.
- Only then remove the old column in a later release.
- Test migrations against a production-sized copy.
Run the practice once without optimizing it. Optimization is how people postpone the first restore drill or the first key rotation. After three honest reps, adjust one friction point — the alert threshold, the runbook command, or the access path — and keep going.
A scene from ordinary operations
You rename a column across millions of rows by adding the new name, backfilling overnight, switching reads, then dropping the old column next week.
That scene is intentionally ordinary. IT skills earn their keep in ticket queues, deploy windows, and quiet Sundays — not only in viral outage postmortems.
If your own version of the scene looks messier, that is fine. Skill is not aesthetic. Skill is whether the next similar moment goes slightly better because you had a plan.
The pitfall that quietly undoes the practice
Rewriting a huge table in one transaction during peak traffic.
If you catch yourself doing that, stop and return to the smallest next verification. Do not scrap the whole system because one change window went sideways. A careless ALTER can pause writes across your app.
Another quiet undoing: collecting more tooling instead of practicing the tooling you already have. Read this once, then open the terminal or console and run step one. Articles do not change uptime. Reps do.
A short checklist
- Decide what “done enough” looks like before you start.
- Remove one friction from the environment or the runbook.
- Run the smallest verification even on a busy day.
- Review once, without courtroom energy.
- Ask yourself: Would this migration still be safe if the table is 10× larger?
Limits and honesty
No single habit fixes underfunding, hostile vendors, or legacy constraints you did not choose. Practices sit on top of reality; they do not erase it. Use what helps, ignore hustle folklore that blames individuals for structural debt, and escalate when the checklist is not enough.
Still, many of the leaks in a week are local and trainable. Access hygiene, DNS checks, backup restores, and clear alerts respond to practice. That is why this belongs under IT on Swiftener: useful moves for systems work, not vibes for performing expertise.
If someone on your team needs a softer entry, share the scene and the checklist — not a lecture. Invitation beats correction.
The one test
Would this migration still be safe if the table is 10× larger?
If the answer is no, shrink the practice until it becomes yes. Expand schema first, dual-write or backfill, then contract old columns.
Then teach the move to someone else in one minute. Teaching is how the skill sticks in your own hands.
Come back to this page when the pager gets loud again. The point is not to memorize a brand of best practice. The point is to leave with a move you can use before the shift ends.