It’s Not Real Learning Until Something Goes Down

...and sometimes that something is production. TL;DR: At my first internship, I broke a test environment—it stayed down for a week. My friend wiped a live database—his team lead spent 5 hours restoring it. In 9 years, I’ve broken (and fixed) prod, test, and everything in between. But never the same thing, the same way, twice. My first dev job was as a Software Developer Co-op at a big MNC in Burlington, Ontario. Day one: I walk into orientation. Room full of 50 co-ops—and 60% of them are women. That had never happened before in my UWaterloo engineering life. (It didn’t happen again until PullFlow, with my current CEO.) Then someone walks in and says, “HR co-ops, follow me.” Almost every woman stands up and leaves. Just me and one other are left behind. The devs. I spent that internship low-key terrified. Always double-checking everything. Asking, “Is this okay?” “Can I run this?” Until one day, a teammate laughed and said: “It’s just the test environment?” “Yeah…” “Then just break it!” So I did. I ran a script with an infinite loop pulling from a database—and accidentally brought down the whole test system. The best part? I didn’t even realize I’d broken it. A week later, someone tried to use it and yelled: “Who used this last? DUDE—it’s all down!” It took 3 more days to fix. I haven’t run a script without loop-spotting since. Two weeks later, my friend Brian shuffled over, pale and sweating. “I think they might fire me.” He’d meant to wipe a test database. It was live. For one of the company’s biggest customers. His team lead (shoutout Emily!) spent 5 hours restoring it from backups. Brian didn’t get fired. He owned it immediately. And he’s never mistaken a connection string again. That first internship? I didn’t build anything world-changing. But I learned how to work. Next co-op, I built the company’s disaster recovery system. It broke, too—took down dev for a day. But I fixed it. With help from every team—engineering, support, sales. When I left, a senior engineer printed my etcd troubleshooting doc and had me sign it like an author. (They asked me back the next term.) Today, I’ve: Broken and restored prod environments Wiped and rebuilt more databases than I can count Taken down a test cluster or three But never the same mistake twice. If you’re just starting out: You will break things. That’s how you learn. Just make sure you own it, fix it, and don’t do it the same way again. That’s how you become someone who’s not afraid of "production is down"—because you know what to do next.

Apr 24, 2025 - 20:09

It’s Not Real Learning Until Something Goes Down

...and sometimes that something is production.

TL;DR:

At my first internship, I broke a test environment—it stayed down for a week.

My friend wiped a live database—his team lead spent 5 hours restoring it.

In 9 years, I’ve broken (and fixed) prod, test, and everything in between.

But never the same thing, the same way, twice.

My first dev job was as a Software Developer Co-op at a big MNC in Burlington, Ontario.

Day one: I walk into orientation. Room full of 50 co-ops—and 60% of them are women.

That had never happened before in my UWaterloo engineering life.

(It didn’t happen again until PullFlow, with my current CEO.)

Then someone walks in and says, “HR co-ops, follow me.”

Almost every woman stands up and leaves.

Just me and one other are left behind. The devs.

I spent that internship low-key terrified. Always double-checking everything. Asking, “Is this okay?” “Can I run this?”

Until one day, a teammate laughed and said:

“It’s just the test environment?”

“Yeah…”

“Then just break it!”

So I did.

I ran a script with an infinite loop pulling from a database—and accidentally brought down the whole test system.

The best part? I didn’t even realize I’d broken it.

A week later, someone tried to use it and yelled:

“Who used this last? DUDE—it’s all down!”

It took 3 more days to fix.

I haven’t run a script without loop-spotting since.

Two weeks later, my friend Brian shuffled over, pale and sweating.

“I think they might fire me.”

He’d meant to wipe a test database.

It was live. For one of the company’s biggest customers.

His team lead (shoutout Emily!) spent 5 hours restoring it from backups.

Brian didn’t get fired. He owned it immediately.

And he’s never mistaken a connection string again.

That first internship? I didn’t build anything world-changing.

But I learned how to work.

Next co-op, I built the company’s disaster recovery system. It broke, too—took down dev for a day.

But I fixed it. With help from every team—engineering, support, sales.

When I left, a senior engineer printed my etcd troubleshooting doc and had me sign it like an author.

(They asked me back the next term.)

Today, I’ve:

Broken and restored prod environments
Wiped and rebuilt more databases than I can count
Taken down a test cluster or three

But never the same mistake twice.

If you’re just starting out:

You will break things.

That’s how you learn.

Just make sure you own it, fix it, and don’t do it the same way again.

That’s how you become someone who’s not afraid of "production is down"—because you know what to do next.