I told you so: Raw Sewage Edition

Many many moons ago I worked for a very successful Kansas City company that syndicates most of the newspaper comics in the US. It’s a really cool company, not just for it’s products, but it’s culture despite being from the early 1970s.

It was the boom of times

Like many companies during the dotcom boom, they were spinning up a ton of web properites. Obviously one per comic strip, but sometimes actually several different related sites for more popular properties like Garfield.

I was a Perl nerd back then and they, like any self respecting web outfit of the time, did everything in Perl. Oh and they had some Flash games, but all the real stuff was in Perl.

They call their comic syndication the Universal Press Syndicate, but the company name is Andrews McMeel Universal. And to be more dotcom-ish, they spun out the Internet portions of their business into a new entity named UClick.com.

I was hired as like developer #2 during this transition so I was given a temporary desk in the corporate HQ for a few weeks while the construction was completed in the new fancy dotcom-ish offices.

How dotcom-ish was this place? Well one of the interview questions was about what kind of beer I liked so they could stock the fridge. And we had a foosball table of course. I had a really long hair at the time and I suspect they put my desk right by the front door for even more edge.

When construction was complete we got to go check out the new office and it was fancy. Polished concrete floors, exposed brick, Aeron chairs, and beer fridges. Everything a bunch of youngsters needed to take over the Internet.

The one ops guy (we called them sysadmins back then) and I went down into the basement to check out our new server room. While I was hired as a developer, some of my previous roles were ops, so I offered to help him get things setup.

Danger! Danger!

The room was ok. I pointed out that basements aren’t ideal and asked if that part of Kansas City was prone to flooding for example. There were sprinklers in the room, which I also pointed out isn’t ideal, but probably acceptable for us considering this was all brand new construction and heavily inspected.

Then I spotted the drains.

Big floor drains the kind you probably have seen in the basements of parking garages or commercial buildings.

I distinctly remember saying, “Well we’re going to fill those in right? So they don’t back up in here?”

“Oh Frank, you’re far too worried about this stuff. It’ll all be fine.”

The move in…

We were promised racks for our servers and they were on order, but were delayed a few weeks. But we still had the entirely pointless internal deadline of being “all moved in” by a certain date. So we were told to move the servers into the basement “data center” anyway.

I asked if we could buy some folding tables to set them on so at least they weren’t on the floor and again was rebuffed for being a worry wort.

So we put them on the floor, as instructed.

Murphy’s Law strikes

Despite being the youngest person and living the furthest away I was often one of the first people into the office in those first few weeks.

A week or so after we moved in I pulled into the parking lot and spotted the ops guy’s car and he NEVER beat me to the office so I assumed one of two things:

1. He left his car at the office after having too many office beers last night
2. We were having some sort of outage

It was the later. As walked through the front door I could SMELL the problem before I even saw him.

I turn the corner and he’s coming up the stairs wearing his hip waders he used for hunting and said, “So you get to tell the boss I told you so.”

2 to 3 inches of sewage had backed up from the drains and filled the basement floor. Just enough to kill every single server and piece of networking gear that wasn’t stacked on top of another server.

Moral of the story? Sometimes a couple hundred dollars of cheap plastic tables can save your company a whole lot of heart ache!

P.S. I left the company a few months later after they made a really bad tech decision. Which allowed me to email him another “I told you so"" on that topic a year or so later when it all fell apart.

Frank Wiles

Founder of REVSYS, Django Steering Council, PSF Fellow, and former President of the Django Software Foundation.

Expert in building, scaling and maintaining complex web applications. Want to reach out? Contact me here or use the social links below.