The Night PlanetSide Nearly Got Cancelled

Starting in May 2000 I was a member of a small team of engineers, game designers and artists working outside St. Louis, MO on a project for Sony Online Entertainment. Together we made PlanetSide, one of the only massively multiplayer first person shooters ever made.

This is one of our stories.

I was at home in my apartment when I got the call. I think it was about 8 in the evening. I can remember that I’d had dinner. I can’t remember what month it was, but that makes sense given the state of things. It was all kind of a blur.

It was Dave on the phone.

Dave was our Producer. He had been brought in to replace our first producer. To make things Better™. Things hadn’t been good for a long time. We were behind schedule and there was a lot of stuff left to do.

Our first Lead Programmer would stop showing up for work for weeks at a time. He liked to talk about infinite coordinate spaces and wrote code as fast as glaciers migrate. They transferred him to San Diego where they discovered he didn’t know a thing about what he was doing.

Our second Lead Programmer quit after four months. I think we’d hired him from some Corporate America back office type place that did mostly Java work. He wanted to make games, but our project was a mess and he probably had no idea what he’d gotten himself into. In my dreams I imagined he ran away screaming into the night, but honestly, he probably just came to his senses.

Now I was the Lead Programmer… of the first game I’d ever worked on.

Go me.

Dave said there was a problem. He’d gotten a call from our lead designer in San Diego. There was a shakeup in Corporate HQ. Maybe it was the .com crash, maybe it was a coup d’etat. Whatever it was, our CEO was out. The bad news was he’d had a few choice things to say on the way out the door. In particular he said our project was crap. It didn’t work. It would never work.

So Sony Pictures was sending a guy to see the game. Tomorrow.

My stomach dropped somewhere into the earth’s mantle.

“OK, I’ll take care of it” I said.

I told my wife I needed to go back into work, and I told her why.

“So you’ll be back… tomorrow?” She asked.

“Hopefully”

I got in our car. I looked at the clock and thought about the fact that I was going to lose 20 minutes on the drive.

Because there was a problem. The game didn’t work. Dave knew that when he called me…

That’s why he called me.

Oh it had worked… once upon a time. Before I took it apart. Before I’d ripped up and refactored a monolithic process into a distributed system so it could scale.

But I hadn’t put it all back together yet.

As I drove towards the office I thought about what to do. I could try to revert the codebase, but there was no way I could revert the data in time, and the old code couldn’t load the new data. Our source control was Visual Source Safe, which was junk. I decided trying to revert would probably fail anyways and cost me precious time.

I had to go forward.

I pulled into the parking lot. It was quiet and dark in the office. I was alone.

When I got to my desk I sat down and thought. Things I needed to do filled my head. I realized there was no way I could track it all. I also knew I’d be at this all night and I’d get punchy. I needed a backstop to keep me moving.

I made a list of everything that needed to work. Starting with logging into the game I listed all the features that needed to work for a demo to happen. Everything from moving and talking to reloading a weapon to dying and re-spawning.

If those worked by morning, it would be enough. Enough to buy time to fight another day.

There’s something about a code base you built yourself that’s very personal and very familiar. PlanetSide’s codebase was like that to me. I built enough of it that I knew where everything was, I knew why everything was. All I had to do was test it until something broke, fix that and repeat.

I just had to do it fast enough.

Running a minimal version of the new distributed PlanetSide cluster required 5 distinct server processes. I also needed at least 2 clients to test anything networked at all. To diagnose any kind of serious problem I knew I’d need to run 1-3 server processes and 1-2 clients under a debugger. All at the same time.

To do this I had one PC with 2 GB of RAM, a 10K RPM SCSI disk for builds and a 7200 RPM disk for everything else, including swap.

At the time a PlanetSide level took up about 800 MB of RAM. To make development possible on only one machine we’d written the 3D engine so that it accessed its assets via read only memory maps which allowed the memory to be shared between processes. The server loaded the level the same way. This meant that as long as the server and both clients were using the same level things would fit, or at least pretty close.

I remember choosing the smallest level we had and I turned my graphics settings way down. I shut down every process I didn’t need.

I started working on the list.

One of the perks software companies often offer their employees is free soda.

In St. Louis we had free soda in our fridge, but as I later discovered this was not true of the San Diego office. Too expensive I guess.

I made good use of our fridge that night. I like to think Sony’s return on their investment in that caffeine was very high.

The next 10 hours were a blur, and I honestly couldn’t tell you everything I had to fix. I just remember it was a lot.

At 6 AM the next morning every item on that list was checked off, the game worked and I was exhausted.

I pushed a new build to San Diego and slept on my desk.

Dave said the demo went great.

Epilogue

PlanetSide was released in 2003 and as of 2014 is still in operation. The distributed server architecture referred to in the article remains PlanetSide’s architecture to this day.

Dave and I are still friends.