Ants on the Melon
• • • • •

All I wanted was sleep.

It was Labor Day and I had been out late the night before playing “nickel and dime” poker with friends. I probably made it to bed around 4am, and the only thing I had planned for my day off was sleep, and maybe catching up on the book I was reading, The Tipping Point. But my slumber would come to an end around 10:30am, and it would be another 3 weeks before I finally would get a decent night’s sleep.

When my phone rang that morning I quickly awoke and jumped out of bed, as I always do. The caller ID on my phone said “Brett’s Cell,” so I figured he was calling to tell me about a cookout or pool party at The Big E Ranch for the holiday, after all, Brett is known for his parties.

I had only been awake for a matter of seconds so I can’t remember exactly what he said, but it was something like “I need a sysadmin, can you help me?” He proceeded to tell me about the Windsor Park Mall shelter that Rackspace had set up and how a better way to track people was desperately needed. “Sure, I’ll be right there.”

When I arrived at The Ranch Brett told me the rest of the story. I already knew that Rackspace CEO Graham Weston had offered up the old Montgomery Wards building at the Windsor Park Mall for use as an evacuee shelter for victims of hurricane Katrina, and that Racker’s had been working all weekend to get it in condition. Graham bought the building a few years earlier with plans that it would become the next Rackspace headquarters. For various reasons, we relocated elsewhere so by this time the Windsor building had been vacant for around 5 years. The real problem was that evacuees had already begun arriving at the shelter, but to everyone’s amazement, neither the Red Cross nor FEMA had any computerized system in place to record information about who was entering or leaving the shelter. I found this hard to understand, mostly because where I work everything is computerized and automated.

Brett told me that he and Dirk had started working on a web-based application to handle this problem and showed me what they had so far. There was a basic page for registering members of a family and checking them in, but not much more. Dirk was no longer working on the code, and Brett does very little programming, so there was a lot of work that needed to be done. To top it off, they had chosen to use the Ruby on Rails framework, with which I had no experience. I spent the rest of that day learning the language and framework, banging my head against the wall, and wondering how to do things in Ruby. With the growing list of tasks that were piling up, it became clear that we needed to recruit teammates. I paged Brandon and called Will, who were both busy for the holiday but would be available the next day. Brett called Lister, who was ready to help immediately. The team was growing and the deadline was shrinking; the race was on.

I left the Ranch around 1am that Monday night, and returned around 9am the next morning. Will came over to help that day, but had to leave around 6pm. Shortly thereafter Brandon arrived and began redesigning the heart of the application: the family page. We worked until around 2am and then headed home to crash for a few hours. The next morning we reconfigured our work area at the Ranch so that we were all sitting at a tall table, facing each other. This worked much better, and for the next week the Ranch was not only Brett’s home, but also all of ours. Wednesday we worked from 10am until 6:30am on Thursday, only breaking for a vendor sponsored sushi dinner and a midnight caffeine run to Starbucks at the Quarry. I had no idea what time it was until Brandon said, “Crap, now we are going to have to deal with morning traffic.” It finally hit me how much fun I was having: I had completely lost track of time, I was exhausted, I was hungry and most importantly, I did not want to stop. Do I really need to sleep? Sigh.

When you are having this much fun, time completely disappears. We each had special skills to contribute, and everyone’s abilities complemented the others. We were all doing what we do best and we were focused on a single, clear goal. In other words, although each of us was playing a different instrument, we were all in the same key, and the result was harmony. Our “Disaster Orchestra” roster:

• Brett – our point man, interface designer, blogger and badgemaster. Unlike me, he does not need sleep.
• Will – DBA extraordinaire, software master, reporting engine. He can ruin your life with a single database view, so don’t tempt him. (Also, he speaks in weird tongues over Google Talk)
• Brandon – blazingly fast coder, ajax lover, image-rotater. Try finding a problem he can’t solve.
• Lister – go-to man for anything and everything, wireless ninja, sushi-destroyer. This man is everywhere at the same time.
• Cory – sysadmin, ruby-newbie turned rails lover, food services. Just don’t ask me to install RT ever again

The extended family included Sharon, Tom, Ed, Marcus, Edwin, Debra, Suzy and a cast of about 200 other Rackers.

Back to our story. It is now Thursday morning and we are ready to unleash our software to the shelters. To hold them over, another Racker had written a small CGI application in Perl with MySQL to capture some basic information on the evacuees who were checking into the shelter. We wanted to import this data, so I wrote a script that logged in to his server, dumped the data, imported it on my laptop, generated a Postgres SQL file, uploaded it to our server and imported it into the database, all in one quick run. Around 2pm on Thursday Sept 8, we ran the conversion scripts, enabled our software, shutdown the old CGI script, and starting teaching the volunteers how to use the new application. 1..2..3..deploy!

It had been a few days since the evacuees had begun arriving in San Antonio, and apparently we were not the only people who were developing software to help organize all the information. We had seen volunteers in the shelters using various other applications to search for missing persons, and other systems for tracking the evacuees at the shelters, but none seemed to be as simple to use as ours. This might not seem like a big deal, but when volunteers are coming and going every hour or so, it is imperative that they be able to sit down at a workstation and immediately begin working without waiting for training, or having to read documentation. Other systems we saw required specific operating systems and browsers (Windows/Internet Explorer/ActiveX), but apparently the people who designed them had no idea what type of machines would be available at these shelters. Most workstations were loaned to the shelters by local businesses or individuals, and we even had a few ancient Pentium 166 machines running Windows 95/98. By requiring the latest in Microsoft technology, these other applications were locking out many people who were using these systems. On the other hand, our system was built to work with any system capable of browsing the web.

We were also faster to respond to the users than anyone else. We often worked right from the shelter so when a volunteer suggested a feature or had a problem, we were immediately able to make the changes and update the application. Sometimes we would watch people using the application to see how we could improve the interface, or speed up repetitive processes. It was a hyper-agile development environment, which Rails made possible. The users knew whom to contact if they had a problem, and most of the time we were within 15 feet. Also, people associated the application with us personally, which was really cool because we were proud of our work, and our users were appreciative; how often does that actually happen?

Over the next few days we made many changes and added more functionality to the application. Brandon and Will both had obligations during the first weekend that the app was live, so when they returned we were able to dive back into things and resume development. We began refactoring, cleaning up the code and performing general maintenance for the next few days, that is, until our next assignment knocked on the door the following Thursday.

We were working from the Ranch when Sharon called to tell us that she had promised that we would have an ID badge system in place by Monday morning. Badges? None of us had any idea what was involved in printing ID badge cards, but then again, just a few days earlier none of us knew how to build software to manage shelters filled with evacuees from a natural disaster. Apparently she had been given 10 minutes to decide if we could do it or not, and after seeing what we had done the previous week, she had no reason to believe we couldn’t do this either. Only this time we were quite out of our element, that element being software. So, first things first, a full featured (not-so-cheap) ID card printer was ordered and shipped via next-day air. Then we started trying to figure out how we were going to actually capture pictures of everyone. Our first idea was to use a webcam and have each person step in front of it just long enough to take a picture and then move on. So, we hooked a webcam up to a linux machine and Brandon put together a system to snap pictures and upload them to our webservers. This looked like it was going to work until we realized that the image quality was pretty bad. After much deliberation we decided to look for another solution. Considering that we were running short on time, our only option appeared to be digital cameras. We hopped in Sharon’s truck and drove to CompUSA to buy a couple digital cameras and SD cards (I don’t think any of us will ever forget that painful trip).

We came up with a system where two people would work the cameras and pass off the SD cards to people sitting in front of the application. In addition, we would be able to verify information about the evacuees in the system as the pictures were uploaded. This helped greatly in cleaning up the data, as our initial import required default values for fields that were not being collected with the previous script.

As the volunteers were working their way through everyone in the shelter, we were trying to figure out how to get the necessary information from our system into the AlphaCard software that came with the printer. Because we were using load balanced web servers, we needed to store the image data in the database so that it would be accessible instantly on both web servers (no rsyncing, etc). As an image was uploaded we rotated it, resized it, and base64 encoded it to avoid storing blobs in the database. As it turns out, this worked perfectly for sending the data to AlphaCard. Our resident querymaster, Will, wrote a funky stored procedure (yay PL/PGSQL! not really), and threw together a PostgreSQL view so that we could access all the information about an evacuee in one straightforward query. Meanwhile, Brett had put together a mock-up template ID card with help from the Rackspace marketing department and connected the printer to a machine at shelter. We allowed access to the database from the shelter, pointed AlphaCard to the correct database view, and before you could say “Gimme Shelter” we were in the ID card printing business. Well, not really a business, but we were doing it, and by Tuesday evening we had around 800 ID cards printed. Top that, DMV.

Soon we were hearing stories about how the ID cards were providing the residents with a way to begin rebuilding their lives. Many people had literally lost everything in the flood, including drivers licenses and other important documents. With the ID cards that we provided, residents were able to open checking and savings accounts at banks. It did not take long for the other shelters in town to hear about what we had done. By this time Hurricane Rita was nearing landfall and evacuees had started arriving from the gulf coast. Most of the shelters still did not have net connectivity at this point, much less a usuable shelter management system with ID card printing capabilities. With an incoming rush of evacuees from the new hurricane, they wisely requested our help.

We were asked to deploy our software at the other shelters and begin preparing to print ID cards for around 2,600 evacuees. No problem! Your order will be ready in 6 days.

Previously, the ID cards were taking about 1 minute each to print (double sided, color). If we were able to print 1 card per minute non-stop (without changing ribbons, reloading cards, cleaning the printer, etc), it would take 43 hours to print all 2,600. This wouldn’t be a problem if the process could be automated to the point that a human was not required. Unfortunately, this was not possible. Each picture had to be adjusted and cropped, and the card loader on the printer could only hold 100 cards. Brett changed the printer configuration so that the reverse side of the ID card was black only, and this one small changed cut the print time from 1 minute down to around 35 seconds – almost half. Still, we spent several full days baby-sitting the printer.

After the last batch of cards was printed, things began to slow down for us as we entered maintenance mode. Occasionally we would receive a request for changes to the software, but for the most part the excitement was over.

The project turned out to be a lot of fun, mostly because it was an opportunity to work on an interesting and important problem, with smart, capable people, in an extremely fast-paced environment. I would work for Ramen and Mountain Dew to be able to do this every day.

A couple nice side effects were that we helped people who really needed help, and also that we learned a new development platform in the process (Rails).

Our efforts also received a little attention from the media. Brett usually posted any news stories on the San Antonio Safelist Blog, but the following articles specifically mention our software:

• Texas Hosting Company Opens Doors To Katrina Victims – Var Business
• FEMA Computers Hampered During Katrina – physorg.com
• Report Warned of FEMA I.T. Problems in Wake of 2004 Hurricanes. – Top Tech News
• As evacuees leave S.A. shelters, they become hard to track (Brett is quoted) – MySA.com
• Home is sweet, even if it’s a Ward store – Denver Post
• Katrina evacuee connections get digital assistance while at shelters (Brett is quoted) – MySA.com

Not only did we have fun and accomplish something, but we also got a little recognition.

The Unifying Rails Dynamo

There have been many nice things said about Ruby on Rails, but there have also been a lot of nice things said about just about every other language and framework as well. It is often difficult to filter out what is actually great from what someone else just thinks is great, but in this case, it was hard to ignore what we were seeing happen right in front of us. In our situation each of the four main developers were previously using a different primary language:

• Brett – PHP
• Brandon – Perl
• Will – Java
• Cory – Python

Brett and Brandon each had some exposure to Rails, although they were both still new to it. Will and I had never coded in Ruby before, much less Rails. To be honest, the first couple days were frustrating, but I realized later that most of the tribulation came not from learning Ruby or Rails, but rather from unlearning what I already knew. Brett recorded his observations of this process on the San Antonio Safelist Blog, and it was picked up by the official Ruby on Rails Weblog. There were a few other critical tools that we built our software upon, and Brett also did a nice job of covering them on the blog.

Also of note, the Agile Web Programming with Rails book by Dave Thomas and David Heinemeier Hannson is exceptional. We had a couple copies and they went with us everywhere.

So, how do you get web developers using PHP, Perl, Java and Python to agree on a common development platform? Well, from our experience the answer appears to be “Ruby on Rails.”