October 16, 2005

“Shelter Famous”, or, How To Build Disaster Relief Software (with Rails)

Filed under: Friends,General,Technology — Cory @ 8:59 pm

All I wanted was sleep.

It was Labor Day and I had been out late the night before playing “nickel and dime” poker with friends. I probably made it to bed around 4am, and the only thing I had planned for my day off was sleep, and maybe catching up on the book I was reading, The Tipping Point. But my slumber would come to an end around 10:30am, and it would be another 3 weeks before I finally would get a decent night’s sleep.

When my phone rang that morning I quickly awoke and jumped out of bed, as I always do. The caller ID on my phone said “Brett’s Cell,” so I figured he was calling to tell me about a cookout or pool party at The Big E Ranch for the holiday, after all, Brett is known for his parties.

I had only been awake for a matter of seconds so I can’t remember exactly what he said, but it was something like “I need a sysadmin, can you help me?” He proceeded to tell me about the Windsor Park Mall shelter that Rackspace had set up and how a better way to track people was desperately needed. “Sure, I’ll be right there.”

When I arrived at The Ranch Brett told me the rest of the story. I already knew that Rackspace CEO Graham Weston had offered up the old Montgomery Wards building at the Windsor Park Mall for use as an evacuee shelter for victims of hurricane Katrina, and that Racker’s had been working all weekend to get it in condition. Graham bought the building a few years earlier with plans that it would become the next Rackspace headquarters. For various reasons, we relocated elsewhere so by this time the Windsor building had been vacant for around 5 years. The real problem was that evacuees had already begun arriving at the shelter, but to everyone’s amazement, neither the Red Cross nor FEMA had any computerized system in place to record information about who was entering or leaving the shelter. I found this hard to understand, mostly because where I work everything is computerized and automated.

Brett told me that he and Dirk had started working on a web-based application to handle this problem and showed me what they had so far. There was a basic page for registering members of a family and checking them in, but not much more. Dirk was no longer working on the code, and Brett does very little programming, so there was a lot of work that needed to be done. To top it off, they had chosen to use the Ruby on Rails framework, with which I had no experience. I spent the rest of that day learning the language and framework, banging my head against the wall, and wondering how to do things in Ruby. With the growing list of tasks that were piling up, it became clear that we needed to recruit teammates. I paged Brandon and called Will, who were both busy for the holiday but would be available the next day. Brett called Lister, who was ready to help immediately. The team was growing and the deadline was shrinking; the race was on.

I left the Ranch around 1am that Monday night, and returned around 9am the next morning. Will came over to help that day, but had to leave around 6pm. Shortly thereafter Brandon arrived and began redesigning the heart of the application: the family page. We worked until around 2am and then headed home to crash for a few hours. The next morning we reconfigured our work area at the Ranch so that we were all sitting at a tall table, facing each other. This worked much better, and for the next week the Ranch was not only Brett’s home, but also all of ours. Wednesday we worked from 10am until 6:30am on Thursday, only breaking for a vendor sponsored sushi dinner and a midnight caffeine run to Starbucks at the Quarry. I had no idea what time it was until Brandon said, “Crap, now we are going to have to deal with morning traffic.” It finally hit me how much fun I was having: I had completely lost track of time, I was exhausted, I was hungry and most importantly, I did not want to stop. Do I really need to sleep? Sigh.

When you are having this much fun, time completely disappears. We each had special skills to contribute, and everyone’s abilities complemented the others. We were all doing what we do best and we were focused on a single, clear goal. In other words, although each of us was playing a different instrument, we were all in the same key, and the result was harmony. Our “Disaster Orchestra” roster:

  • Brett – our point man, interface designer, blogger and badgemaster. Unlike me, he does not need sleep.
  • Will – DBA extraordinaire, software master, reporting engine. He can ruin your life with a single database view, so don’t tempt him. (Also, he speaks in weird tongues over Google Talk)
  • Brandon – blazingly fast coder, ajax lover, image-rotater. Try finding a problem he can’t solve.
  • Lister – go-to man for anything and everything, wireless ninja, sushi-destroyer. This man is everywhere at the same time.
  • Cory – sysadmin, ruby-newbie turned rails lover, food services. Just don’t ask me to install RT ever again :)

The extended family included Sharon, Tom, Ed, Marcus, Edwin, Debra, Suzy and a cast of about 200 other Rackers.

Back to our story. It is now Thursday morning and we are ready to unleash our software to the shelters. To hold them over, another Racker had written a small CGI application in Perl with MySQL to capture some basic information on the evacuees who were checking into the shelter. We wanted to import this data, so I wrote a script that logged in to his server, dumped the data, imported it on my laptop, generated a Postgres SQL file, uploaded it to our server and imported it into the database, all in one quick run. Around 2pm on Thursday Sept 8, we ran the conversion scripts, enabled our software, shutdown the old CGI script, and starting teaching the volunteers how to use the new application. 1..2..3..deploy!

It had been a few days since the evacuees had begun arriving in San Antonio, and apparently we were not the only people who were developing software to help organize all the information. We had seen volunteers in the shelters using various other applications to search for missing persons, and other systems for tracking the evacuees at the shelters, but none seemed to be as simple to use as ours. This might not seem like a big deal, but when volunteers are coming and going every hour or so, it is imperative that they be able to sit down at a workstation and immediately begin working without waiting for training, or having to read documentation. Other systems we saw required specific operating systems and browsers (Windows/Internet Explorer/ActiveX), but apparently the people who designed them had no idea what type of machines would be available at these shelters. Most workstations were loaned to the shelters by local businesses or individuals, and we even had a few ancient Pentium 166 machines running Windows 95/98. By requiring the latest in Microsoft technology, these other applications were locking out many people who were using these systems. On the other hand, our system was built to work with any system capable of browsing the web.

We were also faster to respond to the users than anyone else. We often worked right from the shelter so when a volunteer suggested a feature or had a problem, we were immediately able to make the changes and update the application. Sometimes we would watch people using the application to see how we could improve the interface, or speed up repetitive processes. It was a hyper-agile development environment, which Rails made possible. The users knew whom to contact if they had a problem, and most of the time we were within 15 feet. Also, people associated the application with us personally, which was really cool because we were proud of our work, and our users were appreciative; how often does that actually happen? ;)

Over the next few days we made many changes and added more functionality to the application. Brandon and Will both had obligations during the first weekend that the app was live, so when they returned we were able to dive back into things and resume development. We began refactoring, cleaning up the code and performing general maintenance for the next few days, that is, until our next assignment knocked on the door the following Thursday.

We were working from the Ranch when Sharon called to tell us that she had promised that we would have an ID badge system in place by Monday morning. Badges? None of us had any idea what was involved in printing ID badge cards, but then again, just a few days earlier none of us knew how to build software to manage shelters filled with evacuees from a natural disaster. Apparently she had been given 10 minutes to decide if we could do it or not, and after seeing what we had done the previous week, she had no reason to believe we couldn’t do this either. Only this time we were quite out of our element, that element being software. So, first things first, a full featured (not-so-cheap) ID card printer was ordered and shipped via next-day air. Then we started trying to figure out how we were going to actually capture pictures of everyone. Our first idea was to use a webcam and have each person step in front of it just long enough to take a picture and then move on. So, we hooked a webcam up to a linux machine and Brandon put together a system to snap pictures and upload them to our webservers. This looked like it was going to work until we realized that the image quality was pretty bad. After much deliberation we decided to look for another solution. Considering that we were running short on time, our only option appeared to be digital cameras. We hopped in Sharon’s truck and drove to CompUSA to buy a couple digital cameras and SD cards (I don’t think any of us will ever forget that painful trip).

We came up with a system where two people would work the cameras and pass off the SD cards to people sitting in front of the application. In addition, we would be able to verify information about the evacuees in the system as the pictures were uploaded. This helped greatly in cleaning up the data, as our initial import required default values for fields that were not being collected with the previous script.

As the volunteers were working their way through everyone in the shelter, we were trying to figure out how to get the necessary information from our system into the AlphaCard software that came with the printer. Because we were using load balanced web servers, we needed to store the image data in the database so that it would be accessible instantly on both web servers (no rsyncing, etc). As an image was uploaded we rotated it, resized it, and base64 encoded it to avoid storing blobs in the database. As it turns out, this worked perfectly for sending the data to AlphaCard. Our resident querymaster, Will, wrote a funky stored procedure (yay PL/PGSQL! not really), and threw together a PostgreSQL view so that we could access all the information about an evacuee in one straightforward query. Meanwhile, Brett had put together a mock-up template ID card with help from the Rackspace marketing department and connected the printer to a machine at shelter. We allowed access to the database from the shelter, pointed AlphaCard to the correct database view, and before you could say “Gimme Shelter” we were in the ID card printing business. Well, not really a business, but we were doing it, and by Tuesday evening we had around 800 ID cards printed. Top that, DMV.

Soon we were hearing stories about how the ID cards were providing the residents with a way to begin rebuilding their lives. Many people had literally lost everything in the flood, including drivers licenses and other important documents. With the ID cards that we provided, residents were able to open checking and savings accounts at banks. It did not take long for the other shelters in town to hear about what we had done. By this time Hurricane Rita was nearing landfall and evacuees had started arriving from the gulf coast. Most of the shelters still did not have net connectivity at this point, much less a usuable shelter management system with ID card printing capabilities. With an incoming rush of evacuees from the new hurricane, they wisely requested our help.

We were asked to deploy our software at the other shelters and begin preparing to print ID cards for around 2,600 evacuees. No problem! Your order will be ready in 6 days.

Previously, the ID cards were taking about 1 minute each to print (double sided, color). If we were able to print 1 card per minute non-stop (without changing ribbons, reloading cards, cleaning the printer, etc), it would take 43 hours to print all 2,600. This wouldn’t be a problem if the process could be automated to the point that a human was not required. Unfortunately, this was not possible. Each picture had to be adjusted and cropped, and the card loader on the printer could only hold 100 cards. Brett changed the printer configuration so that the reverse side of the ID card was black only, and this one small changed cut the print time from 1 minute down to around 35 seconds – almost half. Still, we spent several full days baby-sitting the printer.

After the last batch of cards was printed, things began to slow down for us as we entered maintenance mode. Occasionally we would receive a request for changes to the software, but for the most part the excitement was over.

The project turned out to be a lot of fun, mostly because it was an opportunity to work on an interesting and important problem, with smart, capable people, in an extremely fast-paced environment. I would work for Ramen and Mountain Dew to be able to do this every day.

A couple nice side effects were that we helped people who really needed help, and also that we learned a new development platform in the process (Rails).

Our efforts also received a little attention from the media. Brett usually posted any news stories on the San Antonio Safelist Blog, but the following articles specifically mention our software:

Not only did we have fun and accomplish something, but we also got a little recognition.

The Unifying Rails Dynamo

There have been many nice things said about Ruby on Rails, but there have also been a lot of nice things said about just about every other language and framework as well. It is often difficult to filter out what is actually great from what someone else just thinks is great, but in this case, it was hard to ignore what we were seeing happen right in front of us. In our situation each of the four main developers were previously using a different primary language:

  • Brett – PHP
  • Brandon – Perl
  • Will – Java
  • Cory – Python

Brett and Brandon each had some exposure to Rails, although they were both still new to it. Will and I had never coded in Ruby before, much less Rails. To be honest, the first couple days were frustrating, but I realized later that most of the tribulation came not from learning Ruby or Rails, but rather from unlearning what I already knew. Brett recorded his observations of this process on the San Antonio Safelist Blog, and it was picked up by the official Ruby on Rails Weblog. There were a few other critical tools that we built our software upon, and Brett also did a nice job of covering them on the blog.

Also of note, the Agile Web Programming with Rails book by Dave Thomas and David Heinemeier Hannson is exceptional. We had a couple copies and they went with us everywhere.

So, how do you get web developers using PHP, Perl, Java and Python to agree on a common development platform? Well, from our experience the answer appears to be “Ruby on Rails.”

• • •

August 20, 2005

Las Vegas with HSC Friends

Filed under: Friends,Places — Cory @ 2:53 pm

Last weekend, after spending a few days working in Reno, I flew out to Las Vegas to meet up with Andrew, Gerard, and Ben. I was out there two weeks earlier, so I had a pretty good idea of the layout of the town, and felt pretty comfortable getting around.

We stayed at the MGM Grand on the strip, which was really nice (the water in the bathrub actually drained faster than it was coming in – a Las Vegas rarity!). I stayed there when I went to Vegas with my family back in 1995 a year or two after it opened as the largest hotel in the U.S..

Saturday night we had dinner at some ye olde restuarant at The Excalibur. It was an extremely good meal, and for the first time that I can remember, I was the only person at the table who actually finished eating everything I ordered. We finished out the night by wandering around, occasionally playing blackjack, and eventually crashing around 4:30 or 5:00.

The next day we decided to take a trip to downtown Las Vegas. We spent a some time at the $3 blackjack tables at Binion’s, saw the sights, and eventually caught a cab back to the strip. Simply put, don’t ask questions when you meet someone from Eritrea.

All of our flights were delayed that night, and I ended up getting home around 6:30 Monday morning, just in time to completely pass out on my bed. I’ve been struggling to recover all week, which is why I am just now posting an update.

Earlier this week Slashdot covered a story about technology and security in Las Vegas. Interesting stuff, and it really makes you wonder what they aren’t telling.

To sum it all up, I love Las Vegas and am ready to go back whenever you are. :)

• • •

August 11, 2005

Reno, Two Years at Rackspace

Filed under: Friends,Places — Cory @ 4:17 pm

I made it out to Reno today around noon, and will be very busy for the next few days. I’ll be here until Saturday, at which point I will be flying down to Las Vegas to meet up with Andrew, Gerard and Ben for a couple days. WOO!

Two years ago today I started work at Rackspace, it is hard to believe it has been that long.

• • •

August 1, 2005

End of DefCon, Deep Fried Twinkies

Filed under: Events,Friends,Places — Cory @ 3:00 pm

After 3 days of talks and general craziness, DefCon 13 came to a close yesterday. After the closing ceremonies we headed over to In-N-Out Burger for dinner. I had never eaten there before, but I must say it was pretty good. Richard has a picture of my first bite of the burger. We spent the next half hour in the blistering sun trying to get a cab, and wishing we had skipped the milkshakes.

Next, my gracious tour guides took me to downtown Vegas where I was able to see The Golden Nugget, Binions, and eat a deep fried Twinkie. After watching for a while, I got in on a couple games of blackjack and ended up $40 ahead, which made it $80 for the trip, so I was happy. We saw some more sights and headed back to the strip, specifically to Mandalay Bay and The Luxor. The engineering and wow-factor of all these places is amazing, everything is enormous!

I’m heading back to the airport in about 4 hours, so the trip is almost over, but it has been a lot of fun. I will be back out here in 2 weeks to meet up with some friends from college and I can’t wait!

• • •

July 31, 2005

Saturday at DefCon, Blackjack in Vegas

Filed under: Events,Friends,Places,Technology — Cory @ 5:22 pm

Yesterday at DefCon there were a few interesting sessions, but the my favorite of the day was the Introduction to Lockpicking and Physical Security. It was interesting to see how insanely easy it is for someone to unlock a lock within seconds. The only other session I thought was really interesting was the Be Your Own Telephone Company…With Asterisk talk. I’d like to try setting up an Asterisk PBX sometime soon.

Yesterday I ended up hanging out with Dirk and Richard for most of the day. We met up with some friends of theirs for dinner at Wolfgang Puck‘s Bar and Grill at the MGM Grand. I had the Mustard Glazed Salmon with Warm Potato, Granny Smith Apples, and Watercress, and it was excellent. I ate to the point of discomfort, which is not something I often say. After dinner we walked around for a while and saw the sites. Dirk and Richard taught me a lot about blackjack during the night and after several hours of watching and asking questions, I finally decided I was ready to give it a try. I decided that $20 was all I was willing to put on the line, so at a $5 table I got $20 worth of chips and began playing. I was never down, and after about 30 or 45 minutes I had $60 worth of chips. At this point I decided it would be a good idea to cash out and quit while I was ahead.

Las Vegas reminds me a lot of New Orleans because it is very easy to loose track of time if you are not paying attention. I made it back to my room around 5:30, and I didn’t even feel tired. I think I like this town. ;)

• • •

July 14, 2005

Cajun Night at WND

Filed under: Events,Friends — Cory @ 4:55 pm

Last night I headed over to the Wednesday Night Dinner, bearing Beignet’s for cajun night. I got there around 9, after spending a couple hours making a mess in my kitchen with flour, beignet mix and powdered sugar.

Andy grilled up these amazing bacon wrapped jalapeno shrimp that had me sweating for at least an hour. There was a pretty good turnout and among the other things that people brought were fried alligator, crawfish pie, red beans and rice, jambalaya, and fried pickles. Mmmm, good eatin!

After everyone finished eating, we sat around the picnic table and talked for a couple hours, until around 11:00 when Issac, Steve and I started playing music. Our main guitarist, the Professor, was not able to make it this week, so the band consisted of Issac on guitar, Steve singing, and me on mandolin. We played 3 or 4 songs and decided it was too hot, so we headed back to the picnic table and resumed socializing. I left shortly after midnight so I could get ready for my trip to Atlanta. Another great Wednesday night.

• • •

Drew Kennedy and Matt Powell

Filed under: Friends,Music — Cory @ 4:54 pm

Tuesday night I went up to the River Road Ice House in New Braunfels to see Drew Kennedy. Drew was a fellow history major at Hampden-Sydney College, and graduated a year after I did. Several months ago I read in the college newsletter that he was working on a career as a Texas musician. Well, Tuesday I finally had a free night so I made the trip up there to see him play.

I talked with Drew for a little while when I got there, and he mentioned that Matt Powell would be playing with him, and that he was one of the best songwriters around. Everyone has their own idea of greatness and “best”, so sometimes you just have to see for yourself. In this case, Drew was right.

The show was nice and laid back, just the way I like them. As they began getting ready to play, I grabbed a Negro Modelo and assumed a spot at the bar. There was a small, casual crowd there; apparently everyone there knew everyone else.

Drew and Matt alternated playing, with Matt occasionally dropping a lead into Drew’s songs, very nice. They mostly played their original music, with one or two covers mixed in (including “Sitting by the Dock of the Bay”). Not only are they both great songwriters, but the music was also excellent. At times Matt’s music and singing reminded me of Mofro, but then he’d throw a bluegrass song in there and tear it up. Matt is another Virginian (Franklin county) living in south Texas, so it should have been no surprise that what he was playing reminded me of music from back home. Besides the great music Drew and Matt also went the extra mile to perform comedy acts between songs, because as everyone knows, you can never have too many Chappelle’s Show references. :)

If you are into acoustic goodness, go ahead and make it out to one of Drew’s or Matt’s shows. Good listening, indeed.

• • •

July 10, 2005

An Update, Finally

Filed under: Events,Friends,General,Music,Places — Cory @ 1:19 pm

Ok, so it’s been a week and a half since I last posted anything. I’ve been really busy, and haven’t been in the writing mood. Anyway, here’s the news…

Friday, June 1 was my last day with the CORE Team. I was with the group for almost 2 years and I enjoyed working there, but it was time for something new. Last Monday I started at my new position (still at Rackspace) where I will eventually be administering the DNS servers. This is much closer to what I am interested in, and I am excited to have the opportunity to work on something like this.

Last Saturday I went to see War of the Worlds with some friends. I was really disappointed with the movie. That’s all I’ll say about that.

On Sunday Brandon and I went over to Pepper and Ethan’s house to play nickel-and-dime poker. I had never played poker before, but I can totally see how that could become addictive. I think I ended the night down a dollar or two, which bothered me enough to know that I would never want to seriously gamble. Speaking of that…

This week I bought plane tickets to go to DefCon, which is in Las Vegas from July 28 – August 1. DefCon is the big hacker conference where all sorts of interesting ideas make their way to the public. I know of 6 friends that will be out there for the conference, so it should be a good time. I’ll be back in Vegas 2 weeks later for my college roomate’s bachelor party, after spending 3 days in Reno to work. On the subject of travel, I bought plane tickets this week to go to Tampa, Fl to visit Sarah. I’ll be out there Sept 9 – 12.

On July 4th I played music with Evan and Issac for a couple hours, and then headed over to Brett‘s house for the pool party. After 5 or 6 hours in the pool and hot tub, I was exhausted. I went home and crashed early that night.

Tuesday Brandon come over for cigars and movies. We watched two more episodes of Band of Brothers. We have watched through episode 7 now, and I’m really into it. Winters and Spears are my favorites so far, and Lipton is pretty cool as well. We’ve got 4 more episodes to go.

On Wednesday I went over to Issac’s for the Wednesday Night Dinner and had a great time. I took Crawfish Etouffee and rice, which tasted pretty damn good if I do say so myself. :) Evan, Issac and I played music for the crowd for almost an hour, including a soon-to-be-famous acoustic version of “Ice Ice Baby.”

Thursday I had to relax after being social for like 10 days straight. But it started all over on Friday night…

• • •

June 30, 2005

Wednesday Night Dinner

Filed under: Events,Friends,Music,Places — Cory @ 2:16 am

A few months ago when Chris was in town a bunch of people showed up at Rebar to hang out and catch up with him. I ended up talking with his friend Isaac quite a bit that night, and I remembered that he mentioned something about Wednesday Night Dinner. That was all I had heard about it until last night.

Chris is back in town this week, and he got a bunch of people together at Gruene Hall last night for the live DVD that Two Tons of Steel was recording. Isaac was there again and was telling us the story behind the WND. Apparently some people in Austin started the original Wednesday Night Dinner as a way to meet friends of friends, enjoy music, art, and of course, good food. Isaac started up the WND thing in San Antonio earlier this year, and last night he invited us to come over. Everyone brings something: food, drinks, supplies, or if you are Brett, cigarettes. Music is also a big part of the night, so he suggested that I come prepared to play something.

Lately I’ve been trying to make an effort to meet more people outside of work, so this sounded like a good opportunity. After work I stopped and bought some red beans & rice mix and a jalapeno smoked sausage, headed home and cooked it up. I showed up at their place around 9 (when things were supposed to get started) with the beans, my guitar, and my mandolin. Pretty much everyone was there by the time I arrived, which was great because I only knew like 3 or 4 people to begin with. Immediately it was time to eat, and let me just say how great it was to see some real food on the table. Green bean casserole, another type of bean casserole, and about 6 or 7 other types of casseroles (it was casserole night, if you didn’t pick that up). I ate a plateful of casserole and loved it. I topped it off with some sort of Oreo and whipped cream dessert, at Richard‘s recommendation. Whoa, it was all good.

Once the eating slowed down it was time for music. There were several guitar players there, so I grabbed my mandolin and handed my guitar to someone else. We played some Gourds songs (El Paso, Gin and Juice, and Ants on the Melon, of course), Friend of the Devil, and a few other things for about 20 minutes or so until some of the other musicians wanted to play stuff from other genres. That was cool too, because I wanted to hang out with people anyway. The rest of the guys continued to play on and off for the next 3+ hours, and it was fantastic. I really enjoyed just sitting back and listening to the music.

I guess around 20 people showed up this week, maybe more, but overall it was a great night. I hope to attend the WND as often as possible from now on, it’s just the type of social gathering I like: laid back with good food, good music and good people.

• • •

June 28, 2005

The Fanatical CORE Team

Filed under: Friends — Cory @ 1:33 am

Last fall the CORE team won the Rackspace Fanatical Award for the improvements we made to the company’s internal software. We took a picture for it back then, but it was pretty bad. Now that the company is at the new office building, there is a dedicated wall with pictures of those who have won the Fanatical Award. A couple weeks ago we were asked to pose for another picture so that it could go on the wall. Fortunately we had a few days notice, so a few of us decided that we would dress up a little bit that day for the occasion. Unfortunately, only Will and I remembered to do this, but the pictured turned out fairly nice anyway.

• • •
« Previous PageNext Page »
Powered by: WordPress • Template by: Priss