The pace my life has taken, even after tax season has pretty much ended, is enough to bring Chuck Norris to his knees.
I keed. However, I’m managing to stay busy. Check this out:
During last year’s ice storm, Nathan and I started a little Twitter account we called TriStateAlerts. We mainly just used it to share information about shelters, emergency services, and the availability of generators. We covered a few other things, like a chemical leak and also handled severe weather alerts when we could devote some time to watching a Google Earth display for new polygons and manually typing the information into TweetDeck. Naturally, this got old rather quickly.
For the 2010 severe weather season, Nathan and I committed to having the feed automated, and I accomplished this feat on April 7. I would have had it done earlier, but a strange thing is that one has to test in a live environment when doing something like this. You can’t just make it happen. This new system was by no means perfect, it was just a shell script that ran a PHP script every 20 (and later, 10) seconds to check for new warnings, put them in a database, and tweet them. Pretty simple, right?
Our main problem with this system was that its data source was something the National Weather Service still deems experimental — the Common Alerting Protocol. It’s super awesome, though. It’s based on Atom/XML, which makes CAP a dream to work with. I seriously got working code after hacking at it for seven hours. All praise aside, CAP is still experimental. On the night of April 7, the feeds didn’t update for at least 30 minutes. After a chat with the science operations officer at the Paducah, Kentucky NWS office on the subject, I realized that I should really not be using CAP at this point. In fact, during the big April 24 severe outbreak, the CAP feeds never worked. Not once.
By that time, I had started work on the next step in the evolution of our fledgling service. I’ve been playing with a nice little piece of software called PerlEMWIN on and off for about four years now. This mainly consisted of installing it on a server, letting it run while trashing the filesystem all the while. (That writeFile plugin is nasty sometimes.) EMWIN, or the Emergency Managers Weather Information Network, is a tried-and-true method of dispersing weather information. It’s a continuous stream, priority-driven (meaning a warning will interrupt the transmission of something like a radar image), and it’s an automated push technology. This sounded like what I needed! However, I didn’t know a lick of Perl, and how the hell was I going to integrate this whole thing?
Around this time, I had been introduced to the directors of the local emergency management agency, who, from what I gathered, were desperately in need of a solution to disseminate emergency information via the internet, mainly through social networks like Facebook and Twitter. After my first meeting with the deputy director, he was thrilled that we had built something with such a following and told me we had essentially done his work for him. Oh, and please PLEASE sign them up for whatever we’re offering. (More on that later.)
More meetings further developed my plans, and an entirely new platform materialized in my mind. This wasn’t going to be just a service I run out of my bedroom with Nathan somewhere off to the side furiously researching information. My EMWIN integration had just gotten a lot easier, now that I had a perspective of what the whole thing would do.
I spent the better part of a week, mostly late at night, sitting at my kitchen table hacking away at a plugin for PerlEMWIN that would do what I needed. I had absolutely no experience with Perl, and didn’t have a clue where to start. I’ve been doing PHP for over six years, so you’d think that would help. Not with me. I started reading.
One of my biggest obstacles to learning Perl before I just dove into it with this project was that I just, for some reason, could not find out or just couldn’t understand what @_ was. Seriously. That was it. Maybe I’m just older and wiser now, or maybe all those years of doing PHP taught me a little something about programming in general, but when I found the answer to this simple question I couldn’t help but feel pretty stupid. (For the record, @_ is an array that holds the arguments you pass to a function. It’s a lot more flexible than PHP or probably most any other related language, but it’s really confusing.)
After finally learning just what the hell @_ was, and looking at other plugins bundled with PerlEMWIN, I got to work. Line by line, my plugin materialized. When I didn’t understand something or just straight up didn’t know how to do something in Perl, I resisted the urge to just give up and go back to PHP, and pressed on. Over the course of a week, I picked up enough Perl to finish this plugin and even have it drop data into the same MySQL database my old PHP script pointed to. This left out one thing– the social networking side.
I decided to run back to PHP for this one, for two big reasons. First of all, the design of the overall platform had to include a way to post some stuff to some places. For example, if a local emergency management agency subscribed to our platform, we obviously wouldn’t want to have tornado warnings for three counties away posting to their Twitter feed. This mechanism had to be separated from the data ingestion module. Secondly, I had just written the backends for a Twitter app and Facebook app in PHP, and I wasn’t about to learn a completely new client library to do something I had already written code for. I could just as easily have the PerlEMWIN plugin notify the message center that something new had come, and it could do the rest. That’s exactly what I did.
We also needed some dedicated hardware to run this on, with all the plans we had, the ragtag way things were going would just not cut it for a reputable emergency information service like we wanted. I called an old boss of mine and asked if he still had some servers sitting around for sale. I bought a Sun Microsystems SunFire V40z with four processors and 8GB of RAM to run everything on, and worked out hosting with him. It’s always good to not burn bridges– you never know when you’ll get the hook up.
The platform had its first real test during the severe weather event of April 24. I had barely touched our new server haardware, things were working fine the way they were and with a rare high risk area issued for western Kentucky, that was no time for something to go wrong. Nathan and I were working at the emergency operations center that day, fresh from a meeting with a prospective media partner and ready to tweet. As the warnings started going up for the area, all four endpoints we were pushing data to remained silent. I had finished the PerlEMWIN module (sans Twitter/Facebook support) that morning and wanted to make sure it caught all the warnings, and deposited the right information into the database in the correct format before using it. I hadn’t touched the old code since the last severe weather event, so the only logical explanation for the failure was that the CAP feeds weren’t updating at all. Great first impression, guys!
While we were in a controlled panic over our Twitter feed, I started scrambling to get the EMWIN module linked up and running. This was complicated by what could be generously described as an intermittent internet connection at the courthouse. After sticking my SIM card from my phone (which remained disassembled on the table) in my computer, I whipped up a quick script that posted to our sandbox account to test things, and I ran it manually to make sure it would pull the relevant information from the database and post it. It worked! I pointed the script at the right channels and made one final edit to the PerlEMWIN plugin.
More warnings came in, and still the Twitter and Facebook pages were silent. The logfiles showed all the information coming in just fine, and there were numerous warnings in the database. The Perl plugin wasn’t launching this PHP script I had just written. The panic continued. Nathan resorted to posting warnings manually while I worked, slowly but furiously, on the problem at hand. I found the problem (current working directory confusion) and fixed it in time to post the last two severe thunderstorm warnings of the outbreak.
Since then, we’ve moved everything onto our new server and it’s currently colocated in Evansville. The outbreak this past weekend has a few great moments for us; but those warrant another blog entry sometime in the future.