Tick, tick…boom

by Matt Stratton on February 15, 2010

Post image for Tick, tick…boom

Remember yesterday when I was bragging about having 1,700 pageviews of my dumb Facebook user blog post? No, you don’t, since only 19 of you read that post. But if you did…let me just say this. I hadn’t seen nothin’ yet.

The Perfect Storm

The technology

Over the weekend, I was doing some experimenting with stuff on my blog and my server. I host my blog on a VPS, and due to potential memory constraints, was advised to run lighttpd instead of Apache as my web server. If you don’t know what that means, don’t sweat it…all you need to know is that Apache is a very common server to run WordPress on, and Lighty…well, he’s not as popular.

I’d run into some issues using lighty, specifically goopy stuff with .htaccess files and any plugins that rely on it. I gave the matter some thought, and figured hey, I have 512 MB of RAM to work with..let’s try Apache. I putzed with this for a while over the weekend, but never really got it to work the way I wanted to.

Another thing I did this weekend was install the W3 Total Cache plugin. Yeah, I couldn’t use it to its full value, since I don’t have Apache, but it had some neat features. One of them was the ability to use a CDN for things like javascripts, CSS, and images. Since we all know I dig the cloud, I thought it would be cool to migrate this type of content to Amazon CloudFront. I’m not going to lie and say it was super easy, but it was a great learning experience for me, which is really all that matters.

The irony clouds gather

This morning, when I went into work, I told my co-worker Shane about implementing CloudFront on my blog. I said something to the effect of “Do I get the kind of traffic that necessitates serving this content from the edge? No. Was it a cool way to play with Amazon’s CDN and learn about it? Yes.”

Later in the morning, I also thought it time to try Apache one more time. I was 99% sure I knew what I was missing in my config files that kept me from getting it going over the weekend, so I spent about five minutes updating the config, and then fired up Apache (it was listening on a different IP address, so all of my current sites kept running on Lighty).

The part where things start to act weird

About two minutes after starting Apache, I got a weird disconnect error on my WinSCP session (the file transfer program), and it refused to reconnect. I went over to my terminal session in putty, and every command I would enter resulted in a “cannot allocate memory” error. Plus, yeah, all the sites were down.

“Well, this sucks,” I thought to myself. “I know that Apache can be memory hungry, but seriously? After two minutes?”

Since I could not run any commands from the terminal, I couldn’t even stop Apache. This required a nuclear option, which was to log into my VPS control panel and reset the server. Of course, as soon as I do that, I get called away into a meeting. I figure this is no big deal as a) my job is more important than my blog, and b) I didn’t have Apache set to start at startup, so when the server came back online, Lighty would fire up and all would be well.

That’s what I thought, at least. A few minutes later, I got this tweet:

Sure enough, when I fired up my site on my iPhone, it was the default page for nginx. Which is ANOTHER web server software that I’d been playing with a while back. Whoops.

Luckily, I have a nifty app on my iPhone that lets me ssh into my webserver. I was able to quickly stop nginx and start Lighty. And all was well with the world.

The Great Reddit Invasion

When I got back to my desk, I had an email from a consultant friend of mine. The topic was “You’re climbing the Reddit ranks!” and had a screenshot showing a Reddit link with a vote score of 14 on the “upcoming” page on Reddit.

Now, I’d submitted the Hundreds of Facebook users are apparently really dumb post to Reddit myself last week, but got no traction on it. Apparently, someone else submitted it today…and that someone was more popular on Reddit than I was. Out of curiousity, I logged onto Clicky to see the real-time stats on my blog. And discovered that there had been over 3,000 views of that page today so far.

The memory issue, explained

Looking at the comments on the Reddit page, I came accross this:

This actually was posted at the exact time that I was rebooting my server. So it wasn’t just Apache that caused the memory issues…it was Apache PLUS a crapton of Reddit users pounding me.

Wherein the cloud saves my ass

Remember when I said that CloudFront was not necessary on a blog like mine? Today it totally was. As of the time of this writing, which is about 9 PM, there have been over 22,000 visitors to this site. That’s about 50% of all of my traffic for 2009…in one single day.

If I had not offloaded my scripts and images to the CDN, my server would have choked on this. Because I’m too lazy to count, I’m going to estimate that the Facebook post page contains about 8-10 images/CSS/javascripts which had been moved to CloudFront. That means that without this, we’re talking about increasing the HTTP connections to my poor little lighty server to almost a quarter of a million in one day. That’s a lot of freakin’ requests.

I’m also saved by the fact that I moved away from my old host and am on my VPS now. On my old host, as soon as traffic started to climb, I’d start to get CPU usage warnings/threats. And that was just if I got up around 800 visits a day. So yay, VPS. And yay, CloudFront.

Wrap it up, yo

I also added an additional AdSense ad unit just on the Facebook post. I figured all that traffic might be a good time to toss up an ad. Man, did that pay off! I am pretty sure I earned about 98 cents today. That’s almost an entire dollar.

Of course, my initial calculations are that my CloudFront bill for today will be about $5-$6. That sounds like a lot, but I really don’t expect to maintain this kind of traffic over more than a day or two. The buck I made from AdSense doesn’t really counter that. But hey – if I take my lunch to work one day this week, everything will balance out, right?

Have you ever had a post completely blow up and generate WAY more traffic than you ever expected? Did your server have a complete meltdown, or did you come through unscathed?

Image licensed via Creative Commons from Flickr user gmcmullen

Reblog this post [with Zemanta]
  • http://curiousillusion.com curiousillusion

    Well I have to say, you totally lost me. Except for the part where you mentioned me. Oh and the earning nearly a dollar.

    But… sounds very exciting!

    Oh and sorry, it sounded like I was bugging you that we were down, I didn't know if you knew. And sometimes our sites just look down cuz of my crappy work internet.

    • http://mattstratton.com mattstratton

      Oh, if it sounded like you were bugging me, you weren't! That's exactly what I needed to know…because I'd assumed that the reboot would just fix it…but it didn't. And who knows how long it would be before I *did* find out otherwise.

      I have just started playing with some site monitoring software, but since I haven't really gotten it configured properly…I'm not sure how reliable it is. Luckily, I can always count on you and Christina (Stacked) to report outages! :)

  • http://www.dshan.me/blog DShan

    Fucking badass, mate. Seriously.

  • http://www.momontherandom.com/ Cara

    The part where you mentioned @curiousillusion was my favorite part as well, lol. Aside from that it was like reading Chinese. Please tell me you're offering crab rangoon as a special today!

    Oh, but congrats?

  • Pingback: Blazing Minds Interviews Matt Stratton From Good Old Rock

  • http://rachelbaker.me Rachel Baker

    I am working on adding a CDN to my VPS server for speed improvements to my websites and client websites. Thanks for sharing your story, Matt.

  • Pingback: Amazon CloudFront and S3 maximum cost

Previous post:

Next post: