My blog has been down for the better part of a week. It’s now back … almost … well mostly … it’s somewhat back!
Before I tell you the story, let me pre-emptively deal with all the “advice” that starts flying in everytime there’s a problem with my blog:
- moving to not-wordpress is a problem because I’d have to migrate plugins, the layout, and so on
- on using static page generator … my wordpress is configured to do just that, all you see are static pages
- my hosting is free, I like free stuff
- this host has withstood traffic of 500 visitors a second (each visitor translates into 10+ requests)
- I don’t like doing sysadmining
The Story of Being a Free User and Not Buying Enough Cakes
Everything started last Monday when I said that My brain can’t handle OOP anymore. Reddit hated the post, mostly resorting to calling me an idiot, but HackerNews was a far more receptive crowd with 53 upvotes and 53 comments. There was a bit of activity on twitter as well.
What I’m saying is the post was generally taking off a little bit.
Then everything went wrong.
A friend said “Hey, uhm, I’m trying to read your blog, but it’s taking a bit of time to open”.
Meh, that sort of stuff happens sometimes. You can never quite pinpoint it and not like I can do anything about it anyway since I have to rush off to class.
Then the tweets started. “Hey dude, your blog is down!” ”LoL, your site just says It works!” and so on.
Well … fuck.
How could this be? The post wasn’t even that popular? Somewhere near the bottom of hackernews, nothing but hate from reddit … impossible!
One of the fonts was 404-ing. This results in a request that does not get cached and servers go down. Or rather, servers don’t go down but one of the lesser sysadmins goes “You know what, the whole environment is getting a bit slow, and it’s all due to this guy’s blog … I’m sure nobody will mind if I just shut it down for now.”
Then the real fun begins. Later that night the main sysadmin decided that it was finally time to perform that migration to a new server we’ve been planning for months and he’ll do it in the morning since he’s been working on a project ’round the clock and is Too Tired Right Now ™.
Next morning, joy! Bliss! Heaven! He started, ran rsyinc, I switched around some IP addresses and off we go to greener pastures!
In a few minutes accessing the raw domain produced a 403 error and going to /blog gave you a lovely Error establishing database connection.
Progress! The blog is running happily under nginx now and soon enough eveyrthing will be back to normal with just a day’s worth of traffic gone. Not too shabby really, could’ve been much worse.
Then the main sysadmin vanished.
And I couldn’t get him back until Sunday.
I’m just a lowly freeloader with a blog that often brings down whole servers. I don’t even bring him cake or anything! He has better things to do.
Finally, on Sunday, rejoicment and glory and heaven and everything that is nice and lovely and sweet. He says Hello, sorry for the delay. on irc and promptly brings the database over to the new server and hooks it up to the wordpress installation.
Yay my blog is back!
“Okay, everything works, let me just hook up the redirects and page caching back up”
That was the last I’ve heard from him. I can only hope it doesn’t take another week for him to get back to me again. That would suck since this blog is totally vulnerable without a functioning page caching mechanism. There’s just too much traffic when I post anything remotely interesting.
Wish me luck.
- Archie the cake eating dog. (mertonmick.wordpress.com)
- Sysadmin Sunday 78 (serverdensity.com)
- The Most Important things for running a Reliable Internet Service (etbe.coker.com.au)
- The Best-Laid Plans of SysAdmins, or Trading Off Capital Vs. Labor (smallshoptechheavy.wordpress.com)
- Fighting boredom at a sysadmin job by writing a binary grabber (rachelbythebay.com)