My Overloaded Server Story

And the server belongs to me

So as you probably know by now, I converted my blog to Jekyll, yesterday, and it’s been a huge success, in my opinion. I’m motivated more than ever to start blogging more because I have the added tinker factor and using Git and GitHub to keep everything organized rocks.

But that isn’t what this story is about. Tonight’s story is about a server. A lowly VPS floating somewhere in the SFO DigitalOcean…. ocean. I’ve had this server running for many months–probably over a year now and I just don’t know it–and it’s hosted just about anything and everything I’ve tinkered with including many a failed idea.

I use this server a lot and for the most part it always responds well and isn’t sluggish. It’s hosted my former WordPress blog since January and hasn’t made but a peep about it so I’ve spent quite a bit of time thinking nothing’s amiss. I felt, though, that Jekyll shouldn’t be taking as long as it was to build my site but I couldn’t convince myself the server was overloaded.

As it Turns OutTM, I was wrong.

My baby of a server has, for the last five or so months, been filling Newrelic graphs with stuff like this:

CPU

loaded_server_before_cpu

Memory

loaded_server_before_mem

Disk

loaded_server_before_disk

Load

loaded_server_before_load

So I’m just going to go ahead and say that’s not good. The only graph above that’s even remote decent is the Disk graph, but it’s washed over by the CPU > 80% indication so it’s also pretty much hosed.

At this point I’m honestly surprised. I’m a terrible pseudo-sysadmin and I should be fired but there’s no one to fire me and I’m the boss so whatever. If I can fix all this, I’m giving myself a raise.

I started digging. I wanted to see what’s running and who’s sucking up all the juices. I fired up top and waited a few seconds. A few things popped up: a couple instances of node and an instance of ruby.

Hmm. That doesn’t make any sense… I’m not running any node apps and jekyll is the only ruby thing I use right now… oh wait.

See, I tinkered with NodeJS apps late last spring. Ghost was a blogging tool I was considering for a while. I also tinkered with Discourse to see what it was all about. Turns OutTM, neither are suited for my needs.

I don’t really know what happened but I can only assume I forgot about them and they’ve been running all this time. I had a total of five Node instances running and Docker was running Discourse so between the two of them, I was on swap 24/7. So dirty.

This story isn’t super climactic in any way and the fix was easy: kill all the things. I also found this to be a good time to remove Node and Docker since I need neither.

I still couldn’t figure out why my disk usage was so high, though. Leave it to me to forget about a 4GB .iso I left in a folder and leave it to Discourse to fail at sending 5GB of emails through postfix over the last five months and clog up my /var/mail folder. Needless to say, after the almost-winter cleaning, I gained quite a bit of ground:

CPU

loaded_server_after_cpu

Memory

loaded_server_after_mem

Disk

loaded_server_after_disk

Load

loaded_server_after_load

Sorry this story wasn’t more interesting. If you’re curious, my Jekyll build time was cut in half.

That’s about it. Orphaned tools and processes makes my server a dull VPS.


The Stack

(This post is wildly out-dated. The most current info is here.)

Yesterday someone asked me what my stack looked like.

Today I think it makes a good blog post.

I’m always open to share the tech behind my sites and this one is no different. While not incredibly in depth, here’s how it goes:

Layer 1: Infrastructure

The core of it all is DigitalOcean. They provide the resources my site sits on and so far they’re doing amazingly well. If I ever need more space or resources, I just need to click a few buttons and I’m on my way. I could also set up load balancing, too, with little effort if it comes to that. Their claim to fame are their awesome little (or big) droplets that start at $5. For $5 you get 512mb of RAM 20GB of SSD-backed storage, and 1TB of bandwidth. That’s enough for a small, site, for sure. I’ve moved higher than that mainly because of one member of my software stack that naturally uses more RAM. I’ll cover it later.

Layer 2: Operating System

Within my DigitalOcean home, Ubuntu 14.04 sits. I’ve chosen the LTS branch because I’m very much of a set-and-forget mentality when it comes to supporting architecture. I like the idea of not having to worry about my OS becoming outdated in a short window of time. It’ll effectively be supported until 16.04 comes out, and to give you an idea of when that is, 14.10 is currently the newest in line and 15.04 isn’t slated until the end of April for a release date. In essence, I’m good until at least April 2016. 

Layer 3: Web Server

Presenting all of these bits is hard work. That’s why I chose nginx to take up the task. It’s a great server application that I’ve combined with SSL to make every visit to https://jlyman.net secure. Minimal configuration is needed in order to make things hum along smoothly, and I’m glad for that.

Layer 4 1/2: Interpreter

PHP. Given I run WordPress, this is a given. However there are several ways to run it. In my case, I’ve taken then PHP5-FPM route. I could also use FastCGI (the more generic form of FPM) or MOD_PHP (which is actually not that fast in today’s environments). FPM allows me to keep PHP from sucking up too many resources and limiting its worker count. If something gets stuck, I can gracefully shut it down and bring it back up without any major headaches.

Layer 4 2/2: Database

This one is also a given because I run WordPress. MySQL is on the back side hosting the actual content of this site. It’s backed up every hour for good measure and backups are stored in a safe place that can be reached via the outside. There are much more efficient databases however getting WordPress to work with them is rough going if not impossible at this time and I don’t see much changing any time soon.

Layer 5: WordPress

This is the cream on top. WordPress is my CMS of choice and it’s one I’ve used for several years. I remember when I first started using it back on version 2.5. WordPress has come a very long way and I’m grateful that development of such a product continues today with a massive community behind it. 

Layer 6: CloudFlare + SSL

On top of all this, CloudFlare + SSL takes care of all my caching needs. They’re very intelligent about what can be cached and what’s dynamic. I have yet to run into a case where stale content has been served. With their system comes continued use of SSL. This means that from the server to their server is secure, and from their server to you is also secure, no questions. DDoS efforts are limited quickly and geographic-dependencies are removed. Wherever you are, you’ll get quick load times and a lot of burden is taken off my system, which keeps my costs low. I love keeping costs low.


page 1 of 1