Live Streaming with Hardware Acceleration using a Raspberry Pi and RTMP/HLS

If you’ve been following my blog post series on the development of my ever so useful cat cam, powered by a Raspberry Pi, you’ll know I’ve made several attempts at a more stable and scalable streaming solution for my Cat Cam. As it stands today, I’ve been using Motion. While it’s a decent tool, Bandwidth has been my primary concern and I’d like to be able to stream real-time without sucking up what measly bits my ISP gives me if more than a few folks decide to show interest.

Continue reading “Live Streaming with Hardware Acceleration using a Raspberry Pi and RTMP/HLS”

Add RTMP Support to Nginx Installed From Apt

In the process of trying to figure out the best streaming solution for my cat cam, I had to deviate a bit. I combined my RTMP server for my cat cam and Web server for john.ly into one and the latter didn’t have the RTMP module installed. This module is required for my attempts to push H.264 video and have Nginx relay it to whomever is watching, cutting down on the bandwidth of my one-to-one reverse proxy setup I have, now.

Continue reading “Add RTMP Support to Nginx Installed From Apt”

Fighting ffmpeg

Before I begin this seriously long-winded article (I wasn’t expecting it to be nearly 4000 words), I want to let you know that after roughly six hours of poking, yelling at myself, yelling at my screen, yelling at Google for not having the answers I need1, and wishing I had just bought IP cameras, I realized ffserver doesn’t send mjpeg with the right Content-type header for browser viewing, thus said browsers don’t do the right thing to it.2

Continue reading “Fighting ffmpeg”

Blogging, Statically Speaking

The concept of blogging using a staic site generator isn’t news. People have been doing such things for years, now. There are fantastic static site generators out there (Jekyll, Hugo, Pelican to name a few) and all have their unique features, quirks, and ugh moments.

I wrote a post a wrote a post a while back about moving over to Jekyll from WordPress. Within a couple weeks or so, I had moved back. I never posted why, but the reason boiled down to the generation time. It took Jekyll upwards of 90 seconds to two minutes to generate every time I wanted to updated something–anything, really. New to Jekyll 3, the --incremental argument didn’t do anything and I wasn’t comfortable leaving Jekyll running in --serve mode in the background1.

Needless to say my views on running this blog with a static site generator are a little jaded. Recently the idea of using Hugo (written in Go) was suggested and I gave it a whirl. Go was wasy easier to set up and get running. I also found myself looking at several front-end UIs that can be bolted onto Hugo so I don’t have to edit .md files all over the place then work out some git magic to push changes and have them be re-deployed2.

There was one component that I just couldn’t get into this potentially new workflow, though, and that was integrating any of my writing apps without having to also introduce manually saving an editing. I use Desk for writing my blog posts, including this one, and it does a killer job. Within a few clicks, my post is live and I didn’t have to git push a damn thing.

Really it seems as though I’m torn. I like the idea of speed and simplicity. I like the idea of not having to maintain a MySQL database and PHP. I don’t like the idea of trading that work (something I’m already super familiar with and have down pretty well for my needs) for a different kind of maintenance. I can deal with maintaining a server, but if it takes more than minimal effort to get a blog post up, I’m not sure I’d be interested in the alternative.

Did I mention the migration? Never have I ever had my 270 something (as I post this) posts all move over without at least moderate tweaking required. At this point, the amount of effort required to transition would negate any savings. And I’m kind of lazy.

I doubt I’m the only one that feels this way and I’m sure there are a ton of other ways to go about solving this problem. I might never go back to a static site generator, again, and I’m ok with that. If WordPress becomes faster, hooray! If it is no longer written in PHP, I’d feel even better about it. In the meantime, I’ll continue doing things the way I’ve done them since January 2015, almost 18 months ago.

  1. As much as I don’t mind writing custom service scripts for systemd, I don’t feel like I should have to resort to that just to make a quick page update
  2.  I did this with Jekyll, and I’ll share that in a future post.

The CDN Contemplation

About four weeks ago, I bolted a CDN to my blog. The goal: what kind of benefit would I, someone who receives a small but steady stream of traffic, get from such a technological underpinning?

This CDN is nothing fancy. Powered by MaxCDN, It takes my site assets (CSS, JavaScript, and images) and tosses them around the world to their edge servers. In exchange, I can load said assets from cdn.john.ly and an origin server shall provide.

Continue reading “The CDN Contemplation”

Harder Than it Should Be: Jekyll Word Count

When I was running this site through WordPress, I had a plugin that would count the number of words each of my posts contained and give me some metrics. It was a pretty slick plugin and had all sorts of visuals.

With Jekyll, I don’t have such capabilities out of the box (or even remotely close to the box) so I went hunting for a plugin. I found one, and it works, but it’s slow, and I don’t really think there’s much that can be done about it. Given the static nature of Jekyll, it’s not really easy to save persistent information somewhere like in a database without also having a plugin for the database.

The plugin brought down my build to a crawl (140s over 13s) and GitHub didn’t whitelist the plugin, so it was pretty much a non-starter.

Needless to say, I pulled it, but if you want to see the commit where I added it, go here (then the subsequent commits here and hereHere’s where I pulled it).

For the record, before this post: 92,205 words. With this post: 92,392 words.

My Overloaded Server Story

And the server belongs to me

So as you probably know by now, I converted my blog to Jekyll, yesterday, and it’s been a huge success, in my opinion. I’m motivated more than ever to start blogging more because I have the added tinker factor and using Git and GitHub to keep everything organized rocks.

But that isn’t what this story is about. Tonight’s story is about a server. A lowly VPS floating somewhere in the SFO DigitalOcean…. ocean. I’ve had this server running for many months–probably over a year now and I just don’t know it–and it’s hosted just about anything and everything I’ve tinkered with including many a failed idea.

I use this server a lot and for the most part it always responds well and isn’t sluggish. It’s hosted my former WordPress blog since January and hasn’t made but a peep about it so I’ve spent quite a bit of time thinking nothing’s amiss. I felt, though, that Jekyll shouldn’t be taking as long as it was to build my site but I couldn’t convince myself the server was overloaded.

As it Turns OutTM, I was wrong.

My baby of a server has, for the last five or so months, been filling Newrelic graphs with stuff like this:

CPU

loaded_server_before_cpu

 

Memory

loaded_server_before_mem

 

Disk

loaded_server_before_disk

 

Load

loaded_server_before_load

 

So I’m just going to go ahead and say that’s not good. The only graph above that’s even remote decent is the Disk graph, but it’s washed over by the CPU > 80% indication so it’s also pretty much hosed.

At this point I’m honestly surprised. I’m a terrible pseudo-sysadmin and I should be fired but there’s no one to fire me and I’m the boss so whatever. If I can fix all this, I’m giving myself a raise.

I started digging. I wanted to see what’s running and who’s sucking up all the juices. I fired up top and waited a few seconds. A few things popped up: a couple instances of node and an instance of ruby.

Hmm. That doesn’t make any sense… I’m not running any node apps and jekyll is the only ruby thing I use right now… oh wait.

See, I tinkered with NodeJS apps late last spring. Ghost was a blogging tool I was considering for a while. I also tinkered with Discourse to see what it was all about. Turns OutTM, neither are suited for my needs.

I don’t really know what happened but I can only assume I forgot about them and they’ve been running all this time. I had a total of five Node instances running and Docker was running Discourse so between the two of them, I was on swap 24/7. So dirty.

This story isn’t super climactic in any way and the fix was easy: kill all the things. I also found this to be a good time to remove Node and Docker since I need neither.

I still couldn’t figure out why my disk usage was so high, though. Leave it to me to forget about a 4GB .iso I left in a folder and leave it to Discourse to fail at sending 5GB of emails through postfix over the last five months and clog up my /var/mail folder. Needless to say, after the almost-winter cleaning, I gained quite a bit of ground:

CPU

loaded_server_after_cpu

Memory

loaded_server_after_mem

Disk

loaded_server_after_disk

Load

loaded_server_after_load

Sorry this story wasn’t more interesting. If you’re curious, my Jekyll build time was cut in half.

That’s about it. Orphaned tools and processes makes my server a dull VPS.

How I Made Jekyll Post Generation Just a Touch Easier

Now that I’m running my blog using Jekyll, one thing I’ve already found to be rather frustrating is the post generation process. I have a blank .md template I open, save in a new location, then edit, but that seems cumbersome, to me. What I decided to do instead is write a quick Ruby script that generated a post .md file for me based on the information I provide.

Thought Process

I wanted to keep it simple, and just do only what I really needed. I don’t need any fancy logic or checking. I know _posts will be there because I put post_generator.rb inside my Jekyll directory.

Working Example

Here’s my code as it stands inside right now:

It’s functional. It’s not clean and could be refactored a but, but it works.

Improvements

A few things that came to mind after I finished: – Use the system date if none is provided – Re-format the title with title-casing. Without ActiveSupport in Rails, I’ll have to either require it as a gem or write something by hand. I’m thinking the former. – Allow the user to write the post right there in the command line and not have to open a text editor. – Allow the user to choose which text editor to use at the prompt (perhaps with detection?)

It’s a good first draft and it serves the purpose I had in mind. Here’s the GitHub repo.