For a long time, I've run my own infrastructure stuff on FreeBSD jails - this stack included Nginx, PowerDNS, Sentry, PostgreSQL, MariaDB, a lsso stack (Redis + Slapd + Osiris or Epitropos), and various other pieces that I am probably forgetting about at the moment. Thinking back, most of it was for the giggleshits and was a huge learning experience because I hand-jammed all of the configuration for each jail and wrote PF rules on the host machine to do some crazy NAT magic. It all worked fairly well for a long time, but the service bring-up after a reboot was an absolute pain. I used iocage to manage the jails, which was okay for a long time as well, but the usage of ZFS metadata to store jail information made things a little slow. While you could control jail bring-up with iocage, as far as I can remember, there was no way to control the order, which was a bit of an issue.
All the while, I had / have been using Docker containers for various other things, either at work, on my local machine, or other servers. The thought of moving all of those things into immutable containers with un-bundled configuration was exciting - less worry about trash from upgrades, containers are easier to manage, no more iocage overhead, etc.. So I "composed" the important parts. Now I've got a suite of
docker-compose stacks that I can use to bring up those parts. When upgrades are needed, just
docker-compose pull. So awesome. No more worrying about upgrading the system in the jail or any of that cruft. I am happier with the compose pieces, which I aptly call composure.
But, if you look through the composure stuff, you'll notice some... oddities. If one service (such as Sentry) exposes a set of web endpoints that need to be served, not only does the app container have to be linked to the backend network for that stack, but also to an external network for the
lb, which is the
frontend-stack with nginx. But also, the nginx container from the
frontend-stack has to be linked to the other container so that the container name (
sentrystack_app_1) can be used in the upstream config which is created by
confd. It is not horrible, but it is not as smooth and automated as I wished.
confd has to be reloaded to pick up the new template configuration and resource. And if you (or I) forget to generate the SSL certificate with
scripts/bootstrap-domain-ssl.sh from the
frontend-stack, then the nginx container starts flapping and the faulty configuration needs to be removed and the stack needs to be restarted before you can actually do the bootstrap. Phew.
composure is significantly better than hand-jamming all the things in jails, it's not completely what I had hoped for.
The next logical step for me is to attempt to move all the composure stuff to Kubernetes.
That comes after final exams are finished and life settles down. Stay tuned!
In case you're reading this and wondering "why am I even reading this?" - This will be a short series of posts on my move from jails to containers to Kubernetes-orchestrated containers. I hope to learn a bit about the Kubernetes ecosystem and spread some of that knowledge back out in a easily digestible way, because Kubernetes is kinda hard.
If you've got any comments, suggestions, questions, etc., please feel free to drop me an email!
EDIT (06/20/17): This blog runs on Kubernetes. ;)