infrastructure-2017

Posts about our ongoing infrastructure upgrade to a Kubernetes, CoreOS and Docker setup.

Infrastructure 2017: Configuring CoreOS and Kubernetes

With the new infrastructure, I wanted the entire setup to be reproducible from a set of configuration files and scripts. This would mean that I can restore a fresh state anytime in the future, and provide a way to track changes in configuration. This also means that I can quickly spawn another Kubernetes setup to test new features safely.

Before I created installation automation scripts, I spent a while learning about how Kubernetes works by manually running the generic multi-node scripts Kubernetes provides, and failing repeatedly. A while back Sudharshan gave me one of his old OEM desktops, and it became really useful for testing Kubernetes installs. CoreOS is best installed using a Ignition configuration file. It's a JSON file with a specific format that CoreOS would read on the first boot and install the appropriate files or configurations specified. Usually, it is preferred to write these files in YML and then transpile them to JSON with a special tool. A great thing about CoreOS

Infrastructure 2017: A new DNS setup

Previously, DNS has been a pain to maintain. I was using a cloud DNS service, so for every subdomain, I would have to log on to CloudNS and use their web interface to update DNS records. This made it hard to switch DNS providers and easily edit records. And their three domain limit made it impossible to add my other domains, thus I had to run another DNS server. I made the decision to run BIND, but I had to run a service to update my zone files when my IP address changes because I didn't purchase a static IP.

To simplify my complex setup, I decided to host all my DNS locally and solve the dynamic IP problem at once. I made the switch from BIND to CoreDNS due to its extensibility. To ensure that parent DNS servers are always pointing to the right IP, I got a couple of free domains (an example is "ns.makerforce-infrastructure.gq") and pointed it to HE's DNS service, where I set up dynamic DNS updates. In my local zone files, I could also use CNAME records to point to the same d

Infrastructure 2017: Router Setup

After getting the hardware and installing embedded pfSense to a flash drive is configuration. My initial intention was to have my servers and home network (which guests use) on separate VLANs. However, I quickly realised that enabling VLANs caused poorer network performance, so I went back to a single network and used static DHCP allocations.

But I was still getting lower than 500Mbps speeds. My CPU was running at a 100%. While messing with settings, I found an odd solution: Enabling PowerD in System > Advanced > Miscellaneous. With it enabled, I could finally get close-to-gigabit speeds on wired clients!

My guess would be that PowerD allowed the CPU to run at higher clock rates.

After using this machine for a few days, I'm satisfied at the performance it delivers, for only $120 SGD! I'd recommend this to you for building a home server or router as a low-cost, low-power setup.

The case for this PC is not rackmounted, so I created a small shelf using a slab of wood and L brackets

Infrastructure 2017: Router Hardware

October last year, I switched my routing to a virtual machine running pfSense, in the hopes of having better control over my home network. Turns out, many hiccups have occurred since the move. Issues with OpenVPN (which I have since disabled), Linux bridges being reassigned after software updates and other seemingly random issues. The virtual network card also caused a reduction in maximum throughput, saturating at 200Mbps instead of the 800Mpbs previously on the RT-N56U.

Since then, I've also been wanting to make the switch from services (like this blog and GitLab) running in user accounts and virtual machines to containers. Containers are isolated environments for processes to run in, providing the isolation of a virtual machine with close to native performance.

So here's the start to a series on the upgrade of our infrastructure to a new setup powered by containers! I'll also be documenting progress and code on GitHub.

To start off the upgrade, I needed a better router (because my sibl