A Look Inside Our Infrastructure

Published March 14, 2017

Gone are the days of system administrators desperately trying to make sure their servers stay healthy. Services like Amazon Web Services (AWS) allow modern operations staffs to write code to describe the machines and then create new servers on-demand. Here, we will explore how we do just that at Unizin.

Security

We will start with the most crucial component of any serious DevOps practice: security. At Unizin, this takes two forms. First, we have instance-level security which can be thought of as a firewall. In AWS parlance, we call it a security group. Security groups explicitly allow network access to a server or service. That means the default behavior allows nothing to connect to a server. You have to explicitly opt-in to a server being reachable as opposed to opting out of access. The second level of security we have against outside attacks is network-level security. AWS allows you to create a private network within AWS in which all your instances reside. So, for instance, all of our servers reside in the 172.31.0.0/16 network. That means we have roughly 65,536 IP addresses which only servers in that same network can reach. Because all of our instances are in the private network, we can elect to take the public IP address away from our internal services. A malicious party can’t attack our servers if they can’t even reach them.

Infrastructure as Code

One of the guiding principles in modern Developer Operations practice, DevOps, says, “Your server is a phoenix, not a snowflake.” What we mean is that it’s often better to terminate your server and bring a new one up instead of treating your current server like a special snowflake. To do this, we need a way of telling AWS exactly what our servers should look like and how they should behave. Furthermore, we should be able track what our infrastructure looked like in the past so that we can figure out if a specific change caused a bug in the system. Code fits this role precisely. With AWS, we define our infrastructure as code using a service called CloudFormation. CloudFormation takes a JSON or YAML file, parses it, and creates servers and services based off that template. For instance, in a CloudFormation template, you could say that you want a web-facing server to serve data from a PostgreSQL database. CloudFormation then takes that codified description and creates an EC2 server and a RDS database running PostgreSQL. The real key here is that if that EC2 server fails in some way, we can terminate it and use that same code to create a working server in its place. The server is a phoenix, not a snowflake.

Automated Deployment

The second guiding principle in DevOps practice is simple: Automate everything. The most obvious benefit of automation is that it makes your job easier. You build the automation once and then you don’t have to do that job manually ever again. The second benefit is that automation eliminates the possibility of human error. Humans may make mistakes when they do something manually. Computers will never make a mistake. At Unizin, we have completely automated the process of deploying our developer’s code to a development environment which matches the topology of our production environment almost exactly. After a developer has their code merged into the master branch of our source code management system, our deployment server, Jenkins, automatically tests that code and spins up new servers which use that code. This is largely a three-step process. First, Jenkins runs the unit and integration tests locally (on the Jenkins server) to make sure the new code does not break any existing code. Once the new code passes those tests, Jenkins uses a tool called Ansible to send CloudFormation templates to AWS so that AWS can create new servers for us. After the new servers are made, we have written scripts which allows these new servers to automatically download the new code from AWS’s storage service, S3, and install the new code on itself.

Modern system engineering has come a long way from the traditional view of a system administrator creating and maintaining handmade servers. At Unizin, we focus on documented, secure, and reproducible infrastructure. We document our infrastructure as code which allows us to automatically deploy our software with a baked-in, multi-layered security strategy. If you have any questions about the DevOps practice at Unizin, please contact Howie Benefiel.

Howie Benefiel is a Site Reliability Engineer at Unizin responsible for infrastructure, automation, and implementing our DevOps practice across Engineering.