Creating Reproducible Development Environments

Using Vagrant, VirtualBox, and Docker

Photo by Caspar Camille Rubin on Unsplash

Development teams are up against an age old problem: how do you ensure that your development environment is the same across all of the developers on the team? A less frequent but equally important problem is how long does it take to on-board a new developer to a project? Hours? Days? What if I told you that you could do it in minutes!

Works on My Machine ¯\_(ツ)_/¯

Development environments are often fragile. If you just install all of the software directly on your computer, you run the risk of upgrades for one project causing another project to stop working. Even worse, you sometimes get the dreaded “works on my machine” excuse from someone on your team but it doesn’t work on yours, or it works on your machine but not in production. What if you could automate the creation of consistent local development environments right on your laptop or desktop using virtual machines with little or no effort?

Automate Everything: Even the Developer

Carefully and painstakingly creating documentation that gives step-by-step instructions on how to recreate the development environment via cut-n-paste is not very agile. Remember, we value working software over comprehensive documentation. One of the tenets of DevOps is automation in the form of Infrastructure as Code. This is the practice of not hand-crafting infrastructure like servers and networks but rather, using a provisioning tool like Terraform and/or a configuration management solution like Ansible, Chef or Puppet to automatically deploy and configure the infrastructure needed to make your software run using code instead of documentation. But what about the developers? Shouldn’t their environments be automated as well?

I teach a graduate course on DevOps and Agile Methodologies at New York University and this has particular implications in a classroom setting where students show up with Windows PCs, Macs, and even Linux laptops and the requirements for the course may differ from another course that students are taking. They are expected be able to work on projects from both courses simultaneous. This is no different than developers being on multiple projects at the same time. There are various language specific solutions to this problem, but wouldn’t it be great to solve this in a language agnostic way?

Coca-Cola Wars: I can’t take it anymore

Pepsi — Coke, Paper — Plastic, Windows — Mac, so many choices, so little productivity. To solve this problem in the classroom and with my teams at work, I use the trilogy of Vagrant, VirtualBox, and Docker to have all of my developers and students develop on Linux. If Infrastructure as Code is good for production, then why not use it to automate the provisioning of developer environments right on their laptops? Also, it’s good to get developers in the habit of working in a headless server environment because this is what they will encounter while debugging when production workloads in the cloud are misbehaving.

Vagrant is a developer tool for automating the creation of lightweight, reproducible and portable virtual environments via a command-line interface. VirtualBox is a free hypervisor which runs on macOS, Windows, and Linux, that can host virtual machines on your laptop or desktop computer. In the past, I have created virtual machines for my development teams, exported them as OVA files, and had to upload huge multi-gigabyte OVA images that they would have to download in order to use. This would take me hours. When software needed to be updated, I had to create a new image and upload it, and developers needed to download it again taking up even more hours. With Vagrant, I only have to place a small file in each of our git repositories called a Vagrantfile and everything else is taken care of for the developer. Updates are as simple as updating a text file that is tracked via version control.

Where to begin?

To get started, I have my development teams and students install two pieces of software: Vagrant and VirtualBox. That’s it! Of course, they will need a programmer’s editor like Visual Studio Code but for the development environment they only need Vagrant and VirtualBox. This means that the time to on-board a new developer or student is as long as it takes to install Vagrant and VirtualBox and that can be automated with two simple commands using Homebrew on Mac or Chocolatey on Windows:

On macOS using Homebrew:

$ brew install --cask vagrant
$ brew install --cask virtualbox

On Windows using Chocolatey:

C:\> choco install vagrant
C:\> choco install virtualbox

Of course you will need to have installed Homebrew for Mac or Chocolatey for Windows but if you don’t have them, you’re going to love them because they keep your software up to date automatically. IMHO, every developer should be using these command line tools, or tools like them, to manage software installations and increase their productivity. If you don’t want to use these tools, you can always download Vagrant and VirtualBox from their respective web site and install them manually.

Vagrant is the Puppet Master

Vagrant provides the automation that pulls the strings that control everything. It is used to quickly provision complex configurations consistently with repeatability. VirtualBox provides the Virtual Machines (VM) for the developers to work in, and Docker handles all of the middleware without any installation required. I have been using this trifecta for quite a while with great success. It doesn’t matter if your laptop runs on Windows, or macOS, or Linux. Everyone is using a consistent Linux environment that they can’t get wrong, because it’s all automated. No muss, no fuss, no wrong versions of anything. Everyone is working on exactly the same environment.

Once Vagrant and VirtualBox are installed, creating a complete development environment for working on a project or lab is as simple as these commands:

$ git clone https://github.com/rofrano/lab-vagrant.git
$ cd lab-vagrant
$ vagrant up
$ vagrant ssh

That’s it! It doesn’t matter what software needs to be installed for a project, it’s hidden from the developer. It doesn’t matter that different projects have different and sometimes conflicting requirements, it’s all isolated by the virtual machine. Everything the developer needs has been installed and configured consistently and repeatably. You can try the commands above to see for yourself. This is the repository from the tutorials that I have run at several conferences like this one at the Lisa16 conference in Boston on Making Developers More Productive with Vagrant, VirtualBox, and Docker

To start your own empty development environment from scratch, create a folder for your work and use:

$ vagrant init ubuntu/bionic64
$ vagrant up
$ vagrant ssh

This will initialize a new Vagrantfile with the box ubuntu/bionic64 and bring up a virtual machine using the Ubuntu 20.10 Bionic64 image. The Vagrantfile is a ruby file that contains the instructions on how to provision and configure the virtual machine. Initially it contains just the box to use with sample code that you can uncomment or add additional commands to. You can learn more about these commands from the Vagrant documentation. The box is the virtual machine image to start with. Vagrant has a number of images to choose from that you can browse for at Vagrant Cloud

One file to rule them all

The real magic comes from editing the Vagrantfile and adding commands to install, configure and conjure up your own environment. You can use shell commands but Vagrant can also use files from popular configuration management systems like Ansible, Chef, and Puppet. Think about that for a moment. The same files that are provisioning the production servers can configure the developer’s environment as well. That is how you get 12-Factor App Dev/Prod parity.

Below is a simple Vagrantfile that will create a Python 3 development environment with git and a PostgreSQL database using Docker.

Vagrant.configure(2) do |config|
  config.vm.box = “ubuntu/bionic64”
  config.vm.network “forwarded_port”, guest: 8080, host: 8080
  config.vm.network “private_network”, ip: “192.168.33.10”
  config.vm.provider “virtualbox” do |vb|
    vb.memory = “1024”
    vb.cpus = 2
  end
  if File.exists?(File.expand_path("~/.gitconfig"))
    config.vm.provision "file", source: "~/.gitconfig", destination: "~/.gitconfig"
  end
  config.vm.provision “shell”, inline: <<-SHELL
    sudo apt-get update
    sudo apt-get install -y git python3-dev python3-pip
  SHELL
  config.vm.provision :docker do |d|
    d.pull_images “postgres:alpine”
    d.run “postgres:alpine”,
      args: “-d — name postgres -p 5432:5432 -v  postgresql:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres”
  end
end

This Vagrantfile provisions an Ubuntu 20.10 virtual machine, forwards port 8080 on the VM to 8080 on your host computer so that you can use localhost:8080 to get to your application. Substitute whatever port your application listens on. It also gives the VM an IP address, and allocates 1GB of memory and 2 CPUs. It then copies your .gitconfig file into the VM so that git knows who you are, and then installs git and a Python 3 development environment. Finally, it uses Docker to provision a PostgreSQL database in a Docker container. Docker support is native to Vagrant which is why we didn’t need to download and install Docker on our laptops. We use Docker from within the virtual machine so that each of our Docker environments is also isolated from each other.

If you need more software, you simply add the commands to your Vagrantfile and then use:

$ vagrant reload --provision

This will rerun the provisioning blocks again and reinstall the software. It should be noted that once you provision a virtual machine, Vagrant will ignore the provisioning blocks the next time you use vagrant up so the existing virtual machine comes up quickly because everything is already installed.

Committing the Vagrantfile to your git repo each time it changes ensures that every developer on the team will bring up the same consistent environment each and every time.

When you want to bring the environment down, simply use:

$ vagrant halt

This will stop the virtual machine. Then vagrant up will start it up again when you need it. You can also use vagrant suspend and vagrant resume to preserve the current running state instead of rebooting.

When the development environment is no longer needed, or if everything goes horribly wrong, you can delete the virtual machine with:

$ vagrant destroy

That is the beauty of infrastructure as code. When something goes wrong and “it worked yesterday but not today”, you don’t have to waste time trying to fix it. Just delete the VM and provision a new one with:

$ vagrant destroy
$ vagrant up

This will give you a new development environment that is as fresh as the day you first started.

What about my files?

This is where Vagrant really shines. Right about now you might be thinking, “Don’t I have to copy my files into the virtual machine?” or “If I delete the virtual machine won’t I lose all of my work?” The answer to both of these questions is “No”. Here’s why:

Vagrant shares your current working directory inside the virtual machine at the mount point /vagrant. Your project files are never copied into the virtual machine. They are shared with the virtual machine so that any change you make either outside the virtual machine or inside the virtual machine is seen in both places because there really is only on place that the files reside and that’s on your laptop or desktop.

Once you ssh into the virtual machine you can find your project files under the /vagrant folder. In this example you can see the Vagrantfile that you created on your computer from within the virtual machine.

$ vagrant ssh
$ cd /vagrant
$ ls -l
-rw-r—r-- 1 vagrant vagrant  4715 Apr  7 12:52 Vagrantfile

Because the files are shared inside of the virtual machine, you can use your favorite development editor on your computer like Visual Studio Code to edit the files while executing the program from inside the virtual machine. This gives you the best of both worlds, a portable development environment to run your code that is similar to the production environment with the familiarity and convenience of working with your favorite GUI editor.

Conclusion

Hopefully I’ve built the case for using an infrastructure as code solution like Vagrant with your development teams in order to not only automate but have reproducible development environments that eliminate the “works on my machine” syndrome. The productivity boost is cumulatively significant when you consider the time each individual developer spends managing their development environment. This also reduces the friction for onboarding new hires, or when people switch teams.

In addition to using VirtualBox as a provider, Vagrant also supports VMware, SoftLayer, Amazon AWS and Digital Ocean. This means that your developers can instantly build cloud based development environments for themselves as well. If you use the Visual Studio Code Remote Development extension, you can actually develop inside of the cloud virtual machine. I’ll leave that for another blog post.