Developing on Apple M1 Silicon with Virtual Environments

Docker containers all the way down

In my article Creating Repeatable Development Environments, I showed how I use Vagrant as an orchestrator and VirtualBox as a provider of virtual machines for creating consistent development environments for my students and development teams. That worked really well until Apple released their new 2020 Macs with Apple M1 Silicon chips based on the ARM architecture. I had selected VirtualBox because it was free and supports Mac, Linux, and Windows, but it only runs on Intel computers (x86_64 architecture) and Apple Silicon is ARM base (aarch64 architecture). As it turns out, 8 of my students showed up for the 2021 spring semester with Apple M1 Macs which meant that all of my labs based on VirtualBox were not going to work for them. I need another solution and I needed it fast.

Docker on Apple Silicon

I had heard that Docker had released a tech preview of Docker Desktop for Mac that runs on Apple Silicon. I also remembered that Vagrant supports Docker as a provider. This means that Vagrant can control the provisioning of Docker containers just like it controls VirtualBox for provisioning virtual machines. This is somewhat of a unique use case for Docker because the intent of Docker is to provide a consistent, immutable runtime environment; not to be treated like a virtual machine. As with all technology, there are always use cases beyond the original intent and I was about to learn if this use case was viable.

The concept of using Docker as a development environment is not new. In fact, if you use Visual Studio Code, there is an extension called Remote Containers that allows you to develop in containers. I plan to cover that in a future article. The difference here is that normally you would use the docker exec command to establish a shell inside the container but I needed these containers to behave exactly like a virtual machine (VM) because I didn’t want students or developers with Apple M1 Macs to have a different experience than those on an Intel Mac or Windows computer. That means I had to get ssh to work with my container because that is how vagrant expects to establish a shell inside of the VM.

Coercing a Container to behave like a VM

Everyone will tell you to never install an ssh daemon (sshd) inside of a Docker container. In fact, I am one of those people. Normally, you should not do this when using containers for their intended purpose of providing immutable runtimes. To do so would make them mutable and less secure. But this is different. These are development containers.

There was also another problem to overcome. Docker containers don’t normally run a linux init system like a virtual machines does because containers are intended to only run one process, but many of the tools I needed to install expected an init system. Since I was using Ubuntu, I need to get systemd (Ubuntu’s init system) running as the first process (PID 1) in order to get the containers to truly act like a virtual machine. Easier said than done.

Docker makes it very hard to use systemd inside a container but lucky fo me, Matthew Warman had already figured this part out for CentOS and was extremely helpful in getting this to work for me with Ubuntu. With his guidance, I was able to build an Ubuntu 20.04 Docker image that was tailored for my purposes. You can use this image as a vagrant provider to start your own Ubuntu Docker containers that behave like virtual machines for doing development work, compete with an init system.

Building a Multi-Architecture Docker Image

The last hurtle was to build a Docker image that would run on both Intel and ARM based computers so that my students and development teams could use it on either platform. It was surprisingly easier than I thought. I was even able to build a Docker image on my Intel Mac that would run on both Intel and an Apple Silicon Mac by using an experimental Docker feature called buildx.

These are the commands I used to build my vagrant provider Docker image for both amd64 and arm64 architecture.

First you must prepare a buildx builder to use:

$ export DOCKER_BUILDKIT=1
$ docker buildx create --use --name=qemu
$ docker buildx inspect --bootstrap

This creates a buildx container to do the work for you. Then when you build your image, you must specify the platforms that you want build and push them to docker hub (or other registry) to get an image:

$ docker buildx build --platform linux/amd64,linux/arm64 --push --tag rofrano/vagrant-provider:ubuntu .

The command above uses QEMU to build a multi-platform image that will work on both Intel (amd64) and ARM (arm64), and push it to my docker hub account. If you need to support more platforms you can just add them to the --platform parameter separated by a comma.

Final Solution

You can find the final Docker image here on Docker Hub: rofrano/vagrant-provider. If you plan to use it with Vagrant, there is no need to pull it down from Docker Hub because that will happen automatically when vagrant processes your Vagrantfile.

Here is a sample multi-provider Vagrantfile that can use either VirtualBox or Docker as a provider with vagrant:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"
  config.vm.hostname = "ubuntu"
  # Provider for VirtualBox
  config.vm.provider :virtualbox do |vb|
    vb.memory = "1024"
    vb.cpus = 2
  end
# Provider for Docker
  config.vm.provider :docker do |docker, override|
    override.vm.box = nil
    docker.image = "rofrano/vagrant-provider:ubuntu"
    docker.remains_running = true
    docker.has_ssh = true
    docker.privileged = true
    docker.volumes = ["/sys/fs/cgroup:/sys/fs/cgroup:rw"]
    docker.create_args = ["--cgroupns=host"]
  end
  # Provision Docker Engine and pull down PostgreSQL
  config.vm.provision :docker do |d|
    d.pull_images "postgres:alpine"
    d.run "postgres:alpine",
       args: "-d -p 5432:5432 -e POSTGRES_PASSWORD=postgres"
  end
end

To use vagrant with Docker instead of VirtualBox, just specify the --provider parameter with docker as the provider:

$ vagrant up --provider=docker

This will bring up a container that behaves like a virtual machine on Intel or ARM computers. You might have noticed that I’ve added a provision block for Docker in the Vagrantfile. Yes that’s right, we are running Docker inside of our Docker container so that students and developers can work with Docker while inside the virtual environment just like we do with real VMs. In this example we are running PostgreSQL in a Docker container for development work.

To get inside of this development environment you use the same vagrant command as when developing with a VM:

$ vagrant ssh

This will ssh into the Docker contain just as if it were a virtual machine.

Conclusion

Using Docker as a provider for Vagrant has allowed me to create consistent developer environments for my students and development teams regardless of whether they have an Intel computer or an ARM based computer like the new Apple M1 Silicon Macs. Getting a base Docker image to use systemd as its entry point allows the container to behave much like a virtual machine. We even have Docker running inside the container so that we can leverage it for databases and other software during development. Other solutions allow for developing in containers without vagrant, but we’ll leave that for future article.