Fast local development on macOS with ddev and mutagen
Docker for Mac users know: osxfs is just plain slow. NFS is a step forward, albeit with some tradeoffs. I’ve been experimenting with some different file mounting strategies in ddev, and I think I’ve found a winner.
Note: This blog post is heavily targeted at PHP developers. You may be able to apply the same concepts, but this exact setup may not work for you if you’re using Docker for Mac for e.g. nodejs.
The problem
osxfs (the default filesystem used by Docker for Mac) is notoriously and chronically slow for projects with a lot of little files that need to be loaded often. There have certainly been some speedups over the years, but I still wouldn’t call it usable for e.g. large PHP projects, where hundreds of files need to be loaded from disk by the PHP interpreter on every web request.
Many users have moved to NFS as a solution, and while it certainly solves the performance issue (it was about 45% faster than osxfs in my benchmark), it does so with a couple of tradeoffs. One is that processes inside the container can no longer be notified when files change. This functionality is really important for frontend developers who often use tools like webpack, which watches for changes to frontend code and automatically rebuilds the bundled Javascript and CSS files that get sent to the browser. The workaround for the loss of file change notifications is to switch webpack to polling mode, where it will check all of the code files every so often and rebuild if necessary. Unfortunately, this workaround has proven to be problematic as well – sometimes, webpack will just “miss” changes to some files.
There are a number of other attempts at solutions to this problem (including one that I put together a few years back, docker-bg-sync), but many of them seem to fall flat on either performance or reliability, and for a local dev environment, neither of those are really workable.
A solution
I hesitate to call it “the” solution without a lot more testing, but I’ve been using Mutagen as part of my local environment and I’ve been very impressed. For a quick point of comparison, here are the numbers that I’m seeing on my 2019 16” MacBook Pro:
- Drupal 8 install via Drush – osxfs: 51.5 seconds
- Drupal 8 install via Drush – nfs: 23.5 seconds
- Drupal 8 install via Drush – mutagen: 17.5 seconds
Performance-wise, Mutagen is around 26% faster than NFS for this particular benchmark. From a reliability perspective, I haven’t had any problems either. This setup seems to be just the thing that Docker users have needed for years now.
Okay, so how do I set it up?
I’m so glad you asked. If you’re a ddev user, read on – I’ve automated the process for you as much as I can.
Install dependencies
You’ll need to do a one-time installation of a couple pieces of software in order to make this setup work – Mutagen and jq.
Mutagen
Mutagen has helpfully distributed their product through Homebrew, so installation is a one-liner:
brew install mutagen-io/mutagen/mutagen
After installation completes, run this command to tell the Mutagen daemon to run when you log in to your computer:
mutagen daemon register
jq
jq is a command line JSON processing tool. It’s required by the helper script that you’ll add below. You can install jq
by through Homebrew as well:
brew install jq
Configure your ddev project
Note: You will need to use the most recent ddev release (v1.14.0 at time of writing) in order to perform the following steps successfully.
For every project that you want to use Mutagen with, you’ll need to perform the following steps. Make sure that your project is stopped (ddev stop
) before proceeding.
Tell ddev about mutagen
In your .ddev/config.yaml
, you’ll need to add the following lines:
no_project_mount: true
fail_on_hook_fail: true
hooks:
pre-start:
# Make sure we don't already have a session running; it can confuse syncing
- exec-host: "mutagen sync terminate \${DDEV_PROJECT//.} 2>/dev/null || true"
post-start:
# Start the mutagen sync process for this project.
- exec-host: "ddev mutagen start"
pre-stop:
# Terminate the mutagen sync process for this project.
- exec-host: "ddev mutagen stop"
no_project_mount
is a new config option that was recently merged in (thanks, Randy!) that tells ddev to not try to mount any of your project files into the container. This is important because we’re going to sync them in with Mutagen instead, and if the files are already there, weird things happen!
The hooks block is needed in order to automatically execute the helper script that you’ll add below.
Add the helper script
In your ddev project, put the following script into .ddev/commands/host/mutagen
:
#!/bin/bash
function require_program {
if ! type "$1" > /dev/null 2>&1; then
echo "$1 is required. Please install it and try again."
exit 1
fi
}
require_program "mutagen"
require_program "jq"
if [ "$1" == "start" ]; then
# Don't recreate mutagen sync if it already exists.
if [ -f ".ddev/.mutagen-sync-name" ]; then
echo "Mutagen sync appears to already be running"
exit 0
fi
# Clear out the test files that are bundled with the web container.
ddev exec --dir /var/www rm phpstatus.php
ddev exec --dir /var/www rm -r html/docroot/test
# Make sure the mutagen daemon is running.
# If the daemon is already running, this will fail, but that's okay.
mutagen daemon start 2>/dev/null
# Create the sync process from the ddev project name
ddev describe -j | jq -r .raw.name > .ddev/.mutagen-sync-name
if ! mutagen sync list $(cat .ddev/.mutagen-sync-name) 2>/dev/null; then
mutagen sync create . docker://ddev-$(cat .ddev/.mutagen-sync-name)-web/var/www/html --sync-mode=two-way-resolved --symlink-mode=posix-raw --name=$(cat .ddev/.mutagen-sync-name)
fi
# Wait for the initial sync process to complete, watch for errors, and return
# when ready.
echo "Waiting for initial sync to complete"
while true; do
if mutagen sync list $(cat .ddev/.mutagen-sync-name) | grep "Last error"; then
echo "Mutagen sync has errored -- check 'mutagen sync list $(cat .ddev/.mutagen-sync-name)' for the problem"
break
fi
if mutagen sync list $(cat .ddev/.mutagen-sync-name) | grep "Status: Watching for changes"; then
echo "Initial mutagen sync has completed. Happy coding!"
break
fi
sleep 3
done
fi
if [ "$1" == "stop" ]; then
echo "Ending mutagen sync process"
mutagen sync terminate $(cat .ddev/.mutagen-sync-name)
rm .ddev/.mutagen-sync-name
fi
Don’t forget to make the script executable!
chmod +x .ddev/commands/host/mutagen
Run it!
At this point, you should be able to run ddev start
and have everything work. Under the hood, the hooks you added above are calling the helper script, which sets up a mutagen sync session and waits for the initial sync to complete before letting you proceed. Once the sync session has completed, your files should be synced in and out of the container whenever they’re changed on either end of the connection.
One more thing
I went ahead and automated this setup process for you. Head over to ddev-mutagen and see the readme there for info. The script in that repository performs the exact steps outlined above, but read the script anyway before you run it.