Fast local development on macOS with ddev and mutagen

osxfs is just plain slow. NFS is a step forward, albeit with some tradeoffs. I’ve been experimenting with some different file mounting strategies in ddev, and I think I’ve found a winner.

Note: This blog post is heavily targeted at PHP developers. You may be able to apply the same concepts, but this exact setup may not work for you if you’re using Docker for Mac for e.g. nodejs.

The problem

osxfs (the default filesystem used by Docker for Mac) is notoriously and chronically slow for projects with a lot of little files that need to be loaded often. There have certainly been some speedups over the years, but I still wouldn’t call it usable for e.g. large PHP projects, where hundreds of files need to be loaded from disk by the PHP interpreter on every web request.

Many users have moved to NFS as a solution, and while it certainly solves the performance issue (it was about 45% faster than osxfs in my benchmark), it does so with a couple of tradeoffs. One is that processes inside the container can no longer be notified when files change. This functionality is really important for frontend developers who often use tools like webpack, which watches for changes to frontend code and automatically rebuilds the bundled Javascript and CSS files that get sent to the browser. The workaround for the loss of file change notifications is to switch webpack to polling mode, where it will check all of the code files every so often and rebuild if necessary. Unfortunately, this workaround has proven to be problematic as well – sometimes, webpack will just “miss” changes to some files.

There are a number of other attempts at solutions to this problem (including one that I put together a few years back, docker-bg-sync), but many of them seem to fall flat on either performance or reliability, and for a local dev environment, neither of those are really workable.

A solution

I hesitate to call it “the” solution without a lot more testing, but I’ve been using Mutagen as part of my local environment and I’ve been very impressed. For a quick point of comparison, here are the numbers that I’m seeing on my 2019 16” MacBook Pro:

  • Drupal 8 install via Drush – osxfs: 51.5 seconds
  • Drupal 8 install via Drush – nfs: 23.5 seconds
  • Drupal 8 install via Drush – mutagen: 17.5 seconds

Performance-wise, Mutagen is around 26% faster than NFS for this particular benchmark. From a reliability perspective, I haven’t had any problems either. This setup seems to be just the thing that Docker users have needed for years now.

Okay, so how do I set it up?

I’m so glad you asked. If you’re a ddev user, read on – I’ve automated the process for you as much as I can.

Install dependencies

You’ll need to do a one-time installation of a couple pieces of software in order to make this setup work – Mutagen and jq.

Mutagen

Mutagen has helpfully distributed their product through Homebrew, so installation is a one-liner:

brew install mutagen-io/mutagen/mutagen

After installation completes, run this command to tell the Mutagen daemon to run when you log in to your computer:

mutagen daemon register

jq

jq is a command line JSON processing tool. It’s required by the helper script that you’ll add below. You can install jq by through Homebrew as well:

brew install jq

Configure your ddev project

Note: You will need to use the most recent ddev release (v1.14.0) in order to perform the following steps successfully

For every project that you want to use Mutagen with, you’ll need to perform the following steps. Make sure that your project is stopped (ddev stop) before proceeding.

Tell ddev about mutagen

In your .ddev/config.yaml, you’ll need to add the following lines:

no_project_mount: true
hooks:
  post-start:
    # Start the mutagen sync process for this project.
    - exec-host: "ddev mutagen start"
  pre-stop:
    # Terminate the mutagen sync process for this project.
    - exec-host: "ddev mutagen stop"

no_project_mount is a new config option that was recently merged in (thanks, Randy!) that tells ddev to not try to mount any of your project files into the container. This is important because we’re going to sync them in with Mutagen instead, and if the files are already there, weird things happen!

The hooks block is needed in order to automatically execute the helper script that you’ll add below.

Add the helper script

In your ddev project, put the following script into .ddev/commands/host/mutagen:

#!/bin/bash

function require_program {
    if ! type "$1" > /dev/null 2>&1; then
        echo "$1 is required. Please install it and try again."
        exit 1
    fi
}

require_program "mutagen"
require_program "jq"

if [ "$1" == "start" ]; then
    # Don't recreate mutagen sync if it already exists.
    if [ -f ".ddev/.mutagen-sync-name" ]; then
        echo "Mutagen sync appears to already be running"
        exit 0
    fi

    # Clear out the test files that are bundled with the web container.
    ddev exec --dir /var/www rm phpstatus.php
    ddev exec --dir /var/www rm -r html/docroot/test

    # Make sure the mutagen daemon is running.
    # If the daemon is already running, this will fail, but that's okay.
    mutagen daemon start 2>/dev/null

    # Create the sync process from the ddev project name
    ddev describe -j | jq -r .raw.name > .ddev/.mutagen-sync-name
    if ! mutagen sync list $(cat .ddev/.mutagen-sync-name) 2>/dev/null; then
        mutagen sync create . docker://ddev-$(cat .ddev/.mutagen-sync-name)-web/var/www/html --sync-mode=two-way-resolved --symlink-mode=posix-raw --name=$(cat .ddev/.mutagen-sync-name)
    fi

    # Wait for the initial sync process to complete, watch for errors, and return
    # when ready.
    echo "Waiting for initial sync to complete"
    while true; do
        if mutagen sync list $(cat .ddev/.mutagen-sync-name) | grep "Last error"; then
            echo "Mutagen sync has errored -- check 'mutagen sync list $(cat .ddev/.mutagen-sync-name)' for the problem"
            break
        fi
        if mutagen sync list $(cat .ddev/.mutagen-sync-name) | grep "Status: Watching for changes"; then
            echo "Initial mutagen sync has completed. Happy coding!"
            break
        fi

        sleep 3
    done
fi

if [ "$1" == "stop" ]; then
    echo "Ending mutagen sync process"
    mutagen sync terminate $(cat .ddev/.mutagen-sync-name)
    rm .ddev/.mutagen-sync-name
fi

Don’t forget to make the script executable!

chmod +x .ddev/commands/host/mutagen

Run it!

At this point, you should be able to run ddev start and have everything work. Under the hood, the hooks you added above are calling the helper script, which sets up a mutagen sync session and waits for the initial sync to complete before letting you proceed. Once the sync session has completed, your files should be synced in and out of the container whenever they’re changed on either end of the connection.

One more thing

I went ahead and automated this setup process for you. Head on over to ddev-mutagen and see the readme there for info. The script in that repository performs the exact steps outlined above, but read the script anyway before you run it.

Current Projects

Composer Configurable Plugin

Provides a light weight configuration system for Composer plugins. This project is mainly targeted at people writing plugins for Composer.

Composer Patches

A simple plugin that applies patches to Composer dependencies. Patches can be applied from a local or remote patch file. This plugin is widely used in the Drupal, Magento, and Symfony communities.

Past Projects

CHIP-8 Emulator

A CHIP-8 emulator written in Go. It’s not a complete implementation of CHIP-8 but it’s good enough to run some basic ROMs.

ddev mutagen

Docker for Mac has an unbearably slow filesystem, so I wrote a little script that integrates mutagen and ddev to get near-native filesystem performance.

Project Updates

Share on twitter
Twitter
Share on pocket
Pocket

5 Responses

    1. This method does not rely on built-in Mutagen support in Docker. It’s completely standalone and should continue working regardless of what the Docker team does with their filesystem.

  1. Any reason why would I get this? Using Docker 2.5.0.1 and ddev version v1.16.0-rc3. I was following your instructions.

    === Running task: Exec command ‘ddev mutagen start’ on the host (MacBook-Pro-2.local), output below
    Error: unknown command “mutagen” for “ddev”
    Run ‘ddev –help’ for usage.
    task failed: Exec command ‘ddev mutagen start’ on the host (MacBook-Pro-2.local): exit status 255
    A task failure does not mean that ddev failed, but your hook configuration has a command that failed.

    1. I think I solved it. I noticed this in the documentation:
      The command filename is not what determines the name of the command. That comes from the Usage doc line (## Usage: commandname).

      So I added “## Usage: mutagen” at the beginning (in the second line) of the mutagen file.
      After that it worked.

      br,
      Gapa

Leave a Reply

Your email address will not be published. Required fields are marked *

Page Not Found

Like A Sir
I say, old chap: seems you've mucked about a found yourself a page that doesn't even exist!​