The Evolution of The Minotaur

I have a tool called The Minotaur that I just rewrote for the third time, and I think, maybe, it’s done.

The Minotaur is a tool I wrote at mitsi in 2014 to automatically restart services on my server when I modified relevant code. Many web frameworks have something like this, but the tooling tends to be built in to frameworks instead of being general.

The first few versions (visible here) were all variants on the same theme: restart one or more (runit) services when a file in a tree changed. Fairly soon after creating the tool I added debouncing, which means that when a bunch of things change in quick succession they are bundled into a single event, instead of triggering an event per change. This is important since text editors save files in multiple stages to prevent data loss. I used this tool for a little under a year before moving on to ZipRecruiter.

About a year after I started at ZipRecruiter I replaced the built in Plack autorestart with The Minotaur. Part of the reason for doing this is that Plack preloads a bunch of Perl modules, so when it reloads the application it doesn’t reload those, since they are already in memory. One critical difference between The Mitsi Minotaur and The ZR Minotaur is that the latter is a supervisor instead of communicating with another one. I don’t love this, but it made it easier to add it to people’s workflows.

While The Minotaur has been in use at ZipRecruiter it has gained a handful of interesting features. Here’s a brief list:

• --check-script: this script would receive the files that changed and exit 0 or 1 to tell The Minotaur if it should restart the supervised service or not.
• --restart-script: this script runs after restarting the supervised service to allow restarting other, ancillary services.
• --post-ready-script: this allows you to delay the run of the restart script till the supervised script is running successfully.

There were also a number of interesting bug fixes, involving Epoll and sharing the Epoll filedescriptor with Starman, and event loop and inotify interactions causing dropped events. I got help from Ben Grimm, Andrew Ruder, and James Messrie for both the features and fixes above. Thanks!

We are now working towards running production in Kubernetes, and part of that means giving developers a way to work with their code that is similar to production. Docker development workflows vary, and only time will tell if this one turns out well or not, but the idea is that The Minotaur runs outside of the containers and reaches into the container to restart, recompile, or test the actual code.

Because it needs to run on laptops or inside of containers (that might have very minimal runtimes) requiring Perl and an event framework seemed the wrong way to go about it. Furthermore The Minotaur has only worked on Linux for ages due to the direct use of Inotify, which would keep it from working on either OSX or Windows.

So I decided I’d try my hand at reimplementing The Minotaur in Go, but simplifying where I could. Because it cannot be the parent process of the restarted script anymore, I can remove the all the supervision code, which is significant.

My initial thought was that I could get by with just the check and restart scripts. As I wrote the code I realized that the restart script runs right after the check script (I hadn’t gotten around to implementing post ready yet) and that basically it would work just as well if I only ran the restart script, but passed it the arguments so it could exit early if the relevant files haven’t changed.

With the model simplified so much, the interface becomes much simpler, at the expense of a more complex (but more flexible) script. Furthermore the script doesn’t have to deal with threading or asynchrony, so it’s less likely to have bizarre bugs. You can see the current code here. Here’s an example of how I have it set up while working on the leatherman:

minotaur . -- ./internal/build-test

And then the script is:

#!/bin/sh

set -e

for arg in "$@"; do
   echo "$arg"
done | grep -E '\.go$'

go test ./...
go build ./cmd/leatherman/...
echo "Built new leatherman at $(date)"

I mentioned the more complex Minotaur that is currently in place at ZipRecruiter; here’s a script that could tie the pieces together (untested:)

First off we have these scripts that already work well:

• any-perl: if any one of the scripts passed is perl, exits 0
• ready-devserver: when the server is up and ready to serve traffic, exits 0
• restart-devserver-support: restarts other services that support the dev server

Also I’m handwaving a restart-devserver, which would do something as simple as service devserver restart, but inevitably be more complicated because computers.

So these can be combined simply like this:

#!/bin/sh

set -e

any-perl "$@"
restart-devserver
ready-devserver
restart-devserver-support

The Minotaur also has the option to -ignore or -include directories based on regular expressions. Includes happen first, then ignores. The default include is everything, and the default ignore is stuff in a .git directory. Out of the box The Minotaur is quiet, only reporting on errors that cause a crash. You can pass -verbose to get more output.

The following would include any path that contains pkg but not any that include internal, even if pkg is in the full path:

minotaur -include pkg -ignore internal -verbose . -- ./run-tests

Of course depending on your needs you could do some really interesting things with the script, for example you might do something like this to avoid a slow, expensive webserver restart:

#!/bin/sh

set -e

any-perl "$@"

files-compile "$@"
relevant-tests-pass "$@"

restart-devserver
ready-devserver
restart-devserver-support

And files-compile might look like this:

#!/usr/bin/perl

for (@ARGV) {
   my (undef, $file) = split /\t/, $_, 2;
   system $^X, '-c', $file
      or exit $?;
}

That exits non-zero if perl fails to compile any of the passed files. I would write the relevant scripts that The Minotaur runs in either the language being supported (perl scripts for perl code, python scripts for python code) or shell to ease bootstrapping.

I’m sure that there are plenty of file system watch tools out there, but in my experience they tend to accrete features and bugs (like my own did!) Maintaining a simple interface will hopefully prevent this. Hopefully it lasts another three years!

(The following includes affiliate links.)

Have you read The Pragmatic Programmer? It’s one of the few tech books I’ve read all the way through and refer back to once in a blue moon. I strongly suggest checking it out, especially if you are early in your carerr.

Another interesting book, which I intend to read solely based on authorship, is The Practice of Programming, by Kernighan and Pike. It might be a little dated with the focus of C and C++ but much of the concepts should apply everywhere.

If you're interested in being notified when new posts are published, you can subscribe here; you'll get an email once a week at the most.