Eugene

It’s been almost a year since I last posted an update on eugene, the CLI tool I’m building to help people write safer SQL migration scripts for postgres. I announced this tool in Careful with That Lock, Eugene: Part 2. At the time, eugene would execute a single SQL script, recording all the locks acquired and warn about possible downtime due to migrations. It could produce JSON suitable for automated tooling and Markdown suitable for human reading and using in CI comments/checks. That version was already good enough for me to start using in real projects — but it’s improved a lot since then, it’s now easy to run with almost no setup. ...

Database servers are usually long-lived, and important parts of the infrastructure that we build on. We rarely set them up from scratch, because we have to take such good care of them over time. I think this causes a lot of people to think that setting up a database server is some mysteriously difficult ordeal. To be clear, that’s actually true, if you need high availability and a solid recovery point objective. But there are a lot of use cases where that’s overkill, for example short-lived test environments, or CI/CD pipelines. ...

I’ve been busy working on a documentation site for eugene, and I think it’s starting to look pretty good. I wanted to write down some of my thoughts around the process so far, and some of the things I’ve learned. It’s just been a few days since I ported my blog to hugo, so since I was already feeling like I was up to speed on that, I decided I’d try using it for the eugene documentation too. I experimented with a few different setups around the hugo-book theme, but it ended up feeling like I was having to configure a bit too much for my taste. I think it looked really nice, but the setup felt a bit janky and complicated, with having the submodules and everything in the eugene repo itself. I think my problem here is just that doing this with hugo would require me to learn more about hugo than I’m ready for right now. ...

I have been working quite a bit on picking up dangerous migration patterns in migration scripts over at the eugene repository lately. A major feature I’ve added is syntax tree analysis, so that we can pick up some patterns without having to run the SQL scripts. This isn’t quite as precise as running the scripts, but it’s a lot faster and can catch quite a few common mistakes. So let’s take a look at how it works! ...

A while back, I wrote Careful with That Lock, Eugene about an idea for how to check if a database migration is likely to disturb production. That post came about after having an inspiring chat with a colleague about the advantages of transactional migration scripts and the ability to check the postgres system catalog views before committing a transaction. Over the past few weeks, I’ve been experimenting with this idea to test if I can use it to build valuable safety checks for DDL migrations. Kind of like shellcheck, but for database DDL migrations. ...

It is rewarding to work on software that people care about and use all around the clock. This constant usage means we can’t simply take the system offline for maintenance without upsetting users. Therefore, techniques that allow us to update the software seamlessly without downtime or compromising service quality are incredibly valuable. Most projects I’ve worked on use a relational database for persistence, and have some sort of migration tool like flyway or liquibase to make changes to the database schema. This post is about a particular kind of migration situation that, in my experience, most developers who work on such projects will encounter at some point in their career. They will want to apply a simple, and seemingly innocent migration, like adding a column to a table and it’ll cause some number of requests to fail, or maybe even a small outage. There are some tricks we can use here to reduce risk and automatically detect some patterns that cause this problem. ...