If you haven’t already, read Simon Willison’s post on “Vibe engineering”. It perfectly lays out what many people who rely heavily on coding agents observe every day, and the unique challenges in getting the most out of these tools without becoming the bottleneck.
Turning the revered Postgres into a distributed database is a tall order but not a new idea. It has been done in many different ways with different characteristics. Citus requires a fair bit of manual configuration and client setup, but is open-source and distributed as an extension. CockroachDB is simpler to integrate but is not actually Postgres and not open-source. Neon is perhaps the most interesting being a true Postgres extension and having some compelling features, but isn’t fully distributed in the multi-master sense.
I recently released a new version of Reshape, a tool for automating zero-downtime schema migrations in Postgres. I’m quite excited about this new version as it automates schema migrations that require changes across multiple tables, all whilst ensuring zero downtime for both your application and database. I’ll lay out an example of such a complex, multi-table migration here but you might also like to read the introductory blog post for Reshape first.
Renaming a field in your database is perhaps the most fundamental schema change, yet doing so safely is really hard with most databases. Take MongoDB for example. Its schemaless nature means that renaming a field must be done with an update for each record in the database. During this migration, some of your records will have the new name and some the old, and your application will need to be updated to handle them both, unless you simply take your application down for maintenance until all records have been updated.
I’ve been working on a tool called Reshape which aims to automate zero-downtime schema migrations for Postgres. The main thing I aspire for with Reshape is a great developer experience, hopefully as good as other schema migration tools which don’t do zero-downtime. Because of the way Reshape works, it requires some slight changes to your application so lately I’ve been focused on improving that experience with helper libraries.
About six months back, I wrote a post on how to use views in Postgres to encapsulate tables and create a versioned interface to a database. At the end of the post, I mentioned working on a tool to automate zero-downtime migrations using this technique and I’ve finally released something, albeit experimental. It’s open-source and called Reshape.
Schema migrations are tricky, especially when planned downtime is not an option. Changes need to be carefully designed to stay backwards compatible so that both the new and old schema are available simultaneously. They must also be designed to not interfere with any other queries, for example by locking up tables. Plenty has been written on how to this right but one pattern I haven’t seen mentioned often is using views.
The decorator pattern is a convenient way of extending functionality with minimal changes to existing code. The basic premise is to create a new type (the decorator) which wraps an existing one. Our decorator should implement the same interface functions as the wrapped type and forward function calls to the wrapped object with some extra functionality. Because the decorator implements the same interfaces as the type it wraps they are interchangeable.
Apple recently open-sourced FoundationDB, two years after acquiring the company and depriving the world of some promising technology. FoundationDB is a distributed key-value store featuring ACID transactions, painless scaling and easy fault tolerance, all tested with ridiculous thoroughness. In short, a seriously impressive feat of engineering. Before the acquisition its creators often touted the concept of “layers”, meaning stateless programs which add new features to the otherwise simple database. A SQL layer could make FoundationDB behave like a relational database while an AMQP layer could make it function as a message broker.
Like many developers I constantly find myself taking up new programming languages. The learning process is often refreshing and challenging as a result of novelty and it has many times served as a relaxing break from work and studies. At the same time I’ve always associated this curiosity with shame. Instead of diving deeper into what I already know something new grabs my attention and drives me further from a pipe dream of deep knowledge. Lately my feelings have shifted and the shame that I once felt has turned into a feeling that learning new programming languages actually carries a lot of value.