Tuesday, March 7th 2023
Why Turborepo is migrating from Go to Rust
How we're aligning our tools to our work.
As usage has grown and product needs have shifted, we've decided to start an incremental migration from Go to Rust in the 1.7 version. In this article, you'll learn about our motivations for this migration and problems that we are finding Rust solves for our team.
Why are we migrating?
The original decision for Turborepo to use Go followed in the footsteps of
esbuild is fast and avoids much of the initialization overhead of Node.js. Additionally, Go's developer experience is tailored for iteration, something we needed as we learned more and more about what developers wanted from
In the early days of Turborepo, these properties of Go gave us exactly what we needed for the project to be successful. However, as the Turborepo codebase has scaled and merged with Turbopack, Go has begun to underserve both our core team and users in the areas that matter most to Turbo.
Comparing Go and Rust
We've been working on several other migrations lately and have enjoyed the opportunity to refine our approach:
- Doing a mock migration for the BBC's open-source frontend from React to Next.js
- Dogfooding the Next.js 13 App Router for vercel.com
In any major technical migration, there's a lot to consider and the decision shouldn't be taken lightly. In particular, a language migration is quite demanding, asking you to weigh dimensions like the strengths, weaknesses, and community of a given language according to your specific business and technical context.
In our case, we needed to compare Go and Rust to figure out which language was going to serve us best.
Go's language design priorities
Go's strength is network computing in data centers and it excels at this task, powering these workloads at the world's largest scales. The goroutine-per-request model, Context API, and the standard library inclusion of server infrastructure is testament to this community focus.
Additionally, Go favors simplicity over expressiveness. A side effect of that decision means more errors are caught at runtime where other languages might catch them at compilation. With a service running in a data center, you can roll back, fix, and roll forward at your convenience. But, when building software that users install, the cost of each mistake is higher.
For us, it's worth using tools that prioritize up-front correctness. We fully recognize the mismatch of Go's priorities and what we are prioritizing as a problem that we created for ourselves to solve.
Our needs align with Rust
The Rust language and community has prioritized correctness over API abstraction—a tradeoff that we care a lot about when working with:
- Process management
- Other low level OS concepts
- Shipping software to our users' machines
This means additional complexity is surfaced into our codebase, but it's necessary complexity for the problems we're trying to solve.
Rust's type system and safety features allow us to put guardrails in place in our codebase where we need them. The language's expressiveness allows our developers to encode constraints that catch errors at compile time rather than in GitHub issues.
Comparing by example: file permissions
Go's preference for simplicity at the filesystem was creating problems for us when it came to file permissions. Go lets users set a Unix-style file permission code: a short number that describes who can read, write, or execute a file.
1> ls -l turbo.json2-rw-r--r-- 1 anthonyshew users 247 Jan 1 00:01 turbo.json
While this sounds convenient, this abstraction does not work across platforms; Windows actually doesn't have the precise concept of file permissions. Go ends up allowing us to set a file permission code on Windows, even when doing so will have no effect.
In contrast, Rust's explicitness in this area not only made things simpler for us but also more correct. If you want to set a file permission code in Rust, you have to explicitly annotate the code as Unix-only. If you don't, the code won't even compile on Windows. This surfacing of complexity helps us understand what our code is doing before we ever ship our software to users.
Rust's strong (and growing) ecosystem
Rust has a fantastic ecosystem of high-quality, open-source crates (Rust's equivalent to an npm package) that have clear focus on what we care about. An example of where we benefit from this alignment is when we have to interface with native libraries written in C or C++.
As we've built out Turborepo, we've started to rely more often on native C packages like
zstd, a library that helps us compress our cache files. Interoperating with these native libraries in Go requires the use of CGO, which switches us from a pure Go toolchain to a much slower C toolchain. Moreover, this switch is a global process, meaning that if we use a single native library, we have to build our entire codebase with CGO.
In Rust, this interfacing with native C libraries is far more contained. Libraries such as
cxx create safe wrappers and don't require global changes to our builds. Even better, many libraries come with this wrapper already generated.
For example, we ported our
git interface using the
git2 interfaces with the C library
libgit2 underneath the hood, but exposes a safe, idiomatic Rust API. This allows us to get the benefits of both the Rust and C ecosystems while still maintaining a great internal developer experience.
The Turbo ecosystem
Internally, we share a codebase and work closely with the Turbopack team.
Getting aligned means both teams can ship faster by sharing development and maintenance of common utilities in our problem space. For instance, we're taking a lot of inspiration from the Turbopack team when it comes to file-watching so we can build a feature for smart hot-reloading across workspaces sooner.
Rust makes the core team happier
Another great perk: our team wants to write Rust. It's a language that solves what we care about and brings us joy. The fact that we enjoy writing Rust is valuable, by itself, in more ways than one.
- Happier developers deliver better software. Your brain is better at complex problem-solving when it's happy.
- If we're happier while we work, we're much less likely to burn out.
- Rust's efficiency means less energy consumption, letting us do our part in global sustainability.
Rust makes other developers happier
Looking at the past seven years of StackOverflow survey results, we're not alone.
We also made this choice with future developers of Turborepo in mind.
Continuing the migration
We're migrating incrementally, so it's not a complete rewrite overnight.
Right now, we have what we call a "Rust-Go-Rust Sandwich." Rust is the entry point, allowing us to choose whether the implementation for a particular command is in Rust or Go. Our Go code is able to call Rust code, too, giving us paths to keep Go around but always be able to get to Rust. Check out the turborepo-ffi crate and ffi.go to learn more.
We're excited about what Rust has already unlocked for our team and can't wait to finish the oxidation and carcinization of our codebase.
If you're a high-performance engineering team building developer tooling or doing systems work and you're debating Rust or Go, we hope our experience can be a helpful reference for you.
Try out Turborepo today
Turborepo 1.8 was recently released with more features written in Rust. Learn more about what's new to the Turboverse on the 1.8 release post.
If you're looking to create a distributed caching system for your team and CI in three minutes, you can check out our post here.
Turborepo Core Team
Turborepo Core Team
Turborepo Core Team