Building Dynamic Buildkite Pipelines in Ruby
Hello, friends! I started working at Gusto around two weeks ago. During onboarding, Ngan showed me this tool the team built and I thought it was neat. I was surprised to learn that it was open-sourced but unsure if people knew about it. Anyway, I hope you also found this interesting!
One of the first things I learnt about the application infrastructure at Gusto is that the Buildkite CI steps are created dynamically. This means not all commits run the same pipeline steps on CI.
Building dynamic Buildkite pipelines save us on the overall CI's runtime, developers' time, and build costs - all the good stuff.
The pipeline steps are created using a tool created by Gusto's Product Infrastructure team called Buildkite Builder. Buildkite Builder is a specialized YAML builder for Buildkite pipelines. The library for Buildkite Builder was designed to match Buildkite's pipeline convention.
By leveraging the power of Ruby to introspect our commits and build the pipeline, we can determine the appropriate steps to run for each unique pull request. Buildkite Builder also makes the pipeline steps much more readable and reusable than YAML because, well, Ruby!
I won't be sharing the .buildkite/pipeline.rb file from the internal codebase but trust me when I say it was easy to comprehend. Take a look at this pipeline example if you're curious.
Here are some concrete examples of the benefits of dynamically generating the pipeline:
- We add or remove steps to the pipeline based on code analysis done prior.
- We traverse the application's dependency graph and only run tests that are relevant to the change in the commit.
- We easily control the flow and dependencies of setup steps instead of waiting for all of them to be set up sequentially.
- For example, Sorbet type checking shouldn't have to wait for npm to install. A Jest test run shouldn't have to wait for bundler to install.
- Steps can be skipped if there is no need for it.
- For example, RuboCop checks are skipped if there are no changes to Ruby files.
- We set up the build differently based on whether or not the commit is on the main branch or if it's on a PR.
An example of a Buildkite pipeline generated by Buildkite Build - the greyed out steps indicate that they were skipped.
There are guards implemented to avoid semantic merge conflicts from skipped steps between commits - but that'll be a different post altogether!