Maple Ong

Building Dynamic Buildkite Pipelines in Ruby

Hello, friends! I started working at Gusto around two weeks ago. During onboarding, Ngan showed me this tool the team built and I thought it was neat. I was surprised to learn that it was open-sourced but unsure if people knew about it. Anyway, I hope you also found this interesting!

One of the first things I learnt about the application infrastructure at Gusto is that the Buildkite CI steps are created dynamically. This means not all commits run the same pipeline steps on CI.

Building dynamic Buildkite pipelines save us on the overall CI's runtime, developers' time, and build costs - all the good stuff.

The pipeline steps are created using a tool created by Gusto's Product Infrastructure team called Buildkite Builder. Buildkite Builder is a specialized YAML builder for Buildkite pipelines. The library for Buildkite Builder was designed to match Buildkite's pipeline convention.

By leveraging the power of Ruby to introspect our commits and build the pipeline, we can determine the appropriate steps to run for each unique pull request. Buildkite Builder also makes the pipeline steps much more readable and reusable than YAML because, well, Ruby!

I won't be sharing the .buildkite/pipeline.rb file from the internal codebase but trust me when I say it was easy to comprehend. Take a look at this pipeline example if you're curious.

Here are some concrete examples of the benefits of dynamically generating the pipeline:

picture of buildkite pipeline An example of a Buildkite pipeline generated by Buildkite Build - the greyed out steps indicate that they were skipped.

There are guards implemented to avoid semantic merge conflicts from skipped steps between commits - but that'll be a different post altogether!