Don’t Be Too Rigid When Building CI/CD Platforms

TL;DR

The DORA 2024 report emphasizes how CI/CD platforms that automate test execution, reporting, and environment management help teams focus on writing high-quality tests. This is a major advantage, but there’s a slippery slope: overly prescriptive platforms risk reducing autonomy and engagement. Platforms like GitHub Actions strike a balance by offering reusable workflows with “escape hatches” for customization. By adopting simple contracts for test reporting (e.g., xUnit XML files), platforms can provide structure while preserving team flexibility, making it easier to integrate customized workflows into centralized analysis—thus combining the benefits of a golden path while avoiding ’lock-in’ into a prescriptive, rigid delivery pipeline.

A CI/CD platform offering

In the ‘24 DORA report on page 48, an example is provided where a platform has the capability to execute unit tests and report back results directly to development teams, but without that team needing to build and manage the testing execution environment. Then the continuous integration platform feature enables teams to focus on writing high-quality tests.

And I fully agree, this is a significant benefit such a platform can bring. It solves two major problems: The first is about providing the CI server and test runner infrastructure, making sure that this is neatly integrated with the existing SSO solutions, cost reporting, auto-scaling, secure credentials handling, and so forth. This addresses cross-cutting concerns of most teams, an offering that significantly reduces cognitive load. The second is about the CI/CD pipeline or workflow itself. The platform offers a golden path and standardized workflows for all teams, allowing new teams or services to quickly get up to speed with their continuous integration and test setup.

There is one caveat, one slippery slope, though: This can quickly become too prescriptive, too rigid. While managing a CI/CD environment comes with a lot of complexity and cognitive load, this is not so much the case with test execution itself. For many programming languages, this is already part of the project scaffolding, and developers typically know how to run tests in their language of choice—this is significantly closer to their capabilities and expertise than provisioning and operating a CI server. The solution should therefore not be to provide a comprehensive, central, rigid CI/CD workflow that is mandated for all teams. A lack of agency often causes teams to withdraw from CI responsibility entirely. Instead of continuously improving their test and deployment automation, they expect this to be fully covered by the platform team. Instead of fixing problems and making improvements, they start to raise tickets for the platform team to handle. And they start to complain that their aspirations for quality and test automation are hampered by the circumstances.

We build the roads, you drive the car

What the platform should offer instead is a template, a blueprint—a golden path or paved road. Teams can still hit the ground running but have agency in adopting and adapting the workflows to their immediate needs. The platform can support this by providing reusable, standardized workflows for common problems, but still providing ’escape hatches’¹, so that teams can adjust workflows or actions to their needs. They might create their own actions, which could be reused by other teams or could become part of the templates in the long run. In the best case, deviating teams only need to adjust the single aspect they are interested in; everything else can be reused and remains standardized.

The more uniform² the path to production is within an environment, the more the organization benefits from a platform offering. As the platform team can focus on the ubiquitous but similar needs, the risk of creating a too-complex solution that caters to too many requirements can be avoided. This is offset by a lack of (perceived) ownership within the teams for their own path to production but could be warranted in these setups—especially if escape hatches are provided.

By using Pipeline as Code approaches like those offered by GitHub Actions³, teams can still easily integrate new tools, stages, or checks into their setup. This approach allows for ticketless self-service, with no need to involve the platform team to adapt the pipeline.

By standardizing on the reporting format (e.g., xUnit-style XML) and not the tools, it can still be ensured that new tools can be easily integrated into the existing reporting, allowing the teams to focus on the actual problems they want to solve. Similarly, if teams have a need for different reporting or analysis, having the ability to extract the data of a test run in a standardized format allows them the freedom to do so.

This extends to governance and compliance needs: The platform should offer and maintain reusable actions and workflows that can easily be integrated into customized team workflows. Enforcing a specific CI workflow to ensure compliance comes with the consequence of stifling innovation and effectiveness in the affected teams.

The actions and workflows offered by the platform need to be provided as treated and maintained as a product with good developer experience. This becomes especially important when consuming teams initially start off with re-using the templates, as good documentation and examples help to flatten the learning curve and empower teams to take ownership of their pipeline when need arises.

The Risks of Overreach

This is quite nuanced but very easy to get wrong, in my opinion. When platforms become overly prescriptive—mandating fixed workflows or requiring platform team approval for changes—they risk:

• Reducing Autonomy: Teams lose the ability to adapt workflows to their specific needs.
• Eroding Ownership: Teams disengage from pipeline performance and optimization, assuming it’s now someone else’s problem.
• Increasing Complexity: Trying to cater to every team’s specific needs increases the complexity of the workflow template, reducing its usefulness.

This leads to situations where the platform team becomes a bottleneck or is expected to fix failing builds - because the teams themselves are no longer able to understand and reason about the CI pipelines.

Building an Empowering Platform

To avoid these pitfalls, platforms should:

1. Provide Templates: Offer blueprints of build pipelines or automation for common setups via specific reusable workflows and actions. Teams should have the freedom to change and customize them to their needs.
2. Enable Flexibility: Make it easy for teams to enhance their test setup and have different stages or checks or actions.
3. Include Escape Hatches: Allow teams the freedom to deviate where necessary, as with GitHub Actions.
4. Use Simple Reporting Contracts: Ensure that teams can integrate their customized workflows into centralized reporting by adhering to simple contracts like xUnit XML files.

By following these principles, platforms can reduce cognitive load without stifling innovation or ownership.

Developer Independence is Key

In other parts of the DORA report, it is pointed out that developer independence is a key factor to success. This must extend to CI/CD platforms. Automation should empower teams to focus on quality and innovation, not limit their ability to adapt or experiment. The closer a platform comes to the expertise and capabilities of the teams, the more degrees of freedom it needs to provide to ensure teams like the platform and remain autonomous.

Conclusion

The example in the DORA report highlights a valuable concept: freeing teams from infrastructure concerns allows them to focus on creating quality tests and code. But as we embrace automation, we must remain vigilant against the slippery slope of overly prescriptive platforms that stifle autonomy and flexibility.

By designing platforms that provide templates and reusable workflows with escape hatches, and by enforcing simple reporting contracts, we can strike the right balance between automation and flexibility. This approach empowers teams to focus on high-quality development without sacrificing autonomy.

A developer platform should not dictate how the ‘right’ CI/CD pipeline for a team looks like, instead it should make it easy for teams to build the right one.