Hello everyone!
The openSUSE Conference 2025 will happen very soon, and although I’m going to participate remotely this year from the coast of Cambrils, I’m incredibly happy to have been able to contribute with my pre-recorded talk: “Quality on Autopilot: Scaling Testing in Uyuni”.
It’s a fantastic opportunity to share the journey we’ve undertaken at SUSE to scale quality in an open-source project as dynamic and complex as Uyuni. The question I tried to answer is: How do you scale quality in a fast-moving open-source project?
For those who can’t attend, the talk is a tour through our adoption of TestOps practices to bring more automation, visibility, and resilience into our Quality Engineering processes. This wasn’t an overnight change, but a story of small, iterative improvements.
Here are the pillars of our strategy that I discussed in the presentation:
🏗️ Solid Foundations with IaC: We started by automating our infrastructure with Terraform and Salt. This gave us the ability to create consistent, on-demand test environments, eliminating the classic “it works on my machine” problem.
📊 Visibility with Observability: We integrated Prometheus and Grafana to monitor our test trends. This allows us to analyze historical data to detect regressions, identify flaky tests, and understand the behavior of our tests over time.
🏷️ Managing Flaky Tests and Known Issues: We created a custom system that connects our QE workflow in GitHub directly with our test reports. Using a script, we automatically tag Cucumber scenarios, providing immediate visibility into the status of a failed test (if it’s flaky, if a bug has already been reported, etc.).
⚡ Accelerating Feedback with Smart Test Selection: For Pull Requests, we implemented smart test selection based on code coverage. This allows us to run only the relevant end-to-end tests for the affected components, drastically reducing CI times.
🩺 Continuous Synthetic Monitoring: We introduced “Fitness Functions” into our daily pipelines. These are key product metrics (like bootstrap or synchronization times) that give us a continuous view of the system’s overall health and provide early warnings about product-level issues.
💻 Improving the Developer Experience: We designed our own DevContainers to standardize the development environment in IDEs. This has greatly simplified the onboarding process and aligned the entire team.
🤝 Transparency with the Community: We are working on making our AWS test reports public. We believe transparency is key in open-source to build trust and accountability.
This post is just the beginning. In the coming days, I plan to write a bit more detail on some of these topics and, most importantly, share a summary of what I’ve learned from other incredible talks that I’ll be able to follow at the openSUSE Conference 2025.
Also, as soon as the organizers upload the videos to YouTube, I will embed the full recording of my talk right here for you to watch.