As we've previously discussed, Navixy has found a reliable and powerful tool in Playwright for automating our web interfaces. However, as always in the realm of software development and quality assurance, we encountered new challenges and sought ways to enhance our existing systems further.
So, let's turn the page and delve into the latest chapters of our ongoing saga.
Seeking speed beyond Playwright’s capabilities
As the volume of our automated tests soared, the need for parallel test execution became undeniable. However, E2E tests brought their own set of unique challenges. Picture this: several of our tests create entities on their own, meddle with them, and upon conclusion, whisk them away via the API. Now, imagine running two such tests side by side. It's a recipe for conflicts. Our response to this predicament was multifaceted. Not only did we embed the ability to run multiple test suites side by side, but we also fortified the thread safety of our Playwright instances. In other words, we made it possible for them to run free without stepping on each other's toes.
But the quest for speed introduced us to another adversary: Kotlin's compilation time. Though Kotlin can be a joy to work with in many respects, its penchant for leisurely compiling, especially when our tests on CI dwell in isolated Docker containers, meant the compilation dance occurred every single time. To remedy this, we snugly packed the compiled code inside the Docker container and provided signposts for the tests to find it when awakened on CI. Yet, every new infusion of tests into the main branch required a manual reconstruction of the Docker container to accommodate the latest version of the compiled code, a process that threatened to bog down development.
Ultimately, we solved this with a secondary Docker container designed expressly to mold the test container. The fruit of our labors? A time revolution, with the compilation stage, which previously monopolized around six to seven minutes of the process, now occurring almost instantaneously. Now, each code merger into the main branch ignites an automatic ballet of code compilation and Playwright container assembly, ready to be showcased in the repository.
This innovative approach not only obliterated the strenuous compilation time but also carved down the end-to-end test runtime for a batch of 200 tests significantly. Initially staggering at around 40 minutes in a non-parallel setup—a time figure that would have only increased with the additional tests—the process now operates in under 10 minutes in optimal conditions. This time frame, already impressive, carries the potential for further refinement. A slight rearrangement of test suites for better parallel execution could very well shave off even more time, potentially breaking the lightning-fast 10-minute mark. For the moment, we've maintained a slightly slower pace to facilitate debugging and selective test running.
But this isn’t the end of the road. Reports from our DevOps corridors hint at a potential relocation to Amazon's vast expanses, promising even brisker performance could be coming soon, and fueling anticipation for unprecedented speeds and efficiency in our future endeavors.
Integration: a confluence of vision and execution
Every QA engineer knows that simply running tests is just skimming the surface. The real challenge often lies in sifting through the aftermath: the results. Particularly on CI, this can turn into a head-scratching exercise. Now, imagine our delight when we remembered our toolbox had TestRail—a gem we frequently turn to for test management. With its generous suite of integration tools for automated tests, the allure was too strong.
The aftermath was nothing short of a revelation. Now, every time our CI tests draw to a close, a fresh-off-the-press report in the JUnit XML format is seamlessly married with the test cases. Like clockwork, it's then hoisted onto TestRail's welcoming platform and a newly minted report link is sent right into our bustling Slack channel. It's like having the morning newspaper delivered to your doorstep!
Beyond the evident perk of easily digestible test results, the doors to deeper analytics are what truly excite us. We're talking about a time-lapse view of changes, crafting reports tailored to myriad criteria, and the sharp eye to catch those pesky flaky tests.
Onwards and upwards with Navixy
As we pave our path forward, it's not just about setting new milestones, but also refining the journey itself. While we've made impressive strides, our sights are set on expanding our coverage even more, ensuring that the quality of our software isn't just maintained, but consistently elevated.
At the heart of Navixy's ethos is a relentless pursuit of excellence. We're always on the lookout for the next tool or technique that could redefine our software quality assurance. In a landscape that’s forever in flux, we commit to not just keeping pace but setting the pace.
As we gear up for the next leg of our journey, we invite you to be part of the narrative. We're always looking for like-minded professionals to join our ranks, so dive into Navixy's career page and explore the possibilities if you’re curious to learn more.
We have plenty more insights and updates in store. The journey continues, and we hope you'll be there with us every step of the way.
