Many modern workflows aren't amenable to interactive debugging. When a test fails in CI, or a component is failing in a distributed system, it's difficult or even impossible to attach a debugger. Most interactive debuggers need to stop the program to inspect the state, and stopping a program may cause it to be killed by a supervisor or trigger unacceptable cascading failures. rr and Pernosco solve this problem.
We have delivered customized Pernosco integration with CI to users. If you are interested in this kind of integration, contact us.
We built Pernosco integration for Github and Travis CI as a proof of concept. The user installs our Pernosco Github app:
The user triggers a Travis test run which fails. Pernosco receives the failure notification from Github and automatically reruns the failing test, with recording.
If Pernosco successfully reproduces the failure, it updates Github with a link to a Pernosco debugging session.
The user clicks on the "Details" link to enter the Pernosco debugger and diagnose the bug.
Pernosco processing adds latency to the push/test cycle, but only when developers are already waiting for CI results. Compare this to the effort required to rerun and debug the test locally, or to multiple cycles of enabling extra logging and re-pushing to figure out the problem.
Many CI frameworks exist, and many projects have their own. Some work is required to integrate Pernosco with each one.
Firefox developers validate their changes before commit by pushing them to a Mozilla 'Try server' which runs automated tests on them. Pernosco detects test failures on the Try server, then reproduces, records and processes those failures, notifying developers by email when Pernosco sessions are available. Clicking on a link to debug is much more convenient than having to reproduce the test failure locally.
To simplify debugging, our test reproducer stops the recording as soon as the first test in a suite has failed. Sometimes developers want to debug a different test failure. To support that, we offer a dashboard where developers can view each test that failed and select them individually for reproduction and debugging with Pernosco.
Our CI support primarily targets reproduction of deterministic test failures, and re-runs each failing test suite just once by default. We also support rerunning a failed test suite multiple times (with rr chaos mode) to enable reproduction and debugging of intermittent test failures.