1. Overview
  2. Debugging workflow
  3. Debugging workflow: CI
  4. Basic interface
  5. Visualizing control flow
  6. Call stacks
  7. Explaining dataflow
  8. Multiprocess
  9. Search box
  10. Source files
  11. Condition and print expressions
  12. Toolbox
  13. Alerts
  14. Application logs
  15. Callees
  16. View operators
  17. Notebook
  18. Instruction execution
  19. Javascript
  20. Browser UI integration
  21. Screenshots
  22. Additional views
  23. GDB
  24. System debug info
  25. Compiler issues
  26. The Pernosco vision
  27. Related work

Debugging workflow: CI

Many modern workflows aren't amenable to interactive debugging. When a test fails in CI, or a component is failing in a distributed system, it's difficult or even impossible to attach a debugger. Most interactive debuggers need to stop the program to inspect the state, and stopping a program may cause it to be killed by a supervisor or trigger unacceptable cascading failures. rr and Pernosco solve this problem.

We have delivered customized Pernosco integration with CI to users. If you are interested in this kind of integration, contact us.

CI case study: Github and Travis

We built Pernosco integration for Github and Travis CI as a proof of concept. The user installs our Pernosco Github app:

The user triggers a Travis test run which fails. Pernosco receives the failure notification from Github and automatically reruns the failing test, with recording.

If Pernosco successfully reproduces the failure, it updates Github with a link to a Pernosco debugging session.

The user clicks on the "Details" link to enter the Pernosco debugger and diagnose the bug.

Pernosco processing adds latency to the push/test cycle, but only when developers are already waiting for CI results. Compare this to the effort required to rerun and debug the test locally, or to multiple cycles of enabling extra logging and re-pushing to figure out the problem.

Firefox CI

Many CI frameworks exist, and many projects have their own. Some work is required to integrate Pernosco with each one.

Firefox developers validate their changes before commit by pushing them to a Mozilla 'Try server' which runs automated tests on them. Pernosco detects test failures on the Try server, then reproduces, records and processes those failures, notifying developers by email when Pernosco sessions are available. Clicking on a link to debug is much more convenient than having to reproduce the test failure locally.

To simplify debugging, our test reproducer stops the recording as soon as the first test in a suite has failed. Sometimes developers want to debug a different test failure. To support that, we offer a dashboard where developers can view each test that failed and select them individually for reproduction and debugging with Pernosco.

Intermittent test failures

Our CI support primarily targets reproduction of deterministic test failures, and re-runs each failing test suite just once by default. We also support rerunning a failed test suite multiple times (with rr chaos mode) to enable reproduction and debugging of intermittent test failures.

<< Debugging workflow Basic interface >>