1. Overview
  2. Debugging workflow
  3. Debugging workflow: CI
  4. Basic interface
  5. Call stacks
  6. Explaining dataflow
  7. Multiprocess
  8. Search box
  9. Source files
  10. Condition and print expressions
  11. Toolbox
  12. Alerts
  13. Application logs
  14. Callees
  15. View operators
  16. Notebook
  17. Instruction execution
  18. Javascript
  19. Browser UI integration
  20. Screenshots
  21. Additional views
  22. GDB
  23. System debug info
  24. Compiler issues
  25. The Pernosco vision
  26. Related work

Debugging workflow: CI

Many modern workflows aren't amenable to interactive debugging. When a test fails in CI, or a component is failing in a distributed system, it's difficult or even impossible to attach a debugger. Most interactive debuggers need to stop the program to inspect the state, and stopping a program may cause it to be killed by a supervisor or trigger unacceptable cascading failures. rr and Pernosco solve this problem.

We have delivered customized Pernosco integration with CI to users. If you are interested in this kind of integration, contact us.

CI case study: Github and Travis

We built Pernosco integration for Github and Travis CI as a proof of concept. The user installs our Pernosco Github app:

The user triggers a Travis test run which fails. Pernosco receives the failure notification from Github and automatically reruns the failing test, with recording.

If Pernosco successfully reproduces the failure, it updates Github with a link to a Pernosco debugging session.

The user clicks on the "Details" link to enter the Pernosco debugger and diagnose the bug.

Pernosco processing adds latency to the push/test cycle, but only when developers are already waiting for CI results. Compare this to the effort required to rerun and debug the test locally, or to multiple cycles of enabling extra logging and re-pushing to figure out the problem.

Firefox CI

Many CI frameworks exist, and many projects have their own. Some work is required to integrate Pernosco with each one.

Firefox developers validate their changes before commit by pushing them to a Mozilla 'Try server' which runs automated tests on them. Pernosco detects test failures on the Try server, then reproduces, records and processes those failures, notifying developers by email when Pernosco sessions are available. Clicking on a link to debug is much more convenient than having to reproduce the test failure locally.

To simplify debugging, our test reproducer stops the recording as soon as the first test in a suite has failed. Sometimes developers want to debug a different test failure. To support that, we offer a dashboard where developers can view each test that failed and select them individually for reproduction and debugging with Pernosco.

Intermittent test failures

Our CI support primarily targets reproduction of deterministic test failures, and re-runs each failing test suite just once by default. We also support rerunning a failed test suite multiple times (with rr chaos mode) to enable reproduction and debugging of intermittent test failures.

<< Debugging workflow Basic interface >>