1. Overview
  2. Debugging workflow
  3. Debugging workflow: CI
  4. Basic interface
  5. Call stacks
  6. Explaining dataflow
  7. Multiprocess
  8. Search box
  9. Source files
  10. Condition and print expressions
  11. Toolbox
  12. Alerts
  13. Application logs
  14. Callees
  15. View operators
  16. Notebook
  17. Instruction execution
  18. Javascript
  19. Browser UI integration
  20. Screenshots
  21. Additional views
  22. GDB
  23. System debug info
  24. Compiler issues
  25. The Pernosco vision
  26. Related work

Related work

Traditional debuggers such as gdb, WinDbg, and Visual Studio execute programs forwards, stop the program under certain conditions and allow inspecting the current state when stopped. Record-and-replay debuggers such as rr, TTD and UndoDB extend traditional debuggers with support for simulated reverse execution, which solves many problems and enables powerful functionality such as reverse-execution to a data breakpoint. (gdb also has a recording mode, but its recording overhead is about 1000x and has many limitations.) Even with these record-and-replay debuggers, accessing specific program states is still expensive, due to the overhead of creating and resuming process checkpoints and executing forward to the desired program states. These debuggers remain limited by the constraints of traditional debugger interfaces (i.e., inspecting the program state at a "current point in time" which can move forward or backward); building an understanding of program behavior over time still requires techniques such as single-stepping.

The term "omniscient debugging" was coined by Bil Lewis in 2005 for his ODB debugger. Since then there have been several more attempts to make a useful omniscient debugger, notably Chronon, TOD, Chronomancer, Qira and Tetrane's Reven. ODB, Chronon and TOD all work at the JVM bytecode level and aren't applicable to non-JVM applications. Even at the JVM level they're quite slow if you record absolutely everything (TOD quotes 100x slowdown). Chronomancer records at the mafchine level but also has very high slowdown (300x), as does Qira.

Reven is the most similar system to Pernosco that we know of. Not much detailed technical information is available, but like Pernosco it seems to combine a recording phase with a database build phase. Reven is focused on reverse-engineering for security analysis, so its features and interface are more oriented to adversarial machine-level debugging without source or symbols, while Pernosco targets the developers of the software who have those assets and want to debug at a higher level with integration into their workflows.

SteamDrill is a research prototype that performs massively parallel reexecution without building a database. (The authors write "Clearly, materializing the OmniTable for even a simple program is infeasible due to storage and processing overheads" — we disagree.) Pernosco records processes at the machine level, but relies on rr to record application execution with low overhead and minimal disruption, then builds the omniscient database offline and in parallel to reduce the wall-clock processing time dramatically. As far as we know Pernosco is the first omniscient debugger to leverage low-overhead record-and-replay this way.

Other omniscient debuggers have assumed individual developers are responsible for making recordings and processing them on their personal hardware. With Pernosco we recognize getting developers to use debuggers requires automatic production of recordings at useful points in the developer workflow, such as CI test failures, and we've done a lot of work to support that in a scalable way.

As described in this article series, Pernosco goes much further than other omniscient debuggers in leveraging omniscience with new UI approaches to make debugging more efficient.

<< The Pernosco vision