Pernosco supports debugging failures in Github Actions tests. When a test fails, you get a link that takes you directly to debug that failure:
(If you want Pernosco debugging for the Github Actions of your open-source project, contact us.)
Implementing this required reimplementing (some of) Github Actions workflow execution. We have just published our
gha-runner implementation of Github Actions (as a Rust library). You can use
gha-runner to run Github Actions workflows locally, and it has extension points to let projects like Pernosco modify how workflows are executed.
gha-runner contains an example that lets you run Github Actions steps locally under
$ target/debug/examples/gha_local \ --strace /tmp/strace.out \ --image-path ghcr.io/catthehacker/ubuntu:act \ Pernosco github-actions-test 6475d0f048a72996e3bd559cdd3763f53fe3d072 \ .github/workflows/build.yml \ "Build+test (stable, ubuntu-18.04)" ... failures: one test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s error: test failed, to rerun pass '--test one' Step 'cargo test (debug; default features)' exited with code 101, aborting $ head -5 /tmp/strace.out 557 execve("/root/.cargo/bin/cargo", ["cargo", "test"], 0x7ffcb6870200 /* 31 vars */) = 0 557 brk(NULL) = 0x5646cb4bb000 557 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) 557 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) 557 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
gha-runner is far from a complete implementation of Github Actions, but can run simple workflows.)
When a Github Actions workflow fails in a Pernosco-supported project, Github notifies a Pernosco daemon. Our daemon analyzes the failed workflow (using
gha-runner) to check whether the
PERNOSCO_ENABLED environment variable is set in the failing workflow step (we don't want to reproduce and offer to debug build failures, for example). The daemon then spawns a task into our AWS ECS cluster to rerun the failed step under
rr and hopefully reproduce the failure. We use
gha-runner to rerun all job steps up to and including the failing step. We have to run this in our AWS instances because Github's own Azure instances don't provide virtualized hardware performance counters and thus don't support
rr recording. We don't want to require our users to use self-hosted AWS runners (which can be difficult and expensive). The steps before the failing step run normally, but the failing step is run under
rr to get a recording. If we record a test failure, the recording is submitted to Pernosco for analysis. When the analysis is done and the debugger is ready, we update Github with the link to the debugger.
The official Github Actions runner is written in C#, is extremely complicated, and communicates with Github's infrastructure using complex undocumented protocols. It is unsuited to our purposes.
act is much more suitable: it's a self-contained Go project that lets you run Github Actions locally, and looks really good. However, we wanted a library that would integrate easily into our Rust project, and that supports the extension points we need — customizable container-running backends, so we can run GHA jobs as ECS tasks, and the ability to inject
rr into containers and run steps under
rr. So we wrote our own. I'm grateful to the
act author(s) for proving that standalone GHA runners are possible (at least for many workflows) and that I was occasionally able to clarify GHA semantics by reading
act source code.
gha-runner can run simple workflows but is currently missing a lot of functionality. For example, actions that bring their own containers are not currently supported.
post steps are not supported. All actions run in Docker containers and Docker nesting is not supported, so actions that run their own
docker commands won't work. Action expression syntax is quite limited currently. Most of these missing pieces can probably be implemented fairly easily.
We hope that
gha-runner will be useful to other projects and welcome PRs to improve it.
octocrab to access Github APIs. We're in the process of upstreaming some changes to
octocrab; once that is complete we'll rebase
gha-runner to upstream
octocrab and release
gha-runner on crates.io.
Github Actions actions are passed a
$GITHUB_TOKEN that they use to authenticate with Github for checkouts and other API calls. For public projects no authentication is normally needed, but nevertheless a valid token must be provided. To make
gha-runner easy to use for public projects, we have created a personal access token that anyone can use:
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. This token has no access right grants, so can only access public resources. For extra security, it belongs to the
pernosco-unauthorized Github account which itself has no access to anything and exists solely to own this token.