cracked.guide

Blog

Verifying your interpreter with the official test suite in GitHub Actions

On cracked.guide, a Crafting Interpreters chapter is done when the book's own test suite passes against your code. A GitHub Actions workflow in your Lox repository builds your jlox and clox on every push, runs the official tests, and reports which chapters pass. Here is how it works and what we learned getting the old test runner to behave.

By Dane Dolph · · 2 min read

What the workflow does

You add one file, .github/workflows/cracked.yml, to your repository. On every push it:

  1. Checks out your repository and munificent/craftinginterpreters, which has the test runner and the tests.
  2. Installs Dart 2.19.6 and Java 21, and fetches the runner's packages.
  3. Builds your interpreters: by default every .java file under jlox/ (the class in Lox.java runs) and every .c file next to clox's main.c.
  4. Runs the chapter suites from your furthest chapter back until one passes, so a passing chapter 10 also covers 4 to 9.
  5. Reports the results to cracked.guide, which marks those chapters verified.

No secrets

The report is signed by GitHub, not by a token you paste anywhere. The workflow asks for id-token: write, GitHub issues a short-lived token that names your repository, and cracked.guide checks it against the repository you connected. Nothing to configure, nothing to leak.

What it took to make the old runner behave

The test runner was written before Dart 3 and is not null safe, so the workflow pins Dart 2.19.6. A few other things only showed up running the real book interpreters in CI:

  • JAVA_TOOL_OPTIONS must be unset, or the JVM announces it on stderr and every error test fails.
  • A UTF-8 locale, for the string literal tests.
  • Expr.java and Stmt.java are generated from your GenerateAst when you have not committed them.
  • A 10-second limit per test, because the runner has none and a half-built VM can loop forever on a later chapter's test.

If your layout is different

Put a .cracked.json at the root of your repository:

{ "build": "make", "jlox": "build/jlox", "clox": "build/clox" }

Paths are from the root, and jlox and clox are anything that runs a .lox file given as its one argument. jlox_through and clox_through stop at the chapter you are on and save minutes per push.

If you just want to run the tests yourself, see how to run the Lox test suite.