cracked.guide

Interpreters

How to run the official Lox test suite

The Crafting Interpreters repository ships a test runner, tool/bin/test.dart, and a few hundred Lox programs whose expected output is written in comments. You install Dart, fetch the runner's packages, and run dart tool/bin/test.dart <suite> --interpreter <your executable>. Pick a chapter suite like chap10_functions to test a partial implementation.

Updated

Setup

  1. Clone munificent/craftinginterpreters from GitHub.
  2. Install Dart 2. The runner predates Dart 3 (its pubspec asks for a version below 3.0.0), so a current Dart 3 will refuse it. Dart 2.19.6 works.
  3. In the repository's tool directory run dart pub get (or make get at the root, which does the same and more).

Then build your interpreter so it can run a .lox file given as its only argument: a script or binary for jlox, the compiled executable for clox.

Running it

From the repository root:

dart tool/bin/test.dart chap10_functions --interpreter ../my-lox/jlox
dart tool/bin/test.dart clox --interpreter ../my-lox/build/clox

The first argument is the suite: jlox or clox for a finished interpreter, or a chapter name for a partial one. Chapter suite names run from chap04_scanning to chap30_optimization; the full list is in the repository's Makefile. Extra arguments for your interpreter go after --arguments.

Each test is a .lox file with comments like // expect: 3 for printed output and // expect runtime error: ... for errors. The runner compares your output line by line and prints the failures.

Gotchas

  • Early chapters expect early output. Chapters 4 to 7 test tokens and expression output that a finished jlox no longer prints, which is why later suites cover earlier chapters instead.
  • Java noise on stderr. If JAVA_TOOL_OPTIONS is set, the JVM prints a line on stderr and every error test fails. Unset it.
  • Locale. Some string tests need a UTF-8 locale.
  • Infinite loops. The runner has no per-test timeout, and a half-built VM can loop forever on a later chapter's test. Run the suite for the chapter you are on.

Running it on every push

cracked.guide gives you a GitHub Actions workflow that does all of the above on every push: checks out your repo and the book's, installs Dart 2.19.6 and Java, builds your jlox and clox, runs suites from your furthest chapter back until one passes, with a 10-second limit per test, and marks each passing chapter verified on your path. No secrets to configure. How it works, in detail.

Questions

Why does the Lox test runner fail with Dart 3?

The runner was written before Dart 3 and is not null safe; its pubspec requires Dart below 3.0.0. Use Dart 2.19.6.

Can I use the test suite for a Lox written in Rust or Python?

Yes. The runner only needs an executable that takes a .lox file path and prints output and errors the way the book's interpreters do.

How many tests are there?

A few hundred .lox files, grouped by feature (closures, classes, strings and so on). Each chapter suite runs the ones that chapter should pass.