# How to run the official Lox test suite

> How to run the official Crafting Interpreters test suite against your own jlox or clox: Dart setup, test.dart, chapter suites, and running it in CI.

The Crafting Interpreters repository ships a test runner, `tool/bin/test.dart`, and a few hundred Lox programs whose expected output is written in comments. You install Dart, fetch the runner's packages, and run `dart tool/bin/test.dart <suite> --interpreter <your executable>`. Pick a chapter suite like `chap10_functions` to test a partial implementation.

## Setup

1. Clone `munificent/craftinginterpreters` from GitHub.
2. Install **Dart 2**. The runner predates Dart 3 (its pubspec asks for a version below 3.0.0), so a current Dart 3 will refuse it. Dart 2.19.6 works.
3. In the repository's `tool` directory run `dart pub get` (or `make get` at the root, which does the same and more).

Then build your interpreter so it can run a `.lox` file given as its only argument: a script or binary for jlox, the compiled executable for clox.

## Running it

From the repository root:

```
dart tool/bin/test.dart chap10_functions --interpreter ../my-lox/jlox
dart tool/bin/test.dart clox --interpreter ../my-lox/build/clox
```

The first argument is the **suite**: `jlox` or `clox` for a finished interpreter, or a chapter name for a partial one. Chapter suite names run from `chap04_scanning` to `chap30_optimization`; the full list is in the repository's Makefile. Extra arguments for your interpreter go after `--arguments`.

Each test is a `.lox` file with comments like `// expect: 3` for printed output and `// expect runtime error: ...` for errors. The runner compares your output line by line and prints the failures.

## Gotchas

- **Early chapters expect early output.** Chapters 4 to 7 test tokens and expression output that a finished jlox no longer prints, which is why later suites cover earlier chapters instead.
- **Java noise on stderr.** If `JAVA_TOOL_OPTIONS` is set, the JVM prints a line on stderr and every error test fails. Unset it.
- **Locale.** Some string tests need a UTF-8 locale.
- **Infinite loops.** The runner has no per-test timeout, and a half-built VM can loop forever on a later chapter's test. Run the suite for the chapter you are on.

## Running it on every push

cracked.guide gives you a GitHub Actions workflow that does all of the above on every push: checks out your repo and the book's, installs Dart 2.19.6 and Java, builds your jlox and clox, runs suites from your furthest chapter back until one passes, with a 10-second limit per test, and marks each passing chapter verified on your path. No secrets to configure. [How it works, in detail](https://cracked.guide/blog/verify-your-interpreter-with-github-actions).

## Questions

### Why does the Lox test runner fail with Dart 3?

The runner was written before Dart 3 and is not null safe; its pubspec requires Dart below 3.0.0. Use Dart 2.19.6.

### Can I use the test suite for a Lox written in Rust or Python?

Yes. The runner only needs an executable that takes a .lox file path and prints output and errors the way the book's interpreters do.

### How many tests are there?

A few hundred .lox files, grouped by feature (closures, classes, strings and so on). Each chapter suite runs the ones that chapter should pass.

---

Source: https://cracked.guide/learn/lox-test-suite
cracked.guide: A free, dated 145-week path to getting genuinely cracked at programming.