Custom runners

A custom runner runs test files with your own test kit, beside Jest specs and script tests. It is a Luau module that loads one test file and returns what happened to each test in it.

Your first custom runner

These steps build a small test kit with Test, Skip and Equal, and run it with luau-run test. Install Luau Run first, as Getting started shows.

1. Lay out the project

The finished project looks like this:

my-project/
  luau-run.toml
  tests/
    Math.test.luau
  tools/
    Runner.luau

The runner can live anywhere in the project. Test files keep the .test.luau ending.

2. Write a test file

Create tests/Math.test.luau:

tests/Math.test.luau

return function(T)
  T.Test("adds", function()
    T.Equal(1 + 1, 2)
  end)

  T.Test("multiplies", function()
    T.Equal(2 * 3, 5)
  end)

  T.Skip("divides by zero", "not decided yet")
end

The runner calls this function with its kit, T, and the file registers its tests through T. multiplies fails on purpose.

3. Write the runner

Create tools/Runner.luau:

tools/Runner.luau

--!strict
return {
  Protocol = "luau-run/entry-runner@1",
  Run = function(Context)
    local Tests = {}
    local T = {}

    function T.Test(Name: string, Body: () -> ())
      table.insert(Tests, { Name = Name, Body = Body })
    end

    function T.Skip(Name: string, Reason: string)
      table.insert(Tests, { Name = Name, Reason = Reason })
    end

    function T.Equal(Actual: any, Expected: any)
      if Actual ~= Expected then
        error(`expected {Expected}, got {Actual}`, 2)
      end
    end

    local Register = Context.Entry.Load()
    Register(T)

    local Cases = {}
    for _, Test in Tests do
      local Case = { Path = { Test.Name } }
      if Test.Body then
        Case.Failures = { Context.Try(Test.Body) }
      else
        Case.IsSkipped = true
        Case.Reason = Test.Reason
      end
      table.insert(Cases, Case)
    end
    return { Protocol = "luau-run/entry-terminal@1", Cases = Cases }
  end,
}

Run does four things:

1. It builds the kit, T, whose functions collect the file’s tests.
2. It loads the test file with Context.Entry.Load() and calls the function the file returns.
3. It runs each test through Context.Try, which gives back the test’s failure, or nil when it passes.
4. It returns one case per test.

4. Point Luau Run at it

Create luau-run.toml, or add these tables to the one the project has:

luau-run.toml

[test.runners.kit]
entry = "tools/Runner.luau"
runtime = "lute"

[[test.runner-rules]]
glob = "tests/**"
runner = "kit"

[test.runners.kit] declares a runner named kit, and the rule sends every test file under tests to it.

5. Run it

From the project’s folder:

luau-run test

FAIL  tests/Math.test.luau:7
  multiplies
  Failure: expected 5, got 6

FAILED  1 failed, 1 passed, 1 skipped (3)

The failure names the line, the test, and the message T.Equal threw.

6. Fix the test

Change line 7 to T.Equal(2 * 3, 6) and run luau-run test again:

PASS  2 tests, 1 skipped

Congratulations, you got your first Luau Run custom test runner working!

A kit you already have works the same way: pass its functions to the test file, then run each test it collected through Context.Try.

What Run gets

Field	What it does
Context.Entry.Load()	Runs the test file and returns what the file returns. It can be called only once. An error while the file loads fails the file.
Context.Try(Body)	Calls Body, passing on any arguments after it. Returns nil when it passes, or its failure: { Message, Line, Traceback }. The line is found from the error, and Body may yield.

What Run returns

A table with Protocol = "luau-run/entry-terminal@1" and:

Field	What it holds
Cases	One table per test, in order.
Failures	Failures of the file itself, such as a cleanup that failed.
Status	Instead of Cases, for a runner that only knows the whole file: "passed", or "skipped" with an optional Reason.

Each case:

Field	What it holds
Path	The test’s name, outermost group first, such as { "Math", "adds" }.
Failures	What Context.Try returned, or failures the runner made, such as { Message = "timed out", Line = 12 }. A case with any failure fails.
IsSkipped, Reason	true for a skipped test, and an optional reason.

A runner returns names, messages and lines, and Luau Run writes the report. If a runner errors or returns a table that breaks these rules, Luau Run fails the file and prints the reason.

Cleanups and thrown tables

If your kit throws tables instead of strings, give the runner an ErrorMessage function that reads them. This one also returns failed cleanups in Failures:

tools/Runner.luau

return {
  Protocol = "luau-run/entry-runner@1",
  ErrorMessage = function(Error)
    return if type(Error) == "table" then Error.Text else nil
  end,
  Run = function(Context)
    -- Collect Tests and Cleanups, and build Cases, as above
    local Failures = {}
    for _, Body in Cleanups do
      table.insert(Failures, Context.Try(Body))
    end
    return {
      Protocol = "luau-run/entry-terminal@1",
      Cases = Cases,
      Failures = Failures,
    }
  end,
}

A failed cleanup prints as an ERROR line under the file:

ERROR  tests/Store.test.luau:14: could not close the store
  Code: entry-file-failed

Rules

• A runner rule changes how a selected file runs, not which files are selected, so the kit’s files still end in .test.luau or .spec.luau.
• The last rule that matches a file wins. --log-warnings shows files where rules disagree.
• A runner’s name is lowercase letters, digits and hyphens, starting with a letter. jest and script are the built-in runners, and a rule may name them too.
• entry is a .luau or .lua file, as a path from the project’s folder.
• runtime is lute. Each file runs in its own Lute process, started in the project’s folder, and requires resolve as Lute resolves them.
• A project has at most 64 runners and 256 rules. A file has at most 1,024 tests, 1,024 failures, 64 failures in one test and 16 names in a Path.