helpers/mkResult

The primary way to define a result check.

Runs command in a derivation with set +e. Exit code, stdout, and stderr are captured in separate outputs (exitCode, stdout, stderr) rather than failing the build; the derivation always succeeds regardless of the command’s outcome.

For extra derivation attributes (e.g. nativeBuildInputs), use mkResultWith directly.

Type

mkResult :: String -> String -> Derivation

Arguments

name

Check name. Becomes the derivation name result-<name>.

command

Shell command to run as the check body.

Example

mkResult "hello" "hello --version"

mkResultWith {
  name = "result-grep-output";
  nativeBuildInputs = [ pkgs.ripgrep ];
  buildCommand = mkResult.buildCommand "rg 'pattern' somefile";
}

helpers/mkResultWith

Low-level result check builder. Prefer mkResult for most use cases.

Produces a derivation with four outputs: out (sentinel), stdout, stderr, and exitCode. The command attribute is a shell script body whose stdout, stderr, and exit code are captured automatically. The derivation always exits successfully.

For full control over the build script, use buildCommand directly (bypasses capture wrapping).

If passthru.skip is true, delegates to mkSkip automatically.

Type

mkResultWith :: AttrSet -> Derivation

Arguments

attrs

Attribute set passed to stdenvNoCC.mkDerivation. Required keys: name and either command or buildCommand. All other mkDerivation keys are supported.

Example

mkResultWith {
  name = "result-my-check";
  nativeBuildInputs = [ pkgs.hello ];
  command = ''hello --greeting "hi"'';
}

helpers/mkSnapshot

Assert the outputs of a result check match expected values.

Compares exitCode, stdout, and/or stderr of the wrapped check against expected strings. Any mismatch is reported to the snapshot’s own stderr output. At least one of exitCode, stdout, or stderr must be provided.

For extra derivation attributes, use mkSnapshotWith directly.

Type

mkSnapshot :: String -> AttrSet -> Derivation -> Derivation

Arguments

name

Check name. Becomes the derivation name snapshot-<name>.

expectations

Attribute set with the following keys:

exitCode (optional)

Expected exit code string, e.g. "0" or "1".

stdout (optional)

Expected stdout content.

stderr (optional)

Expected stderr content.

resultCheck

The result check derivation to test.

Example

mkSnapshot "hello-snapshot" { exitCode = "0"; stdout = "hi\n"; }
<| mkResult "hello" "echo hi"

helpers/mkEval

Declare a suite of pure Nix evaluation tests as an eval check.

An eval check is plain data — no derivation, no store access. Tests are evaluated lazily, one attribute per test, so runners such as nrc can shard them across nix-eval-jobs workers. Use mkEntries to compute the per-test results.

Registered in resultChecks.checks, an eval check displays as a suite with one entry per test. Use this for testing pure Nix functions. For testing shell commands or build-time behaviour, use mkResult or mkSnapshot.

Type

mkEval :: AttrSet -> EvalCheck

Arguments

tests

Attribute set of test cases. Each entry must have expr (the value under test) and expected (the expected value). Unlike lib.debug.runTests, every attribute is a test regardless of its name.

Example

resultChecks.checks.my-lib = mkEval {
  testAdd = {
    expr = myLib.add 1 2;
    expected = 3;
  };
};

helpers/mkSkip

Mark a check as skipped.

For result check derivations, clears all build inputs, replaces the build command with a no-op, and sets passthru.skip = true, which generators use to report the check as skipped rather than passed or failed.

For eval checks, sets skip = true; mkEntries then reports every test as skipped without forcing its expression.

Type

mkSkip :: (Derivation | EvalCheck) -> (Derivation | EvalCheck)

Arguments

check

A result check derivation produced by mkResult, mkResultWith, or mkSnapshot, or an eval check produced by mkEval.

Example

mkResult "my-check" "echo hello" |> mkSkip

helpers/mkEntries

Compute per-test result entries for an eval check.

Each test in the check becomes an entry of the form

{ kind = "eval"; status; exitCode; stdout; stderr; }

where status is "pass", "fail", or "skip". Failures carry a formatted report in stdout and a summary in stderr, matching the conventions of derivation-based checks. Entries are computed lazily: a test’s expression is only forced when its entry is, and skipped tests are never forced at all.

Skipping is controlled by skip = true on the check (set by mkSkip) or by listing test names in skipTests on the check.

Type

mkEntries :: EvalCheck -> AttrSet

Arguments

check

An eval check produced by mkEval.

Example

mkEntries (mkEval {
  testAdd = {
    expr = 1 + 2;
    expected = 3;
  };
})
=> { testAdd = { kind = "eval"; status = "pass"; exitCode = "0"; stdout = ""; stderr = ""; }; }

helpers/mkReport

Generate the report package for the derivation checks in a check set.

Takes checks in the same shape as resultChecks.checks — flat derivations, suites, and eval checks — and reports on the derivation half. Eval checks are not derivations; they are covered by mkEvalChecks instead.

Type

mkReport :: AttrSet -> Derivation

Arguments

checks

Attribute set of checks. Flat: name = drv. Suite: name = { checkName = drv; ... }. Eval checks are ignored.

Example

mkReport {
  my-test = mkResult "my-test" "exit 0";
  db.schema = mkResult "db-schema" "exit 0";
}

helpers/mkEvalChecks

Compute the entries tree for the eval checks in a check set.

Takes checks in the same shape as resultChecks.checks and computes per-test entries for the eval half, keyed by check then test name. Entries are lazy, so runners can force them in parallel. This is the evalChecks half of the value nrc consumes, behind the resultChecks.<system> flake output and the --file convention.

Type

mkEvalChecks :: AttrSet -> AttrSet

Arguments

checks

Attribute set of checks. Derivation checks are ignored; they are covered by mkReport.

Example

mkEvalChecks {
  my-lib = mkEval {
    testAdd = {
      expr = 1 + 2;
      expected = 3;
    };
  };
}
=> { my-lib.testAdd = { kind = "eval"; status = "pass"; exitCode = "0"; stdout = ""; stderr = ""; }; }

generators/json

Generate a newline-delimited JSON report from result check outputs.

Prefer mkReport, which accepts checks in their natural shape; this generator takes the normalized form it produces.

Each line of the output is a JSON object with the following fields:

• name: the attribute name of the check
• suite: suite name, or null for flat checks
• kind: the check type ("result", "snapshot", or "eval")
• status: "pass", "fail", or "skip"
• exitCode: the raw exit code string
• stdout: captured stdout
• stderr: captured stderr
• drvPath: path to the check derivation in the Nix store

kind reflects passthru.kind on the result check derivation. status is "skip" when exitCode is empty (set by mkSkip).

Type

json :: AttrSet -> Derivation

Arguments

checks

Attribute set of { check, suite } pairs, keyed by entry key.

Example

pkgs.resultChecks.json.override { inherit checks; }

Options

resultChecks.enable

Enable Result monad checks.

Type: boolean

Default:

true

resultChecks.enableFlakeChecks

Whether to add an aggregate Result check to flake.checks.

The single resultChecks flake check depends on every derivation check (built in parallel by the scheduler) and bakes in eval check verdicts. Its log carries the full per-check report; it fails if any check failed.

Type: boolean

Default:

true

resultChecks.checks

Checks to run.

Values are a derivation (flat check), an attrset of derivations (suite), or an eval check (mkEval). Suite and eval checks are grouped under a named header in the TUI and keyed as "suite:name" in reports.

Type: attribute set of (package or (eval check (mkEval)) or attribute set of package)

Default:

{ }

resultChecks.evalChecks

Per-test result entries of all eval checks, keyed by check then test name. Entries are computed lazily so runners can force them in parallel (e.g. via nix-eval-jobs).

Type: lazy attribute set of lazy attribute set of raw value (read only)

Default:

pkgs.resultChecks.mkEvalChecks cfg.checks

resultChecks.report

The generated report package.

Covers derivation checks only; eval check results are exposed through evalChecks so that runners can evaluate them in parallel without the report forcing them.

Type: package (read only)

Default:

pkgs.resultChecks.mkReport cfg.checks

resultChecks.skipChecks

Check keys to skip.

Flat checks are identified by name (e.g. "lint"). Suite checks and eval tests are identified as "suite:name" (e.g. "db:schema").

Skipped derivation checks are replaced with placeholder derivations; skipped eval tests are never evaluated. Both are marked as skipped in the report.

Type: list of string

Default:

[ ]