Conformance suite

A runtime conforms to Rule Cascade 1.0 when it passes the cases in this directory for the level it claims. Every runtime in this repository runs exactly these files, on every operating system.

Two ways to run it

In process. Each runtime has a test that reads these files directly:

python tools/rulecheck.py conformance                 # Python reference
npm test -w @yarlisaisolutions/rule-cascade           # TypeScript
make java                                             # Java, no Maven needed
make go                                               # Go

Over the engine protocol. One driver certifies any engine in any language, as long as it speaks the JSON Lines protocol of specification section 13:

python tools/rulecheck.py conformance --engine "rule-cascade engine --conformance-operators"
python tools/rulecheck.py conformance --engine "node packages/typescript/dist/cli.js engine --conformance-operators"

This is the quickest way to bring up a new runtime: implement the protocol, point the driver at it, and fix what it reports.

The conformance operators

Every runner registers three custom operators. They exist to test the custom-operator mechanism and are small enough to write in any language in a few minutes.

Writing a runner for a new runtime

1. Expressions. For each case evaluate expr with the roots in env, the function table in functions and the conformance operators registered. Expect expect, or an evaluation error when error is true. Compare numbers numerically and everything else by type and value.
2. Bundles and golden tests (evaluator). Read each file in rulesets.json > bundles. Run the tests of the matching document in documents: decision, the exact set of finding rules, every listed member of each expected finding against some finding of that rule, each listed effect as a subset match, and the command names in order. A test's given plus its entity and operation is the request; channel defaults to server.
3. Evaluations. For each case evaluate request against the bundle named by ruleset on channel, drop detail from findings, and require deep equality with result.
4. Rulesets (compiler). Load each document in documents with the others as its registry and schemaDocuments answering entity $ref lookups. Compare checksum, the SHA-256 of the canonical JSON of the bundle (bundleChecksum), the client manifest's rule ids and its param names. The bundle you produce must equal the published one.
5. Load errors (compiler). Apply the parent and child merge patches to base, load the child with both as the registry, and require that expectError is among the reported codes, or that the load succeeds when it is null.

Where the expected values come from

Expression, load-error and protocol expectations are written by hand and checked against the reference when they are added. Checksums, bundles and the evaluation corpus are produced by the reference implementation (python tools/rulecheck.py sync), so they prove that runtimes agree with the reference, not that the reference is right. The golden tests and the hand-written cases are what tie the reference to the specification.

Rendered from conformance/README.md in the repository. Edit it there.