ADR 0007: The engine as a program: JSON Lines protocol, single-file command, WASI module

Status: accepted

Context

Four native libraries cover Python, TypeScript, Java and Go. Programs in Ruby, PHP, Rust, C#, a shell script or anything else need the same engine without a port, on Windows, macOS and Linux. The conformance suite had the mirror problem: each runtime needed its own runner in its own language.

Decision

An engine may be offered as a program that speaks JSON Lines on standard input and output: one request object per line in, one response per line out, in order (specification section 13). The commands are version, load, manifest, evaluate, expression and compile.

The Go runtime is built as that program: rule-cascade, one static binary per operating system and processor, and the same code as a WebAssembly module for WASI preview 1, rule-cascade.wasm. Every native library offers the protocol too, and one driver, rulecheck conformance --engine "<command>", runs the whole suite against any of them.

Consequences

• Any language that can start a process, or host a WASI module, and read and write JSON has the engine. examples/engine-clients shows nine.
• A new runtime is certified by implementing the protocol and pointing the driver at it; it needs no runner of its own.
• One process answers one request at a time. Parallel work needs a pool of processes. A request costs a line of JSON each way; the measurements are in the engine-clients README.
• Custom operators cannot cross a process boundary. The command and the module have only what was built into them, and rules that need another operator fail closed.
• A WASI command module runs from its first input line to its last, so in-process use is a batch; a conversation needs the module in a child process.
• The protocol is a second public interface to keep stable, next to the HTTP API.

Alternatives considered

• A C library and FFI bindings. Each language needs a binding and each platform a shared library; memory ownership crosses the boundary, and a fault in the engine takes the host down.
• gRPC or a local HTTP daemon. Needs a port, a lifecycle and a client library with generated code in every language. The HTTP rule server already serves remote callers; a local caller needs nothing a pipe does not give it.
• A WebAssembly module with a custom call interface. Every host would write glue for memory and strings. Standard input and output work with every WASI runtime as it is.
• A port per language. The cost this project exists to avoid.