Incident: RULE-EVALUATION-ERROR spike
Blocking RULE-EVALUATION-ERROR findings are rising. Find the rule and the cause from the finding's detail, reproduce offline, and fix the host or roll back the bundle.
A rule that cannot be evaluated never passes silently: it produces a blocking finding with code
RULE-EVALUATION-ERROR, the generic message "This rule could not be evaluated." and a detail that
says why, and the decision is deny. Users see operations refused that should succeed. Treat a rise
as an incident.
Triage
Measure the blast radius
From the decision logs (every decision is logged with ruleset id, version, checksum and finding
codes), group the RULE-EVALUATION-ERROR findings by rule, ruleset checksum and service. Note
when the rise started.
| What you see | Most likely |
|---|---|
| It started with a deployment of the rules (new checksum) | A rule meets data it was not written for, or needs an operator the host lacks |
| It started with a deployment of the application (same checksum) | The application now sends data of another type, or no longer sends ctx.now |
| One service only, same checksum elsewhere is fine | That host's custom operators or request building |
Read the detail
Each finding names the rule and the reason. Real examples from the example rulesets:
detail | Cause | Fix |
|---|---|---|
number expected, got "120" | A value of the wrong JSON type reached the rules: a string where the schema says number | Validate the shape of the body before evaluating and answer 400; convert form input to the entity's types |
string expected, got null on a rule that reads ctx.now | The request carries no ctx.now | Pass the server clock in every request (ctx: { now }), in the UI too |
custom operator x-luhn is not registered | The ruleset declares a custom operator this engine does not have | Register it; compare the manifest's operators with the registered ones at start-up and refuse to start on a gap |
| Any other detail, on a rule that changed recently | The rule's expression does not handle data it now meets (for example a missing optional field) | Roll back, then fix the rule with a golden test (below) |
The engine protocol, the command and the rule server have no custom operators of yours: a ruleset that needs one fails closed there by design.
Reproduce offline
Evaluation is a pure function of the bundle and the request, so the failing request reproduces exactly with the bundle whose checksum the log names:
echo '{"entity":"Transfer","operation":"create","data":{"id":"t-5","type":"domestic","amount":"120","currency":"USD","memo":"rent","beneficiary":{"name":"Jo","country":"US"}}}' |
rule-cascade evaluate --bundle acme.payments.transfer.bundle.json - |
jq -c '.findings[] | select(.code == "RULE-EVALUATION-ERROR") | {rule, detail}'{"rule":"org.transfer.amount-limit","detail":"number expected, got \"120\""}
{"rule":"transfer.amount.positive","detail":"number expected, got \"120\""}
{"rule":"transfer.large.review-warning","detail":"number expected, got \"120\""}Mitigate
If a rule change caused it, roll back the bundle
Bundles are immutable: deploy the previous one. A service that embeds a runtime rolls out the previous bundle; the rule server gets the previous ConfigMap and a rolling restart (Kubernetes playbook). Browsers follow at their next manifest fetch.
If the host caused it, fix the host
Restore shape validation, the type conversion, ctx.now or the operator registration, and deploy
the application. Do not weaken the rule to make the error go away: the error is the rule refusing to
guess.
Fix forward with a golden test
Add the failing request as a golden test of the ruleset, so the case is pinned before the fix
ships, and follow ship a rule change. If the rule must accept a
missing value, guard it with when: { op: exists, ... }; the cookbook has
the patterns.
Done when the rate of RULE-EVALUATION-ERROR findings is back to its baseline (normally zero),
the request that triggered it is a golden test or a shape-validation test, and an alert on the
finding code is in place.
Prevent it
- Alert on any
RULE-EVALUATION-ERRORfinding; report itsdetailto error tracking (architecture: operating it). - Validate the shape of every request before evaluating (
400), so rules only see data of the type the schema gives it (enforcement guide step 7). - Check custom operators at start-up in every runtime, the browser included.
- Pass
ctx.nowin every request.
Source: site/content/docs/playbooks/incident-rule-evaluation-error.mdx