Added in 0.3. The matching producer capability is signals.
An attempt’s outcome says how that attempt ended. Signals say how the run’s result was judged, by whom and when: a test verdict, a reviewer’s decision, a runtime measured, a revert two days later. The acceptance rule says which signals make a run a success for this user, so two teams with different bars can record the same kind of run and each read success their own way.

​
Signals

One timestamped observation about the run’s result: a verdict, a score or a late event. Value types follow the kind (E195).
​
Id
required
​
enum
required
One of: verdict, score, event.
​
string
required
Recommended: verdicts tests, referee, pair_referee, merge, ci; scores bench_score, runtime_s, quality; events issue, incident, revert, hotfix.At most 200 characters.
​
string | number | null
required
verdict: one of accept, reject, pass, fail, error. score: a number, or null for a declared score not measured yet. event: a reference string. Only a score may be null (E195).At most 500 characters.
​
string
Unit of a score.At most 100 characters.
​
enum
Which direction of a score is better.One of: higher, lower.
​
number
Optional threshold that turns a score into pass or fail.
​
enum
How the model treats a score. linear (default); log for positive ratio-like quantities (runtime, memory, latency); fraction for values in [0, 1] (pass rates; E195).One of: linear, log, fraction.
​
string
The attempt after which the verdict applies; it locates the attempt at which the run was accepted.At most 200 characters.
​
Timestamp
required
When the signal was observed. Late events are appended with their own time.
​
object
​
OutcomeTier
required
​
Ext
A signal’s kind fixes what its value may be (E195):
  • Only a score MAY be null. A score with scale: "fraction" MUST lie in [0, 1]. So MUST acceptance_rule.score.target when that rule’s scale is fraction (E195).
  • scale tells a model how to treat a score: linear (the default), log for positive ratio-like quantities such as runtime, memory or latency, and fraction for rates.
  • at_attempt names the attempt after which a verdict applies, so a reader can tell how many attempts it took to reach acceptance.
  • tier says how the signal is known, from verified (a check ran) to asserted (nothing backs it). See evidence tiers.
  • A late event, such as a revert, is appended to the run’s signals with its own observed_at. An event about a commit is matched to its run through the run’s commit or merge artifact.

​
Acceptance rule

run.acceptance_rule is the rule in effect when the run was recorded. It travels with the run, so a document can be read the same way after the user’s rule changes. The user’s acceptance rule in effect when the run was recorded. The outcome function reads success through it. A rule with only score expresses runtime targets, benchmark scores and quality bars; a binary rule is the special case with score null.
​
string
required
For example tests+referee or perf>=2400.At most 200 characters.
​
ShortText
required
For people: tests pass and referee accepts.
​
string[]
Verdict names that must pass.
​
ScoreTarget | null
A score target. With requires empty, success is score reaches target; with both, success needs both.
​
string[]
Events that turn success into failure inside the window. [“revert”, “incident”] when absent.
​
integer
14 when absent.
​
Ext
​
string
required
A score signal name (E195: never a verdict or event name).At most 200 characters.
​
number
required
​
enum
required
One of: higher, lower.
​
enum
One of: linear, log, fraction.
​
Ext
A run is accepted under the rule when all of these hold:
  1. Verdicts. Every name in requires has a passing verdict (accept or pass).
  2. Score. When score is set, the named score reaches its target: at least the target when better is higher, at most when lower. With requires empty the score alone decides; with both, success needs both. A rule whose score is null is a plain pass-or-fail rule.
  3. Late events. No event named in excludes_events was observed within window_days of the run’s end. Such an event turns a success into a failure. excludes_events is ["revert", "incident"] when absent, and window_days is 14.
acceptance_rule.score.name MUST NOT name a verdict or event signal, and requires MUST NOT name a score or event signal (E195).
How loopmath reads it. For each name, the latest verdict by observed_at counts. An error verdict is neither a pass nor a fail, so that part of the rule is unknown, as is a missing verdict or a score still null. The result is only as trustworthy as the weakest tier among the signals it used.

Privacy