How many scenarios and observations make enough evidence?

Judgment is how well a person decides once AI has given the answer. AIRS-J/1 is the standard AIR APAC publishes for it. A certification result under AIRS-J/1 rests on evidence that is counted as well as scored. In short, each of the nine dimensions needs at least three observations, so a certifying Assessment has at least 27 (Sections 8.3 and 8.4). They come from at least three scenarios (Sections 8.4 and 10) that span at least three consequence families (Section 6.4). This section gives those counting rules in the Standard's own words. It says why they exist and what the confidence line on a result tells the person reading it. Section numbers refer to AIRS-J/1 v2.3.

What is an observation?

The Standard defines an observation as "A single scored instance of one dimension, arising from one assessed decision point, supported by identified evidence in the transcript" (Section 3). A decision point is "A moment in an assessment at which the candidate must commit to an action, position or artifact that carries cost" (Section 3). Each observation is scored on the published levels 0 to 4 (Section 7). It must be traceable to the transcript: "An observation whose evidence cannot be pointed at does not exist" (Section 8.8).

How many observations does each dimension need?

"Each dimension requires a minimum of three independent observations arising from distinct decision points" (Section 8.3). The rule applies to all nine dimensions in all four domains. A dimension below the minimum "is scored Insufficient Evidence and cannot be certified" (Section 8.3). Rule 4 of the six decision rules requires Section 8.3 to be "satisfied for all nine dimensions" (Section 12.2). One dimension below the minimum therefore stops certification, however strong the other eight are, because all six rules must be met (Section 12.2).

Why must observations come from distinct decision points?

The Standard says: "Two observations from the same decision point are one observation" (Section 8.3). A plain reading: one moment of decision shows how a person acted once. Counting that moment twice would make a single act look like a pattern. Three distinct decision points mean that each dimension is read from at least three separate moments, each with its own cost, before it can count towards certification.

How many observations does a certifying Assessment need in total?

"The minimum implies 27 observations" (Section 8.4). That is three for each of the nine dimensions. It is the lowest count the Standard accepts. The confidence rules in Section 10 ask for more before they make a stronger statement about the evidence. Moderate confidence needs three to four observations per dimension. High confidence needs five or more per dimension.

How many scenarios does a certifying Assessment use?

The Standard leaves the exact number to the instrument. "The official instrument's coverage per scenario, and the number of scenarios a certifying assessment needs, are set in its specification" (Section 8.4, Annex F). Section 8.4 states that the official instrument's rule is at least three held-back Assessment scenarios. Confidence adds its own scenario counts: three or more for Moderate, four or more for High (Section 10). Certifying scenarios are held back for Assessment. A scenario is retired from certification use when its exposure exceeds the published threshold, or "when it appears in public preparation material". It is also retired when performance shifts in a way consistent with content leakage (Section 9.3). Retired scenarios may be released for practice.

What range of consequences must the scenarios cover?

Section 6 opens with the reason: "Judgment scores are not comparable across situations of different consequence" (Section 6). Every assessment instance declares a stakes profile, and consequence families are recorded, not scored (Sections 6.1, 6.2). "A certifying assessment must span at least three distinct primary consequence families" (Section 6.4). At least one of them "must involve irreversible or non-financial harm (C1, C4, C5b or C6)". Section 6.1 names these as people safety and welfare (C1), information and privacy harm (C4), institutional and public trust, reputation and legitimacy (C5b), and environmental and community-system harm (C6). A candidate assessed only on financial and operational scenarios has not met this requirement. Section 12.3: "Rule 5 stops a candidate clearing on three financially-framed scenarios."

What does the confidence line tell a reader about the evidence?

"Confidence describes evidential sufficiency, not the candidate's ability" (Section 10). It states how much evidence there is and how widely it was gathered:

• High. Five or more observations per dimension, from four or more scenarios, scored by rules that meet Section 11.2. Stakes coverage under Section 6.4 is exceeded.
• Moderate. Three to four observations per dimension, from three or more scenarios. Stakes coverage is met.
• Provisional. The minimum count is met, but the observations cluster in a single scenario, or stakes coverage is not met.
• Insufficient. The minimum is not met for one or more dimensions.

"Confidence is printed on the credential and shown in the register" (Section 10).

Why can a result with too little evidence not certify?

Section 12.2 has six decision rules, and all six must be met. Rules 4 to 6 concern the evidence: Section 8.3 met for all nine dimensions, stakes coverage met, and confidence Moderate or High. Section 12.3 states what they are for: "Rules 1 to 3 govern level; rules 4 to 6 govern whether the level is knowable". A plain reading: a level read from too few moments, or from one kind of consequence, cannot be relied on to describe the person. "Certification may be issued at Moderate or High only" (Section 10). Provisional and Insufficient results are development readings with no band (Section 10.1).

Does more evidence mean a higher level?

No. A dimension score "is the arithmetic mean of its valid observations" (Section 12.2.1). A plain reading: more observations give a better-supported reading, but they do not raise it by themselves. Band and confidence answer different questions. The band reports where the composite falls, with its floors (Section 12.4). Confidence reports how much evidence stands behind it. A High-confidence result and a Moderate-confidence result can carry the same band. Section 10 states: "An employer can distinguish a High-confidence Certified from a Moderate-confidence Certified."

What does not count as evidence?

"Self-assessment, survey response, declared experience, training attendance and manager endorsement generate no observations and contribute nothing to a score" (Section 8.2). A plain reading: a long career or a completed course adds nothing to the count. The transcript keeps "decisions, artifacts, sources consulted, stated rationale, revisions and timings" (Section 8.7). Keystroke capture, affect inference, video analysis and biometric signals "are never scoring inputs" (Section 8.7). The Standard "reads what a person did, which can be shown to them and explained" (Section 8.7).

Who can see the evidence behind a result, and is verification free?

The candidate receives "the stakes profiles assessed" and "the evidence cited for each observation", whether or not the candidate certifies (Section 12.5). An employer that commissions assessments of its own people receives each result by name with "the evidence behind each observation" (Section 12.6). For applicants, the employer receives each employment-grade result by name, in the form of Section 12.5, for that hiring decision. Where the result is Provisional or Insufficient, the employer is told only that no employment-grade result was obtained (Section 12.6). An independent candidate's result goes only to parties the candidate names. The register "does not record employer, dimension scores, stakes profiles, or any transcript content" (Section 14.1). Verification is free and needs no account (Section 14.3).

What this does not claim

• It gives no count of decision points per scenario and no length for an Assessment. The official instrument's specification sets its coverage per scenario (Section 8.4, Annex F).
• It does not claim that more observations raise a level. Confidence describes the evidence, not ability (Section 10).
• It does not claim that experience, training or a manager's view adds evidence (Section 8.2).
• It does not claim that the register shows evidence or scores (Section 14.1).
• It describes no certifying scenario, and reading it does not change anyone's chance of certifying (Sections 9.3, 20.2).
• It gives no person's result as a percentage or a rank. Results are reported as levels and bands (Section 12.4).

AIR APAC publishes the standard for judgment in AI-enabled work, certifies people against it through its official instrument, and stands behind every result.

Verify a certificate