Why does the Standard score from published rules and not from impressions?
AIRS-J/1 scores only what a person did in a recorded assessment, by rules that anyone can read. This FAQ explains why, what the Standard leaves out, and how a reader can check a reading.
Judgment is how well a person decides once AI has given the answer. AIRS-J/1, the AIR APAC Standard for Judgment in AI-Enabled Work, scores it only from what a person did in an assessment, by rules that anyone can read. The answers below explain why, what the Standard leaves out, and how a reader can check a reading.
What does "published rules" mean?
The complete rubric of AIRS-J/1 is public and versioned (Section 4.1). AIR APAC "applies no undisclosed criterion, weighting, adjustment or override". The limit is stated plainly: "Any criterion not in this document, or in the instrument specification it incorporates (Annex F), does not exist" (Section 4.1). A rule that is not published is not used to score anyone. An instrument is the assessment tool that records what a person does; Annex F is its written specification.
What counts as an impression, and why does it not count?
The Standard does not use the word "impression". In this FAQ it means a view of a person that does not come from what the person did in the assessment.
Section 8.2 names five such sources: self-assessment, survey response, declared experience, training attendance and manager endorsement. They "generate no observations and contribute nothing to a score". Section 7.3 adds that an assessor "may not score on inferred motivation, attitude, confidence or personality".
Section 1.1 describes the human contribution of deciding whether AI work is fit to act on. It says: "Training attendance, tool proficiency certificates and self-reported confidence do not measure it". This is a rule about evidence for a score. It does not say that a manager's view is wrong, or that it has no value for other purposes.
Where does a score come from instead?
A score comes from observations, and every observation comes from evidence in the transcript. Section 3 defines an observation as "a single scored instance of one dimension, arising from one assessed decision point, supported by identified evidence in the transcript".
The transcript is "the complete retained evidence record of an assessment instance, from which every observation must be independently derivable" (Section 3). Each observation is read against an anchor, the published description of a level for that dimension. The test is plain: "An observation whose evidence cannot be pointed at does not exist" (Section 8.8).
What is never used as evidence?
Keystroke capture, affect inference, video analysis and biometric signals are never scoring inputs (Section 8.7). The transcript keeps "decisions, artifacts, sources consulted, stated rationale, revisions and timings". In the words of Section 8.7, "the Standard reads what a person did, which can be shown to them and explained."
Why does this matter when two people are compared?
Section 1 lists the purposes of the Standard. One is that "Two individuals assessed in different organisations, countries or delivery settings receive comparable results" (purpose 2). Another is that "Any party (a candidate, an employer, a regulator, a competitor) can inspect the criteria against which a decision was made" (purpose 3).
Section 1 adds that "Comparability is the governing design constraint". Scenario settings may be localised, but criteria may not (Sections 4.5 and 8.5). So the same published rules apply to every candidate, in every organisation and country, and anyone can read them.
Does the same record always give the same reading?
For rule-based scoring, the Standard requires it. Section 11.2 states that "the same transcript always yields the same observations and levels", and requires this to be "tested on every release".
Where human assessors score, independent and authorised assessors must agree on a common set of transcripts. Section 11.2 sets the level of agreement they must reach for each dimension. A dimension below that threshold has its anchor or its rule rewritten in the next version.
Where does a model fit?
Under Section 11.4, rule-based scoring by an approved instrument generates observations directly, and "every level cites the events and the rule that produced it". A model-generated score "is not an observation and never stands alone": a qualified human assessor confirms or overrides any level a model proposes, with reasons recorded.
Every reading is scored by published rules and traces to what the person did. No model's opinion stands alone as a reading.
Can an employer or trainer add its own criteria?
No. Under Section 4.5, delivery partners and licensed trainers "may not vary, extend, reweight or supplement the dimensions, anchors, thresholds or decision rules in this Standard". An employer describes the role; under Section 12.10 it "does not set, weight or add criteria".
What does the candidate see so they can check?
Every candidate receives a full report under Section 12.5. It includes "the anchors applied" and "the evidence cited for each observation". It also gives every dimension score that can be reported separately, the four domain scores, the band, the confidence level and the stakes profiles assessed. The report "is provided whether or not the candidate certifies" (Section 12.5). A candidate can set each level beside the published anchor and the act it cites.
What if a rule was applied wrongly?
The candidate may appeal on the grounds in Section 18.6, which include "misapplication of an anchor" and "an observation not supported by transcript evidence" (how to appeal).
What if the rule itself seems wrong?
That is a comment on the Standard, not an appeal. Section 18.6 states: "Disagreement with an anchor itself is not a ground for appeal; it is a comment on the Standard, and is routed to the review board."
The rules can change. Under Section 4.8, where monitoring shows that a claim needs strengthening, AIR APAC acts on it in the next release. In the words of Section 4.8, "it fixes the instrument, the rule or the anchor, and the validation report records what changed".
Does a published rule mean the reading is proven?
No. A published rule shows how a reading was reached. Section 5.1 calls the nine dimensions "a reporting hypothesis, not an established factor structure". In plain words, the evidence does not yet show that the nine dimensions are separate from one another (Section 5.1). Section 11.7 sets out the checks, V1 to V8, that the validation report must show before certain claims in Section 2.1 may be made.
Every reading is scored by published rules and traces to what the person did. The validation report is published as the checks in the Standard are met.
What this does not claim
- It does not claim that the rules or the official instrument have passed the checks in Section 11.7.
- It does not claim that a manager's view, a training record or a person's own account of their work is wrong. The Standard does not count them as evidence for a score.
- It does not claim that no person reads a transcript. A qualified human assessor confirms or overrides any level a model proposes (Section 11.4), and only an authorised assessor may do so (Section 11.6).
- It does not claim that a result decides employment. A result is one input to an employer, and the employment decision is the employer's (Section 12.6).
AIR APAC publishes the standard for judgment in AI-enabled work, certifies people against it through its official instrument, and stands behind every result.