How much of a reading is model-assisted, and who checks it?
Under AIRS-J/1, a language model may propose a level for an observation, and a qualified human assessor must then confirm or override it. This page explains how each observation is scored, how the share of model help is recorded, and which people check it.
Judgment is how well a person decides once AI has given the answer. AIRS-J/1 is the standard for judgment in AI-enabled work, and it sets out how each observation is scored. On this page, "reading" means AIR APAC's reading in a result. Section 12.6 describes an employment-grade result as "AIR APAC's expert reading of the person's judgment in AI-enabled work, which the employer may rely on as one input."
How an observation is scored
A result is built from observations. Each observation is scored on the 0 to 4 scale of Section 7. Each must be traceable to identified evidence in the transcript (Section 8.8). The record of every observation states its scoring mode: by a rule-based instrument, by a human assessor, or by a model with a human assessor confirming (Annex C).
Scoring by rule. An approved instrument may score by applying published rules to the retained record of what the person did. Section 11.4 says such scoring "generates observations directly". In its words, "every level cites the events and the rule that produced it, and no human confirmation is needed". Section 11.2 adds that "the same transcript always yields the same observations and levels, tested on every release". Where an approved instrument scores by rule, AIR APAC issues the certification decision under Section 12 (Section 11.6).
Scoring with a model's help. Section 11.4 sets a different rule for language models. A model-generated score "is not an observation and never stands alone".
When a model helps
Section 11.4 sets six conditions. In plain words:
- A model may propose a level, and it must cite the evidence in the transcript.
- A qualified human assessor then confirms or overrides the proposal, and the reasons are recorded.
- The proposal and the assessor's decision count as one observation, attributed to the human assessor.
- Where the model and the assessor differ by more than one level, the observation goes to a second human assessor. That second assessor's decision is final.
- The model-assisted proportion is recorded on the result.
- The model version and the prompt version are fixed and recorded in the transcript ("pinned", in the Standard's word).
Section 9.6 adds that every transcript records the scenario version, rubric version, instrument version and scaffolding level. In its words: "A result that cannot state these is void."
No model's opinion stands alone as a reading.
How much of a reading is model-assisted
AIRS-J/1 sets no fixed share. Each result records its own model-assisted proportion (Section 11.4). The candidate's report states it, together with the dimension scores, band, confidence and evidence (Section 12.5). The candidate receives this report whether or not they certify.
Who checks the model's part
- Authorised assessors. Only an assessor authorised under AIR APAC's assessor standard may score or confirm an observation (Section 11.6). That assessor standard covers entry competence, training, calibration, monitoring, conflict controls, suspension and removal.
- Testing again at each change. Model-assisted scoring is tested again against a set of transcripts kept aside and scored by people ("re-benchmarked", in the Standard's word). This happens on each model or prompt change, and at least once a quarter (Section 11.5).
- Suspension on failure. If model-assisted scoring fails to meet the agreement requirement of Section 11.2 against human scores, automated scoring is suspended for the affected dimensions (Section 11.5).
What is never a scoring input
Section 8.7 names four kinds of signal that are never scoring inputs: "Keystroke capture, affect inference, video analysis and biometric signals". The reason is given in the same sentence: the Standard reads what a person did, "which can be shown to them and explained". No scoring mode may use them, whether a rule, a human assessor or a model proposes the level.
If a person thinks a reading is wrong
A candidate may appeal, among other decisions, a certification-status decision or an employment-grade result released to an employer, within 30 days (Section 18.5). A development reading decides nothing and is not appealable; a comment on it is a complaint under Section 18.7. Section 18.5 also says that "re-running the rule is not the review", because rule-based scoring is deterministic. The review looks for a wrongly applied rule or anchor, evidence the instrument extracted wrongly, a technical failure or a procedural failure. The grounds in Section 18.6 include "an observation not supported by transcript evidence". The route is on the appeals page.
What this does not claim
- It gives no typical, average or highest model-assisted share. Each result records its own.
- It does not say whether the official instrument uses model-assisted scoring, or which model it would use.
- It does not say whether the public register entry shows the share.
- It does not say that assessors never make mistakes. The appeal grounds let a candidate ask for a review of how an anchor was applied, or of an observation the transcript does not support (Section 18.6).
- It reports no validation finding. The validation report is published as the checks in the Standard are met.
AIR APAC publishes the standard for judgment in AI-enabled work, certifies people against it through its official instrument, and stands behind every result.