How are appeals kept independent of the original decision?
AIRS-J/1 keeps the people involved in a result out of the review of that result. This page sets out who reviews an appeal and what the reviewer works from. It names the interests the Standard keeps away from a result, and what an employer does while an appeal is open.
Judgment is how well a person decides once AI has given the answer. AIRS-J/1, the Standard for Judgment in AI-Enabled Work, sets the rules for certifying it.
A candidate may appeal four kinds of decision under the Standard, within 30 days (Section 18.5). They are a certification-status decision, an employment-grade result released to an employer, an accommodation decision that affects access to assessment, and a suspension. An employment-grade result is a result from an approved instrument at Moderate or High confidence (Section 12.8).
The reviewer stands outside the original decision
Section 17 says the appeals reviewer is "A review board member, or an authorised assessor under §11.6". The reviewer is also "not involved in the original decision or delivery, and not employed by or paid through Exeter Labs".
Exeter Labs is a company that the owner of AIR APAC also owns (Section 4.9). The Standard does not place the reviewer outside AIR APAC. A review board member is drawn from AIR APAC's standing councils, and an assessor is authorised under AIR APAC's assessor standard (Sections 17 and 11.6). AIR APAC names the reviewer in its acknowledgement of each appeal.
Why the rule is not simply run again
Rule-based scoring by an approved instrument must be reproducible: "the same transcript always yields the same observations and levels" (Section 11.2). Section 18.5 states: "Because rule-based scoring is deterministic, re-running the rule is not the review".
The review looks instead for four things (Section 18.5):
- a wrongly applied rule or anchor;
- evidence the instrument extracted wrongly;
- a technical failure;
- a procedural failure.
The grounds in Section 18.6 also apply (Section 18.5).
What the reviewer works from
"The review considers the transcript, published anchors, relevant access-support record and applicable procedures" (Section 18.5).
Two other rules apply to the decision under review. Every observation must be traceable to identified evidence in the transcript: "An observation whose evidence cannot be pointed at does not exist" (Section 8.8). The rubric is public, and AIR APAC "applies no undisclosed criterion, weighting, adjustment or override" (Section 4.1).
The decision under review therefore rests on traceable evidence and published rules. "Outcome and reasoning are provided in writing" (Section 18.5).
Interests the Standard keeps away from a result
- What was bought, and the owner. "Certification outcomes are decided by the rules in Section 12 alone, never by what a candidate or employer bought. The owner of AIR APAC takes no part in any individual result." (Section 4.3)
- How people are paid. "No assessor, delivery partner, trainer or licensee is paid on a basis that varies with candidate outcomes." (Section 4.4)
- Assessor conflicts. Where a human assessor scores or confirms an observation, the assessor declares conflicts before each assessment, and the declarations are retained. No one may be sole assessor for a candidate they have personally trained in the preceding twelve months. No one may be sole assessor for a candidate in their own reporting line, or where they hold a financial interest in the outcome (Section 4.6).
- A hidden conflict. An undisclosed assessor conflict is itself a ground for appeal (Section 18.6).
Separate questions go by separate routes
An appeal asks whether a ground in Section 18.6 is met for one candidate. Other questions have their own routes.
- Disagreement with an anchor. It is not a ground for appeal: "it is a comment on the Standard, and is routed to the review board" (Section 18.6).
- A complaint. A complaint is distinct from an appeal. It "does not change certification status except through the due process in Sections 18.4 to 18.6" (Section 18.7). See the complaints procedure.
- A suspension. It is decided by "a panel of three drawn from the review board", with written reasons and a right of appeal under Section 18.5 (Section 18.4). The reviewer of that appeal was not involved in the suspension decision (Section 17).
- A development reading. A reading from practice, a training session or a result below Moderate confidence (Section 3). The Standard states: "A development reading decides nothing and is not appealable; a comment on it is a complaint under Section 18.7." (Section 18.5)
What the employer does while an appeal is open
Every licence under which an organisation commissions assessments must contain a term on appeals (Section 12.11). Under it, where an applicant appeals within five working days of release, the organisation "does not communicate an adverse decision resting on the result until the appeal is decided".
An appeal on a result released for hiring is decided within ten working days (Section 18.5). The parties who received the result are sent the decision. Where the result changed, they also receive the corrected result, "which replaces the first for every recipient" (Section 18.5).
The employment decision stays with the employer, which "is responsible for its own decision" (Section 12.11).
> What this does not claim > > - It does not say that an appeal will change a result, or how often appeals succeed. > - It does not place the appeals reviewer outside AIR APAC. > - It names no appeals reviewer. > - It does not say that an employer must reverse a decision it has already communicated. Section 12.11 covers an appeal lodged within five working days of release. > AIR APAC publishes the standard for judgment in AI-enabled work, certifies people against it through its official instrument, and stands behind every result.