For boards and executives

Comparable across people, teams and markets

Two managers, the same AI output, two different decisions. Which was right, and would the Jakarta team have decided as the Seoul team did? Until judgment is scored on one published scale under like stakes, nobody can answer either question.

8 min read By AIR APAC

A regional head receives the same AI-drafted pricing recommendation in two offices. In one, the country manager accepts it with a note on the assumptions she checked. In the other, the country manager rejects it and escalates. Both are experienced. Both are confident. The regional head now has two decisions on one output and no way to say which was the better-judged. She cannot tell whether the difference lies in the managers, the markets or the morning they each had. This scenario is illustrative, and AIR APAC holds no team data of this kind yet.

Variation between decision-makers predates AI, and AI's effect on it is an open question

The argument that people in the same role reach different decisions on the same facts is well made and older than the current tools. Kahneman, Sibony and Sunstein called it noise: variability in judgments that should be identical. Two judges in the same court handing down different sentences for the same crime. Two doctors in the same city reaching different diagnoses for identical patients. The variation was invisible because nobody compared the decisions, and it was costly because the organisation paid for every wrong one.

What AI does to that variation across a team is not answered in the research reviewed for this series. Two hypotheses are plausible and point in opposite directions. The first is that a shared AI recommendation anchors everyone to the same answer and reduces the spread. The second is that the recommendation is accepted by some and overridden by others on grounds that differ by person, and the spread widens, now with the tool's confidence attached to one side. The research on overreliance reviewed for this series is almost entirely about individuals on tasks. None of it compares teams. AIR APAC poses this as the open question it is.

For a board, the open question is the practical one. Whether the organisation's judgment with AI in the room is consistent across its markets is something a board might reasonably want to know and currently cannot. Not because the answer is hidden, but because there is no common scale on which the Seoul decision and the Jakarta decision could be placed.

Comparability is the Standard's governing design constraint

AIRS-J/1 says in its own text that comparability is its governing design constraint. The nine dimensions, the anchors for every level, the evidence rules and the decision rules are public in full. No scoring criterion applied in certification exists outside the published Standard and the specification of the official instrument that it incorporates by reference. A person assessed in one organisation, country or industry is read against the same anchors as a person assessed in any other. That is what makes a result portable for the person and usable for the organisation. It is also the reason the Standard is single: tracks differ in scenario complexity, stakes density and stakeholder count, never in dimensions, anchors, thresholds or decision rules.

Scenario settings may be localised. The decision points may not. Sector, role titles, organisational norms, regulatory context and language can all be adapted for a market. The decision points, the dimensions they elicit, the stakes profile and the anchors applied are fixed, and a localised scenario needs AIR APAC's approval before it is used in certification. The Jakarta scenario can look like Jakarta. The decision it tests is the same one Seoul faced.

For HR and L&D: localisation is where a global programme usually loses comparability. The Standard's rule is a usable line: adapt the setting freely, never the decision point.

Like stakes, or the comparison is false

A Level 3 verification in a low-stakes exercise is not the same behaviour as a Level 3 verification where a person could be harmed. The Standard states that judgment scores are not comparable across situations of different consequence, and it builds a control for it. Every assessment instance declares a stakes profile: one primary consequence family and at most one secondary, drawn from six. People safety and welfare. Financial and fiduciary loss. Operational and service continuity. Information and privacy harm. Trust, reputation and legitimacy. Environmental and community-system harm.

Consequence families are recorded, never scored. They exist so that observations are compared within like stakes, and so that an employer reading a Certified result can see the range of consequence it was earned across. A certifying assessment must span at least three distinct primary families, and at least one must involve irreversible or non-financial harm. A candidate assessed only on financial and operational scenarios is scored Insufficient Evidence, because judgment shown only in those scenarios has not been shown to transfer.

The stakes profile is what makes the Seoul and Jakarta comparison honest. If both country managers were assessed on decisions carrying the same consequence family, the comparison is between behaviours. If one faced a privacy harm and the other a pricing margin, the comparison is between situations, and the Standard says scores from such situations are not comparable.

Strength in one domain never offsets weakness in another

The pass rule is non-compensatory. Strength in one domain does not offset weakness in another, and domain scores are not averaged away into a single number for the pass decision. Six thresholds must all be met: no dimension below 2.0, no domain below 2.0, a composite at or above 2.3, the evidence minimum satisfied for all nine dimensions, stakes coverage satisfied, and confidence moderate or high. Failure of any one results in Not Certified.

Each rule does separate work, and the Standard says what. A flat profile at exactly 2.0 across all nine yields a composite of 2.0 and fails the composite rule, so a candidate who sits exactly at the floor everywhere does not certify. The dimension and domain floors stop a candidate clearing the composite on three high dimensions while sitting at 1.0 on, for instance, decision rights. A person who frames brilliantly, tests rigorously and escalates everything, and so sits below 2.0 on decision rights, is not certified, because a hero in three dimensions and an absence in one is exactly the profile a non-compensatory rule exists to catch.

A hero in three dimensions and an absence in one is the profile the pass rule exists to catch.

Bands, not scores, and counts, not rankings

Certified results are reported in bands. Certified, from a composite of 2.3 to 2.9 with all floors met. Certified with Merit, 3.0 to 3.4 with no dimension below 2.5. Certified with Distinction, 3.5 and above with no dimension below 3.0. The reason is stated: bands prevent false precision and league-table misuse. A composite of 2.74 invites comparison with a 2.71 as if the gap meant something. A band does not.

The organisation that commissions an assessment receives a team view. It reports counts of levels and bands across the people assessed. A team view covers ten people or more; below ten, none is given. Where an employer commissions an assessment of its own people, it also receives each result by name, with the evidence behind each observation: its facilitator receives every result, and each line manager receives the results of their own people. Before an Assessment, candidates are told who receives their result and that an employment-grade result may be used for development, assessment, promotion and hiring. What AIR APAC does not publish is any individual league table or employer-comparative candidate score. The regional head in the opening scenario would learn, from a team view covering ten people or more, how many of the people assessed sat at each level and in each band. She would not receive a ranking, because a ranking is what turns a development tool into a weapon.

For trainers and facilitators: the team view is also the honest answer to "how did the cohort do". After an Assessment, counts by band and level, with the stakes profiles stated, is a result a trainer can stand behind. After a training session, the team view counts decisions and how the published rules read them, not levels or bands. A ranked list is not an answer in either case.

One record that travels across five regimes

Five regimes in and around the region now address human oversight or accountability, each in its own words. Korea's AI Basic Act has been in force since 22 January 2026. It asks operators of high-impact AI to provide for human management and supervision, and to keep documents that show the measures taken (Article 34). Vietnam's Law on AI has been in force since 1 March 2026. It sorts AI systems into high, medium and low risk, and it applies to foreign organisations involved in AI activities in Vietnam. For high-risk systems, it asks deployers to supervise the system, keep the ability to intervene and answer to the authorities for how the system is run (Article 14). Singapore's Model AI Governance Framework for Agentic AI, version 1.5, is a framework of recommended practice, not a statute. It sets out that organisations should consider clear allocation of responsibility and regular audit of the effectiveness of human oversight. The EU AI Act's Article 14 lists what the people assigned to oversee a high-risk system must be enabled to do, and it applies to the high-risk systems in Annex III from 2 December 2027. Article 4 of the Act, as replaced in July 2026, asks providers and deployers to take measures to support AI literacy among their staff. Australia's Guidance for AI Adoption, published in October 2025, lists deciding who is accountable and maintaining human control among its six practices.

None of these texts sets a common scale for how well a person judges AI output, and an organisation operating across them faces five local records or one portable one. This is AIR APAC's inference about how a level-based credential could be used, not a claim that any regulator recognises it. The credential is recorded on a public register with its track, band, confidence, standard version and dates. Verification is free, by credential code, with no account and no commercial relationship with AIR APAC. The credential belongs to the individual and survives a change of employer. A board asked by a regulator in any of the five markets what its people can do with AI output has, in principle, one answer to give in all five.

Three questions to ask on Monday

  1. If two of your managers in two markets reached different decisions on the same AI output, could you say today which was better judged, and on what basis?
  2. When your organisation reports on AI capability across markets, is it reporting the same thing in each, or five local versions of the same word?
  3. Which of the five regimes above asks something of your organisation now, and what evidence of oversight would you show it?

Two decisions, one output. Until they sit on one scale under like stakes, the question of which was right has no answer, and the Standard was built to give it one.

Commission a team view

What this does not claim

AIRS-J/1 assesses how a person decides, not what they know. A result is not a measure of intelligence, personality, AI technical skill or domain expertise, and it is not a substitute for supervision, professional licensing or regulatory authorisation. AIR APAC holds no team data yet; the opening scenario is illustrative and invented. The effect of AI on variation across teams is stated as an open question, not a finding. The portability of the credential across regulatory regimes is AIR APAC's inference about use; AIR APAC does not claim that any regulator recognises it. Nine Dimensions is a reporting hypothesis; the Standard says it will reduce the number if the evidence requires it.

Sources

  1. Kahneman, D., Sibony, O., Sunstein, C. R. (2021). Noise: A Flaw in Human Judgment. Cited for the argument that judgment varies between people on identical facts, the definition of noise, and the examples of judges and doctors. Limitation: predates generative AI; cited for the concept, no figures used; the book's own site was read, not the book. https://readnoise.com/ Opened 19 October 2026.
  2. AIR APAC (2026). AIRS-J/1 v2.2, AIR APAC Standard for Judgment in AI-Enabled Work, effective 18 October 2026. Sections 1, 2.1, 5.1, 6.1 to 6.6, 8.5, 12.1 to 12.7, 13, 14.1 to 14.4, 20.3. Text checked against the v2.2 file AIR_APAC/standards/AIRS-J-1.md; public page https://airapac.org/standard/v2-2/ Opened 19 October 2026.
  3. Republic of Korea (2025). Framework Act on the Development of Artificial Intelligence and Establishment of Trust, Law No. 20676, enacted 21 January 2025, enforced 22 January 2026. Article 34, high-impact AI business operators' responsibilities. Read in the English translation by the Center for Security and Emerging Technology, Georgetown University, of the text from the Korean Ministry of Government Legislation; an unofficial translation. https://cset.georgetown.edu/wp-content/uploads/t0625_south_korea_ai_law_EN.pdf Opened 19 October 2026.
  4. Socialist Republic of Vietnam (2025). Law on Artificial Intelligence, Law No. 134/2025/QH15, passed by the National Assembly on 10 December 2025, in force from 1 March 2026. Articles 2, 9, 14 and 34. Read in an English text hosted by a law firm, not the official gazette. https://apolatlegal.com/wp-content/uploads/2026/08/Law-134-2025-QH15.pdf Opened 19 October 2026.
  5. Infocomm Media Development Authority, Singapore (2026). Model AI Governance Framework for Agentic AI, version 1.5, published 20 May 2026, updated 5 June 2026. Section 2.2, Make humans meaningfully accountable. https://www.imda.gov.sg/-/media/imda/files/about/emerging-tech-and-research/artificial-intelligence/mgf-for-agentic-ai.pdf Opened 19 October 2026.
  6. European Union. Regulation (EU) 2024/1689 (AI Act), Article 14, human oversight, read on the Future of Life Institute's AI Act site, not the Official Journal. Article 4, AI literacy, as replaced by Regulation (EU) 2026/1744 of 8 July 2026 (Official Journal, 24 July 2026), which also sets 2 December 2027 as the application date for Chapter III Sections 1 to 3 for Annex III high-risk systems. https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng and https://artificialintelligenceact.eu/article/14/ Opened 19 October 2026.
  7. Australian Government, National AI Centre (2025). Guidance for AI Adoption, six essential practices, first published in October 2025. The judgment claim in the body is limited to the names of two of the six practices. Not opened; cited from web-search result summaries that list the practices (the ai.gov.au foundations PDF and the industry.gov.au page timed out when fetched, and the Australian Policy Online record at https://apo.org.au/node/332616 was seen in search results only).
Read next

More articles on judgment in AI-enabled work

AIR APAC publishes these articles in the institution's voice. Every figure carries its source and its limit. All articles.

For legal, risk and compliance

A poor decision or an unlucky outcome? Running the AI incident review

When an AI-assisted decision goes wrong, the first reflex is to blame the tool. The Standard scores that at Level 0. The second is to abandon every decision made the same way. That is Level 1. Good correction tells a poor decision from an unlucky one.

8 min read

For heads of department and line managers

The answer is not the decision

AI now produces the answer. It does not carry the consequence of acting on it. What is left for your people is a short list of decisions, and organisations have no common language for it.

9 min read

For legal, risk and compliance

Singapore's courts have already decided: verification is personal

As far as AIR APAC can establish, no regulator in the region has yet fined an organisation for acting on unverified AI output. The courts moved first. In Singapore the penalty landed on named lawyers, and the duty to check was held to be non-delegable. Courts in Victoria and Hong Kong criticised unverified AI material too.

7 min read