{"id":1384,"date":"2026-11-16T18:30:00","date_gmt":"2026-11-16T13:00:00","guid":{"rendered":"https:\/\/www.graveiens.com\/blog\/?p=1384"},"modified":"2026-11-12T00:07:54","modified_gmt":"2026-11-11T18:37:54","slug":"ai-generated-assessment-items","status":"publish","type":"post","link":"https:\/\/www.graveiens.com\/blog\/ai-generated-assessment-items\/","title":{"rendered":"AI-Generated Assessment Items: 8 Failure Modes and the Human Review That Fixes Them"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>Last updated: November 2026<\/em><\/p>\n\n\n<div class=\"wp-block-group key-takeaways\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n\n<h2 class=\"wp-block-heading\">Key takeaways<\/h2>\n\n\n<ul class=\"wp-block-list\">\n\n<li><strong>AI-generated assessment items<\/strong> can speed up first drafts, but research, testing standards and regulators still expect qualified people to approve every item before learners see it.<\/li>\n\n\n<li>AI drafts fail in predictable ways: wrong or double keys, ambiguous stems, weak distractors, cueing, bias, blueprint drift, reading-level drift and unclear IP.<\/li>\n\n\n<li>A six-gate workflow (specify, draft, solve, fairness, access, field-test) keeps a named human accountable at every step except drafting.<\/li>\n\n\n<li>AI moves effort rather than removing it. Drafting gets faster, while verification, documentation and disclosure need more care.<\/li>\n\n\n<li>Use the CLAIMS check (Correct, Level, Aligned, IP-clean, Misconceptions, Screened) to pass, fix or reject each draft.<\/li>\n\n<\/ul>\n\n<\/div><\/div>\n\n\n<p class=\"wp-block-paragraph\">Generative AI can produce hundreds of draft test questions in an afternoon. The hard part is knowing which ones are safe to use.<\/p>\n\n\n<p class=\"wp-block-paragraph\">This guide is for publishers, EdTech assessment teams, test-prep companies and university assessment units working with <strong>AI-generated assessment items<\/strong>. It shows where AI drafts tend to fail and how a human-in-the-loop workflow catches those failures. It also gives you a simple checklist for judging each item.<\/p>\n\n\n<p class=\"wp-block-paragraph\">You will also see where effort shifts when AI joins item writing, and what to settle on disclosure and IP before you scale.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>Already drafting with AI?<\/strong> <a href=\"https:\/\/www.graveiens.com\/#contact\">Send us 20 AI-drafted items<\/a> and get a free human review sample, scored item by item.<\/p>\n\n\n<h2 class=\"wp-block-heading\">Can AI-generated assessment items be used without human review?<\/h2>\n\n\n<p class=\"wp-block-paragraph\">No, not for any test that informs decisions about learners. AI-generated assessment items can be fluent and neatly formatted yet still carry a wrong key, a biased context or a skill the blueprint never asked for. Research, testing standards and regulators point the same way: AI drafts, and qualified people decide what ships.<\/p>\n\n\n<p class=\"wp-block-paragraph\">A peer-reviewed mini review of automatic item generation by <a href=\"https:\/\/www.frontiersin.org\/journals\/education\/articles\/10.3389\/feduc.2023.858273\/full\" target=\"_blank\" rel=\"noopener\">Circi, Hicks and Sikali (2023)<\/a> lists the promises clearly. They include less time and cost per item and support for large item pools. The same review notes that reviewing and approving items is still a time-consuming step.<\/p>\n\n\n<p class=\"wp-block-paragraph\">The authors also conclude that data-driven models that do not use templates still have a long way to go before they meet the quality expected in operational testing. That gap is why human review of AI-generated items is the core of the process, not an extra.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Regulators say the same. As of November 2026, <a href=\"https:\/\/www.gov.uk\/government\/publications\/ofquals-approach-to-regulating-the-use-of-artificial-intelligence-in-the-qualifications-sector\/ofquals-approach-to-regulating-the-use-of-artificial-intelligence-in-the-qualifications-sector--2\" target=\"_blank\" rel=\"noopener\">Ofqual\u2019s policy paper on AI<\/a> (updated 16 July 2026) sets the bar for awarding organisations in England. They must show that any AI use gives valid, reliable and fair outcomes. That includes \u201cappropriate expert human involvement and oversight, quality assurance and accountability for outcomes.\u201d<\/p>\n\n\n<p class=\"wp-block-paragraph\">In the US, the Department of Education\u2019s 2023 report on AI in teaching and learning made \u201cEmphasize Humans-in-the-Loop\u201d its first recommendation, according to its <a href=\"https:\/\/www2.ed.gov\/documents\/ai-report\/ai-report-core-messages.pdf\" target=\"_blank\" rel=\"noopener\">official summary handout<\/a>.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>In short:<\/strong> AI can write the first draft. Only qualified reviewers can confirm that an item is correct, fair and aligned, and that is what buyers, regulators and candidates rely on.<\/p>\n\n\n<h2 class=\"wp-block-heading\">What does generative AI do well and badly in item writing?<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Generative AI is good at volume and variation: first drafts, parallel versions, rewrites to a style guide and draft rationales. It is weak wherever judgment is needed. That includes proving there is exactly one correct answer, writing distractors from real misconceptions, holding a reading level and matching the exact skill a blueprint demands.<\/p>\n\n\n<h3 class=\"wp-block-heading\">Where AI drafts help<\/h3>\n\n\n<ul class=\"wp-block-list\">\n\n<li><strong>First drafts at volume.<\/strong> Subject experts spend their time editing instead of facing a blank page.<\/li>\n\n\n<li><strong>Parallel versions.<\/strong> AI can turn one approved item into variants with new numbers or contexts that test the same skill.<\/li>\n\n\n<li><strong>Reformatting.<\/strong> It can move items into a house template with stem, options, key and rationale fields.<\/li>\n\n\n<li><strong>Draft rationales and solutions.<\/strong> A reviewer then checks every line, since a confident explanation can still be wrong.<\/li>\n\n\n<li><strong>Suggested metadata.<\/strong> Topic or cognitive-level tags are a starting point for a human to confirm.<\/li>\n\n<\/ul>\n\n\n<h3 class=\"wp-block-heading\">Eight failure modes to screen for in generative AI test items<\/h3>\n\n\n<p class=\"wp-block-paragraph\">Most of these flaws are not unique to AI. Human writers make them too. The difference is scale: a model can repeat the same flaw across hundreds of generative AI test items before anyone notices.<\/p>\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Failure mode<\/th><th>What it looks like<\/th><th>How reviewers catch it<\/th><\/tr><\/thead><tbody><tr><td>1. Wrong or double key<\/td><td>The marked answer is wrong, or two options can be defended<\/td><td>A second expert solves the item without seeing the key<\/td><\/tr><tr><td>2. Ambiguous stem<\/td><td>The question can be read two ways or misses a needed condition<\/td><td>The reviewer answers the stem before reading the options<\/td><\/tr><tr><td>3. Implausible distractors<\/td><td>Wrong options no learner would pick, or options that are partly true<\/td><td>Each distractor is mapped to a named misconception<\/td><\/tr><tr><td>4. Cueing<\/td><td>The key is the longest option, repeats stem words or sits in the same position too often<\/td><td>Check option length, word overlap and key positions across the set<\/td><\/tr><tr><td>5. Bias and sensitivity<\/td><td>Stereotypes, regional knowledge or upsetting contexts<\/td><td>A separate, trained bias and sensitivity reviewer<\/td><\/tr><tr><td>6. Blueprint misalignment<\/td><td>The item tests a nearby skill or a lower cognitive level than tagged<\/td><td>Check against the standard code and DOK or Bloom\u2019s level<\/td><\/tr><tr><td>7. Reading-level drift<\/td><td>Language load above the target grade, or wordy stems<\/td><td>A readability check plus an editor who knows the grade band<\/td><\/tr><tr><td>8. Copyright, IP and disclosure<\/td><td>Text close to published or released items, or AI use not recorded<\/td><td>Similarity checks, a provenance log and an AI-use statement<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"820\" src=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-failure-modes.webp\" alt=\"Eight failure modes of AI-generated assessment items: wrong key, ambiguous stem, implausible distractors, cueing, bias, blueprint misalignment, reading-level drift and IP risk\" class=\"wp-image-1380\" srcset=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-failure-modes.webp 1600w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-failure-modes-300x154.webp 300w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-failure-modes-1024x525.webp 1024w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-failure-modes-768x394.webp 768w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-failure-modes-1536x787.webp 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">Screen every AI draft for these eight failure modes before it reaches a reviewer queue.<\/figcaption><\/figure>\n\n\n<p class=\"wp-block-paragraph\">Watch the last row closely. In test prep, a model can echo the style or wording of released exam questions. That creates legal and brand risk even when the item is technically correct.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>In short:<\/strong> Use AI for speed and variation. Expect to catch key errors, weak distractors, bias, drift and IP problems in review, and plan reviewer time for them.<\/p>\n\n\n<h2 class=\"wp-block-heading\">How does a human-in-the-loop review workflow for AI items work?<\/h2>\n\n\n<p class=\"wp-block-paragraph\">A human-in-the-loop workflow puts a named person in charge of every gate that decides the quality of AI-generated assessment items. AI handles one step: drafting. People specify the item, solve and edit it, screen it for fairness and accessibility, and judge field-test data. Items that fail any gate go back for repair or rejection, never forward.<\/p>\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"760\" src=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-review-workflow.webp\" alt=\"Six-gate human-in-the-loop review workflow for AI-generated assessment items: specify, AI draft, SME solve, fairness review, accessibility check and field test, with rejected drafts looping back to the draft step\" class=\"wp-image-1381\" srcset=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-review-workflow.webp 1600w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-review-workflow-300x143.webp 300w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-review-workflow-1024x486.webp 1024w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-review-workflow-768x365.webp 768w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-review-workflow-1536x730.webp 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">Only the draft step is automated. People own the other five gates, and failed items loop back.<\/figcaption><\/figure>\n\n\n<h3 class=\"wp-block-heading\">1. Specify: blueprint, item model and prompt pack<\/h3>\n\n\n<p class=\"wp-block-paragraph\">AI-generated assessment items need the same inputs a human writer gets: the blueprint, the target standard, cognitive level, item type, style guide and sample items. Then add an item model, which is a template that fixes what may vary in an item.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Circi and colleagues report that most large-scale work in this field uses template or rule-based item models, because they keep items aligned to the intended construct. Record the tool, version and prompt for every batch so each item\u2019s history can be traced.<\/p>\n\n\n<h3 class=\"wp-block-heading\">2. Draft: generate more than you need<\/h3>\n\n\n<p class=\"wp-block-paragraph\">Ask for more drafts than your final count, because some will be rejected. Generate in small batches by skill, which makes duplicates and coverage gaps easier to spot. Keep drafts in a holding area that is clearly labeled as unreviewed.<\/p>\n\n\n<h3 class=\"wp-block-heading\">3. Solve and edit: the subject-expert gate<\/h3>\n\n\n<p class=\"wp-block-paragraph\">A subject-matter expert (SME) solves each item cold, without seeing the key. If their answer differs from the AI\u2019s key, or two options work, the item is fixed or rejected. The SME then checks alignment, tightens the stem and rewrites weak distractors.<\/p>\n\n\n<p class=\"wp-block-paragraph\">The <a href=\"https:\/\/www.testingstandards.net\/open-access-files.html\" target=\"_blank\" rel=\"noopener\">AERA, APA and NCME Standards for Educational and Psychological Testing<\/a> (2014 edition) cover this step. Standard 1.9 says that when expert judgment supports validity, the experts\u2019 qualifications and review procedures should be described. So keep a reviewer log with names, roles and decisions.<\/p>\n\n\n<h3 class=\"wp-block-heading\">4. Fairness: bias and sensitivity review<\/h3>\n\n\n<p class=\"wp-block-paragraph\">A separate trained reviewer screens for stereotypes, cultural or regional assumptions and contexts that may upset or distract test takers. The Standards\u2019 commentary notes that different judges may rate items for different purposes, such as cultural sensitivity and reading level. The item editor should not be the only fairness reviewer.<\/p>\n\n\n<h3 class=\"wp-block-heading\">5. Access: accessibility and editorial checks<\/h3>\n\n\n<p class=\"wp-block-paragraph\">Editors check reading load, grammar, notation and house style. Accessibility reviewers confirm three things. Images have alt text that conveys what is needed to answer without giving away the key. Tables are structured. Math is encoded so screen readers can read it.<\/p>\n\n\n<h3 class=\"wp-block-heading\">6. Field-test: let student data decide<\/h3>\n\n\n<p class=\"wp-block-paragraph\">Expert review predicts how an item will behave. Field testing shows it. Circi and colleagues describe how traditional item development pilots items. Teams then evaluate difficulty, discrimination and differential item functioning (DIF). DIF flags items that work differently for groups of equal ability.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Common signals include the p-value (the share of students who answer correctly), the point-biserial correlation and distractor choice rates. Items that misbehave are revised or retired. For a view of how these gates fit a full bank build, see our guide to <a href=\"https:\/\/www.graveiens.com\/blog\/question-bank-development-services\/\">question bank development services<\/a>.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>In short:<\/strong> AI drafts in step two. People own the other five steps, and any failed gate sends the item back.<\/p>\n\n\n<h2 class=\"wp-block-heading\">How do you review AI-generated items? The CLAIMS item writing review checklist<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Score every AI draft against six checks before it enters your bank. They are Correct key, reading and cognitive Level, Aligned to the blueprint, IP-clean and disclosed, Misconception-based distractors, and Screened for bias and accessibility. An item passes only when all six pass. A single failure means fix or reject, not \u201cgood enough.\u201d<\/p>\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"820\" src=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-claims-check.webp\" alt=\"The CLAIMS item writing review checklist for AI-generated items: Correct key, Level, Aligned, IP-clean, Misconception-based distractors and Screened for bias and accessibility\" class=\"wp-image-1382\" srcset=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-claims-check.webp 1600w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-claims-check-300x154.webp 300w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-claims-check-1024x525.webp 1024w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-claims-check-768x394.webp 768w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-claims-check-1536x787.webp 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">The CLAIMS check: an AI-drafted item passes only when all six checks pass.<\/figcaption><\/figure>\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Check<\/th><th>Question the reviewer answers<\/th><th>Fix or reject if\u2026<\/th><\/tr><\/thead><tbody><tr><td>C: Correct<\/td><td>Does an expert solving cold reach the same single key?<\/td><td>Two defensible answers, or the key is wrong<\/td><\/tr><tr><td>L: Level<\/td><td>Do reading load and cognitive demand match the target?<\/td><td>Wordy stem, grade-band drift, or a recall item tagged as analysis<\/td><\/tr><tr><td>A: Aligned<\/td><td>Does the item measure the tagged standard and blueprint cell?<\/td><td>It tests a nearby skill or duplicates another item<\/td><\/tr><tr><td>I: IP-clean<\/td><td>Is it original, logged and disclosed as the contract requires?<\/td><td>Close match to published items, or no provenance record<\/td><\/tr><tr><td>M: Misconceptions<\/td><td>Does each distractor reflect a real learner error?<\/td><td>Options no one would choose, or options that are partly true<\/td><\/tr><tr><td>S: Screened<\/td><td>Has it passed bias, sensitivity and accessibility review?<\/td><td>Stereotypes, regional assumptions or missing alt text<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p class=\"wp-block-paragraph\">Give each item one of three outcomes: pass, fix or reject. Record a reason code for every fix and reject. These codes turn your review of AI-generated assessment items into item quality assurance you can measure, batch by batch.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Watch the pattern, not just the items. If one CLAIMS letter keeps failing, fix the prompt or item model at the source. For example, repeated \u201cM\u201d failures usually mean the prompt never listed the misconceptions you want distractors to target.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>In short:<\/strong> CLAIMS gives reviewers one shared standard. Six checks, three outcomes and reason codes make review faster, fairer and easier to audit.<\/p>\n\n\n<div class=\"wp-block-group cta-inline\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n\n<p class=\"wp-block-paragraph\"><strong>Want CLAIMS as a one-page reviewer sheet?<\/strong> <a href=\"https:\/\/www.graveiens.com\/#contact\">Request the CLAIMS checklist<\/a> (mention \u201cCLAIMS checklist\u201d in your message and we\u2019ll email the one-page PDF), or send 20 AI-drafted items and we will return them scored against all six checks.<\/p>\n\n<\/div><\/div>\n\n\n<h2 class=\"wp-block-heading\">AI-drafted vs. fully human item writing: where does the effort go?<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Using AI-generated assessment items shifts effort rather than removing it. Drafting gets faster. Verification, documentation and disclosure need more care, because reviewers must check fluent text that no person wrote. Fairness review and field testing cost about the same either way, since every new item needs them no matter who drafted it.<\/p>\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"780\" src=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-effort-comparison.webp\" alt=\"Relative effort of AI-assisted versus human-only item writing across drafting, verification, fairness review, field testing and documentation\" class=\"wp-image-1383\" srcset=\"https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-effort-comparison.webp 1600w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-effort-comparison-300x146.webp 300w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-effort-comparison-1024x499.webp 1024w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-effort-comparison-768x374.webp 768w, https:\/\/www.graveiens.com\/blog\/wp-content\/uploads\/2026\/11\/ai-generated-assessment-items-effort-comparison-1536x749.webp 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">AI cuts drafting effort but raises verification and documentation work. Fairness review and field testing stay about the same.<\/figcaption><\/figure>\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Task<\/th><th>AI draft + human review<\/th><th>Fully human<\/th><\/tr><\/thead><tbody><tr><td>First drafts<\/td><td>Fast; many drafts per prompt<\/td><td>The slowest step; writer time per item<\/td><\/tr><tr><td>Key verification<\/td><td>Every item solved cold; more rejects to handle<\/td><td>A second-expert solve is still needed<\/td><\/tr><tr><td>Distractor quality<\/td><td>Often needs rewriting<\/td><td>Writer builds from known misconceptions<\/td><\/tr><tr><td>Fairness review<\/td><td>Same full screen<\/td><td>Same full screen<\/td><\/tr><tr><td>Field testing<\/td><td>Same; required for new items<\/td><td>Same; required for new items<\/td><\/tr><tr><td>Documentation and disclosure<\/td><td>Provenance log, tool terms and AI-use statement<\/td><td>Standard reviewer logs<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p class=\"wp-block-paragraph\">These are relative effort levels, not prices. Your real savings depend on a few drivers, so weigh them before you choose a model:<\/p>\n\n\n<ul class=\"wp-block-list\">\n\n<li><strong>Stakes.<\/strong> Low-stakes practice can accept lighter review than items that feed grades, placement or certification.<\/li>\n\n\n<li><strong>Subject complexity.<\/strong> Advanced STEM, medicine and law drafts need senior reviewers, which narrows the time saved.<\/li>\n\n\n<li><strong>Item type.<\/strong> Simple multiple-choice gains the most. Technology-enhanced items and passage sets still need heavy human design.<\/li>\n\n\n<li><strong>Draft reject rate.<\/strong> If reviewers reject many drafts, the savings shrink. Measure the rate in a pilot before you scale.<\/li>\n\n\n<li><strong>Reviewer calibration.<\/strong> Reviewers need a shared rubric, such as CLAIMS, so decisions stay consistent across people.<\/li>\n\n\n<li><strong>Documentation needs.<\/strong> Clients and regulators may ask how items were made, so logging takes real time.<\/li>\n\n<\/ul>\n\n\n<p class=\"wp-block-paragraph\">Adaptive banks add one more driver: difficulty coverage. Our guide to <a href=\"https:\/\/www.graveiens.com\/blog\/digital-sat-question-writing\/\">digital SAT question writing<\/a> shows how difficulty tiers multiply the item count, whoever drafts the items.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>In short:<\/strong> Expect faster drafting and heavier checking. A 20-item pilot with tracked reject rates will tell you more than any general estimate.<\/p>\n\n\n<h2 class=\"wp-block-heading\">Disclosure, IP and governance: what should US and UK teams check?<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Check four things before you scale AI-generated assessment items: your AI tool\u2019s terms on inputs and outputs, whether secure item content may be entered into it, how AI use will be disclosed, and who owns the final items. Settle each in writing, then keep a provenance record for every item.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>England (Ofqual).<\/strong> Ofqual applies its existing, outcomes-based rules to AI rather than new AI-specific rules. Its July 2026 update lists item and assessment generation among the priority use cases awarding organisations raised. The same workshops flagged risks around accuracy and reliability, data security and confidentiality, and staff capability.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Ofqual adds that AI in assessment design raises risks relating to accuracy, bias and the appropriateness of AI-generated content. It says it is doing research to build the evidence base on good practice. Its remit covers England only.<\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>United States.<\/strong> The Department of Education\u2019s 2023 report asks for humans in the loop and for AI systems that can be inspected, explained and overridden. The AERA, APA and NCME Standards add a documentation duty. Standard 1.11 says that when validity rests on test content, the procedures used to specify and generate it should be described and justified. Any AI in assessment development, including a drafting step, is part of that procedure.<\/p>\n\n\n<p class=\"wp-block-paragraph\">The 2014 Standards are open access, and the sponsoring organizations named a joint committee to revise them in 2024. Watch for the new edition.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Use this governance checklist for bias and accuracy checks, disclosure and ownership:<\/p>\n\n\n<ul class=\"wp-block-list\">\n\n<li><strong>Tool terms.<\/strong> Confirm whether the provider may store or train on your inputs, and what rights you hold in outputs.<\/li>\n\n\n<li><strong>Confidentiality.<\/strong> Never paste live, secure or unreleased items into a public tool.<\/li>\n\n\n<li><strong>Provenance.<\/strong> Log the tool, version, prompt, date, reviewers and edits for each item.<\/li>\n\n\n<li><strong>Disclosure.<\/strong> Agree in the contract whether and how AI use is disclosed to you, and by you to your customers.<\/li>\n\n\n<li><strong>Originality.<\/strong> Run similarity checks against released and published items, especially for test prep.<\/li>\n\n\n<li><strong>Ownership.<\/strong> Get an IP assignment that covers items, rationales, graphics and metadata.<\/li>\n\n<\/ul>\n\n\n<p class=\"wp-block-paragraph\"><em>This section is general information, not legal advice. Ask your counsel how AI-assisted content affects ownership and disclosure in your markets.<\/em><\/p>\n\n\n<p class=\"wp-block-paragraph\"><strong>In short:<\/strong> Governance is mostly paperwork done early: tool terms, confidentiality rules, provenance logs, disclosure wording and IP assignment.<\/p>\n\n\n<h2 class=\"wp-block-heading\">Common mistakes when adopting AI-generated assessment items<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Most problems with AI-generated assessment items trace back to process, not to the model. These are the mistakes to avoid:<\/p>\n\n\n<ul class=\"wp-block-list\">\n\n<li><strong>Treating fluent text as correct text.<\/strong> Polished wording hides wrong keys and vague stems.<\/li>\n\n\n<li><strong>Letting the AI\u2019s key stand.<\/strong> Every item should be solved cold by a qualified person.<\/li>\n\n\n<li><strong>Reviewing in one pass.<\/strong> Content, fairness and accessibility need different reviewers and different checklists.<\/li>\n\n\n<li><strong>Scaling before a pilot.<\/strong> Without a measured reject rate, you cannot plan reviewer time or cost.<\/li>\n\n\n<li><strong>Fixing items instead of prompts.<\/strong> Repeated failures point to the prompt or item model, so fix the source.<\/li>\n\n\n<li><strong>Skipping field testing.<\/strong> Items that \u201clook good\u201d can still be too easy, too hard or unfair to a group.<\/li>\n\n\n<li><strong>Forgetting the paper trail.<\/strong> If you cannot show how an item was made and checked, you cannot defend it.<\/li>\n\n<\/ul>\n\n\n<p class=\"wp-block-paragraph\">The cost of these mistakes rises with the stakes. Our explainer on <a href=\"https:\/\/www.graveiens.com\/blog\/summative-assessment\/\">summative assessment types and examples<\/a> covers how end-of-course tests differ from low-stakes practice.<\/p>\n\n\n<h2 class=\"wp-block-heading\">How GravEiens EduServices approaches AI-drafted items<\/h2>\n\n\n<p class=\"wp-block-paragraph\">GravEiens EduServices has developed education content since 2017 and works within an ISO 9001 certified quality system. Our network of 700+ experts serves 350+ global clients. When a client wants human review of AI-generated assessment items, or AI in the drafting step, we fit that work into our four-stage QA workflow:<\/p>\n\n\n<ol class=\"wp-block-list\">\n\n<li><strong>Content creation.<\/strong> Subject experts write items, or edit AI drafts, to your blueprint and style guide.<\/li>\n\n\n<li><strong>Internal review.<\/strong> Our team reviews items against the checks agreed in scope, such as CLAIMS, before you see them.<\/li>\n\n\n<li><strong>Client review.<\/strong> Your team reviews each batch and returns comments for resolution.<\/li>\n\n\n<li><strong>Final delivery.<\/strong> Approved items are delivered in your agreed format.<\/li>\n\n<\/ol>\n\n\n<p class=\"wp-block-paragraph\">Tool choice, provenance logging, disclosure wording and field-test support depend on your program, so we scope this with you. Learn more about our <a href=\"https:\/\/www.graveiens.com\/assessment-authoring\/\">assessment authoring and item writing services<\/a>, our <a href=\"https:\/\/www.graveiens.com\/k12-education\/assessment-development\/\">K-12 assessment development<\/a> work and our <a href=\"https:\/\/www.graveiens.com\/test-prep-content\/\">test prep content development<\/a> team.<\/p>\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n<div id=\"rank-math-faq\" class=\"rank-math-block\">\n<div class=\"rank-math-list \">\n<div id=\"faq-question-17950000000160\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">Are AI-generated assessment items as good as human-written items?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>They can be, after expert review, but raw drafts vary widely. A peer-reviewed review of automatic item generation found that evaluation methods differ a lot across studies. It also concluded that data-driven models still fall short of operational testing quality. Judge your own drafts with blind expert review and field-test data, not first impressions.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-17950000000161\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">Do we need to disclose AI use in assessment items?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>It depends on your contracts, clients and regulator, so settle it in writing either way. In England, Ofqual expects awarding organisations to show that AI use gives valid, reliable and fair outcomes with expert human oversight. A short AI-use statement plus a provenance log answers most questions. This is general information, not legal advice.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-17950000000162\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">Who owns AI generated test questions?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Ownership depends on your contract, the AI tool\u2019s terms and the law where you operate. Protect yourself with a vendor IP assignment that covers items, rationales, graphics and metadata. Keep records of the human edits made to each item. Ask counsel about your markets, because this is general information, not legal advice.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-17950000000163\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">Can AI write good distractors for multiple-choice items?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>AI can suggest distractors, but they are a common weak point. Drafts often include options no learner would choose, or options that are partly correct. Reviewers should tie each distractor to a named misconception and write a short rationale. Research on automatic item generation has long treated distractors as a challenge, though methods have improved.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-17950000000164\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">How many reviewers should check each AI-generated item?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>At least two people beyond the drafting step: a subject expert who solves the item cold and a separate fairness reviewer. Add an editor and an accessibility reviewer for high-stakes items. Record each reviewer\u2019s role and qualifications, as the AERA, APA and NCME Standards recommend when expert judgment supports validity.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-17950000000165\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">Does Ofqual allow AI to write exam questions?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Ofqual\u2019s July 2026 update lists item and assessment generation among priority use cases and says it is building evidence on good practice. It applies existing rules, expects expert human involvement and oversight, and notes that certain uses do not meet its requirements. Its remit covers England. Check the latest guidance before you plan regulated use.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-17950000000166\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">Should AI-generated items be field-tested?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Yes. AI-generated assessment items need field testing like any new item. Expert review predicts behavior, while field testing shows it. Check difficulty (p-value), discrimination (point-biserial) and distractor choice rates. Where sample sizes allow, add differential item functioning checks. Revise or retire any item whose statistics fall outside your program\u2019s rules.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n<div class=\"wp-block-group cta-box\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n\n<h2 class=\"wp-block-heading\">Send us 20 AI-drafted items for a free human review sample<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Share 20 items your team or tool has drafted, plus the blueprint they should match. Our reviewers will check each one against the CLAIMS check and return pass, fix or reject decisions with reason codes. <a href=\"https:\/\/www.graveiens.com\/#contact\"><strong>Request your free review sample<\/strong><\/a>.<\/p>\n\n<\/div><\/div>\n\n\n<h2 class=\"wp-block-heading\">Sources<\/h2>\n\n\n<ol class=\"wp-block-list\">\n\n<li>AERA, APA and NCME. Standards for Educational and Psychological Testing, 2014 edition (open access files). Accessed November 2026. <a href=\"https:\/\/www.testingstandards.net\/open-access-files.html\" target=\"_blank\" rel=\"noopener\">https:\/\/www.testingstandards.net\/open-access-files.html<\/a><\/li>\n\n\n<li>Circi, R., Hicks, J. and Sikali, E. (2023). Automatic item generation: foundations and machine learning-based approaches for assessments. Frontiers in Education, 8:858273. <a href=\"https:\/\/www.frontiersin.org\/journals\/education\/articles\/10.3389\/feduc.2023.858273\/full\" target=\"_blank\" rel=\"noopener\">https:\/\/www.frontiersin.org\/journals\/education\/articles\/10.3389\/feduc.2023.858273\/full<\/a><\/li>\n\n\n<li>Ofqual. Ofqual\u2019s approach to regulating the use of artificial intelligence in the qualifications sector. Policy paper, updated 16 July 2026. Accessed November 2026. <a href=\"https:\/\/www.gov.uk\/government\/publications\/ofquals-approach-to-regulating-the-use-of-artificial-intelligence-in-the-qualifications-sector\/ofquals-approach-to-regulating-the-use-of-artificial-intelligence-in-the-qualifications-sector--2\" target=\"_blank\" rel=\"noopener\">https:\/\/www.gov.uk\/government\/publications\/ofquals-approach-to-regulating-the-use-of-artificial-intelligence-in-the-qualifications-sector\/ofquals-approach-to-regulating-the-use-of-artificial-intelligence-in-the-qualifications-sector&#8211;2<\/a><\/li>\n\n\n<li>U.S. Department of Education, Office of Educational Technology. Handout: AI and the Future of Teaching and Learning (core messages of the 2023 report). Accessed November 2026. <a href=\"https:\/\/www2.ed.gov\/documents\/ai-report\/ai-report-core-messages.pdf\" target=\"_blank\" rel=\"noopener\">https:\/\/www2.ed.gov\/documents\/ai-report\/ai-report-core-messages.pdf<\/a><\/li>\n\n<\/ol>\n\n\n<h3 class=\"wp-block-heading\">About the author<\/h3>\n\n\n<p class=\"wp-block-paragraph\"><strong>GravEiens EduServices Editorial Team.<\/strong> Our editorial team includes assessment specialists, subject-matter editors and instructional designers who write and review items for publishers, EdTech platforms and test-prep companies. We base every guide on hands-on item development and review work, and we check facts against published standards and regulator guidance.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>AI-generated assessment items need expert review. See 8 failure modes, a 6-gate workflow and the CLAIMS check. Send 20 items for a free review.<\/p>\n","protected":false},"author":1,"featured_media":1379,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[174],"tags":[214,215,177,175,178],"class_list":["post-1384","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-assessment-question-banks","tag-ai-in-assessment","tag-assessment-quality","tag-item-writing","tag-question-banks","tag-test-prep-content"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/posts\/1384","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/comments?post=1384"}],"version-history":[{"count":1,"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/posts\/1384\/revisions"}],"predecessor-version":[{"id":1385,"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/posts\/1384\/revisions\/1385"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/media\/1379"}],"wp:attachment":[{"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/media?parent=1384"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/categories?post=1384"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.graveiens.com\/blog\/wp-json\/wp\/v2\/tags?post=1384"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}