As AI becomes more deeply embedded in decision-making, organizations must rethink what effective governance and accountability require.
By Baskar Sundararajan and Srivathsan Karanai Margan
Editor’s Note: Artificial intelligence is transforming decision-making and creating new oversight responsibilities. In Part 1 of this two-part series, the authors explain why human oversight has become essential, the regulatory frameworks shaping it, and what it means for business subject matter experts.
You can’t fully grasp AI if you don’t do at least a little coding yourself and take a closer look at the technical aspects; you also can’t fully understand AI without closely examining the philosophical and ethical aspects.
—Murat Durmus
In recent years, discussion about artificial intelligence (AI)—especially its generative and agentic variants—has dominated conversations across every industry. Questions about how to use it, for what purpose, and to what effect have driven companies to experiment with the technology through pilots and deployments.
Generative and agentic AI systems have yet to overcome epistemic unreliability, loss of controllability, security vulnerabilities, and goal misalignment. These unresolved challenges can lead to extremely risky outcomes, and the opacity of AI decision-making limits interpretability and accountability, compelling regulators worldwide to mandate human oversight.
This article discusses what human oversight means and how it applies to business subject matter experts (SMEs) responsible for making day-to-day AI-driven decisions in the insurance industry.
Humans in AI Oversight
Three modes of human oversight are defined for AI-driven decision-making, namely human-in-the-loop (HITL), human-on-the-loop (HOTL), and human-in-command (HIC). The choice depends on the required level of oversight, decision rights, and risk ownership.
HITL: The HITL model applies when a task is high-risk, low-volume, and irreversible, requiring nuanced judgment and accountability. HITL validates each decision, and no automated decision is implemented without explicit approval. The AI provides a recommendation or initial analysis, and the workflow proceeds only after human review and authorization. This model provides maximum control and is designed to catch errors before they take effect.
HOTL: The HOTL model applies where individual errors are reversible and not catastrophic, either because volume necessitates automation or because errors can be corrected afterward. Transactions are time-sensitive and high-volume, making per-transaction review impractical while making unguided automation too risky. The AI operates independently within preset thresholds, and humans monitor behavior and outcomes as supervisors, intervening when the AI encounters uncertainty, exceptions, or errors. Monitoring occurs in real time or through periodic review. This approach controls system behavior through intervention thresholds and catches errors during or after execution.
HIC: Unlike HITL and HOTL, which govern human involvement during decision execution, HIC operates as a governance-level control framework. HIC functions at a strategic level, establishing objectives, scope, constraints, and accountability arrangements that shape how an AI system is conceived, deployed, and controlled. Under this model, senior leaders, designated authorities, or formal committees retain responsibility for authorizing AI use, determining permitted purposes and contexts, setting limits on autonomy, and defining the conditions under which reliance on the system is acceptable. HIC also encompasses escalation pathways, role-based accountability, and the authority to suspend or withdraw AI systems when risk exceeds defined thresholds.
Beyond these three models, there is an outlier, human-out-of-the-loop (HOOTL) model that removes the human from the execution path entirely, allowing the system to operate autonomously. It is appropriate for low-risk, well-tested processes where errors are harmless and result in a retry rather than an escalation. The SME’s role under HOOTL is confined to the system design stage and includes defining process boundaries, setting autonomous execution thresholds, and documenting the conditions under which human intervention is not required.
Existing literature on HITL, HOTL, and HIC focuses mostly on technically trained practitioners. In practice, however, AI system effectiveness depends on how business SMEs who manage customer requests and processes on the operational floor perform each oversight role.
Business SMEs in Oversight
A business SME holds deep knowledge of a specific domain, interprets business rules, evaluates outputs against real-world context, and exercises authority to approve, modify, or reject decisions for correctness, compliance, and operational relevance. In insurance, the SME may be an actuary, underwriter, claims adjudicator, fraud analyst, regulatory analyst, or customer service executive. Depending on the oversight model applied, SME involvement varies across the AI lifecycle phases, spanning development (design, data preparation, model training, and evaluation), deployment (contextualization and defining guardrails), and operation (monitoring, output validation, and feedback).
The rise of generative and agentic AI is closing the gap between business and technology, yet an SME rooted in business knowledge may find this capability gap increasingly difficult to manage. The practical requirement is to build a working familiarity with AI concepts and engage directly in selected technical activities. In the development phase, SMEs provide domain expertise to identify use cases, label training data or validate data quality, and define business rules that the model must respect. They validate models by testing edge cases and flagging violations of business or regulatory expectations. After deployment, SMEs review, supervise, or intervene in AI recommendations and actions depending on the risk profile and operational context.
In personal insurance lines, AI use cases span customer onboarding, data extraction, triage, risk assessment and scoring, underwriting, policy verification, coverage analysis, damage assessment, fraud detection, claims adjudication, settlement, and customer servicing. In commercial lines, the scope extends to account-level risk aggregation, loss review, manuscript policy generation, excess and surplus lines placement, project monitoring, and trade credit or surety bond evaluation.
HITL and HOTL are not mutually exclusive and often work together within a given process, with some steps requiring HITL and others better suited to HOTL. High-impact or less-validated use cases may begin with HITL, while lower-risk and tightly bounded ones may begin with HOTL. As predictability and accuracy improve, oversight may shift from HITL to HOTL and from HOTL to HOOTL.
Oversight in Practice
The oversight model determines SME engagement. Under HITL, the SME is a required gate, and no recommendation proceeds without explicit review and approval. Under HOTL, the SME operates as a supervisory control, intervening when defined thresholds are breached or performance indicators signal drift. The sections below describe how each oversight mode applies across insurance functions, with differences for commercial lines noted where exposure, contractual complexity, or regulatory requirements differ materially from personal lines.
Underwriting
In personal lines underwriting, the underwriter must validate and be accountable for every decision involving declinations, significant premium loadings, and policy exclusions. AI can suggest risk categories and decisions, but human validation is required to ensure adherence to underwriting standards and regulatory requirements. The underwriter reviews edge cases, overrides anomalies, and applies judgment where data is insufficient. In commercial lines, HITL requirements are broader, with complex or large exposures, manuscript endorsements, and risks outside filed forms requiring underwriter sign-off before binding.
Under HOTL in personal lines, the underwriter monitors the submission queue for volume anomalies such as risk concentrations in emerging categories, new construction types, or geographic clusters affected by perils for which the model was not trained. These patterns may signal concept drift and require model recalibration. The underwriter also monitors the triage queue to assess how submissions are categorized and prioritized, and reviews cases outside standard parameters or above defined complexity thresholds. At a portfolio level, the underwriting manager monitors average risk scores by segment and deviations from expected loss ratios. In commercial lines, the underwriting manager additionally monitors line-of-business concentration limits, treaty reinsurance utilization rates, and submissions referred to specialist markets or declined.
Claims
In personal lines claims, initial notifications of bodily injury, fatality, or high‑severity loss require human contact. HITL applies when the claim involves coverage or benefit interpretation, assessment of loss or injury, or the applicability of policy constraints such as exclusions, riders, sub-limits, and caps. AI may make claims decisions only when the claim amount falls below a preset threshold and within defined confidence bands. Claims that breach those thresholds or fall outside those confidence bands escalate to HITL, as do disputes over loss assessments, benefit determinations, or settlement recommendations. In commercial lines, the HITL threshold is typically lower relative to total insured value. Large property losses, business interruption claims, and third-party liability matters require a dedicated claims handler or senior adjuster at first notification. Coverage interpretation on commercial policies, particularly where manuscript wording or bespoke endorsements apply, must be reviewed by a qualified professional before any coverage position is communicated to the insured.
Under HOTL, the operations manager monitors first notice of loss (FNOL) queue in real time for intake volume, loss type classification, severity estimation, triage accuracy, policy verification, and coverage analysis, and intervenes when submissions are misrouted or severity distributions indicate an event the AI has not encountered before. The claims supervisor monitors straight-through-processing throughput and tracks the proportion of AI decisions that adjusters reverse as a leading indicator of model error. Continuous divergence between AI-estimated and actual settlement amounts indicates model drift. In commercial lines, the claims manager additionally monitors incurred but not reported (IBNR) development patterns on long-tail liability lines, tracks litigation rates for AI-settled claims, and reviews whether AI triage correctly distinguishes between primary and excess layer exposures.
The rise of generative and agentic AI is closing the gap between business and technology, yet an SME rooted in business knowledge may find this capability gap increasingly difficult to manage.
Fraud Detection
AI systems identify fraud patterns, but the special investigation unit (SIU) investigator must evaluate the broader context and evidence AI may miss. When a claim is flagged, the investigator must independently assess the evidence before recommending referral to law enforcement or initiating a suspicious activity report, as these actions carry legal consequences and must be made by a named human. In commercial lines, fraud indicators can involve staged contractor invoices, inflated business interruption losses, or collusive broker-claimant arrangements, and the investigator must review financial records and vendor relationships before reaching a conclusion.
Under HOTL, fraud detection systems analyze every claim for duplicate submissions, inconsistent damage patterns, billing anomalies, and network linkages at a scale that no human team can match. The SIU manager monitors the alert queue and initiates dedicated investigations when alerts spike from a specific provider network or geographic cluster, or when claims of a particular type signal an emerging fraud ring. In commercial lines, the SIU manager additionally monitors for coordinated business interruption claims, inflated contractor invoices across multiple accounts, and abnormal claims frequency within a specific industry segment or broker portfolio.
Actuarial and Pricing
HITL applies to rate filing and certification, where AI output serves as input to the formal actuarial attestation process. Any decision involving a protected class or demographic characteristic requires heightened review for disparate impact, and this responsibility cannot be discharged by algorithmic compliance checking. The actuary must attest that the AI system meets all applicable regulatory standards, including fairness, nondiscrimination, and documentation requirements. In commercial lines, the underwriting actuary or pricing analyst must review AI-generated loss cost projections and exposure assessments before pricing and terms are finalized.
Under HOTL, actuaries monitor pricing models in production for performance and compliance, comparing actual and expected loss ratios by segment and detecting concept drift resulting from changes in the risk environment, new perils, shifts in claims patterns, or regulatory changes. They also monitor whether AI models or external data act as proxies for protected classes or produce adverse effects over time. In commercial lines, the pricing actuary additionally monitors rate adequacy by industry class and account size, and tracks whether AI-generated rate indications diverge from manual benchmarks on complex or large accounts.
Humans must be able to justify decisions, explain outcomes, and remain accountable under regulatory and institutional scrutiny.
Policy Servicing
In policy servicing, an AI-generated endorsement containing a coverage error creates contractual risk at the time of loss. HITL is required for changes that materially affect coverage, such as adding or removing a named insured, changing limits or deductibles, adding exclusions, modifying riders or endorsements, and reinstating or canceling coverage. A human must confirm that the change has been correctly implemented and that the updated policy reflects the original service request. In commercial lines, mid-term changes to insured values, the addition of newly acquired locations, changes to the named insured schedule, and modifications to umbrella or excess layers all require a qualified underwriter or account manager to review and confirm before the endorsement is issued.
Under HOTL, policy servicing covers address changes, payment processing, routine endorsements, certificate issuance, and standard renewals. These are high-volume, low-risk transactions where supervisory oversight is more appropriate than per-item review. The operations manager monitors queue throughput, exception rates, and customer satisfaction signals, such as repeat contacts on the same policy, which may indicate that AI is not resolving queries on first contact. In commercial lines, the account manager monitors automated certificate issuance for accuracy against the underlying policy schedule, tracks changes to scheduled locations or vehicles on fleet and property policies, and reviews whether AI-generated renewal packages reflect negotiated conditions for accounts with bespoke endorsements.
Customer Communications
HITL applies to any customer communication constituting a formal complaint, whether related to a coverage dispute, benefit determination, underwriting decision, policy cancellation or nonrenewal, premium increase, denial of service, or claim settlement. The response must be drafted by, or reviewed and approved by, a qualified professional before it is sent. In commercial lines, communications involving coverage declinations, material changes to structures, or responses to reservation-of-rights letters must be reviewed by a senior underwriter or legal counsel before being sent.
Under HOTL, standard customer responses cover FAQs, coverage and claims status updates, billing and payment queries, policy document issuance, and routine correspondence. The contact center supervisor monitors sentiment scores, first-contact resolution rates, escalation rates by query type, and complaints received after AI-handled interactions. In commercial lines, the account manager monitors whether AI-generated responses to broker queries accurately reflect coverage terms and premium allocations, and flags cases where the AI response may have introduced ambiguity into the coverage position.
Compliance and Regulatory Reporting
An AI‑generated regulatory filing containing an error creates institutional liability that far exceeds the cost of human review. HITL is compulsory for all regulatory submissions and adverse outcome disclosures, including rate submissions, adverse decision notifications, and supervisory responses. A human compliance officer must personally review and sign off on every submission made to the regulator.
Under HOTL, compliance and regulatory reporting focus on continuous model governance and supervisory monitoring. The compliance team monitors model outputs for bias by reviewing decision rates across demographic segments against expected baselines and flagging statistical deviations that may indicate emerging disparate impact. They also monitor model changes, data updates, and threshold adjustments to ensure continued regulatory alignment. In commercial lines, the compliance team additionally monitors market conduct requirements for large commercial policyholders, surplus lines regulations, and whether AI model changes require regulatory notification or refiling in multiple state jurisdictions.
SME in Command
Under HIC, domain experts authorize each AI use case before deployment, supported by documented risk assessments, clear accountability, and defined bias testing protocols. They establish requirements for model validation, ongoing oversight, and emergency intervention. In commercial lines, the SME on the board or at senior leadership level must define AI governance standards applicable to AI‑assisted broker interactions, delegated underwriting authority arrangements, and managing general agent operations where AI tools are used by a third party on behalf of the carrier.
At the enterprise level, the SME in command approves AI use cases and defines the deployment scope. They set the risk appetite and escalation thresholds and retain authority to intervene at a system level. They also retain accountability for AI-driven outcomes under regulatory and institutional scrutiny.
The Boundary of Human Oversight
Human oversight in an AI‑enabled organization is not a temporary safeguard. It is a control requirement arising from the fundamental characteristics of generative and agentic AI systems. These systems operate probabilistically, produce outputs that are not always verifiable at source, and cannot assume responsibility for their decisions. The burden of correctness, compliance, and accountability remains with humans, and the HITL, HOTL, and HIC models define how that burden is distributed across decision‑making layers.
Whether a human intervenes before execution, supervises ongoing operations, or governs system design and deployment, the expectation is the same. Humans must be able to justify decisions, explain outcomes, and remain accountable under regulatory and institutional scrutiny. Across insurance processes, AI systems cannot independently establish correctness in ambiguous situations, interpret regulatory nuances, or accept responsibility for adverse outcomes. These functions remain the responsibility of domain experts who serve as the control points within the system. AI supports human decision-making at scale but does not replace the need for human oversight.
Baskar Sundararajan is chief technology officer for BFSI at Tata Consultancy Services. Srivathsan Karanai Margan is an insurance domain consultant at Tata Consultancy Services.
References
DeepScribe. (2024, July 9). Optimizing human-AI collaboration: A guide to HITL, HOTL, and HIC systems.
Endsley, M. R. (1996). Automation and situation awareness. In R. Parasuraman & M. Mouloua (Eds.), Automation and human performance: Theory and applications (pp. 163–181). Lawrence Erlbaum.
FinTech Global. (2026, January 14). Why human-in-the-loop is key for insurance AI. FinTech Global.
Gouraud, J., Delorme, A., & Berberian, B. (2017). Autopilot, mind wandering, and the out of the loop performance problem. Frontiers in Neuroscience, 11, Article 541.
Ito, A. (2025, September 7). From HITL to HOOTL—A complete guide to human involvement models in the AI era. Medium.
National Association of Insurance Commissioners. (2023, December 4). NAIC model bulletin: Use of artificial intelligence systems by insurers.
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410.
Starmind. (2025, August 28). What human in the loop really means in insurance. Starmind.
Tech Jacks Solutions. (2026). Human-in-the-loop. Tech Jacks Solutions.