Making AI Risk Assessment More Objective: Addressing Sociotechnical Challenges

Abstract

Current AI risk assessment methods in industry rely heavily on subjective judgment. Yet recent U.S. AI policy frameworks, including Executive Order 14319 and America's AI Action Plan, mandate that AI systems be "free from ideological bias" and pursue "objective truth". While NIST provides systematic risk evaluation guidance, current AI governance frameworks are not designed to meet such objectivity requirements, instead offering flexibility that accommodates implementation across various contexts. Organizational AI risk practices rely heavily on subjective likelihood and impact scoring, compounded by subjectivity introduced when results are translated from technical teams to executives. Literature addressing this gap spans three disconnected streams: (1) AI governance frameworks relying on subjective risk assessment, (2) technical ML bias measurement focused on algorithmic fairness, and (3) organizational implementation approaches failing to translate governance principles into operational practice. Previous approaches lack links between risk analysis, objectivity requirements, and actionable guidelines, while policy frameworks remain too broad for practical application. Missing is research systematically bridging policy objectivity requirements with sociotechnical challenges of organizational risk assessment. To address this gap, we propose a framework transforming observable organizational factors into measurable risk indicators. Drawing from socio-technical systems theory and established risk taxonomies, we decompose traditional likelihood and impact metrics into specific, observable criteria replacing subjective estimates vulnerable to biases, addressing organizational tendencies to underestimate risks.