Logo image
Back
Human intelligence in AI: a socio-technical perspective on why partners and calibration matter in talent intelligence systems
Dissertation

Human intelligence in AI: a socio-technical perspective on why partners and calibration matter in talent intelligence systems

Joshua A. Bellis
Doctor of Business Administration (D.B.A.), Drexel University
May 2026
DOI:
https://doi.org/10.17918/00011436
pdf
Bellis_Joshua_20261.21 MB
PDF Embargoed Access, Embargo ends: 31 Dec 2026

Abstract

Artificial intelligence adoption Human resources technology Information systems Talent intelligence Technology adoption
Despite significant investments in artificial intelligence (AI) within the field of Human Resources and Talent, organizations often encounter an "execution gap" between technical capability and operational performance. This gap typically presents itself from a deterministic approach to software implementation that neglects the socio-technical reality of probabilistic AI systems, which require "Human-in-the-Loop" (HITL) configuration. Grounded in Socio-Technical Systems Theory, Reference Class Forecasting and Absorptive Capacity, this quantitative study investigated whether External Implementation Partners (the "Outside View") serve as a structural intervention to drive multidimensional project success (efficiency and Effectiveness), and whether internal user calibration mediates this relationship. Utilizing a sample of 135 organizations deploying the Eightfold.ai platform, multiple linear regressions, stepwise regressions, and mediation analyses were conducted. The findings reveal a decoupling of structure and behavior in AI adoption. The engagement of an External Implementation Partner significantly improved baseline operational efficiency, reducing Time-to-Fill by an average of 10.1 days, by effectively navigating the macroeconomic constraints impacting the organization's particular industry. However, mediation testing was rejected (p = 0.475), demonstrating that External Implementation Partners do not causally induce internal human calibration behavior. Additionally, a stepwise regression revealed that while organizational controls and External Implementation Partners account for 37.3% of the variance in technical effectiveness, internal calibration behavior independently explains a massive 66.2% of the variance in algorithmic Quality (Match Score). This study empirically establishes the theoretical limits of the "Outside View" in AI deployment. The data proves that while operational speed is structural, algorithmic quality is behavioral. Achieving digital transformation requires a dual-pronged strategy: outsourcing structural configuration to external experts to build the system, while aggressively managing internal "Realized Absorptive Capacity" to train it. Ultimately, organizational leaders must consider if their organizational structure to utilize deterministic technologies should be adapted to address the needs and requirements of probabilistic technologies, meaning the infrastructure and teams we built to get us through the last 30 years, can't look the same for the next 30. Leaders should also budget not only for the external software and partner scaffolding, but also for the internal human capacity required to maximize the socio-technical return on investment.

Metrics

1 Record Views

Details

Logo image