Presentation
Teaching Hands-On Skills with AI: Exploring the Feasibility of Automated Physical Skills Training
DescriptionMission readiness increasingly depends on the ability to train personnel faster, more consistently, and with fewer expert instructors. In domains such as Tactical Combat Casualty Care (TCCC), advanced manufacturing, and maintenance operations, growing workforce shortages and increasing task complexity are creating major challenges for hands-on training.
This presentation explores the development of an AI-enabled system designed to automatically assess skilled physical task performance and generate individualized feedback. The system combines egocentric video capture, multimodal AI analysis, and agentic feedback workflows to evaluate trainee performance against standardized procedures and support after-action review.
Using immersive TCCC training scenarios, the research examined the system’s ability to identify task steps, classify performance using Go/No-Go criteria, and generate explanatory feedback aligned with expert judgment. Results demonstrated strong performance in recognizing correctly executed procedures while also revealing important challenges related to subtle error detection, spatial reasoning, and interpretation of implicit expert knowledge.
The presentation will discuss lessons learned from assessing the feasibility of using multimodal AI in hands-on training environments and highlights broader human factors considerations for building scalable, trustworthy, and operationally relevant AI-enabled training technologies in high-consequence domains.
This presentation explores the development of an AI-enabled system designed to automatically assess skilled physical task performance and generate individualized feedback. The system combines egocentric video capture, multimodal AI analysis, and agentic feedback workflows to evaluate trainee performance against standardized procedures and support after-action review.
Using immersive TCCC training scenarios, the research examined the system’s ability to identify task steps, classify performance using Go/No-Go criteria, and generate explanatory feedback aligned with expert judgment. Results demonstrated strong performance in recognizing correctly executed procedures while also revealing important challenges related to subtle error detection, spatial reasoning, and interpretation of implicit expert knowledge.
The presentation will discuss lessons learned from assessing the feasibility of using multimodal AI in hands-on training environments and highlights broader human factors considerations for building scalable, trustworthy, and operationally relevant AI-enabled training technologies in high-consequence domains.
Event Type
Lecture
TimeThursday, October 22nd5:10pm - 5:30pm PDT
Location