Summary
Rashi Agrawal, who leads AI and ML at Hinge Health, delivers a practitioner’s architecture talk at the AI Engineer conference on what it actually takes to ship member-facing healthcare AI safely. She opens with concrete production failures that set the stakes: a man hospitalized after an LLM instructed him to substitute table salt with sodium bromide (bromide levels reached 200 times the safe limit), a Mount Sinai study finding a consumer health AI undertriaged life-threatening emergencies including diabetic ketoacidosis and respiratory failure 50% of the time, and ECRI naming AI chatbot misuse the number one health technology hazard of 2026.
Agrawal’s framework rests on three distinct layers. The first is architecture-level PHI protection: stripping patient data at the pipeline ingestion boundary before it reaches storage, maintaining complete separation between production and non-production environments, and enforcing access controls tied to both role and geographic region under HIPAA and FDA good machine learning practice requirements. The second layer is deterministic rules that execute above the model in code — clinical versus non-clinical routing, emergency escalation triggers, and identity verification all happen before a single token is generated, because probabilistic systems are inherently unsuitable for decisions that cannot be wrong.
The third layer is continuous production evaluation: not a pre-launch checklist but always-on automated judges scoring live traffic across 30-plus dimensions (clinical accuracy, escalation appropriateness, drift, refusal rate), member thumbs-up/thumbs-down signals per message, and 100% sampling of high-stakes conversation traces. Her closing point is that the bottleneck in production health AI is not compute or model capability — it is having enough human reviewers to read the monitoring signals and act before failures compound.
📺 Source: AI Engineer · Published August 19, 2026
🏷️ Format: Deep Dive
