source: Google Research: Advancing AMIE towards expert-level audio-visual clinical consultations
level: research
Google Research advanced its AMIE medical AI system to perform real-time video consultations. The new AMIE (Video) configuration processes visual and auditory cues, guides patients through virtual physical exams, and reasons diagnostically during live interactions. Built on Gemini and Project Astra, it uses a multi-agent architecture with separate talker, planner, and perception agents to maintain natural conversation while performing deep clinical reasoning and audio-visual analysis in parallel.
In a randomized study with 100 clinical scenarios and 300 consultations, AMIE (Video) matched board-certified primary care physicians on history-taking, diagnostic accuracy, management, and communication. It outperformed both physicians and a text-only AMIE in eliciting physical signs and guiding virtual exams. Patient actors preferred the video interface over text chat, rating it easier to use and more effective for communicating health concerns, and gave AMIE (Video) favorable empathy and rapport scores.
The study used trained patient actors in simulated settings, not real patients with their own conditions. Scenarios were limited to conditions that can be acted out, omitting presentations where audio-visual perception is critical. Automated evaluations revealed occasional perceptual and reasoning errors, and the system still has intermittent technical issues. Real-world feasibility studies are underway, but validation with real patients and broader clinical presentations is needed before any conclusions about practical utility can be drawn.
why it matters: This shows that AI can handle the visual and auditory dimensions of clinical practice, potentially expanding telehealth access and freeing physicians for complex cases.
source: Google Research: Advancing AMIE towards expert-level audio-visual clinical consultations