Audio Intelligence

Apple:

Audio Intelligence is a brand-new class of Apple Intelligence features that use the microphones of Apple Watch, the power of Apple silicon, and on-device and cloud-based AI models to help you detect sounds, identify music, and stay present and engaged — all while protecting your privacy at every step.

Hartley Charlton:

• Siri Recap: Uses ambient listening to generate high-level notes from conversations, such as takeaways from a team meeting or a parent-teacher conference, producing an Apple Intelligence-generated title, summary, and key points reviewable in the Siri app.
• Live Rewind: Lets users “go back in time for 15 seconds,” transcribing and displaying what was just said in a conversation with a double press of the Digital Crown.
• Shazam: Automatically identifies music playing nearby and surfaces the song title and artist on the Music Recognition widget in the Smart Stack.
• Sound Recognition: Brings sound alerts to Apple Watch for users who are deaf or hard of hearing, listening for sirens, alarms, doorbells, a baby crying, and more, even when their iPhone isn’t nearby.

Lily Hay Newman:

Apple is emphasizing that all of them are built to prioritize privacy and security using the extensive infrastructure the company has developed to protect its other Apple Intelligence and Siri services. As more and more of these types of features debut, though, the audio intelligence announcements are a reminder that AI-powered tools are becoming increasingly universal in all areas of computing and life. “These features do not create or store audio recordings, and raw audio used for processing is completely inaccessible to the operating systems, apps, the user, or Apple,” the company wrote in a report shared with WIRED. Apple emphasizes that its on-device capabilities for Apple watches have expanded thanks to the company’s new S11 chips. The chips include a special memory-protected space that Apple calls the Secure Exclave. Designed specifically for sensor data, the exclave is an isolated buffer that is inaccessible to the rest of the operating system where Apple Watch can hold and process audio data in a protected space. To try to mitigate blatant privacy violations, Apple says that the watch produces an audible chime when a watch owner initiates a Live Rewind transcript to alert other people in the area—even if it is on silent mode or paired with headphones. And the features are built to omit or filter out potentially identifying information about speakers. But the sheer scope and breadth of the tools is extremely noteworthy.

Zac Hall:

If you want to really take a deep dive on privacy and the new features, Apple has published an 11-page privacy paper that goes even further.

Previously:

• Apple Watch Ultra 4
• Apple Watch Series 12
• Pebble Index