Ambient Listening

If you arrived to dinner with a friend and they said “I’m going to record our entire dinner conversation” would you happily go along with this or would you instead ask that your friend not record your conversation? And if you did go along with it, would you speak more carefully? Share less? Wonder where that recording might end up?

What if the friend was simply wearing a device that recorded everything in their life and didn’t mention that the conversation would be recorded, would you be offended?

Our expectations of privacy are going to be tested over the next few years as a new type of device, powered by generative AI, begins to infiltrate every aspect of our lives. Ambient listening and “companion AI” are emerging device categories which employ “always-on” microphones and AI that can process ambient audio in real time. While initially we will certainly have some backlash from a privacy perspective, as with cameras on phones, many people will likely come to see the benefits as worth the trade-offs. But that outcome isn’t automatic, it will depend on norms, safeguards, and power dynamics we’re still figuring out. In my opinion we will soon become accustomed to listening devices in our offices, schools, hospitals, and even our homes. And the value delivered will outweigh the perceived loss of privacy.

Let’s start with the easy to justify use cases of listening devices throughout hospitals. Several companies, including Microsoft, have focused on this market. Providing an “autonomous digital scribe” which can transcribe patient conversations freeing the doctor to focus attention on the patient. Taken to the next level, passive always-on listening throughout the hospital can help to identify and correct for a huge share of serious adverse events which can be traced to communication failures during handoffs. When nurses change shifts, patients move from ER to recovery, specialists consult and then disappear misunderstandings or miscommunication can occur. Ambient listening could create a continuous ‘tape’ of instructions and clarifications that reduces these gaps. In addition to Microsoft (Nuance) startup companies like Abridge, DeepScribe, and Heidi are all providing ambient listening products for healthcare.

For a few years now a related technology from companies like Gong has been used by sales teams who often report as much as 25% higher close rates by analyzing successful pitch patterns. Customer service centers have also used the technology to reduce average handle time by providing agents with real-time AI coaching based on real-time conversation analysis. But this creates new dilemmas: should your annual review include analysis of every interaction you’ve had? Metrics derived from ambient listening will require careful human oversight.

In the consumer space companies like Bee, friend, Limitless, and omi are providing wearable devices which create a record of everything that happens during a wearer’s day. Described as providing a companion, personal journaling, memory aid, and emotional support these devices combine the creation of a transcript of all conversations with unprompted commentary, advice, and reminders. Imagine being able to ask, “what was that product my friend mentioned at dinner last night?” And your always-on wearable device can search the transcript of that conversation.

Of course there will be negative use cases as well. When you and your roommate or partner argue about what happened the prior night or what was agreed to, there is a transcript waiting to bear witness (or to muddy the water further with different interpretations). And if law enforcement and border agents can look at everything on your phone, can they look at all of the transcripts of your conversations?

The value proposition of ambient listening is compelling: enhanced productivity, improved safety, augmented memory, and unprecedented insights. But value realization depends on thoughtful implementation that respects human agency and privacy. We stand at an inflection point similar to the introduction of email, smartphones, and video conferencing. Success won’t come from asking “Should we adopt ambient listening?” but rather “How do we adopt it while preserving what makes us human?” This means maintaining spaces for spontaneous creativity, protecting vulnerable conversations, and ensuring that efficiency gains don’t come at the cost of psychological safety.