16 reasons to Turn Off the AI Transcript in Session
With Soleil Merroir
1. Confidentiality is more complex now
Confidentiality is specific legal architecture built around one assumption: that what a client says stays inside a relationship bound by licensure, mandatory reporting exceptions, and a supervising body that can actually hold a human accountable. The second a client's disclosure is transcribed, it enters a different architecture — a vendor's terms of service, server logs, and API calls. A signed Business Associate Agreement (BAA) makes an arrangement HIPAA-compliant. It does not grant evidentiary privilege. In civil litigation, criminal proceedings, custody battles, or immigration hearings, raw transcripts and server logs held by third parties can be subpoenaed in ways a therapist’s synthesized, high-level summary progress notes never would be. Compliant and confidential are not synonyms.
2. You can’t "de-identify" human stories
"De-identified," "aggregated," and "used to improve services" show up constantly in privacy policies with opaque definitions and retention windows. In therapy, true de-identification of unstructured dialogue is nearly impossible. Unique life histories, specific relational dynamics, and intersectional identities cannot be stripped by an automated anonymization pass without destroying the clinical context. Furthermore, research on AI tools in mental health consistently highlights severe gaps between marketing promises and operational reality regarding data privacy and model training (JMIR Mental Health 2025;12:e79838).
3. Clients self censor for safety, when a machine listens
People self-censor around machines the same way they self-censor around anyone they aren't sure they can trust — and the material most likely to get edited out in real time is exactly the material we address in therapy spaces- deep vulnerability, shame, kinks, fears around identity and belonging. Not information most clients would want a tech company, or most notably, their employer or landlord, knowing. A client doesn't have to consciously distrust the recording to change how they engage with you, because of distrust of a known privacy risk. (Note: While some neurodivergent clients report reduced anxiety when a therapist isn't actively hand-writing notes, for marginalized and criminalized clients, the presence of a constant acoustic listener creates a tangible chilling effect.)
4. Transcripts destroy protective documentation discretion for marginalized clients
A skilled clinician exercises intentional narrative judgment when writing notes — particularly for clients who are criminalized, undocumented, sex-working, trans, or living under restrictive state surveillance. The safest clinical note is often one that purposefully omits specific identifying details that could be weaponized in custody disputes, court orders, or state investigations. An AI transcription tool captures everything indiscriminately. It lacks the moral and political discretion required to protect vulnerable clients from carceral systems.
5. Algorithmic bias reproduces pathology-first framing
Generative models are trained on dominant medical and psychiatric literature that heavily favors diagnostic reductionism, DSM taxonomies, and pathology first for ease and speed, not for care. As a result, auto-generated summaries systematically nudge clinical formulations back toward standardized diagnostic categories. This actively undermines non-pathologizing, relational, and liberatory therapeutic frameworks.
6. Consent is dynamic, not once on a terms of service
Consent isn't a static gate you pass through; it is continuous and renegotiated as the terms of the exchange change, we know this pattern in sexuality and kink because of the nature of the power exchange and risks of harm requiring high levels of consent. Similarly a tech company is in a position of a lot of power, holding the data of millions of vulnerable users, so if the terms aren’t reviewed over time, there’s likely things happening outside of the client’s awareness. A client signing an intake form agreeing to a "documentation tool" in session one has no way to consent to that same tool transcribing a disclosure about childhood sexual abuse or stigmatized behavior in month eight. The stakes shift dramatically over the course of treatment, but technical consent rarely keeps pace on platforms and telehealth.
7. The tool's incentives and your client's interests are fundamentally misaligned
Every professional ethics code prohibits dual relationships for a reason: one structure cannot serve two masters without eventually favoring one. A commercial vendor's business model depends on system engagement, retention, valuation, and scale — not on your client's relational safety. The same system holding someone's vulnerability is built within a market structure designed to extract value from data. The note-taker is the next iteration of this problem- now our real therapy data is a commodity for extraction, without us attached to it.
8. It normalizes machine surveillance as acceptable in intimate spaces
Every choice made in the therapy room conveys implicit information about safety, boundaries, and witnessing. If a therapist's record-keeping relies on an always-on automated listener, the subtle lesson is that having a commercial machine present for your most vulnerable moments is normal. It dissolves the boundary between AI as backend infrastructure and AI as a relational participant.
9. It reproduces the same extraction pattern found in all digital labor
Whenever a market system profits from users, someone's vulnerability, attention, or uncompensated expression is where the value comes from. Sex workers' bodies trained all the new sex imagery without consent or compensation, and now therapists are experiencing the same extraction. Every word you speak is now a possible data point for LLM’s to consume and regurgiate for their next word prediction in a synthesized therapy bot model. A client's vulnerabilities, and ours as professionals, is now the most valuable human data for AI. Turning intimate human vulnerability into raw material for commercial technology stacks is NOT within the scope of the work I agreed to perform, or consented to performing.
10. Capturing clinical struture accelerates commodification of therapeutic labor and automation
Every time a transcription tool captures how a clinician structures a session, times an intervention, or frames a complex reflection, that skill is captured as structural data. This isn't neutral. It takes embodied clinical expertise and converts it into data points that can train automated mental health tools. It asks clinicians to actively participate in the commodification and potential replacement of their own skilled labor.
11. There's no one inside the transcript to hold what it heard
Human vulnerability is valuable and protected because it has weight, emotional, energetic, communal on those who experience it. The machine can take in millions of traumas a day, but what happens to the weight of those experiences, the ripple across the humans who shared and held those words? In the clinical world, the therapist and the client move through the stories, the meaning, the paths to self and other connection, after session and into ever human interaction thereafter. We are impacted by one another and learn to hold what that means again. This isn’t data to be categorized and mined.
12. Scope of competence applies to technology, and most clinicians cannot audit their stack
Ethical codes (e.g., NASW 1.04, APA Standard 2.01) require clinicians to practice within their demonstrated competence. Most clinicians using AI note-takers cannot verify which underlying model processed a specific session, whether intermediate caching occurred, whether vendor BAA terms extend to sub-processors, or how server logs are handled during a vendor acquisition or insolvency. Offering confidentiality through a system you cannot independently audit or verify is practicing outside your scope of competence. Not knowing *or caring, about the impact of AI and the tools on your client, would indicate telehealth is outside your scope.
13. Your client didn't consent to the whole vendor chain
Most AI transcription tools aren't running self-contained models on local hardware. They call out to third-party Large Language Model (LLM) APIs to do the language processing. Your client's disclosure doesn't just go to "the app"; it flows to the app, its hosting providers, and its underlying model vendors. Even when enterprise grade Zero Data Retention (ZDR) endpoints are configured correctly, the technical architecture relies on an interconnected supply chain. The gap between "I told my therapist's note-taking tool" and "I told my therapist's note-taking tool, which passed my voice through a multi-vendor cloud processing pipeline" reinforces that client’s need that information stated clearly, to consent to participation of data release.
14. A BAA with the front-end app doesn't guarantee security across sub-processors
Many specialized clinical wrappers rely on enterprise API endpoints provided by third-party model developers. While enterprise zero-data-retention agreements exist, a therapist's BAA is typically signed with the front-end vendor, not the underlying foundation model provider. If that vendor misconfigures their API calls, defaults to standard endpoints, or uses unvetted third-party sub-processors, HIPAA compliance breaks down silently at the infrastructure level. "HIPAA Compliant" on a homepage refers to the front door; it says nothing about the pipes behind it.
15. Real-Time Hallucinations Create Permanent E-Discovery & Liability Traps
Modern transcription architectures (such as Whisper-based audio models) are prone to hallucinating text during pauses, "dead air," or background noise — sometimes generating aggressive or nonsensical phrases that were never spoken. If a clinician manually edits out a hallucinated phrase in the final progress note, the raw audio log or original auto-generated transcript stored on vendor servers remains electronically stored information (ESI). In civil litigation, discrepancy between an auto-generated transcript and a modified progress note opens the door to claims of evidence spoliation or record tampering. By introducing an unedited verbatim middle layer, you create a permanent legal liability trap that human-written summary notes never produce.
16. It Removes Human Friction and Gatekeeping of Client Records
Historically, when an attorney, insurance auditor, or law enforcement agency issued a records request, it landed directly on the clinician’s desk. That friction was a feature, not a bug. It gave the therapist time to consult the client, explain what was being requested, seek legal counsel, assert privilege, or narrow the scope of the release. As therapy management platforms integrate automated note-generation directly into client portals and cloud EMRs, record releases are increasingly automated. Verbatim transcripts can be exported or subpoenaed without the clinician ever acting as a protective boundary between the client and the institution requesting their data.
We are all struggling under overwhelming administrative tasks, and the promise of relief from AI is tempting. But "it saves time" has never been a good clinical, legal, or ethical justification for altering the core privacy and context of the therapeutic relationship.
Turn it off. It’s hard to oppose systems that move faster than we can understand, but it’s not too late to regain clinical oversight, repair the relationship with your client and be scrupulous the future of our work in the technological landscape.
