July 14, 2026 · 10 min read

The AI security review for healthcare AI startups: what hospital and payer reviewers ask

Your prior-auth agent works. The pilot went well and your clinical champion is pushing to sign. Then the hospital's security office sends the review, and next to the HIPAA and SOC 2 sections you prepared for sits a block of questions about the AI itself. Who sees PHI at inference time. Whether another health system's data can leak into your embeddings, and what your agent is allowed to write to the EHR. Here is what that section asks and what actually satisfies each question.

How the AI section differs from the HIPAA and SOC 2 review

The standard review checks your organization. The HIPAA part maps to the Security Rule safeguards: access control, encryption, workforce training, contingency plans, plus a signed BAA. The SOC 2 part attests that your controls operated over a period. Both matter. Neither says anything about what your model or agent actually does.

The AI section checks the product. It asks where PHI flows when a request hits your inference endpoint, which third parties sit behind that endpoint, how tenants are separated in your vector store, whether agent actions are logged one by one, and who approved the agent's write access to clinical systems. Your SOC 2 report cannot answer these questions because SOC 2 does not cover model or agent controls. That gap is why hospitals bolted this section on.

The questions are converging on a template. The Cloud Security Alliance's AI-CAIQ, roughly 320 questions with version 1.1 shipped in June 2026, is the source many procurement teams now copy from. Hospital reviewers add a healthcare layer on top: BAAs, minimum necessary, audit trails, EHR access. Payers add a model-risk flavor, asking about validation and monitoring the way they would for an underwriting model.

One more difference: the audience. A hospital review often pulls in the privacy officer and clinical informatics alongside the CISO's team. Write answers a privacy officer can act on. "PHI is redacted before any third-party model call, here is the mechanism and the log" lands. "We take privacy seriously" does not.

The seven questions hospital and payer reviewers ask

Wording varies between health systems, but the same seven themes keep recurring in reviews of prior-auth, patient-communication, RCM, and clinical-documentation products. Here is each one, with the evidence that closes it.

1. What stops PHI from reaching your model provider?

Usually the first question, often phrased exactly like that. The reviewer wants to see the mechanism itself. Evidence that satisfies: a data-flow diagram at the inference level showing every hop a prompt takes, the named redaction or de-identification step and where it runs, the routing rule that decides which model a request can reach, and logs showing the control firing in production. If PHI legitimately reaches the provider, say so and point to the BAA and retention settings. If your answer is "our prompt template doesn't include PHI," expect a follow-up. Free-text fields carry PHI whether the template invites it or not.

2. Does the BAA chain reach your model provider?

Hospitals live by the BAA chain. HIPAA's business associate rules flow down: you sign a BAA with the covered entity, and every subcontractor of yours that touches PHI signs one with you. The model provider is the link reviewers now zoom in on. Bring a subprocessor table listing every party that can touch PHI, with the BAA status and data-handling terms of each. Name the model API tier you use, because BAA coverage differs by tier and consumer products are typically excluded. If a provider will not sign, show the compensating control: redaction before the call, or routing PHI workloads to a model you host.

3. Can one health system's data bleed into another's, embeddings and caches included?

Hospitals compete, and payers hold data about each other's members. So reviewers push past "logical separation" boilerplate and ask about the specific places cross-tenant leakage happens in AI systems: shared vector indexes, shared caches, fine-tuning on pooled data. Evidence: an isolation note that covers each store by name. Use per-tenant vector collections, because a single index behind a tenant filter you promise to always apply invites the follow-up question. Per-tenant encryption keys. Cache keys scoped by tenant. A flat statement that you do not train or fine-tune on customer data, or the consent and isolation terms if you do. And a deletion path that reaches embeddings and caches along with the source documents.

4. Show us a per-action audit trail we could hand to OCR

The Security Rule requires audit controls (45 CFR 164.312(b)), and hospital compliance teams treat the audit trail as the artifact they will need if OCR ever investigates. An AI agent raises the bar: the actor is software, so the log has to attribute each action to a named agent and the human authority behind it. What closes it: a sample log export for one real workflow, built on synthetic patient data, showing per-action records with timestamp, agent identity, the human or policy that authorized the action, the record touched, the model used, and the result. Then integrity: how you would prove the log was not edited after the fact. A tamper-evident chain the reviewer can verify beats a retention-policy paragraph. We wrote more on what evidence reviewers actually accept.

5. What can your agent write to the EHR, and what stops a bad write?

Write access gets a much harder look than read access, because a bad write lands in a chart or a claim. For prior-auth and RCM agents, writes to the EHR or claims system get their own review, sometimes their own committee. Evidence: a scope table listing every action the agent can take against the EHR, the credential it uses, and the guard on each. Give each integration its own scoped service account, since one shared user with broad rights is the pattern reviewers flag first. An allowlist of write actions, human approval gates on the consequential ones, and a log of every write with enough detail to reverse it. If everything the agent does today is read-only, say so, because read-only scope closes this question faster than any table.

6. What happens when the model is wrong?

Healthcare reviewers assume the model will fail and want to know what happens next. For clinical documentation, a wrong output reaches a chart. For prior auth, it reaches a coverage decision. Evidence: the human oversight map showing where a person reviews output before it takes effect and what triggers escalation, your production monitoring for error rates and drift, and an incident path that includes AI-specific failure modes with your notification commitments to the customer. NIST's AI Risk Management Framework gives you shared vocabulary here, and mapping your answer to it saves the reviewer translation work.

7. Where do models run, and how long does our data live in your system?

Hospitals ask because their own policies pin data to regions and set retention schedules, and your stack inherits those obligations. Lay out your deployment options in plain terms (shared SaaS, single tenant, customer VPC, on-prem), the regions available, and a retention schedule per data type: prompts, outputs, embeddings, logs. If you offer a self-hosted or VPC option, lead with it for conservative buyers. It converts several of the hardest questions above into one-line answers.

The honest-gap playbook

You will not have every control. No startup does, and healthcare reviewers know it. What kills deals is not the gap. It is the overstatement they find three weeks later, because in a hospital that discovery lands on the privacy officer's desk and the review starts over with less trust. The playbook is three lines per gap: current state, compensating control, dated fix.

Keep a gap register as part of your review pack and update it as items close. Reviewers reward the vendor who volunteered a gap before they found it, and they remember the one who claimed a control that did not exist.

Why a standing AI-section pack collapses review time

Every hospital asks these same seven questions, in different words and a different order. Answer them from scratch each time and you pay the full cost per deal: engineers pulled off the roadmap to reconstruct data flows, and weeks of back-and-forth as each answer spawns the next question. In public write-ups about hospital procurement, founders describe the AI section alone adding weeks to reviews they thought were nearly done.

Pre-writing the AI section as a standing pack collapses that. Eight artifacts cover it:

Map each artifact to the frameworks reviewers check against, the AI-CAIQ and NIST AI RMF, so they can tick their boxes without scheduling a call. Then reuse it. The second review becomes a diff of what changed since the first. Keep the pack versioned and update it when your architecture changes, because a stale pack that contradicts your live system is worse than no pack. For the question set beyond healthcare, see the 20 questions the AI section will ask you and how to answer each of the 20.

FAQ

Do we need a BAA with our model provider?

If PHI reaches the model provider, yes. HIPAA requires a business associate agreement with every party that creates, receives, maintains, or transmits PHI on behalf of a covered entity, and the obligation flows down to subcontractors. Some model providers sign BAAs on their enterprise API tiers, usually with retention turned off. Consumer tiers generally are not covered. If you cannot get a BAA, keep PHI out of those calls with redaction or local routing, and show the reviewer how you enforce that.

We have SOC 2. Does that cover the AI section?

No. SOC 2 attests to your organizational security controls: access, change management, incident response. It was never designed to cover model or agent behavior, PHI flow at inference time, tenant isolation of embeddings, or per-action agent logs. Hospital reviewers ask the AI questions precisely because SOC 2 does not answer them.

Are embeddings PHI?

Treat them as if they are. Embeddings computed from patient records derive from PHI, and they do not meet HIPAA's Safe Harbor de-identification standard just because they are vectors. Reviewers ask where embeddings live, which tenant they belong to, and how they get deleted. The safe posture: per-tenant vector stores, per-tenant encryption keys, and a deletion path that reaches the vectors themselves.

What does a HIPAA audit trail need to show for an AI agent?

The HIPAA Security Rule requires audit controls that record activity in systems containing ePHI. For an agent that means every consequential action: what the agent did, on which record, under whose authority, with which model, and what changed. Aggregate API-call counts are not enough. Reviewers want per-action records they could reconstruct an incident from, with integrity protection so the log itself is trustworthy.

Can we pass a hospital security review without self-hosting the model?

Often, yes. What reviewers actually require is control over PHI flow. A hosted model with a BAA, retention off, and demonstrable redaction can pass. A self-hosted or VPC deployment removes the hardest questions entirely, which is why many healthcare AI vendors offer it as an option for their most conservative buyers.

How long does a first hospital security review take?

Founders describe reviews stretching to months when they answer ad hoc, because each AI question triggers a new round of internal digging and the reviewer queue resets with every reply. Arriving with a prepared AI section, evidence attached, can compress rounds from weeks to days.

Facing a hospital or payer review now? Run the free 2-minute readiness checker to see which of the seven questions you can already answer. The AI Security Review Kit turns a 30-minute intake into the full standing pack.

Written by Dmitrii Karataev. Twenty years in infrastructure and security. Sources linked inline. Corrections welcome: hello@daylite.ai.