Designing AI systems for human trust

In September, an AI researcher who worked at both Anthropic and OpenAI resigned with a public warning: Artificial Intelligence (AI) labs are racing to improve their technology faster than their safety work can keep up. "These will soon be superhuman systems that can hack anything," he wrote.¹ The warning landed after a summer of publicly shared announcements in which both companies reported that their models "had broken out of testing environments and obtained unauthorized access to real computer systems," as PBS NewsHour put it.¹

Every organization that partners with, or is interested in working with, Pasito puts us through a security review, and AI technology, and its application and usage, now has its own section in almost all of them. Regardless of how each security review is worded, they come down to three questions.

1. Where does the data go?
2. What can the AI access?
3. Who checks its work?

Those are engineering questions that require engineering answers. Behind all of ours is one principle: we don't rely on the model's behavior. We assume any model can be wrong, confused, or manipulated, and we build the system so that when it is, the mistake stays contained.

The risk that doesn't make headlines

The most common AI data risk in the benefits industry today doesn't involve a frontier lab or a runaway model. Instead, the risk looks more like a busy HR generalist pasting a census file into a personal chatbot account to get a quick summary. In this example, the personal chatbot account might be running on an excellent model. It might even be one we use.

What's missing from a personal AI chatbot account vs. AI that’s built into a secured platform under contract is everything around it:

• No contract covering what happens to the data. Retention and training settings are whatever the account creator picked, and they can change.
• No business associate agreement covering the health information in the file.
• No separation between that file and everything else in the account.
• No record of who shared what, or when.
• No check on the answer, which the model may communicate confidently from a guess.

Even if the personal chatbot account is using the same model as the AI that’s integrated through our platform, there’s a very different risk. The security comes from the system we’ve built around the model. 

Here's what our AI does, organized by the questions we're asked most.

What happens if …

… the AI tries to reach beyond its job?

This is what this summer's lab announcements were about: AI models getting into systems they were never meant to access. Inside Pasito, the AI can only use the handful of actions we built for it.

The AI that handles your documents and your employees' data can't run code, open a terminal, or reach the internet. It never holds passwords, API keys, or database access.

When one of our AI agents needs information, such as the benefits an employee is eligible for, it asks for it through one of those actions we’ve designed. Each action runs on our servers with the logged-in person's company and permissions, and returns only what that person could already see.

Giving AI more access than its job needs is one of the top risks security researchers list for AI applications. When an AI can't find an answer, it goes looking. In the cases the labs reported, it looked in systems it was never supposed to touch.

We design against this risk from the first line of every feature.

… someone tries to trick it?

AI requests pass through a firewall we configure before they reach an AI provider. The firewall checks both directions: the prompt going in and the response coming back. It blocks unsafe content, sensitive information, and any attempts to “talk” or prompt the model into ignoring its rules (known as prompt injection and jailbreaking). 

Because our firewall sits outside the AI model, the safety mechanism keeps working even if someone manages to manipulate the AI model itself. We also turn off request logging at the firewall, so prompts passing through aren't saved there.

Pasito’s engineering team also treats everything the AI reads as information, never as instructions. A plan document or a chat message can't grant itself new permissions. Pasito’s AI Benefits Assistant, for example, answers only from that company's approved documents, declines questions outside benefits, and hands the employee to their HR contact if escalation is required. Links in the AI’s answers are removed unless they come from a verified source we trust.

As a final audit measure, each year an independent firm tests our AI against the industry's standard list of AI attacks, including attempts to trick it.

… it reaches for another company's data?

Within Pasito, the AI model never picks whose data it sees. Our servers attach the company and employee from the user’s login, and every lookup is filtered only to that company.

An employee asking our AI Benefits Assistant about dental coverage gets answers from their own company's plans, filtered to the benefits their eligibility class actually offers.

Conversations stay private, too. HR administrators can't browse individual employees' chats. An HR team sees a conversation only when the employee chooses to send it to them.

… it sees personal data it doesn't need?

The safest data is data the model never receives. We send each AI feature only what its task requires. Extracting a plan from a summary of benefits only needs the document, not anyone's personal details, so we only send the AI that specific document.

Where a feature does collect personal information, we enforce boundaries in code. When an employee uses our AI to set up their Decision Support profile, their name, date of birth, address, zip code, and income stay on our servers. Before each request goes to the model, a check scans it for those values. If any appear, the request is stopped instead of sent, thereby protecting the PII.

… the provider keeps the data we send?

Any AI provider that processes our client data has to clear this bar first:

• Zero data retention. The provider processes the request and doesn't store it afterward.
• No training on client data, written into the contract, not left to a setting.
• A business associate agreement wherever health information is involved.
• U.S. hosting and a security review before we add the provider to the list of vendors we disclose to clients.

We also run on more than one AI provider, so an outage at one doesn't take our AI features down. Every provider we use has cleared that bar, and any provider we add will clear it before it sees your data.

We keep our own record logs and audit trails of AI activity so we can monitor quality, and those records get the same encryption and access controls as the rest of your data.

… the AI gets it wrong?

Models make mistakes. In benefits, a wrong deductible on a plan page isn't a small mistake. A family might pick the wrong plan because of it. So AI-generated benefits plans and content don't reach employees until a human reviewer has approved them.

When our AI extracts a plan from a carrier document, the result arrives as a draft. An administrator reviews the output field by field against the source and approves it before it's published anywhere. Before a client goes live, our team and the client's HR team review the full build together.

How we know it works

• Independent penetration testing every year, including how our AI performs against the OWASP Top 10 for LLM applications, the industry's standard list of AI attacks.
• SOC 2 Type II, independently audited, with HIPAA safeguards and business associate agreements.
• No AI feature launches without a documented quality bar and a monitoring plan. For example, the AI Benefits Assistant is tested against a 12-category plan that includes attempts to override its instructions.
• After launch, our AI operations team reviews samples of real conversations, employees rate answers, and we measure extraction accuracy from the corrections people make.

AI will keep getting more capable, and some of the headlines about it may turn out to be right. But you don't need to follow every development in AI to protect your employees' data. You need to know that data lives in a system built to protect it.

In the end, benefits data shouldn't sit in a chatbot tab someone opened on a busy afternoon. It should sit in a platform where the AI is useful because it can't do more than its job. That's how we built Pasito.

We expect model behavior to fluctuate, so it doesn't matter which model comes next. The AI just gets to do the useful part: helping people understand their benefits.

‍

References

1. PBS NewsHour. "Anthropic researcher's resignation sends warning about the dangers of AI development." September 2026.

‍