Who is the AI agent working for in financial and insurance services? A brief from Stanford University.
Really interesting brief from Stanford University examining the conflict-of-interest risks posed by AI agents and calling for imposing a duty of loyalty on developers and deployers.
Key Takeaways
- AI agents may be steered to prioritize the best interests of their developers and deployers over those of their users.
- Currently, there are no requirements or standardized mechanisms for disclosing developers and deployers’ conflicts of interest — potentially causing invisible harm to consumers.
- Developers and deployers of AI agents operating in consumer contexts, particularly those in higher-stakes or regulated areas such as financial services or healthcare, should be subject to a duty of loyalty.
Introduction: What Agents Are and Why They Matter
AI is undergoing a fundamental shift from reactive generative tools to autonomous agentic systems. An agent runs on the same large language model (LLM) that powers a chatbot, but where the chatbot returns text and waits for the next prompt, agents are capable of performing multistep tasks to achieve defined objectives, calling external tools, retaining information across contexts, querying databases, and coordinating with other agents. This all happens with minimal human intervention — the user delegates the goal, and the agent chooses what actions to take.
For example, instead of merely drafting a travel itinerary on request, an agent (or group of agents) can access data, connect with APIs to navigate booking sites, cross-reference prices against a user’s learned preferences and willingness to pay, and execute purchases using stored payment details. Because these agents also have persistent memory and retain context across sessions, tools, and platforms, privacy frameworks built around app-specific data silos are no longer sufficient.
Today, agent autonomy — the extent to which an agent operates without user involvement — exists on a spectrum. Some systems pause for user approval before each action, while others execute entire workflows unsupervised. Where an agent falls on that spectrum is a design choice, and who makes that choice — the developer, deployer, or end user — has direct consequences for individual users and enterprises once deployed.
Since early 2025, Amazon, Google, Anthropic, OpenAI, Perplexity, Meta, and Microsoft have embedded proprietary agents directly into browsers and apps. Launched across domains and contexts of varying consequence, these agents mark a structural shift in how we browse and operate online. There are also a number of open-source AI agents on the market, including OpenClaw and Hermes Agent, and new open-source governance tools aimed at creating frameworks for agentic AI safety testing.
To function effectively — whether ordering groceries, rescheduling doctor’s appointments, or managing personal finances — these agents can request access to intimate personal data. Some may require login credentials, financial records, health information, communication histories, and real-time control over computing environments. Broad, cross-domain data access makes an agent more useful: The more data and context an agent has access to, the more effectively it can act on a user’s behalf. But that same broad data access creates a privacy risk that is invisible to the user. By accessing data across domains, an agent can infer highly sensitive information about a user’s physical health, cognitive decline, financial distress, or immigration status — categories the user never explicitly disclosed but that the agent can piece together from the data it gathered for entirely unrelated tasks.
As agent deployment expands, we raise two critical questions: will an agent act in the user’s best interest when its developer’s interests diverge from theirs, and who is liable if the user suffers harm?
In the absence of consumer AI regulation and adequate federal and state data privacy laws, agents risk becoming extensions of platforms that prioritize corporate gain over the best interests of their users — despite consumers’ expectations when they call on agents. In this brief, we argue that developers and deployers of agents — meaning both the entity that trained the underlying models that power an agent and the entity that operationalizes agents for users — should be classified as fiduciaries, especially those operating in high-stakes contexts. That designation would impose a duty of loyalty that requires entities to act in the best interests of their users within the scope of the delegated task, free from undisclosed conflicts of interest. Implementing an effective fiduciary framework requires coordinated action across three levels: technical standards bodies, federal regulators, and Congress.
Read the full report:
