[Understanding Agency] It's Agents All The Way Down
Unpacking agency within AI systems to help better understand AI Agents and better communicate about them.
Introduction
When we interact with AI Agent applications, what appears to be a single, coherent system is actually a series of interacting, goal-driven agent behaviors. Like nested boxes, they operate at different scales and impact our overall system in different but inter-connected ways.
Understanding these layers of agency1 is not just an academic exercise. It is crucial for being able to properly diagnose issues and to better communicate about our systems.
When things go wrong with AI agents, the problem can lie in the misalignment between these layers. A language model might exhibit perfect understanding of a task, but if the control layer isn't properly designed to harness this capability, the end user experiences a disconnected or incompetent agent. Conversely, a beautifully designed user interface suggesting high-level agency can crumble when the underlying systems aren’t sophisticated enough to fulfill user expectations.
It is also important to consider when different roles within an organisation are discussing an AI Agent. A developer that works with agent frameworks like AutoGen or LangChain ReAct is thinking of agents as the composition of discrete capabilities in the control layer. A product owner referring to an AI Agent may be focussing more on end-user experience through a front-end interface. An AI researcher may be giving more weight to the capabilities of an underlying language model, and so on.
So let’s explore what the different levels of agency are and how better understanding them will help us design better systems and communicate more efficiently with others about them.
Levels of Agency
These levels of agency exist because our application is made up of at least three different layers. A foundational AI capability layer (typically a large language model), a control layer that directs the foundational AI layer and, finally, the actual user interface through which a user interacts with the system. As shown in the figure below we term the different levels of agency Emergent Agency, Structured Agency and Perceived Agency.
Emergent Agency
At the foundational level lies Emergent Agency - the agent-like behaviors that emerge from the architecture and training of large language models (or other AI capabilities) themselves. This type of agency manifests in several ways:
First, through the model's ability to maintain coherent dialogue and reasoning across a conversation. When a language model engages in step-by-step thinking or maintains consistent persona traits, we're seeing emergent agency at work. This includes the model's capacity to:
Parse and understand requests
Maintain context and coherence
Exhibit consistent behavioral patterns
Demonstrate reasoning capabilities
Second, through architectural features like Mixture of Experts (MoE), where different "expert" sub-systems specialize in different types of tasks, creating a kind of distributed agency within the model itself. The model's router network acts as a coordinator, deciding which experts to consult for different inputs.
Structured Agency
Structured Agency is where we, more directly, shape and constrain the raw capabilities of the language model into useful patterns through system design and engineering. This is the realm of:
Tool use and function calling
Memory systems and state management
Planning and orchestration
Error handling and recovery
This level is particularly important because it's where we transform the emergent capabilities of the model into reliable, practical behaviors. It's also where frameworks like AutoGen, LangChain or agent management systems like my own OpenDialog operate, providing structured ways to compose agent capabilities.
A key aspect of Structured Agency is how it manages the interaction between different components. For example, when an AI assistant checks a calendar, it's not just making an API call - it's coordinating between:
Natural language understanding
Planning and decision making
External tool integration
State management
Response generation
Something to watch out for is that depending on the framework the reference may be to either agent capabilities of even multi-agent systems where disparate capabilities are represented as single agents within the control layer that are co-operating to solve a problem.
Perceived Agency
At the highest level, Perceived Agency is what users actually experience when interacting with the system. This is where all the underlying complexity needs to come together into a coherent, understandable interface.
Perceived Agency is shaped by:
The system's communication style and personality including interface elements such as avatars, branding, etc
Error handling and recovery from a user perspective
Transparency about capabilities and limitations
Use of multiple interaction modes from text, voice and other UI elements.
This level is crucial because it's where user trust and understanding are built or broken. A system might have sophisticated underlying capabilities, but if the Perceived Agency isn't well-designed, users won't be able to effectively utilize or trust the system.
Consider a calendar assistant again: users don't need (or want) to know about the API calls, state management, or language model prompting happening behind the scenes. They just want to experience a helpful, competent agent that can understand their needs and take appropriate action.
Conclusion
Understanding these three levels of agency - Emergent, Structured, and Perceived - gives us a powerful framework for thinking and communicating about AI systems. It helps us:
Debug more effectively by identifying at which level problems are occurring
Design better systems by ensuring alignment between the layers and thinking of the system in a more holistic way
Communicate more clearly across teams about what we mean when we talk about "agents"
Set appropriate expectations for what our systems can and cannot do
As AI systems become more sophisticated, these distinctions will only become more important. Future developments in language models or other AI techniques will expand what's possible at the Emergent Agency level. New frameworks and tools will enhance our ability to create robust Structured Agency. And advances in user interface design will reshape how Perceived Agency manifests.
But perhaps most importantly, this layered view of agency reminds us that AI systems are not monolithic entities. They are carefully orchestrated combinations of different types of agency, each with its own characteristics and challenges. Success in building AI systems requires understanding and respecting these different layers, ensuring they work together harmoniously to create experiences that are both powerful and trustworthy.
Agency is taken to mean goal-directed behaviour as described in our framework for agency and autonomy.