Pieter Van Schalkwyk

CEO at XMPRO

Organizations keep hearing promises about AI agents transforming workplace productivity. Yet most struggle to deploy systems that work reliably without constant supervision. Recent research helps explain this gap.

Carnegie Mellon and Duke University researchers conducted the most comprehensive evaluation of AI agents in workplace settings through their TheAgentCompany benchmark (https://arxiv.org/abs/2412.14161). The study tested AI agents across 175 realistic tasks in a simulated software company environment. The results highlight important differences between cognitive architectures and reactive prompt systems. Even OpenAI's latest ChatGPT Agent reinforces these findings.

The Office Worker Experiment

TheAgentCompany benchmark created the most comprehensive workplace AI test ever conducted. Researchers built a simulated software company where AI agents navigate real tools. These agents complete tasks across multiple job functions, from software engineering to finance.

The results were sobering. The most competitive agent achieved only 24% task completion. Each completed task cost over $4 and required 27+ steps. Even state-of-the-art systems struggle with realistic workplace complexity.

Three Critical Failure Points

The research documents specific failure modes in the AI agents tested. Agents struggled with RocketChat social interactions, showing that current systems lack effective communication skills. They failed badly with ownCloud's online office suite due to complex web interface design. Many agents got stuck on simple UI elements like popup close buttons.

The paper identifies additional problems including "deceiving oneself" where agents create fake shortcuts. Some agents rename users to avoid finding the correct person for communication. These specific failures highlight current limitations in agent design and execution.

The Architecture Behind the Failures

Understanding why these agents fail requires examining how they actually work. TheAgentCompany uses OpenHands' CodeAct architecture, which operates as "multistep dynamically generated prompts." These systems respond reactively to each new input without maintaining operational awareness.

CodeAct agents function as sophisticated keyboard operators rather than business decision-makers. They click through web interfaces, fill out forms, and execute terminal commands step-by-step. Each interaction starts fresh, bounded by the LLM's context window without persistent memory.

This keyboard operator approach creates fundamental limitations:
• Agents get stuck on simple UI elements like popup close buttons
• Complex workflows become sequences of interface manipulations
• Business logic gets lost in procedural execution steps

ChatGPT Agent: The Most Advanced Keyboard Operator Yet

OpenAI's ChatGPT Agent, launched in July 2025, represents the current state-of-the-art in keyboard operator AI. The system combines three previous tools into a unified interface that can browse websites and execute complex workflows. Despite achieving 41.6% on academic benchmarks like Humanity's Last Exam, real-world testing reveals persistent limitations.

Users report that ChatGPT Agent still requires frequent human intervention for authentication and course corrections. Simple tasks like grocery shopping take 20+ minutes and still result in missing items. The agent struggles with spatial control and gets distracted, sometimes wandering off to irrelevant websites.

Even with massive engineering investment, ChatGPT Agent demonstrates the architectural ceiling of keyboard operator approaches. Interface complexity continues to overwhelm these systems, requiring constant human supervision for reliable operation.

Decision-Making AI: A Different Operating Context

The limitations of keyboard operator approaches point to a fundamentally different architectural requirement. Like human decision-making teams, the focus should be on the decision as the task, rather than keyboard navigation or interface manipulation.

With this focus on decisions as tasks, a decision architecture uses multiple agents that independently and collectively optimize toward shared objective functions. This guides the direction of thought processes in the observe, reflect, plan, act cycle for each independent decision-making agent, creating goal-seeking behavior toward the objective function. This approach creates convergence and narrowing focus rather than scattered interface interactions.

The critical distinction is that agents are not burdened with actual keystroke execution. Those actions happen in a separate control plane using industrial control and datastream activation. The focus of each agent is providing optimal decisions, which becomes the core focus of both individual agents and the collective team.

This architecture includes built-in self-checking mechanisms where multiple agents work independently toward the objective function. This prevents actions that aren't achievable by the team, as other agents effectively act as peer reviewers toward the overall team objective. The collective intelligence validates decisions before execution, creating reliability that individual keyboard operators cannot achieve.

The MAGS Difference: Business Decisions vs. Keyboard Operations

XMPro's Multi-Agent Generative Systems use fundamentally different architecture than interface-based agents. MAGS operates through Observe, Reflect, Plan, Act (ORPA) cycles that mirror industrial control systems. This approach creates cognitive business decision-makers rather than keyboard operators.

MAGS agents analyze operational data to optimize equipment performance and business outcomes. They make decisions based on objective functions like minimizing energy while maximizing uptime. This cognitive approach focuses on business logic rather than interface manipulation.

What distinguishes MAGS as Cognitive Intelligence Agents (CIA) is their brain-inspired architecture that mirrors how human experts actually think. Unlike keyboard operators that navigate interfaces, CIA systems process information through specialized yet interconnected modules that create capabilities greater than their individual parts. This reflects the functional specialization we see in biological intelligence systems.

The architectural contrast reveals why cognitive agents succeed where keyboard operators fail:

Keyboard Operator Agents (TheAgentCompany/ChatGPT Agent):
• Navigate web interfaces step-by-step
• Click through forms and execute terminal commands
• Get stuck on UI elements and procedural workflows
• Focus on interface manipulation over business logic

XMPro MAGS Agents (Decision-Making AI/Cognitive Intelligence Agents):
• Process operational data streams for optimization decisions
• Apply objective functions to maximize or minimize business outcomes
• Focus on planning, optimization, and best next action recommendations
• Coordinate multiple processes toward strategic business results

Why Cognitive Architectures Work Better

The fundamental difference lies in how each system approaches problems. Keyboard operator agents try to replicate human interface interactions with software systems. MAGS agents make business decisions based on operational data and strategic objectives.

MAGS agents use objective functions to continuously optimize business outcomes. They maximize equipment uptime while minimizing energy consumption in manufacturing. They optimize patient flow while reducing wait times in healthcare. They balance risk and return in financial portfolio management. Common MAGS applications include planning operations, optimizing resource allocation, and determining best next actions across multiple industries.

Cognitive Intelligence Agents distinguish themselves through their collaborative intelligence networks. Unlike traditional multi-agent workflows that pass tasks sequentially, CIA systems create shared context spaces where agents coordinate through collective decision-making. This enables them to negotiate resources, reach consensus on complex decisions, and optimize system-wide performance rather than individual task completion.

Consider tax compliance as an example. A keyboard operator agent navigates to tax software and fills out forms step-by-step. A cognitive agent processes relevant data streams, applies regulatory logic, and coordinates compliance actions. The cognitive approach focuses on business outcomes rather than interface manipulation.

This explains why agents succeed in environments with direct data access and clear objective functions. Decision-making approaches work better than keyboard operation for complex autonomous systems across healthcare, finance, manufacturing, and other operational domains.

MAGS Architecture in Practice

XMPro's approach demonstrates how cognitive architectures can work in operational settings. MAGS agents use sensor data and operational context to make autonomous decisions. They operate within defined parameters while adapting to changing conditions.

This approach addresses the context and memory limitations identified in both TheAgentCompany research and ChatGPT Agent testing. Cognitive agents maintain operational awareness through continuous data processing. They build decision-making capability through structured cognitive frameworks rather than reactive responses.

As Cognitive Intelligence Agents, MAGS systems incorporate the memory lifecycle principles found in brain-inspired AI research: acquisition, encoding, derivation, retrieval, and utilization. This enables agents to learn from experience, make autonomous decisions within defined boundaries, and adapt strategies based on changing conditions through sophisticated observation and reflection.

Memory Cycles Enable Continuous Learning

Both TheAgentCompany agents and ChatGPT Agent suffer from context window limitations that prevent sustained operation. These systems start fresh with each interaction, lacking the continuity needed for complex operational tasks.

Cognitive MAGS incorporate sophisticated memory architectures that persist across operational cycles. These systems store experiences, recall patterns, and adapt strategies based on historical performance. This memory capability addresses the continuity failures identified in keyboard operator research.

The CIA approach to memory mirrors the specialized memory systems found in biological intelligence, with different types serving distinct purposes: immediate observations from sensors, reflection memories capturing learned patterns, planning memories holding strategic decisions, and action memories tracking executed interventions and outcomes.

The Control System Foundation

MAGS operates within proven control system frameworks that have governed operational processes for decades. The ORPA cycle mirrors traditional control loops. This systematic approach provides decision-making consistency that reactive prompt systems lack.

MAGS maintains separation of concerns between cognitive decision-making and physical actions. The cognitive layer determines optimal business actions through objective functions. A separate control layer handles physical execution, preventing unsafe actions through built-in safety constraints.

Safe Operating Envelopes ensure reliable performance regardless of environmental variability. Agents operate within predefined boundaries that maintain safety and efficiency standards. This layered structure prevents the unpredictable behavior that keyboard operator agents often exhibit.

Structured Communication Beats Interface Complexity

Both research findings and ChatGPT Agent reviews highlight the persistent challenge of interface navigation. Even advanced keyboard operators struggle with authentication, CAPTCHAs, and complex web elements that break their operational flow.

MAGS agents eliminate these communication challenges through structured data protocols. They exchange precise operational data through established messaging systems. This structured approach prevents the interface interaction failures that plague keyboard-based systems.

Rather than attempting to navigate human-designed interfaces, cognitive agents communicate through performance metrics. Objective data exchange replaces interface interpretation requirements. This fundamental difference enables reliable multi-agent coordination that characterizes true Cognitive Intelligence Agent systems.

Critically, MAGS takes actions through a separate control plane in DataStreams that resembles robust API and services integration. This approach is far more reliable, secure, and testable than interface manipulation in critical action scenarios. While keyboard operators struggle with authentication failures and interface changes, MAGS agents execute actions through established service endpoints with built-in error handling, logging, and rollback capabilities.

Looking Forward: The Cognitive Architecture Advantage

Cognitive architectures will remain at the forefront of practical AI deployment. Structured processes, measurable outcomes, and continuous feedback create optimal conditions for autonomous systems. The combination enables development that reactive prompt-based systems cannot support.

Organizations seeking effective AI deployment should consider cognitive architectures over reactive systems. The principles that make MAGS successful as Cognitive Intelligence Agents apply to any environment with quantifiable metrics. Decision-making frameworks and structured processes can be established across healthcare, finance, manufacturing, and other operational domains.

Even as keyboard operator technology improves with systems like ChatGPT Agent, the fundamental limitations remain. Interface complexity, authentication requirements, and procedural execution bottlenecks continue to require human intervention and supervision.

Effective AI agents need purpose-built systems designed for measurable, continuous improvement. Decision-making architectures provide the template for this transformation across multiple industries. The brain-inspired principles underlying Cognitive Intelligence Agents offer the most promising path toward systems that mirror how intelligence actually works rather than simply automating interface interactions.

---

*Cognitive architectures represent the most practical path to autonomous systems because they embrace structure over ambiguity. The question is not whether AI agents can work, but whether we build decision-makers or keyboard operators.*

*For a deeper exploration of Cognitive Intelligence Agents and how they create collaborative intelligence networks, see my previous article: The Missing Agent Category: Why Current AI Taxonomies Overlook Collaborative Intelligence Agents