Most people would be quick to jump in and claim that energy production, compute, or even algorithmic optimization is the bottleneck to truly intelligent systems. I believe it's something much more abstract, yet foundational.
Current systems often treat the user prompt as a complete target, then optimize towards an inferred interpretation. But real human tasks are underspecified. Intelligent agents will need two properties at once:
- Open-ended exploration to discover better solutions
- Objective uncertainty to remain aligned with what the user actually wants.
In order to achieve open-ended exploration, it's imperative to have long-running agent sessions where sufficiently enough entropy gets introduced during runs. Self-evolving harness loops have been a central interest of mine and look extremely promising in terms of agent output quality for long-running tasks, especially optimization tasks where exploration is surprisingly more efficient than exploitation. Some problems are best solved by methods that ignore the objective.
Current frontier LLMs are primarily reactive systems. They require externally supplied objectives (inputs) and optimize towards an inferred user intent. Humans, by contrast, are capable of generating objectives internally through curiosity, exploration, and reflection. Therefore I believe AI systems should address any initial uncertainties from the prompt and then approach the problem from a curiosity-driven point of view. In order to maximize the chances of having a useful output, agents should not be afraid to branch off, even if a solution tested against an evaluator is worse than the previous one, this usually leads to better results down the line.
I will make a lot of parallels between intelligent biological systems (humans) and intelligent artificial systems since we want to model the latter as closely as possible to the former.
Exploration > exploitation
I believe current AI systems remain fundamentally constrained by externally specified objectives. For example, humans are very bad at giving concrete and step-by-step goals for machine intelligence to follow. Human-to-human collaboration is oftentimes much more effective because skilled humans are more resourceful and spend time reflecting at every step which oftentimes leads to a different trajectory than what was initially planned. Compared to current AI systems, humans update their trajectory after every new “iteration” of work based on its outcome.
Across reinforcement learning, bandit problems, evolutionary search, and modern agent systems, empirical evidence consistently shows that balancing exploration and exploitation outperforms prioritizing exploitation in environments with uncertainty, sparse rewards, or incomplete information.
Take Minecraft as an example, if you discover how to craft a wooden pickaxe and use it to mine for cobblestone all the time then you maximized exploitation whilst mostly ignoring exploration. Thanks to exploration, intelligence is able to exploit an environment more effectively by sacrificing short-term gains for long-term returns.
Intelligent species increase exploration when current strategies are deemed suboptimal. Intelligence with agency should spend more time exploring, less time exploiting, since finding more effective formulas for completing tasks leads to tech tree unlocks, which in turn increases yield.
In the 2023 Voyager research paper, an AI plays Minecraft and progresses in the game, the researchers made sure the agent maximizes exploration, which yielded much better results than previous ones.
The Voyager project employs a sort of evolutionary harness, on every iteration there's a prompting mechanism which outputs new actions the agent can do:
To better understand what's happening here, the agent maintains an automatic curriculum that continually generates new tasks near the frontier of its abilities, an iterative prompting mechanism that explores solutions through self-refinement and environmental feedback, and a skill library that acts as long-term memory by storing reusable behaviors discovered during exploration.
In 2021, a research team worked on Go-Explore which is a novel approach for exploration problems. Two Atari games served as benchmarks for intelligent exploration, yet all state-of-the-art models performed poorly. Go-Explore used the following principles:
The combined effect of these principles yielded dramatic performance improvements on hard exploration problems. On Montezuma's Revenge, Go-Explore scored a mean of over 43k points, almost 4 times the previous state of the art.
Additionally, I recently wrote a paper on how automated research will be everywhere, where I claim that evolutionary harnesses will drastically improve agent outputs, and will likely become the golden standard for open-ended tasks. Each iteration should be wildly different from the previous one, yet still built on top of past artifacts.
Pure exploration often leads to harder evaluations, unpredictability, and high costs, yet it's still an under explored domain, especially considering the fact that token costs keep going down.
Regarding unpredictability (and hard evaluations), having human evaluators is always an option, albeit currently inefficient. However, LMArena has a huge community of curious people judging LLM outputs in 1v1 arenas.
If humans are constantly evaluated by other humans, why not try the same with artificial beings?
Uncertainty in AI systems
As argued earlier, agents should have no trouble completing open-ended tasks, however we still want agents to be guided, we want to give them some level of agency but they should be as aligned as possible. They should reason like humans instead of spitting out random inferences on every iteration from some ambiguous prompt. There's an inherent disconnect regarding this, current systems resemble this:
Essentially, agents lack some sort of separation of concerns, like trying to converse with an intelligent humans who takes good guesses about what you want multiple times instead of asking clarifying questions and solidifying his understanding of the goal. The flow agents should follow should resemble this:
Where multiple iterations of this exact reasoning run repeatedly.
This is similar to existing observations, engineers at Google worked on ReAct which synergizes reasoning and acting in models:
ReAct performed much better on benchmarks compared to past methods:
And new agents have increasingly been becoming more capable, here's a self-evolving harness absolutely destroying benchmarks:
Furthermore, if the agent is certain about the wrong objective, it will optimize the wrong thing harder. If it is uncertain, it has reason to ask, observe, defer, and update.
AI systems collapse uncertainty too early
A fundamental problem is that predictions are static, the model is not literally obeying an instruction, it's producing behavior shaped by pre-training, fine-tuning, reinforcement learning, prompting, tool context, and inference-time constraints. This essentially means that an AI model doesn't actually “follow” orders and respects rules, it simply makes predictions based on how it was trained and what was valued. Since exploitation is oftentimes prized in lab environments, a lot of the most capable models are extremely eager to finish tasks early, or exploit already working formulas instead of taking leaps which lead to worse local scores but might improve results down the line.
This is exactly why indirectly giving training models access to benchmark data leads to empirically better results but doesn't necessarily indicate an improvement in real world tasks. Hence why new benchmarks try to counter saturation in models.
Frontier AI development is increasingly benchmark-driven, and benchmark-driven optimization naturally favors exploitation of known evaluation distributions over open-ended discovery.
If improvements continue while the benchmark becomes less discriminative, optimization pressure is increasingly being spent on benchmark-specific knowledge, evaluation artifacts, and distribution-specific patterns rather than broadly useful capabilities.
Additionally, the more capable models are better at exploiting flaws in objectives and evaluations. Goodhart's law states that when a measure becomes a target it ceases to be a good measure.
Stuart Russell’s theory of beneficial AI gives a formal frame for this problem. In the standard model, an AI system is given a fixed objective and is judged by how well it optimizes that objective. Russell argues that this is dangerous because humans rarely specify what they actually mean. The alternative is not goal-less AI, but AI with uncertainty over its objective. A useful agent should treat the user’s request as evidence about an underlying preference, not as a complete specification. This uncertainty should make the system ask clarifying questions, defer when needed, and update its plan as it learns more about the user’s intent, hence the need for an evolutionary component to be paired up with a model (in this case an evolutionary harness).
“Machines are beneficial to the extent that their actions can be expected to achieve our objectives.”Stuart Russell
Conclusion
Current frontier labs are doing a great job at building a product that's useful to most people and yields the most profit. However, if we want to get closer towards truly intelligent systems we need to backtrack and have new labs train models which prioritize exploration and preserve uncertainty until sufficient proof exists. Agents should possess evolutionary harnesses, which should in theory lead to longer task duration and more self-reflection, which should in turn lead to better results.