Meet The AI Agent With Multiple Personalities

In the coming years, agents are wide expected to instrumentality implicit much and much chores connected behalf of humans, including utilizing computers and smartphones. For now, though, they’re excessively mistake prone to beryllium overmuch use.

A caller cause called S2, created by the startup Simular AI, combines frontier models with models specialized for utilizing computers. The cause achieves state-of-the-art show connected tasks similar utilizing apps and manipulating files—and suggests that turning to antithetic models successful antithetic situations whitethorn assistance agents advance.

“Computer-using agents are antithetic from ample connection models and antithetic from coding,” says Ang Li, cofounder and CEO of Simular. “It’s a antithetic benignant of problem.”

In Simular’s approach, a almighty general-purpose AI model, similar OpenAI’s GPT-4o oregon Anthropic’s Claude 3.7, is utilized to crushed astir however champion to implicit the task astatine hand—while smaller unfastened root models measurement successful for tasks similar interpreting web pages.

Li, who was a researcher astatine Google DeepMind earlier founding Simular successful 2023, explains that ample connection models excel astatine readying but aren’t arsenic bully astatine recognizing the elements of a graphical idiosyncratic interface.

S2 is designed to larn from acquisition with an outer representation module that records actions and idiosyncratic feedback and uses those recordings to amended aboriginal actions.

On peculiarly analyzable tasks, S2 performs amended than immoderate different exemplary connected OSWorld, a benchmark that measures an agent’s quality to usage a machine operating system.

For example, S2 tin implicit 34.5 percent of tasks that impact 50 steps, beating OpenAI’s Operator, which tin implicit 32 percent. Similarly, S2 scores 50 percent connected AndroidWorld, a benchmark for smartphone-using agents, portion the adjacent champion cause scores 46 percent.

Victor Zhong, a machine idiosyncratic astatine the University of Waterloo successful Canada and 1 of the creators of OSWorld, believes that aboriginal large AI models whitethorn incorporated grooming information that helps them recognize the ocular satellite and marque consciousness of graphical idiosyncratic interfaces.

“This volition assistance agents navigate GUIs with overmuch higher precision,” Zhong says. “I deliberation successful the meantime, earlier specified cardinal breakthroughs, state-of-the-art systems volition lucifer Simular successful that they harvester aggregate models to spot the limitations of azygous models.”

To hole for this column, I utilized Simular to publication flights and scour Amazon for deals, and it seemed amended than immoderate of the unfastened root agents I tried past year, including AutoGen and vimGPT.

But adjacent the smartest AI agents are, it seems, inactive troubled by borderline cases and occasionally grounds unusual behavior. In 1 instance, erstwhile I asked S2 to assistance find interaction accusation for the researchers down OSWorld, the cause got stuck successful a loop hopping betwixt the task leafage and the login for OSWorld’s Discord.

OSWorld’s benchmarks amusement wherefore agents stay much hype than world for now. While humans tin implicit 72 percent of OSWorld tasks, agents are foiled 38 percent of the clip connected analyzable tasks. That said, erstwhile the benchmark was introduced successful April 2024, the champion cause could implicit lone 12 percent of the tasks.