I Survived Agents vs Zombies

I wonder if the folks at OutSystems were familiar with the concept of the "philosophical zombie" and the mild discourse taking place amongst internet nerds trying to decide if today's LLMs fit the bill when they set up the seasonal coding challenge/game, Agents vs Zombies. This is very much not the point of this article, but as an aside: LLMs are definitely not p-zombies and nothing ever will be because they're just an apparently useful tool for philosophers to engage in thought experiments on the internet. If you want to know more, maybe check out this article on Experience Machines by Robert Long:

AI systems are not p-zombies

In a boring but nonetheless important terminological sense

Experience MachinesRobert Long

These are the inevitable rabbit holes that one finds oneself shoulder-deep in when sitting down on a caffeinated Saturday morning to write about a fun game they played. I guess I was prone to it since I have been celebrating the holidays by watching scary movies and just watched Cargo:

Anyway. Agents vs Zombies. Let's start with what it was, exactly. On October 23rd, Rodrigo Coutinho from OutSystems posted the exciting news on the forums that a Halloween-themed AI coding challenge had begun and the entire global development community was welcome to join in on the fun. Your character, the endlessly cute Agent character OutSystems created to represent agentic AI in a non-threatening way, has to find its way through maps of increasing size with more obstacles in order to find its way to an antidote on the map to save Neo, the OutSystems mascot, from succumbing to their fate of becoming a zombie. I could go on another tangent here about the fact there's never an antidote for being a zombie so the premise is flawed, but I will not because the game was really good and you're here to read about that, not my seasonal interests.

From a technical perspective, the whole thing boiled down to an experiment in leveraging LLMs for pathfinding in an abstract environment. If you know me, then you know I am highly critical of the applicability of LLM technologies and wear a "show me" attitude on my sleeve. I am not a blind hater, I just need the use cases to be proven out before I will accept LLMs as the technical revolution we are being told to believe they are. As such, I admit that I rolled my eyes when I first saw this come up, but it was definitely something I had to experience. I can't say, "show me" and then close my eyes.

I got signed up, downloaded the game files off the Forge, and ended up getting hopelessly addicted to building the best LLM-based pathfinding algorithm I possibly could, and not even because I wanted the cash prize. I didn't realize there were prizes until I was well into the game. The folks at OutSystems just did a really good job with this one and clearly poured a lot of time and care into making something fun for the development community at large. This was an investment, clearly, and I believe the intended ROI is a more engaged and happy base of their most important users: the developers.

Yes, I said the developers are the most important part of the OutSystems ecosystem, get over it. It's a development platform and a ship doesn't run without sweaty, unspeakably attractive people shoveling coal into the engine.

The release of Agents vs Zombies happened against the backdrop of new leadership at the company and a number of decisions aimed at improving the lives of their customers and users. The new CEO, Woodson Martin, announced at ONE in Lisbon that they would be reducing the barriers to entry that have marked ODC since its public release almost 3 years ago. In a huge move to make the cloud-native product more accessible, O11 and ODC would no longer require separate contracts and O11 would receive ongoing support indefinitely. In the same breath, they also announced that personal environments, which are used for everything from training to community contributions, would now be ODC instead of O11.

It may be funny to describe tech company announcements at their own global conference as "jaw dropping," but I am a nerd and a longtime OutSystems developer and I was equally stunned and thrilled at the move. I've wondered for a long time why it was so hard to get access to ODC and I always assumed it was because they were worried about platform stability and cost associated with hosting an entire cloud-native development infrastructure for every individual who wants it. Clearly, someone heard whatever the actual concerns were (I am in no way an "insider" at OutSystems) and said, "I don't care. Send it."

And send it they did! Not even a month after making ODC available to all developers, they released Agents vs Zombies and said, "have at it." I can't speak to any actual numbers, but there were around 100 people in the Discord server and you have to imagine that's just a small percentage of the total population of developers that played the game. Oh, and by the way: really big move setting up a Discord server for players to get individual support for when things went wrong. Honestly, all around a good move which was executed rather well.

The game itself looked really good. The art direction was whimsical and fun, and it really did sort of feel like playing a video game. Like I said, I was hopelessly addicted to making this thing work in pretty short order, even though I still think LLMs are not ideal for the use case of pathfinding. After all, pathfinding is kind of a solved exercise in AI, especially in small, defined spaces like the ones we were working with in Agents vs Zombies. The environment was intentionally unclinical with the API opting to verbally describe the safety heuristics of all available moves when your agent was determining where to move next so you would be encouraged to use an LLM to examine the safety of those moves.

I will do a technical breakdown of the whole experience in a later article. The big day was Halloween when someone from OutSystems unveiled the final map and ran everyone's agents against it. I was not surprised to hear that 87% of agents managed to at least run when poked, but I was somewhat surprised to find that only 37% of those actually managed to reach the goal after an average of about 4 runs. Regrettably, I forgot Halloween starts in Portugal quite a bit earlier than it does in Minneapolis so I had some untested changes that got used in the big show, so none of my agents ended up finding the antidote. I am in that 63% that failed, and I am still pretty mad about it considering the last map didn't even appear to be that hard.

As I said, I won't complain about LLMs as a use case for this kind of pathfinding right now and will instead wait until I write the technical article for that. For now though I will say it is wild that with the full might of any foundation model of choice, only 37% of developers managed a working solution.

I digress. OutSystems: great job. This was a load of fun and went a long way toward engaging developers and getting them familiar with LLMs as a functional technology. I am looking forward to more games in the future!