cashew | Distracted by AI (Reply)

So. Somewhere along the line of me working on my project, I got sidetracked and went down the rabbit hole of AI game master projects. Like...wow, I sank way, way too many hours into playing AI Dungeon.

TL;DR summary before I get into the details: AI, even the most advanced versions, are not good enough to write fiction. I've tried both the dedicated DM AIs (aka, ones where a scripting platform is layered over the AI generated text) and free-form chatbot as a GM, and it all fell short. We are so not ready for AI-assisted creative writing yet, unless you're satisfied with a ton of repetitive nonsense.

My journey down the rabbit hole began when I wondered about whether I can wrangle an AI into a DM for game testing purposes. I read the idea off of Reddit and thought, "Well, why not? Can't be worse than those text-based adventures of yesteryear."

I was wrong. It is so much worse and so much more illogical.

After some extensive Google searching and testing, I've found that the there are several severely limited free trial options available, probably due to the cost of running AI-based chat bots as a GM. Out of these, three were fairly promising options: AI Dungeon, Friends & Fables, and Perchance AI RPG. (I've tried also free-forming on chatbots, but the problem with repetition is still pretty bad.)

I'm going backwards in terms of usability and depth of experimentation.

Friends & Fables

Friends & Fables is a D&D inspired AI DM game. The service provides limited number of tokens (the currency used for generating responses from AI) per day on a free account. The website offers a fairly expansive system that have already ported all of the D&D 5E items/rules/spells/class/etc. into the UI so the "D&D" session can get up and running in a few clicks. It also has built in tools that allow the player to design their own custom class & spells, provided it follows the basic D&D mechanics (hit roll, attack roll, spell slots, etc.). So, some thoughts:

The biggest problem I have with F&F is it's inability to actually deal with the many nuances of 5E game mechanics (the core materials are split across three 400-paged books!) and it has no way to handle "custom" mechanics to deal with the fact that currently F&F has a really really bare-bones understanding of how 5E works. A huge oversight is how the combat system (which is run by AI but calculated by script) has no room for support/utility skills, which easily takes up half of the D&D spells/skills and probably the reason a gamer would choose TTRPG over a video game.
After exhausting the 50 credits I've been given at the beginning (F&F only offers a few credits a day on the free tier after the initial 50 trial offering), I found that the things I can do are seriously limited. 50 credits is not enough to get one through a reasonably sized combat session, let alone even a one-shot campaign.
Combat is bugged AF. As mentioned in the first bullet point, the game has no real way of handling any support skills. The player can only target themself with healing spells, with the numbers being all wonky (seems like the AI just pulls a number out of thin air without consulting the rules at all). The combat UI also does not allow multiple targets (despite AoE spells) nor track damage over time nor splash damage nor...well, you get the idea. So, given that this is D&D and combat is like 70-90% of the game, this is...no bueno.
Ignoring that this is supposed to be a "D&D" game, the UI is actually quite slick and pretty well designed if one thinks of it as something that's derived from the D&D core system but actually is an original game (like Pathfinder or ICRPG). Pare down the complexity of D&D combat to "hit thing to make HP bar go down" and ignore all the utility spells, or treat utility spells as RP-only flavor that can do whatever the player wants, then it is effectively a reasonably slick semi-structured free-form RP platform.
Of the "pretending to be a GM" systems, I would say F&F has the AI sounding the most...like a GM. The ideas are still generic, no doubt, but it is able to generate far, far more complex responses and keep the world's lore straight. I think this is thanks to the structure of how F&F feeds the information to the AI, so that the AI can keep track of multiple characters, locations, and the general quest line. It helps that quest progression goals are pre-defined by a human rather than generated by the AI based on previous story content.
The ability to separate IC and OOC is really helpful. This is a system that every AI DM game needs to figure out because giving AI OOC instructions to get to the next scene or skip fluff dialogue is really key to keeping the story going, as AI usually get hung up on a single detail and refuse to let go, resulting in endless loops around the same topic.

In conclusion, F&F's AI GM is pretty good at keeping track of world lore, but terrible at understanding D&D mechanics beyond "hit to make HP go down". Also number of turns per day is too limited, so the player can't really make any meaningful progression on the free tier.

Perchance AI RPG (and the AI Text Adventure)

Perchance have two versions of the AI GM. There's AI RPG, which is completely free-form, and AI Text Adventure, which generates 3 options for you to pick from at the end of each turn. Otherwise, they function about the same.

The thing that sticks out about Perchance's project is that it generates a lot of text and the entire log is saved locally in the browser's cache. This is...good? I think? Point is, if one has a really nice little story going on, one can keep a draft easily.
The problem with Perchance's AI is that the story is written in such a way that it'll go for many, many paragraphs before asking the player for input. That means the story often goes way, way off rails before the player can intervene. Often I felt like I'm more along for the ride and giving writing direction rather than role-playing.
The biggest drawback is that there's no way for the player to design anything original with Perchance. The prompt box gives about 500 words in which to explain the general outline of what kind of RP the player wants (name, class, quick backstory, starting quest), but the player can't instruct the AI on the exact nature of the world lore. If someone wants to RP in an already published world, like Strixhaven or Naruto, the AI can generate some reasonable output. But if, like me, you're looking for a custom world where demons are actually nerdy scholars, angels talk in song, dragons are politicians, and sentient moss runs the bakery, well...you're kind of out of luck.
The AI is really verbose. Like to the point of word vomit. If the player wanted the AI to just get on with the action already, it's going to be an uphill battle. However, the AI is very willing to do all kinds of RP, including NSFW and manages to remember BL is male-male romance, which is apparently an achievement given the uh...weird places I've been taken elsewhere (*cough*AIDungeon*cough*).
Perchance's projects do have the advantage of being infinitely free to use. No need to sign up, no limits on token generation, just keep going forever, assuming one isn't bored to tears by the tenth turn or so. I've found that the speed of text generation slows to an absolute crawl around turn 10-ish and the story line starts going in circles, never progressing further. Granted, I was also trying to see what types of content the AI is capable of responding to, so I wasn't trying to push it nearly as hard to stay on track with the plot. Overall, I found the experience the most meh out of the three.

So, Perchance plays mostly like a randomly generated choose-your-own-adventure style RP. But it really fails at generating the back-and-forth experience where the player describes how the PC reacts and then the GM tells the player how the world reacts. Overall, Perchance feels more like an automated creative writing bot with the player designing the Main Character, but the player doesn't really have any fine-tuned control over each and every little interaction as one would in an actual RP game.

AI Dungeon

Full disclosure, I played AI Dungeon the most extensively out of the three. The free version is usable if one is willing to battle the wonkiness generated by the limited 2000 token context (~1500 words or a little over 3 pages of narrative). However, the major problems encountered in the free version (repetition) does not get fixed by using premium services based on the feedback I was reading.

AI Dungeon is a good balance between generous free offering against the amount of customization one is able to do. It can do completely free-form response (like Perchance AI RPG), but instead of spitting back 5 very long paragraphs, the AI responds with only one or two paragraphs. This way, the player can quickly course correct when the AI (inevitably) starts getting repetitive.
The ability to set up story cards (a keyword based trigger system that injects relevant information into the AI context to remind the AI of what you're talking about rather than letting the AI guess) allows for very extensive world building. Combined with custom AI instructions, author's notes, and plot essentials (which are all ways to give AI the appropriate context for what the story is about), the AI can be wrangled into staying on topic. Using the "story" mode entry allows the player to forcibly move the story on when the AI gets stuck on a topic and refuses to let go.
The memory system (an AI summary of past plot points) is...bad. AI thinks more text means the subject is important, which means AI has no grasp of foreshadowing, misdirection, red herring, subversion, implication, or even lying. Which means every turn or so, I'm dipping into the memory database to re-write the summary for the AI so it is correctly interpreting what is happening. This...really slows down the speed of role-playing a lot.
AI Dungeon is compatible with scripting, so if one is inclined, one can write some scripts to re-contextualize input and reformat the memory...and effectively go as far as coding dice rollers and stat trackers and rudimentary mini-games into the creative writing the AI is spitting back out. I've tried a few of these scripts, but the battle of remembering the text commands to trigger the script makes it feels more like a poorly coded text-based MUD and at that point...well, MUDs run so much faster and better and...not really the kind of experience I was looking to get.
While tedious, story card and memory management has given me enough control over the world lore that I can essentially have fun pretending I'm a slime running around in a fantasy world matchmaking poor unwitting souls. Granted, the AI has a poor understanding of how non-government organizations are structured and what the difference between an NGO and government is, but like...I guess I can forgive that given I'm running around in a world that's completely made up and not following typical fantasy tropes.
My recent experimentation project is "running" D&D by doing the combat stuff manually (with dice and everything!) then input the results into AI Dungeon to see how the AI will narrate the combat. This has been far more successful than trying to fight scripts or webUI that don't understand the intricacies of D&D 5E rules. With my brain occupied by the fiddly bits of 5E combat, I'm not paying nearly as much attention to the constant repetition.
Like all AI I've tried, as soon as the story hits the 1000-word mark, the repetition starts kicking in. There's only ever so many descriptors the AI seems to cycle through ("suspicious hooded figure", "black cloak", "places his hand on the hilt", "we have to proceed carefully", "it might be dangerous" are all really common phrases). Sadly, at this point, I can't tell if this is an AI fault or if the fantasy genre is just really repetitive. I suspect it's a little of both.
AI Dungeon also likes to go into NSFW territory really fast. Look at a stranger for too long and the AI will be convinced this is true love and throw the PC into a wall to get ravished. (And yes, the word "ravish" also appears way, way too much.) So far the only way to stop this from happening is a) play as a dude or b) play as a slime.

So, AI Dungeon is definitely the most highly customizable system with the most amount of player control, at the cost of having very high repetition problems and an overall kind of clunky user experience. However, with infinite turns a day using an AI bot for the price of signing up for an account, I would say it's pretty reasonable.

For free-form RP, AI Dungeon requires immense amounts of hand holding. For a pseudo-D&D experience, running the combat mechanics off-site then interpreting the combat results into natural language (aka none of this "I do 3d4 damage"), it sort of works. Clunky for sure, but more authentically D&D than trying to coax a pre-made UI to recognize the interaction between feats, always-prepared spells, free casting, and advantage-disadvantage rolls. And for those who want a really hard-coded limitation...well, scripts exists.

So...that's what I've been doing. Now back to playing as a helpless slime.