Making D&D Character Portraits With ChatGPT

Alright, this one's pretty nerdy, even by my standards. Brace yourselves.

I feel like I should mention up top that I have cautious feelings about the impact of generative AI on the livelihoods and esteem of artists who have traditionally designed and illustrated characters for tabletop games. Also, none of this is endorsed by the companies who make the tools I used. At the professional level, I favor having artists involved, and I hope we find good ways to integrate these tools into their arsenal. However, I happen to be artistically challenged, and I can't visualize characters without some help, so my table and I spent some time trying to get inspiration from ChatGPT.

We ended up with this process that produced pretty cool art that matched our (developing) vision for our characters in an impressive if sometimes inconsistent way.

The tl;dr is this:

1. Start with some visual inspiration, either art found online or AI generated portraits.
2. Use that art as a reference for making a 3D model using Hero Forge (a character builder tool for making D&D minis).
3. Tweak the pose, equipment, and colors of the 3D model in Hero Forge, then have ChatGPT translate those screenshots into high quality character art.

Looking at that list, you may be wondering why I ran the art back though ChatGPT if I started out with an AI generated portrait as inspiration. The main reason is control over details. When having AI generate the image purely from a text description, it has very few constraints which makes it hard to make small tweaks without ending up with a very different style. For example, you might want to change the character's pose or what weapon they're holding, or the color of a specific item. It's possible to do this iteratively by selectively regenerating parts of the image, but this is time consuming and cumbersome, and I think that it tends to drift away from your original style over time. By giving the AI a reference image and asking it to target a certain art style, I found it easier to control the details of the character.

Step 1: Visual Inspiration

• For some characters, we used existing art that we found online as a placeholder character portrait. This is common, especially for NPCs. None of us have a strong sense of what these characters look like, so it's easy to find something acceptable and just move on.
• For characters that had established visual elements or that were important enough for us to develop custom looks for, we used ChatGPT interactively to discuss the characters, and it generated concept art for us using its integration with DALL-E.

This is the starting prompt I used:

Hey can you help me design a character for D&D? I've been playing this character for a while but I don't know what he looks like. I'm going to tell you a bit about him, and then I'd like you to ask me questions to refine how he looks. Finally, I'll ask you to generate some images based on our conversation. Sound good?

I think just saying it was supposed to be a character portrait for D&D did a lot of heavy lifting in terms of establishing the visual style. It implies things like "fantasy", "painterly", "portrait framing", etc.

The results were helpful in three ways:

1. Chatting with ChatGPT about the character's appearance helped us develop them iteratively. For someone who doesn't think visually about characters, it was helpful to have the AI prompt me back with questions about details.
2. Even though the generated art doesn't perfectly capture the tone of the characters or could contain extraneous details that are hard to remove, it looks nice. It's evocative and makes a character sheet/token feel more personal. The goal of the art is to aid in roleplaying and immersion, so it doesn't have to be perfect, it just has to set the tone and remind us of key details about the characters.
3. The faces are detailed and unique. Ok, this one is more subtle, but it's important for the next step. Interactive character builders like the one we used to create 3D miniatures ("minis") of the characters are like videogame character creation screens. There's often a limited number of stock faces and body parts to use as a base, so often characters will look same-y. The tool we used, Hero Forge, has very detailed customization controls that affect the face mesh, but since I'm not a sculptor, I can't use them right without a reference image. That's where the generated art comes in: the faces are pretty unique and give me a target to aim for.

Step 2: Turning It Into 3D Modeled Minis

So the next step was to use the reference images to create models in Hero Forge.

Hero Forge has a set of stock face shapes. They tend to be a bit cartoonish, (probably to make them easier to see when they're on a 1" tall mini), and if you over use the same ones, characters won't look very unique. Humans are very sensitive to the geometry of faces, so even small tweaks can lead to unique looking characters. For that, I delved into the advanced tools in the Hero Forge face editor and tweaked the dimensions of each part of these characters' facial anatomy until they were close to the reference images.

Because I backed one of their Kickstarters, I have access to a Hero Forge feature that lets me make spinny GIFs of the models. Take a look:

And introducing some new characters:

This is Callum, an NPC who became an Oath of Glory Paladin through the tragic loss of three of his friends in an ill-fated encounter with a fire elemental. We started with art from internet for him since prior to the encounter with the elemental he was a minor NPC. As a result, I don't have his source art to share.

...well once our characters (Bud and Gliran) heard about the demise of the rest of the NPCs (Callum's friends), we had to go clean up after them, so we went to fight the elemental. That fight felt so unwinnable that we turned to superstition.

See, the dice give and they take away. They're fickle. But they have one weakness: reverse psychology. If we wanted our characters to live, they'd definitely die, and we had just invested a few hours in developing art for them, so the dice knew we were invested. Not good. To convince them otherwise, we devised a scheme:

• My friend asked me to come up with the class and lineage of his next character. Knowing him, I picked a combo that he'd never pick for himself: a genasi bard. Genasi are half-elemental people, so like fire hair, or water hair, or wind hair, or crystals for hair. (Sounds like lazy writing when I say it out that way, doesn't it?) Turns out he had never even opened the players handbook to the bard page, so mission accomplished.
• Then, during the fight, every time he had to roll for something important, I'd whip out Hero Forge and start working on a model for the genasi bard while we loudly proclaimed how much more fun it would be to play that character than the ones we were playing now. Because of course, the only way to convince the dice that we were even more serious about this character than the ones we just made art for was to pretend to make and buy a physical mini, the next level of commitment.

By making the 3D models, we could control the details of their faces, items, colors, and motifs. Working in Hero Forge can feel unconstrained in a different way, where without reference art, it's easy to make unnatural looking faces by playing with all the sliders. Starting with some generated art fixes that. On the other hand, it's a lot easier to experiment in Hero Forge when customizing the way the character feels because you get immediate visual feedback when adding items, changing body proportions, tweaking their stance, and fiddling with colors.

Step 3: Pose The Models and Regenerate the Art

I tried to constrain the type of image that got generated by starting with a posed image of the mini against an appropriate background.

Let's start with Callum.

Here's the prompt I gave ChatGPT:

I have a 3D rendered character token for a D&D character, but I'd like help generating a more realistic, illustrated portrait of him. His name is Callum and he is an Oath of Glory paladin. He's 18 or 19 years old, and despite his youth, he's become more serious lately due to the loss of some friends.

And this is what it gave me:

And then for our edgy-boy, Bud:

I have a 3D rendered character token for a D&D character, but I'd like help generating a more realistic, illustrated portrait of him. His name is Bud and he's a servant of the goddess of darkness. In this portrait, he's standing in a fiery volcano casting a dark spell. He's about 35 years old but his skin in a pallid white and his face is gaunt and bony, like a skull. His eyes are dark in their sockets and emit a faint aura of night.

The generated image didn't make his eyes as dark as I'd wanted. In the lore, Bud has "Eyes of the Void", and we're not sure if that means he has obsidian eyeballs or black-hole voids. I got ChatGPT to tweak just the eyes by selectively regenerating just that part of the image. Because so much of the image stayed the same, it was very consistent, keeping the details and motifs from the Hero Forge design.

Experimenting With Changing Poses

One of my hopes was that by starting with 3D models that I could modify and re-pose, I could get more consistent portrayals of the same characters in different situations.

I started by editing Callum's model in Hero Forge to depict him at a bar, mourning the loss of his friends. This would be shortly before Bud and Gliran picked him back up.

So I tried to get ChatGPT to generate a new image based on this, keeping in mind the awesome paladin illustration it did earlier:

Can you make me another illustration with the same style based on another source image? This image shows Callum a few months earlier, grieving at a bar after losing his friends in a fight with a fire elemental. He's looking at his injured left hand. Please take care to match the image you just generated as closely as possible in terms of Callum's facial features.

Here are a few of the results it created after several tweaking prompts attempting to get the visual style to match the paladin pic:

These results weren't as impressive as the original ones. The last image in the gallery wasn't bad, but even then it was clearly a different style. It sorta had that fan art look where many different artists draw the same character in different styles, but I wanted to do better.

So I did the only reasonable thing in this age of AI, and I asked ChatGPT to compare the really cool paladin image with the others, and asked it for help describing the difference between the styles. After a lot of back and forth (it was really obsessed with the backgrounds being different), we landed on some descriptors: "photorealistic", "painterly", and "natural lighting".

After some reprompting using those keywords, this is what I got:

That's a lot better. Not quite perfect, but close enough in style and depiction to be useful.

Iterating On the Prompt

I decided to try this regeneration approach on one more character and decided it'd be cool to see what it did with the genasi bard.

From talking to friends and experimenting a bit, these are the elements of the prompt that I wanted to include:

• Add stylistic notes: "photorealistic", "painterly style", "natural lighting"
• Include a name and age, to get it to focus on making a more realistic person.
• Some personality or biographical details, to let it fill in the blanks with tonally appropriate things.

So my friends and I had to come up with some details for this bard on the fly since our characters did in fact survive the encounter with the fire elemental and the genasi bard had not yet seen service.

We named him Kalani, using a d100 table I found on Reddit.

The prompt:

I have a render of a D&D mini and I'd like you to help me generate a portrait based on it. The portrait should be done in a painterly style with photorealistic details and natural lighting. This character's name is Kalani and he is an air genasi bard, riding a griffin or a hippogriff. He's in his early 20's, and either as a consequence of his air genasi heritage or his youthful inexperience, he's pretty prone to seemingly random outbursts. He sometimes hears voices and music in the wind as it passes by. This leads him to sometimes vacillate between saying insightful or helpful things, and then doing more questionable things like egging on bar patrons, dishing hot takes, or making jokes that cut too close to truth. However, given his high charisma and good looks, he often gets a pass.

The results:

Dang. Kalani's looking good on his first try. I legitimately want to see this character happen now.

Conclusion

Yeah, that was fun. I'm convinced that generative AI is a really great tool for inspiration and ideation, especially for folks like me that lack visual art skills. I think that if I had bigger plans for these characters, I'd still want to commission an artist that could work with me to develop them further, both visually and as a character. That said, these characters feel so much more real when I can see them illustrated this way, knowing that they're custom and unique.

I think I'll try this technique again whenever I get a chance to make a new character (hopefully because I join a second campaign and not because Gliran kicks the bucket).