In this post on AniFiguLa I would like to record how I set up appropriate programs locally on my computer to use an AI language model to imitate a chat with my favorite anime character and show you how you can replicate it. In the post we will go into the choices we have for this, the installation, showing you how to add the first character and show you how to connect to Automatic1111 SD Web UI.
Selection options and decision
There are various graphical user interfaces (GUI) for text generation user interfaces (GUI):
I have dealt with all three and finally decided on SillyTavern, as it is the most intuitive interface and is regularly updated by with the latest features by the community.
Next, you have to ask yourself whether you want to generate the texts locally or whether you prefer to run it online via a provider such as OpenAI or OpenRouter.
Necessary considerations:
- How well equipped is your home computer?
- Somewhat reasonable models start at 13 billion parameters (13B). However, this also means that 12 GB of VRAM are due. You would not be able to load the very largest models with a single consumer graphics card (see Nvidia GeForce RTX 4090 mit 24 GB RAM) as even 24 GB RAM can be exhausted.
- How often do you generate texts?
- The power consumption is in addition to the very expensive hardware. The graphics card uses its VRAM in the models up to the end and if this is not sufficient, there is also the option of using the system memory (RAM). All this also means that the RAM (and therefore the CPU) and graphics card are under load and consume power. This is not insignificant and could come close to or even exceed the consumption during gaming. Unfortunately, I don't have any exact measured values.
- How high must the degree of "common sense" be that the
AI has?
- 13B models or models with even fewer parameters will usually formulate very clumsy and unintelligent answers. You should probably not discuss the existence of the universe or ask them for helpful tips.
- Since GPT3.5 and GPT4 run exclusively at OpenAI and are therefore are not available as a local model and you would probably need a whole server cabinet with hardware to run these models locally, the only option for GPT4 is to go to a provider. However, GPT4 is currently also the model that can generate the best answers.
- How much money do you want to spend?
- If you already have suitably powerful hardware, you can of course try to run the local models on your hardware. But keep the power consumption in mind if you are generating texts over longer periods of time.
- The use of GPT4 for approx. 40 messages already costs 2,50$. This is of course a variable value, as it depends on how many tokens you have calculated per message. But you can imagine the order of magnitude. For GPT3.5 it will be cheaper, as the model cannot offer quite as much quality.
Personally, after a few attempts with local models, I've settled on my Nvidia GeForce RTX 3080with 10 GB VRAM. I switched to using the API access from OpenAI . The VRAM with 10 GB RAM allows to run only a few models functionally and then unfortunately mostly only 6B or 7B models. In comparison, the responses of GPT3.5 and GPT4 are also better across the board than those of local models.
When using OpenRouter there were limitations and some requests could not be processed due to policies. When using the OpenAI API, these did not exist.
Installation of SillyTavern
First of all, for the latest installation instructions, take a look at the official docs.
In summary, all you have to do is install NodeJS, clone the Github repository and run Start.bat. Alternatively you can of course run SillyTavern in Docker Container.
Then open the URL http://127.0.0.1:8000/ in your browser and the SillyTavern interface should welcome you:
Get your favorite AI anime character
Of course you can also define a character yourself. But since this is not quite trivial and involves a lot of effort I recommend that you search the following sources:
- chub.ai (Characterhub)
- botprompts.net
You can of course also edit the definitions there after downloading and adapt them according to your wishes. The definitions provide a good starting point and you don't have to start from scratch.
For the JSON format of botprompts.net you can use the character editor from https://zoltanai.github.io/character-editor/ . There you can import the JSON, find a suitable image for the character and then save it under "Export" as a "Character Card". This can then be transferred 1:1 to Silly Tavern.
As an example, I'll use Holo my favorite character from the anime and light novel Spice and Wolf, to start an AI chat.
After successful import, your character will automatically appear in the list of characters. Select your character from there.
If you are using a provider, you must first generate an API key which you can then enter in SillyTavern. Without this API key it is not possible to use paid language models such as GPT3.5 and GPT4.
Generate AI images from the chat context with your favorite character
You can connect SillyTavern with Automatic1111’s Stable Diffusion Web UI and generate images from the context created with the character. This way the AI can then generate the face of your favorite anime character and show you in the chat.
Of course you need a working installation of the SD Web
UI. Follow the installation instructions and
then start the SD Web UI with the additional command line argument
"-api" to make it accessible via the network.
If you have not yet installed a model, you can download various models from CivitAI for all possible
styles.
Then there's just one final question to answer: Who is your favorite anime character? Feel free to write it in the comments.
* The links used are affiliate links. By a purchase via the link I will participate in the turnover. This has no effect on no effect on the price for you.