Tokens

 

¶ Why is there a token limit?

There is a limit to the amount of text MonAI can process in a prompt. That limit is expressed in tokens. 

 

¶ What, exactly, is a token?

A token is a character or sequence of characters that represents a single unit of meaning. A word might equal 1 token, or it might be 2 or more. 
Tree = 1 token
House = 1 token
Treehouse = 2 tokens
Navigate = 2 token
Circumnavigate = 4 tokens

It's hard to predict how many tokens any given prompt will use, even for the token counter. The counter may be off on its estimation.

 

¶ What is the token limit for MonAI?

150 tokens is the current limit.

 

¶ Why do tokens exist?

Stable Diffusion changes words (which it doesn't understand) into tokens (which it does understand), and then into numbers. It generates art based on those numbers.

 

¶ Why do some embeddings have so many tokens?

An embedding is actually an AI model trained from images. Some of them have fewer tokens and some have more, depending on how the embedding was made. MonAI currently has a selection of negative embeddings and more information about them HERE.