| Components | All | New | MacOS | Windows | Linux | iOS | ||||
| Examples | Mac & Win | Server | Client | Guides | Statistic | FMM | Blog | Deprecated | Old | |
Llama.Tokenize
Convert the provided text into tokens.
| Component | Version | macOS | Windows | Linux | Server | iOS SDK |
| Llama | 16.1 | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
MBS( "Llama.Tokenize"; LlamaSession; Text; add special; parse special ) More
Parameters
| Parameter | Description | Example |
|---|---|---|
| LlamaSession | The session identifier. | $$session |
| Text | The text to process. | "Hello World" |
| add special | Allow to add BOS and EOS tokens if model is configured to do so. | 1 |
| parse special | Allow tokenizing special and/or control tokens which otherwise are not exposed and treated as plaintext. Does not insert a leading space. | 1 |
Result
Returns OK or error.
Description
Convert the provided text into tokens.Examples
Tokenize some text:
Let ( [
tokens = MBS("Llama.Tokenize"; $$session; "Hello World.");
text = MBS("Llama.Detokenize"; $$session; tokens)
]; tokens & ¶ & text )
Example result:
[2, 9259, 4109, 236761]
Hello World.
tokens = MBS("Llama.Tokenize"; $$session; "Hello World.");
text = MBS("Llama.Detokenize"; $$session; tokens)
]; tokens & ¶ & text )
Example result:
[2, 9259, 4109, 236761]
Hello World.
Tokenize with and without the BOS token:
Let ( [
r1 = MBS("Llama.Tokenize"; $$session; "Hello World."; 0; 0);
r2 = MBS("Llama.Tokenize"; $$session; "Hello World."; 1; 1)
]; r1 & ¶ & r2 )
Example result:
[9259, 4109, 236761] [2, 9259, 4109, 236761]
r1 = MBS("Llama.Tokenize"; $$session; "Hello World."; 0; 0);
r2 = MBS("Llama.Tokenize"; $$session; "Hello World."; 1; 1)
]; r1 & ¶ & r2 )
Example result:
[9259, 4109, 236761] [2, 9259, 4109, 236761]
See also
Created 10th February 2026, last changed 10th February 2026
