Skip to content

Commit db12b95

Browse files
authored
Update API.md
1 parent 0c6e966 commit db12b95

File tree

1 file changed

+78
-3
lines changed

1 file changed

+78
-3
lines changed

API.md

Lines changed: 78 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,11 +42,24 @@ SELECT ai_log_info(1);
4242

4343
**Description:**
4444
Loads a GGUF model from the specified file path with optional comma separated key=value configuration.
45+
If no options are provided the following default value is used: `gpu_layers=99`
46+
47+
The following keys are available:
48+
```
49+
gpu_layers=N (N is the number of layers to store in VRAM)
50+
main_gpu=K (K is the GPU that is used for the entire model when split_mode is 0)
51+
split_mode=N (how to split the model across multiple GPUs, 0 means none, 1 means layer, 2 means rows)
52+
vocab_only=1/0 (only load the vocabulary, no weights)
53+
use_mmap=1/0 (use mmap if possible)
54+
use_mlock=1/0 (force system to keep model in RAM)
55+
check_tensors=1/0 (validate model tensor data)
56+
log_info=1/0 (enable/disable the logging of info)
57+
```
4558

4659
**Example:**
4760

4861
```sql
49-
SELECT llm_model_load('./models/llama.gguf', 'n_predict=512,n_gpu_layers=99');
62+
SELECT llm_model_load('./models/llama.gguf', 'gpu_layers=99');
5063
```
5164

5265
---
@@ -71,8 +84,13 @@ SELECT llm_model_free();
7184
**Returns:** `NULL`
7285

7386
**Description:**
74-
Creates a new inference context with optional with optional comma separated key=value configuration.
75-
If no context is explicitly created, one will be created automatically when needed.
87+
Creates a new inference context with comma separated key=value configuration.
88+
89+
Context must explicitly created before performing any AI operation!
90+
91+
The following keys are available:
92+
```
93+
```
7694

7795
**Example:**
7896

@@ -82,6 +100,63 @@ SELECT llm_context_create('n_ctx=2048');
82100

83101
---
84102

103+
## `llm_context_create_embedding()`
104+
105+
**Returns:** `NULL`
106+
107+
**Description:**
108+
Creates a new inference context specifically set for embedding generation.
109+
110+
It is equivalent to `SELECT llm_context_create('generate_embedding=1,normalize_embedding=1,pooling_type=last');`
111+
112+
Context must explicitly created before performing any AI operation!
113+
114+
**Example:**
115+
116+
```sql
117+
SELECT llm_context_create_embedding();
118+
```
119+
120+
---
121+
122+
## `llm_context_create_chat()`
123+
124+
**Returns:** `NULL`
125+
126+
**Description:**
127+
Creates a new inference context specifically set for chat conversation.
128+
129+
It is equivalent to `SELECT llm_context_create('context_size=4096');`
130+
131+
Context must explicitly created before performing any AI operation!
132+
133+
**Example:**
134+
135+
```sql
136+
SELECT llm_context_create_chat();
137+
```
138+
139+
---
140+
141+
## `llm_context_create_textgen()`
142+
143+
**Returns:** `NULL`
144+
145+
**Description:**
146+
Creates a new inference context specifically set for text generation.
147+
148+
It is equivalent to `SELECT llm_context_create('context_size=4096');`
149+
150+
Context must explicitly created before performing any AI operation!
151+
152+
**Example:**
153+
154+
```sql
155+
SELECT llm_context_create_textgen();
156+
```
157+
158+
---
159+
85160
## `llm_context_free()`
86161

87162
**Returns:** `NULL`

0 commit comments

Comments
 (0)