Skip to content

Commit 0d1b1c9

Browse files
Merge pull request #9658 from liamsommer-mx/toolchoice
Toolchoice & OpenAI Document Chat
2 parents 1c9b20b + a616542 commit 0d1b1c9

File tree

5 files changed

+54
-9
lines changed

5 files changed

+54
-9
lines changed

content/en/docs/appstore/use-content/platform-supported-content/modules/aws/amazon-bedrock.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,17 @@ Some capabilities of the chat completions operations are currently only availabl
228228

229229
* **Document Chat** - This operation supports the ability to chat with documents for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). To send a document to the model add a `FileCollection` with a `System.FileDocument` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/genai-for-mx/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/genai-for-mx/commons/#add-file-to-collection) operation. For Document Chat, it is not supported to create a `FileContent` from an URL using the above mentioned operations; Please use the `System.FileDocument` option. Make sure to set the `FileType` attribute to **document**.
230230

231+
##### Tool Choice
232+
233+
All [tool choice types](/appstore/modules/genai/genai-for-mx/commons/#enum-toolchoice) of GenAI Commons for the [Tools: Set Tool Choice](/appstore/modules/genai/genai-for-mx/commons/#set-toolchoice) action are supported. For API mapping reference, see the table below:
234+
235+
| GenAI Commons (Mendix) | Amazon Bedrock |
236+
| --- | --- |
237+
| Auto | Auto |
238+
| Any | Any |
239+
| None | Tools removed from request |
240+
| Tool | Tool |
241+
231242
#### RetrieveAndGenerate {#retrieve-and-generate}
232243

233244
The `Retrieve and Generate` activity can be used for conversations leveraging Retrieval Augmented Generation through a knowledge base. This operation corresponds to the *RetrieveAndGenerate* microflow.

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/_index.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -71,8 +71,7 @@ Mendix connectors offer direct support for the following models:
7171
|--------------|---------------------|---------------------|-------------------|-----------|-------------------------|
7272
| Mendix Cloud GenAI | Anthropic Claude 3.5 Sonnet | Chat Completions | text, image, document | text | Function calling |
7373
| | Cohere Embed English, Cohere Embed Multilingual | Embeddings | text | embeddings | |
74-
| Azure / OpenAI | gpt-3.5-turbo | Chat completions | text | text | Function calling |
75-
| | gpt-4, gpt-4-turbo, gpt-4o, gpt-4o mini, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4.5-preview | Chat completions | text, image | text | Function calling |
74+
| Azure / OpenAI | gpt-4, gpt-4-turbo, gpt-4o, gpt-4o mini, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4.5-preview | Chat completions | text, image, document (OpenAI only) | text | Function calling |
7675
| | DALL·E 2, DALL·E 3, gpt-image-1 | Image generation | text | image | |
7776
| | text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large | Embeddings | text | embeddings| |
7877
| Amazon Bedrock | Amazon Titan Text G1 - Express, Amazon Titan Text G1 - Lite, Amazon Titan Text G1 - Premier | Chat Completions | text, document (except Titan Premier) | text | |
@@ -104,4 +103,4 @@ In addition to the models listed above, you can also connect to other models by
104103

105104
* To connect to other [foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-features.html) and implement them in your app, use the [Amazon Bedrock connector](/appstore/modules/aws/amazon-bedrock/).
106105
* To connect to [Snowflake Cortex LLM](https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex) functions, [configure the Snowflake REST SQL connector for Snowflake Cortex Analyst](/appstore/connectors/snowflake/snowflake-rest-sql/#cortex-analyst).
107-
* To implement your connector compatible with the other components, use the [GenAI Commons](/appstore/modules/genai/commons/) interface.
106+
* To implement your connector compatible with the other components, use the [GenAI Commons](/appstore/modules/genai/commons/) interface and follow the how-to [Build Your Own GenAI Connector](/appstore/modules/genai/how-to/byo-connector/).

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/mendix-cloud-genai/Mx GenAI Connector.md

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -161,9 +161,7 @@ For [Chat Completions (without history)](/appstore/modules/genai/genai-for-mx/co
161161
In the entire conversation, you can pass up to five documents that are smaller than 4.5 MB each. The following file types are accepted: PDF, CSV, DOC, DOCX, XLS, XLSX, HTML, TXT, and MD.
162162

163163
{{% alert color="info" %}}
164-
When adding a document to the `FileCollection`, you can optionally use the `TextContent` parameter to pass the file name. Ensure the file name excludes its extension before passing it to the file collection.
165-
166-
Note that the model uses the file name when analyzing documents, which could make it vulnerable to prompt injection. Depending on your use case, you may choose to modify the string or not pass it at all.
164+
The model uses the file name when analyzing documents, which may introduce a potential vulnerability to prompt injection. To reduce this risk, consider modifying file names before including them in the request.
167165
{{% /alert %}}
168166

169167
### Knowledge Base Operations
@@ -254,6 +252,17 @@ The **Documentation** pane displays the documentation for the currently selected
254252

255253
{{< figure src="/attachments/appstore/platform-supported-content/modules/technical-reference/doc-pane.png" >}}
256254

255+
### Tool Choice
256+
257+
All [tool choice types](/appstore/modules/genai/genai-for-mx/commons/#enum-toolchoice) of GenAI Commons for the [Tools: Set Tool Choice](/appstore/modules/genai/genai-for-mx/commons/#set-toolchoice) action are supported. For API mapping reference, see the table below:
258+
259+
| GenAI Commons (Mendix) | Amazon Bedrock |
260+
| -----------------------| ----------------------------- |
261+
| auto | auto |
262+
| any | any |
263+
| none | tools removed from request |
264+
| tool | tool |
265+
257266
## Implementing GenAI with the Showcase App
258267

259268
For more guidance on how to use microflows in your logic, Mendix recommends downloading the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of example use cases and applies almost all of the Mendix Cloud GenAI operations. The starter apps in the [Mendix Components](/appstore/modules/genai/#mendix-components) list can also be used as inspiration or simply adapted for a specific use case.

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/reference-guide/external-platforms/openai.md

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,20 @@ When you use Azure OpenAI, it is recommended to set the optional `MaxTokens` inp
219219

220220
For more information on vision, see [OpenAI](https://platform.openai.com/docs/guides/vision) and [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision) documentation.
221221

222+
#### Document Chat {#chatcompletions-document}
223+
224+
Document chat enables the model to interpret and analyze PDF documents, allowing it to answer questions and perform tasks based on the document content. To use document chat, you can send an optional [FileCollection](/appstore/modules/genai/genai-for-mx/commons/#filecollection) containing one or more documents along with a single message.
225+
226+
For [Chat Completions (without history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-without-history), `OptionalFileCollection` is an optional input parameter. For [Chat completions (with history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-with-history), a `FileCollection` can optionally be added to individual user messages using [Add Message to Request](/appstore/modules/genai/genai-for-mx/commons/#chat-add-message-to-request).
227+
228+
You can send up to 100 pages across multiple files, with a maximum combined size of 32 MB per conversation. Currently, processing multiple files with OpenAI is not always guaranteed and can lead to unexpected behavior (for example, only one file being processed).
229+
230+
{{% alert color="info" %}}
231+
Azure OpenAI does not currently support file input.
232+
233+
Note that the model uses the file name when analyzing documents, which may introduce a potential vulnerability to prompt injection. To reduce this risk, consider modifying the string or not passing it at all.
234+
{{% /alert %}}
235+
222236
#### Image Generations {#image-generations-configuration}
223237

224238
OpenAI also provides image generation capabilities which can be invoked using this connector module. The `OpenAIDeployedModel` entity is compatible with the [image generation operation from GenAI Commons](/appstore/modules/genai/genai-for-mx/commons/#generate-image).
@@ -257,11 +271,11 @@ OpenAI-specific exposed microflow actions to construct requests via drag-and-dro
257271

258272
This microflow changes the `ResponseFormat` of the `OpenAIRequest_Extension` object, which will be created for a `Request` if not present. This describes the format that the chat completions model must output. The default behavior for OpenAI's models currently is `Text`. This operation must be used to enable JSON mode by providing the value `JSONObject` as input.
259273

260-
#### Files: Initialize Collection with OpenAI File {#initialize-filecollection}
274+
#### Files: Initialize Collection with OpenAI Image {#initialize-filecollection}
261275

262276
This microflow initializes a new `FileCollection` and adds a new `FileDocument` or URL. Optionally, the `Image Detail` or a description using `TextContent` can be passed.
263277

264-
#### Files: Add OpenAI File to Collection {#add-file}
278+
#### Files: Add OpenAI Image to Collection {#add-file}
265279

266280
This microflow adds a new `FileDocument` or URL to an existing `FileCollection`. Optionally, the `Image Detail` or a description using `TextContent` can be passed.
267281

@@ -280,6 +294,17 @@ The **Documentation** pane displays the documentation for the currently selected
280294

281295
{{< figure src="/attachments/appstore/platform-supported-content/modules/technical-reference/doc-pane.png" >}}
282296

297+
### Tool Choice
298+
299+
All [tool choice types](/appstore/modules/genai/genai-for-mx/commons/#enum-toolchoice) of GenAI Commons for the [Tools: Set Tool Choice](/appstore/modules/genai/genai-for-mx/commons/#set-toolchoice) action are supported. For API mapping reference, see the table below:
300+
301+
| GenAI Commons (Mendix) | OpenAI |
302+
| -----------------------| ------- |
303+
| auto | auto |
304+
| any | required|
305+
| none | none |
306+
| tool | tool |
307+
283308
## GenAI showcase Application {#showcase-application}
284309

285310
For more inspiration or guidance on how to use those microflows in your logic, Mendix recommends downloading the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of example use cases.

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/reference-guide/genai-commons.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,8 +141,9 @@ This is a file in a collection of files that belongs to a message. Each instance
141141
| `FileContent` | Depending on the `ContentType`, this is either a URL or the base64-encoded file data. |
142142
| `ContentType` | This describes the type of file data. Supported content types are either URL or base64-encoded file data. For more information, see the [ENUM_ContentType](#enum-contenttype) section.
143143
| `FileType` | Currently only images and documents are supported file types. In general, not all file types might be supported by all AI providers or models. For more information, see the [ENUM_FileType](#enum-filetype).
144-
| `TextContent` | An optional text content describing the file content or giving it a specific name. This can be used to refer to specific files in the prompt of the message. |
144+
| `TextContent` | An optional text content describing the file content. |
145145
| `FileExtension` | Extension of the file, e.g. *png* or *pdf*. Note that this attribute may only be filled if the ContentType equals *Base64* and can be empty. |
146+
| `FileName` | If a FileDocument is added, the `Filename` is extracted automatically. |
146147

147148
#### `ToolCollection` {#toolcollection}
148149

0 commit comments

Comments
 (0)