Added support for the "think" for Ollama #3386
Conversation
1. Added the `think` field to Ollama's `ChatRequest`
2. Added the `thinking` field to Ollama's `Message`
3. Added the `think` property to `OllamaOptions`, allowing users to specify whether to enable or disable thinking
Signed-off-by: Sun Yuhan <[email protected]>
Signed-off-by: Sun Yuhan <[email protected]>
…fault behavior, thereby ensuring compatibility with older versions of Ollama calls. Signed-off-by: Sun Yuhan <[email protected]>
|
@tzolov @ilayaperumalg @markpollack Could you please help review this PR? Thank you. |
…tainer image version of ollama to 0.9.0 Signed-off-by: Sun Yuhan <[email protected]>
|
Yes, we will review. Thanks |
| * If this value is not specified, it defaults to null, and Ollama will return | ||
| * the thought process within the `content` field of the response, wrapped in `<thinking>` tags. | ||
| */ | ||
| @JsonProperty("think") |
There was a problem hiding this comment.
It seems like 'think' is not a part of the options map in ollama, but a 'top level' field in the request object.
In https://ollama.com/blog/thinking there is the example
curl http://localhost:11434/api/chat -d '{
"model": "deepseek-r1",
"messages": [
{
"role": "user",
"content": "how many r in the word strawberry?"
}
],
"think": true,
"stream": false
}'
and the golang type supporting this feature also shows the same structure.
https://github.com/ollama/ollama/blob/45f56355d557b7130c7c07bbd6e1b634a758d946/api/types.go#L91
So it shouldn't be added to the options map.
There was a problem hiding this comment.
The comment in the code
and Ollama will return
* the thought process within the `content` field of the response, wrapped in `<thinking>` tags.
seems to contradict what was documented on the ollama web site that shows the 'think' response as a separate field from 'content', and not nested inside the 'content' field.
There was a problem hiding this comment.
Maybe we want to also expose it in OllamaOptions as that right now would be the only way to pass in this feature flag when making calls via ChatModel or ChatClient. I think if the feature of enabling thinking mode is implemented from a ChatModel or ChatClient level the right solution will present itself. Can you improve this PR to handle this scenario please?
| @@ -260,6 +262,7 @@ public Flux<ProgressResponse> pullModel(PullModelRequest pullModelRequest) { | |||
| public record Message( | |||
| @JsonProperty("role") Role role, | |||
| @JsonProperty("content") String content, | |||
| @JsonProperty("thinking") String thinking, | |||
There was a problem hiding this comment.
We should propagate the 'thinking' response back into the ChatGenerationMetadata so that it is accessible in the response when using OllamaChatModel. This would be inside the internalCall method of OllamaChatModel and add add a test for that in OllamaChatModelIT
Fixes #3383
As mentioned in issue: #3383 , Ollama added support for "think" in its latest 0.9.0 version:
https://github.com/ollama/ollama/releases
https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-chat-completion.
This PR implements support for that attribute and includes the following key changes:
thinkfield to Ollama'sChatRequestthinkingfield to Ollama'sMessagethinkproperty toOllamaOptions, allowing users to specify whether to enable or disable thinkingActually, there is currently another issue:
As stated in Ollama's API documentation here, during requests to Ollama, the
messagefield supports sending the model's own reasoning (thoughts) back to it. However,AssistantMessagedoes not currently support transmitting this field, which means the model will not be aware of its previous thoughts.Therefore, perhaps we need to add a specialized
Messageimplementation for Ollama, such asOllamaAssistantMessage. I'm not sure whether this would be considered a significant change.