Assistant: Basic Anthropic prompt caching

[seeM]

This PR uses Anthropic's prompt caching API to reduce costs and alleviate organization rate limit pressure – particularly in Databot.

By default, we add a cache control point after the system prompt. Callers (e.g. Databot) can disable that if needed.

This PR also adds more Assistant logging:

• The entire Anthropic API request/response at trace level, and summaries (including token usage) at debug level.
• Elements added to the user context message.

Release Notes

New Features

• Assistant caches prompts for Anthropic models (Assistant: Prompt caching #8077).

Bug Fixes

• N/A

QA Notes

Try out Assistant and Databot and check the cache read/write debug logs in the "Assistant" output channel.

[github-actions]

E2E Tests 🚀
This PR will run tests tagged with: @:critical

^{readme  valid tags}

[seeM]

I seem to have broken the context stuff a bit, will fix.

EDIT: Fixed.

[wch]

If I'm understanding the code right, I think that the cache point on the last user message will not actually be read from, for a couple of reasons:

• Suppose for the user message 1, it writes to the cache. Then when Positron sends the next request (with user message 1, assistant message 2, and user message 2), the cache point is only set on user message 2. But I believe that for this request there must be cache point on user message 1, if you want to restore from the cache. So I think it makes sense to put cache points on the two most recent user messages: The cache point on the most recent message will cause

• The cache point is put on the last part of the last user message. However, when the request is made, the structure of the messages is roughly this (relevant code here):

  • User and Assistant messages from context.history. This includes only the content shown in the UI.
  • User message: Positron context like whether the active session is R or Python, and the session history.
  • User message: The user's text entered by the user in the chat box.

  Note that the Positron context is ephemeral -- it is not persisted across requests. So this means that if you put the cache breakpoint on the last message, there will never be another request that has that content as the prefix, and so there will never be a cache hit. In order for there to be a cache hit, I think you would have to swap the order of the last two messages: the user's actual text would have to come before the Positron context, and the cache breakpoint would have to be on the message with the user's text. And even then, you'd have to be sure that the exact same text available in the next turn when you get it out of context.history -- I could imagine that there might be some small changes, like to whitespace, when the previous text is pulled from context.history.
  Also, I am not 100% sure about this, but I think that if a message is sent with content: [{ type: "text", text: "Hello", cache_control: {type: "ephemeral"}}], as opposed to content: "Hello", then it should always be sent with the more complex JSON structure no matter where it appears in the history. Probably safest to always convert it.

I don't think it's necessary to cache the tools separately from the system prompt. I think it makes sense to just put a cache point at the end of the system prompt.

I believe that the cached content always contains the entire request up the breakpoint. Even if there are multiple breakpoints, each one will store in the cache everything from the beginning to that breakpoint, not just the content between breakpoints. The Anthropic documentation isn't super clear about this, so I asked Claude about this and pointed it to the docs, and it says the same thing: https://claude.ai/share/a85ae7e7-c3df-46be-9308-9a35a4a96705

Finally, for chat participants that follow the VS Code API, they can't change the real system prompt. Instead, they are supposed to insert a first user message with the same content that would go in a system prompt. It would be nice if the extension author could designate that message as being a cache breakpoint -- without breaking compatibility with the VS Code API. I understand that it may be outside the scope of this PR, but I just want to mention that.

[seeM]

Thanks for the detailed feedback @wch!

1. Oops, you're right re user message caching, I've removed that for now. Given the complexity required there, I agree that may be best handled by updating the extension API and letting callers define cache controls directly.
2. You're right that a cache control is not needed in the last tool, I've removed that too.

At this point, this PR only adds a cache control point after the system prompt, which should be a big win for Databot users, as you initially suggested. I'd like to get that merged ASAP in time for RC. Will sync up with you about further improvements.

[jmcphers]

LGTM!

2025-06-25 10:01:41.634 [debug] [anthropic] Adding cache control point to system prompt
2025-06-25 10:01:45.975 [debug] [anthropic] SEND messages.stream [req_011CQVJGyw7CudpFoPpSQXEg]: model: claude-sonnet-4-20250514; cache options: default; tools: executeCode, getAttachedPythonPackages, getAttachedRPackages, getInstalledPythonPackageVersion, getInstalledRPackageVersion, getPlot, getProjectTree, inspectVariables, notebook_install_packages, notebook_list_packages, positron_editFile_internal, positron_findTextInProject_internal, positron_getFileContents_internal, vscode_fetchWebPage_internal, vscode_searchExtensions_internal; tool choice: default; system chars: 26495; user messages: 2; user message characters: 6724; assistant messages: 0; assistant message characters: 2