Skip to content

Conversation

@fbzhong
Copy link

@fbzhong fbzhong commented Jan 25, 2026

Fix issue where extra content like images were being filtered out in streaming responses due to empty-check happening before modifier execution.

Key changes:

  • Move ResponseChunkMessageModifier execution before empty message check to allow modifiers to attach extra content (e.g., images) to otherwise empty chunks
  • Update empty message validation to include AssistantGenMultiContent field
  • Refactor JSON parsing from jsoniter to sonic/ast for better performance and type safety
  • Add proper null/type validation for choices array using AST navigation
  • Add test coverage for choices null scenarios in both message modifiers

This ensures that streaming responses can include images and other extra content that modifiers add to messages, even when the main content field is empty.

What type of PR is this?

fix

Check the PR title.

  • This PR title match the format: <type>(optional scope): <description>
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Attach the PR updating the user documentation if the current PR requires user awareness at the usage level. User docs repo

(Optional) Translate the PR title into Chinese.

修复(model/openrouter): 修复 SSE 流式响应中图片等额外内容丢失的问题

(Optional) More detailed description for this PR(en: English/zh: Chinese).

en:
Problem: When using OpenRouter's streaming API, extra content (such as images) added by response modifiers were being incorrectly filtered out. This occurred because the empty-message check ran
before modifiers had a chance to attach their content.

Root Cause: The streaming pipeline checked if messages were empty (Content == "" and ToolCalls == 0) before executing ResponseChunkMessageModifier, causing chunks with modifier-added
content to be discarded.

Solution: Reorder the execution pipeline to run modifiers first, then check for emptiness. Additionally, include AssistantGenMultiContent in the empty-check to properly handle multimodal
content.

Technical Improvements:

  • Migrated from jsoniter to sonic/ast for more robust JSON parsing
  • Added proper null and type validation for the choices array
  • Improved error handling for malformed API responses
  • Added comprehensive test coverage for edge cases

Impact: Users can now receive images and other extra content in streaming responses without data loss.

zh(optional):
问题: 使用 OpenRouter 流式 API 时,ResponseModifier添加的额外内容(如图片)被错误地过滤掉了。这是因为空消息检查在Modifier有机会附加内容之前就执行了。

根本原因: Chunk Stream在执行 ResponseChunkMessageModifier 之前就检查消息是否为空(Content == ""ToolCalls == 0),导致包含Image内容的消息块被丢弃。

解决方案: 调整Stream的执行顺序,先运行Response Modifier,然后再检查是否为空。此外,在空检查中包含AssistantGenMultiContent 以正确处理多模态内容。

技术改进:

  • jsoniter 迁移到 sonic/ast 以实现更稳健的 JSON 解析
  • choices 数组添加适当的空值和类型验证
  • 改进对格式错误 API 响应的错误处理
  • 为边界情况添加全面的测试覆盖

影响: 用户现在可以在流式响应中接收图片和其他额外内容,不会丢失数据。

(Optional) Which issue(s) this PR fixes:

N/A

(optional) The PR that updates user documentation:

N/A

Fix issue where extra content like images were being filtered out in streaming responses
due to empty-check happening before modifier execution.

Key changes:
- Move ResponseChunkMessageModifier execution before empty message check
  to allow modifiers to attach extra content (e.g., images) to otherwise empty chunks
- Update empty message validation to include AssistantGenMultiContent field
- Refactor JSON parsing from jsoniter to sonic/ast for better performance and type safety
- Add proper null/type validation for choices array using AST navigation
- Add test coverage for choices null scenarios in both message modifiers

This ensures that streaming responses can include images and other extra content
that modifiers add to messages, even when the main content field is empty.
@CLAassistant
Copy link

CLAassistant commented Jan 25, 2026

CLA assistant check
All committers have signed the CLA.

@fbzhong
Copy link
Author

fbzhong commented Jan 26, 2026

It seems like I can do nothing on the error of 'EinoExt CI / check-submodule-changes'

RequestError [HttpError]: Resource not accessible by integration
    at /home/runner/work/_actions/peter-evans/create-or-update-comment/v4/dist/index.js:4695:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  status: 403,
  response: {
    url: 'https://api.github.com/repos/cloudwego/eino-ext/issues/666/comments',
    status: 403,

@fbzhong
Copy link
Author

fbzhong commented Jan 30, 2026

@meguminnnnnnnnn can you review this pr?

fbzhong and others added 2 commits January 30, 2026 03:00
…ypes

Remove incorrect validation that only allowed text parts with cache control.
OpenRouter supports image_url and other content types just like OpenAI.
The previous strict check was incorrectly rejecting valid image content.

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants