-
Notifications
You must be signed in to change notification settings - Fork 50
optimized dsv3.2 missing underscore #847
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: ganyi/optimized_dsv3.2
Are you sure you want to change the base?
optimized dsv3.2 missing underscore #847
Conversation
Signed-off-by: ganyi <[email protected]>
Signed-off-by: ganyi <[email protected]>
Signed-off-by: ganyi <[email protected]>
Signed-off-by: ganyi <[email protected]>
Signed-off-by: ganyi <[email protected]>
Signed-off-by: ganyi <[email protected]>
Signed-off-by: ganyi <[email protected]>
Signed-off-by: ganyi <[email protected]>
ROCMAiterMLASparseMetadataBuilder sets cudagraph_support, but the actual checked name is _cudagraph_support. This causes the default value AttentionCGSupport.NEVER to be used instead of the intended AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE Signed-off-by: Stig-Arne Grönroos <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
|
It's critical that DS3.2 has full graphs on decode. Some perf numbers: Single decode step went down from 59ms to 29ms. This is pretty much 100% speedup in this particular case on this particular node (MI300X) with a slow CPU . There are some additional warnings in the logs so the real PR may need to retest and change things now that the graphs actually work, but they simply must work before it goes in as it's so critical for performance. |
e68d9a5 to
85ef10d
Compare
85ef10d to
5d6bf0f
Compare
ROCMAiterMLASparseMetadataBuildersetscudagraph_support, but the actual checked name is_cudagraph_support.This causes the default value
AttentionCGSupport.NEVERto be used instead of the intendedAttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODEPurpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.