Using generic implemntation for 16-bit activations and 8 bit weights for Conv2D in Backends #16007
+512
−6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… for linear in backends Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Reviewed By: hsharma35 Differential Revision: D87946776
…for Conv2D in Backends Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Differential Revision: D87993325
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16007
Note: Links to docs will display an error until the docs builds have been completed. ❗ 2 Active SEVsThere are 2 currently active SEVs. If your PR is affected, please view them below:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for 16-bit activations with 8-bit weights (W8A16) in quantized operations for the Cadence HiFi backend. Previously, 16-bit support required both activations and weights to be 16-bit.
- Extends
quantized_linear_out,quantized_linear_per_tensor_out,quantized_conv2d_nchw_out, andquantized_conv2d_nhwc_outto handle W8A16 heterogeneous types by delegating to generic implementations - Adds comprehensive unit tests for the new W8A16 support in linear and conv2d operations
- Updates build configuration to include necessary dependencies for generic implementations
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| backends/cadence/hifi/operators/tests/test_op_quantized_linear_out.cpp | Adds unit test for quantized linear with int16 activations and int8 weights |
| backends/cadence/hifi/operators/tests/test_op_quantized_conv2d_out.cpp | Adds unit tests for quantized conv2d (NCHW/NHWC) with int16 activations and int8 weights |
| backends/cadence/hifi/operators/targets.bzl | Updates build targets to add generic implementation dependencies for W8A16 support |
| backends/cadence/hifi/operators/op_quantized_linear_out.cpp | Adds W8A16 type check and delegates to generic implementation for linear operations |
| backends/cadence/hifi/operators/op_quantized_conv2d_nhwc_out.cpp | Adds W8A16 type check and delegates to generic implementation for NHWC conv2d |
| backends/cadence/hifi/operators/op_quantized_conv2d_nchw_out.cpp | Adds W8A16 type check and delegates to generic implementation for NCHW conv2d |
| backends/cadence/aot/quantizer/quantizer.py | Adds new quantizer class for 16-bit conv activations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+238
to
+239
| } | ||
| else if (out.scalar_type() == executorch::aten::ScalarType::Byte) { |
There was a problem hiding this comment.
[nitpick] The else if should be on the same line as the closing brace according to standard C++ formatting conventions. Change to } else if for consistency with typical C++ style.
Suggested change
| } | |
| else if (out.scalar_type() == executorch::aten::ScalarType::Byte) { | |
| } else if (out.scalar_type() == executorch::aten::ScalarType::Byte) { |
Comment on lines
+296
to
+297
| } | ||
| else if (out.scalar_type() == executorch::aten::ScalarType::Byte) { |
There was a problem hiding this comment.
[nitpick] The else if should be on the same line as the closing brace according to standard C++ formatting conventions. Change to } else if for consistency with typical C++ style.
Suggested change
| } | |
| else if (out.scalar_type() == executorch::aten::ScalarType::Byte) { | |
| } else if (out.scalar_type() == executorch::aten::ScalarType::Byte) { |
RahulC7
added a commit
to RahulC7/executorch
that referenced
this pull request
Nov 28, 2025
…for Conv2D in Backends (pytorch#16007) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Differential Revision: D87993325
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Context
We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions.
Current Behavior
Right now, we're composing two macros together, the
ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16macro:https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25
and the function macro(
quantized_linearchosen for example):https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41
so together, it just becomes a switch statement, calling the
quantized_linearfunction with the correct template parameter.However, note that it assumes that both the input activations and weights are the same dtype, which is not the case.
This Diff
We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests.
Differential Revision: D87993325