[Inductor][float8] Support qlinear for float8 in inductor #2565
Conversation
Add fp8 dequant promotion
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2565
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
| return ( | ||
| len(node.all_input_nodes) == 2 | ||
| and node.all_input_nodes[1].target == torch.tensor | ||
| ) |
There was a problem hiding this comment.
will add return False
| for bias in [True, False]: | ||
| self._qlinear_test_helper((torch.randn((2, 4)),), bias=bias) | ||
| for is_fp8 in [True, False]: | ||
| for bias in [True, False]: | ||
| self._qlinear_test_helper( | ||
| (torch.randn((2, 4)),), bias=bias, is_fp8=is_fp8 | ||
| ) |
There was a problem hiding this comment.
It would be better to fp8 stuff in separate tests, i.e., keeping test_qlinear_cpu and adding test_fp8_qlinear_cpu. Same for other tests.
| @@ -1804,13 +1940,166 @@ def test_qlinear_add_int8_mixed_bf16(self, use_relu, is_qat, is_dynamic): | |||
| is_dynamic=is_dynamic, | |||
| ) | |||
|
|
|||
| def _fp8_qlinear_add_test_helper( | |||
There was a problem hiding this comment.
What's the difference between the int8 version and the fp8 version? Can we merge them?
| lambda x, y: x.add_(y), | ||
| lambda x, y: y.add_(x), | ||
| ] | ||
| is_fp8 = True |
There was a problem hiding this comment.
If we are defining a dedicated helper for fp8, is this still needed?
| @parametrize("dtype", [torch.float32, torch.bfloat16]) | ||
| @parametrize("input_dim_exceeds_two", [True, False]) | ||
| @parametrize("check_reuse_input", [True, False]) | ||
| def test_fp8_qlinear( |
There was a problem hiding this comment.
What's the difference between this test case and the test_qlinear_cpu above?
| for is_fp8 in [True, False]: | ||
| for original_pattern_output_dtype in [torch.float32, torch.bfloat16]: | ||
| is_bf16 = original_pattern_output_dtype == torch.bfloat16 | ||
| for x_scale_zp_are_tensors in (False, True): |
There was a problem hiding this comment.
Use itertools.product maybe?
For float8_e4m3fn, support
on inductor.
For FP8, there are following issues
Based on these issues,