Repository navigation
[DAGCombiner] Fix overflow in rotate of rotate fold for non power of 2 types - #239933
robinbishtt wants to merge 4 commits into
Conversation
|
Hello @robinbishtt 👋 Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description. Frequently asked questionsHow do I add reviewers? This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically. You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using What if there are no comments? If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers. Are any special GitHub settings required to contribute to LLVM? We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details. If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse. Thank you, |
|
@llvm/pr-subscribers-backend-x86 @llvm/pr-subscribers-llvm-selectiondag Author: Robin Bisht (robinbishtt) ChangesFixes #239932 The fold in Example: This patch skips the fold when that arithmetic can overflow the shift amount type. Power of 2 widths keep the fold. A version that does the arithmetic in a wider type and keeps the fold is also possible. I picked the smaller change and am happy to switch if reviewers prefer. Testing:
Full diff: https://github.com/llvm/llvm-project/pull/239933.diff 2 Files Affected:
diff --git a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
index 196111ea8e1f7f..78492bcfcc4e0e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
@@ -11146,6 +11146,15 @@ SDValue DAGCombiner::visitRotate(SDNode *N) {
bool C2 = DAG.isConstantIntBuildVectorOrConstantInt(N0.getOperand(1));
if (C1 && C2 && N1.getValueType() == N0.getOperand(1).getValueType()) {
EVT ShiftVT = N1.getValueType();
+ // The amount arithmetic below is done in ShiftVT and can reach
+ // 3 * Bitsize - 2. If that does not fit in ShiftVT the result wraps
+ // around. This is harmless only when Bitsize is a power of 2 (the
+ // wrap-around is then a multiple of Bitsize), so skip the fold otherwise.
+ unsigned AmtBits = ShiftVT.getScalarSizeInBits();
+ APInt MaxAmt = APInt::getMaxValue(AmtBits);
+ bool MayOverflow = !isPowerOf2_32(Bitsize) && MaxAmt.ult(3 * Bitsize - 2);
+ if (MayOverflow || MaxAmt.ult(Bitsize))
+ return SDValue();
bool SameSide = (N->getOpcode() == NextOp);
unsigned CombineOp = SameSide ? ISD::ADD : ISD::SUB;
SDValue BitsizeC = DAG.getConstant(Bitsize, dl, ShiftVT);
diff --git a/llvm/test/CodeGen/X86/rotate-fold-overflow.ll b/llvm/test/CodeGen/X86/rotate-fold-overflow.ll
new file mode 100644
index 00000000000000..a3ed92289fa6f1
--- /dev/null
+++ b/llvm/test/CodeGen/X86/rotate-fold-overflow.ll
@@ -0,0 +1,61 @@
+; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s
+; Nested rotates on a non-power-of-2 integer must not be folded using an amount
+; that overflowed the shift amount type (i8 on X86).
+
+define i193 @rot_of_rot_i193(i193 %a) {
+; CHECK-LABEL: rot_of_rot_i193:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: andl $1, %r8d
+; CHECK-NEXT: leal (%r8,%rsi,2), %edi
+; CHECK-NEXT: shldq $56, %rcx, %rdi
+; CHECK-NEXT: movq %rsi, %r8
+; CHECK-NEXT: shrdq $8, %rdx, %r8
+; CHECK-NEXT: shldq $56, %rdx, %rcx
+; CHECK-NEXT: shrl $7, %esi
+; CHECK-NEXT: movq %rcx, 8(%rax)
+; CHECK-NEXT: movq %r8, (%rax)
+; CHECK-NEXT: andl $1, %esi
+; CHECK-NEXT: movb %sil, 24(%rax)
+; CHECK-NEXT: movq %rdi, 16(%rax)
+; CHECK-NEXT: retq
+ %s1 = shl i193 %a, 1
+ %r1 = lshr i193 %a, 192
+ %rot1 = or i193 %s1, %r1
+ %s2 = shl i193 %rot1, 184
+ %r2 = lshr i193 %rot1, 9
+ %res = or i193 %s2, %r2
+ ret i193 %res
+}
+
+define i100 @rot_of_rot_i100(i100 %a) {
+; CHECK-LABEL: rot_of_rot_i100:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq %rsi, %rax
+; CHECK-NEXT: movq %rdi, %rdx
+; CHECK-NEXT: shrdq $1, %rsi, %rdx
+; CHECK-NEXT: movl %eax, %ecx
+; CHECK-NEXT: shrl %ecx
+; CHECK-NEXT: shldq $55, %rdx, %rcx
+; CHECK-NEXT: shlq $26, %rdi
+; CHECK-NEXT: shrq $10, %rax
+; CHECK-NEXT: movl %eax, %edx
+; CHECK-NEXT: andl $67107840, %edx # imm = 0x3FFFC00
+; CHECK-NEXT: orq %rdi, %rdx
+; CHECK-NEXT: movq %rax, %rsi
+; CHECK-NEXT: shldq $63, %rcx, %rsi
+; CHECK-NEXT: shrl %eax
+; CHECK-NEXT: shldq $55, %rsi, %rax
+; CHECK-NEXT: shlq $26, %rcx
+; CHECK-NEXT: shrq $10, %rdx
+; CHECK-NEXT: andl $67108863, %edx # imm = 0x3FFFFFF
+; CHECK-NEXT: orq %rcx, %rdx
+; CHECK-NEXT: retq
+ %s1 = shl i100 %a, 90
+ %r1 = lshr i100 %a, 10
+ %rot1 = or i100 %s1, %r1
+ %s2 = shl i100 %rot1, 90
+ %r2 = lshr i100 %rot1, 10
+ %res = or i100 %s2, %r2
+ ret i100 %res
+}
|
|
I have read the LLVM Developer Policy, the Code Review Policy, and the LLVM AI Tool Use Policy, and I have noted the AI usage (for formatting the PR description) in the PR description. |
|
Per the LLVM Developer Policy, please squash the two commits into a single commit and use standard LLVM commit headline conventions (e.g., [DAGCombiner]...). |
|
Thanks for investigating and catching the overflow bug in
I found these 3 while checking this issue and PR. |
|
|
|
@GiTaDi-CrEaTe Thanks for the review. Squashed into a single commit and moved the tests into combine rotates.ll. Note that for i256 with an i8 shift amount, the new guard returns before BitsizeC is created so that case is not affected. |
| ; CHECK-NEXT: retq | ||
| %s1 = shl i193 %a, 1 | ||
| %r1 = lshr i193 %a, 192 | ||
| %rot1 = or i193 %s1, %r1 |
There was a problem hiding this comment.
Use fshl/fshr intrinsics instead of expanded patterns
There was a problem hiding this comment.
Thanks for the review. I tried the fshl/fshr form first but those tests do not reach this fold so they would pass without the fix.
On an unfixed llc (20.1.2) nested llvm.fshl/llvm.fshr on i193 i100 and i255 with large amounts for example fshl by 1 then by 184 on i193 and fshl by 90 twice on i100 all give the correct result. The expanded shl/lshr/or form gives the wrong result for the same widths and amounts.
From reading DAGCombiner.cpp, fshl(x, x, c) only becomes a rotate when ROTL is legal or custom for the type (visitFunnelShift) and i193 is not. MatchRotate turns the expanded pattern into ROTL/ROTR before legalization which is the path that hits the overflow in visitRotate.
So I kept the expanded form, so the test fails without the fix. I can add fshl/fshr tests as extra coverage if you want. Is that OK?
Also, would you prefer that I keep the fold by doing the amount arithmetic in a wider type, instead of skipping it for non power of 2 widths as this patch does now?
| return SDValue(); | ||
| bool SameSide = (N->getOpcode() == NextOp); | ||
| unsigned CombineOp = SameSide ? ISD::ADD : ISD::SUB; | ||
| SDValue BitsizeC = DAG.getConstant(Bitsize, dl, ShiftVT); |
There was a problem hiding this comment.
I don't understand why all these calculations are done on SDValues instead of APInt. Surely C1 and C2 and BitsizeC are all compile time constants here? Am I missing something?
There was a problem hiding this comment.
N1 and N0.getOperand(1) are checked with isConstantIntBuildVectorOrConstantInt, so they can be non-splat constant BUILD_VECTORs as well as scalar constants e.g. combine_vec_rot_rot in combine-rotates.ll has per lane amounts <1, 2, 3, 4> and <12, 13, 14, 15>. Using FoldConstantArithmetic on SDValues handles both scalars and per lane vector constants including undef lanes without manual lane iteration.
If you and @RKSimon prefer keeping the fold for non power of 2 types instead of bailing out we could either do the FoldConstantArithmetic sequence in a wider integer or vector VT and truncate back to ShiftVT or compute it per lane with APInt. Happy to implement whichever approach you prefer.
There was a problem hiding this comment.
Ugh. Thanks for explaining. It's a shame we don't have a better way of handling vector constants than calling FoldConstantArithmetic (which is allowed to fail) on SDValues.
Fixes #239932
The fold in
DAGCombiner::visitRotatethat mergesrot(rot(x, c1), c2)into one rotate computes((c1 % BW) +/- (c2 % BW) + BW) % BWin the shift amount type. That type is 8 bits wide here. When the bit width BW is not a power of 2 (i100, i193, ...), the value can reach 3*BW-2 and wraps around, so the merged rotate amount is wrong. A power of 2 BW is not affected, because 256 is a multiple of it.Example:
rot(rot(x, 1), 184)on i193 computes 1 + 184 + 193 = 378, which wraps to 122 in 8 bits. The result is a rotate by 122 instead of 185.This patch skips the fold when that arithmetic can overflow the shift amount type. Power of 2 widths keep the fold. A version that does the arithmetic in a wider type and keeps the fold is also possible. I picked the smaller change and am happy to switch if reviewers prefer.
Testing:
llcfrommainwith this change and compared 15 nested rotate cases i64 to i256 with Python big integers. All 15 pass. Without the change, 8 of them give the wrong result (details in the issue).