Skip to content

[DAGCombiner] Fix overflow in rotate of rotate fold for non power of 2 types - #239933

Open
robinbishtt wants to merge 4 commits into
llvm:mainfrom
robinbishtt:main
Open

robinbishtt wants to merge 4 commits into
llvm:mainfrom
robinbishtt:main

Conversation

@robinbishtt

@robinbishtt robinbishtt commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #239932

The fold in DAGCombiner::visitRotate that merges rot(rot(x, c1), c2) into one rotate computes ((c1 % BW) +/- (c2 % BW) + BW) % BW in the shift amount type. That type is 8 bits wide here. When the bit width BW is not a power of 2 (i100, i193, ...), the value can reach 3*BW-2 and wraps around, so the merged rotate amount is wrong. A power of 2 BW is not affected, because 256 is a multiple of it.

Example: rot(rot(x, 1), 184) on i193 computes 1 + 184 + 193 = 378, which wraps to 122 in 8 bits. The result is a rotate by 122 instead of 185.

This patch skips the fold when that arithmetic can overflow the shift amount type. Power of 2 widths keep the fold. A version that does the arithmetic in a wider type and keeps the fold is also possible. I picked the smaller change and am happy to switch if reviewers prefer.

Testing:

  • Added tests to llvm/test/CodeGen/X86/combine-rotates.ll i193 and i100. The CHECK lines were generated with update_llc_test_checks.py using a compiler that has this change.
  • I built llc from main with this change and compared 15 nested rotate cases i64 to i256 with Python big integers. All 15 pass. Without the change, 8 of them give the wrong result (details in the issue).
  • AI Tool Use: AI assistance was used only to structure and format the PR description text. The bug discovery, patch, and test cases are my own work.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

Hello @robinbishtt 👋

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

  • All contributions to LLVM must follow our LLVM AI Tool Use Policy. In particular, if you used AI while working on this PR, remember to add a note to the PR description.
  • The LLVM Code-Review Policy and Practices document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
  • Our LLVM Developer Policy describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.


Frequently asked questions

How do I add reviewers?

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using @ followed by their GitHub username.

What if there are no comments?

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

Are any special GitHub settings required to contribute to LLVM?

We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details.


If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse.

Thank you,
The LLVM Community

@llvmorg-github-actions

llvmorg-github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

@llvm/pr-subscribers-backend-x86

@llvm/pr-subscribers-llvm-selectiondag

Author: Robin Bisht (robinbishtt)

Changes

Fixes #239932

The fold in DAGCombiner::visitRotate that merges rot(rot(x, c1), c2) into one rotate computes ((c1 % BW) +/- (c2 % BW) + BW) % BW in the shift amount type. That type is 8 bits wide here. When the bit width BW is not a power of 2 (i100, i193, ...), the value can reach 3*BW-2 and wraps around, so the merged rotate amount is wrong. A power of 2 BW is not affected, because 256 is a multiple of it.

Example: rot(rot(x, 1), 184) on i193 computes 1 + 184 + 193 = 378, which wraps to 122 in 8 bits. The result is a rotate by 122 instead of 185.

This patch skips the fold when that arithmetic can overflow the shift amount type. Power of 2 widths keep the fold. A version that does the arithmetic in a wider type and keeps the fold is also possible. I picked the smaller change and am happy to switch if reviewers prefer.

Testing:

  • New test llvm/test/CodeGen/X86/rotate-fold-overflow.ll (i193 and i100). The CHECK lines were generated with update_llc_test_checks.py using a compiler that has this change.
  • I built llc from main with this change and compared 15 nested rotate cases i64 to i256 with Python big integers. All 15 pass. Without the change, 8 of them give the wrong result (details in the issue).

Full diff: https://github.com/llvm/llvm-project/pull/239933.diff

2 Files Affected:

  • (modified) llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (+9)
  • (added) llvm/test/CodeGen/X86/rotate-fold-overflow.ll (+61)
diff --git a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
index 196111ea8e1f7f..78492bcfcc4e0e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
@@ -11146,6 +11146,15 @@ SDValue DAGCombiner::visitRotate(SDNode *N) {
     bool C2 = DAG.isConstantIntBuildVectorOrConstantInt(N0.getOperand(1));
     if (C1 && C2 && N1.getValueType() == N0.getOperand(1).getValueType()) {
       EVT ShiftVT = N1.getValueType();
+      // The amount arithmetic below is done in ShiftVT and can reach
+      // 3 * Bitsize - 2. If that does not fit in ShiftVT the result wraps
+      // around. This is harmless only when Bitsize is a power of 2 (the
+      // wrap-around is then a multiple of Bitsize), so skip the fold otherwise.
+      unsigned AmtBits = ShiftVT.getScalarSizeInBits();
+      APInt MaxAmt = APInt::getMaxValue(AmtBits);
+      bool MayOverflow = !isPowerOf2_32(Bitsize) && MaxAmt.ult(3 * Bitsize - 2);
+      if (MayOverflow || MaxAmt.ult(Bitsize))
+        return SDValue();
       bool SameSide = (N->getOpcode() == NextOp);
       unsigned CombineOp = SameSide ? ISD::ADD : ISD::SUB;
       SDValue BitsizeC = DAG.getConstant(Bitsize, dl, ShiftVT);
diff --git a/llvm/test/CodeGen/X86/rotate-fold-overflow.ll b/llvm/test/CodeGen/X86/rotate-fold-overflow.ll
new file mode 100644
index 00000000000000..a3ed92289fa6f1
--- /dev/null
+++ b/llvm/test/CodeGen/X86/rotate-fold-overflow.ll
@@ -0,0 +1,61 @@
+; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s
+; Nested rotates on a non-power-of-2 integer must not be folded using an amount
+; that overflowed the shift amount type (i8 on X86).
+
+define i193 @rot_of_rot_i193(i193 %a) {
+; CHECK-LABEL: rot_of_rot_i193:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    andl $1, %r8d
+; CHECK-NEXT:    leal (%r8,%rsi,2), %edi
+; CHECK-NEXT:    shldq $56, %rcx, %rdi
+; CHECK-NEXT:    movq %rsi, %r8
+; CHECK-NEXT:    shrdq $8, %rdx, %r8
+; CHECK-NEXT:    shldq $56, %rdx, %rcx
+; CHECK-NEXT:    shrl $7, %esi
+; CHECK-NEXT:    movq %rcx, 8(%rax)
+; CHECK-NEXT:    movq %r8, (%rax)
+; CHECK-NEXT:    andl $1, %esi
+; CHECK-NEXT:    movb %sil, 24(%rax)
+; CHECK-NEXT:    movq %rdi, 16(%rax)
+; CHECK-NEXT:    retq
+  %s1 = shl i193 %a, 1
+  %r1 = lshr i193 %a, 192
+  %rot1 = or i193 %s1, %r1
+  %s2 = shl i193 %rot1, 184
+  %r2 = lshr i193 %rot1, 9
+  %res = or i193 %s2, %r2
+  ret i193 %res
+}
+
+define i100 @rot_of_rot_i100(i100 %a) {
+; CHECK-LABEL: rot_of_rot_i100:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rsi, %rax
+; CHECK-NEXT:    movq %rdi, %rdx
+; CHECK-NEXT:    shrdq $1, %rsi, %rdx
+; CHECK-NEXT:    movl %eax, %ecx
+; CHECK-NEXT:    shrl %ecx
+; CHECK-NEXT:    shldq $55, %rdx, %rcx
+; CHECK-NEXT:    shlq $26, %rdi
+; CHECK-NEXT:    shrq $10, %rax
+; CHECK-NEXT:    movl %eax, %edx
+; CHECK-NEXT:    andl $67107840, %edx # imm = 0x3FFFC00
+; CHECK-NEXT:    orq %rdi, %rdx
+; CHECK-NEXT:    movq %rax, %rsi
+; CHECK-NEXT:    shldq $63, %rcx, %rsi
+; CHECK-NEXT:    shrl %eax
+; CHECK-NEXT:    shldq $55, %rsi, %rax
+; CHECK-NEXT:    shlq $26, %rcx
+; CHECK-NEXT:    shrq $10, %rdx
+; CHECK-NEXT:    andl $67108863, %edx # imm = 0x3FFFFFF
+; CHECK-NEXT:    orq %rcx, %rdx
+; CHECK-NEXT:    retq
+  %s1 = shl i100 %a, 90
+  %r1 = lshr i100 %a, 10
+  %rot1 = or i100 %s1, %r1
+  %s2 = shl i100 %rot1, 90
+  %r2 = lshr i100 %rot1, 10
+  %res = or i100 %s2, %r2
+  ret i100 %res
+}

@robinbishtt

Copy link
Copy Markdown
Contributor Author

I have read the LLVM Developer Policy, the Code Review Policy, and the LLVM AI Tool Use Policy, and I have noted the AI usage (for formatting the PR description) in the PR description.

@GiTaDi-CrEaTe

Copy link
Copy Markdown
Contributor

Per the LLVM Developer Policy, please squash the two commits into a single commit and use standard LLVM commit headline conventions (e.g., [DAGCombiner]...).

@GiTaDi-CrEaTe

Copy link
Copy Markdown
Contributor

Thanks for investigating and catching the overflow bug in visitRotate. A few thoughts from looking at the PR:

  1. Both rotate amounts are compile-time constants, and the final combined rotate amount is guaranteed to fit in [0, Bitsize - 1]. In your i193 example, rot(rot(x, 1), 184) is rot(x, 185), which fits in i8. The overflow only happened because of intermediate math in ShiftVT. Bailing out permanently turns off the fold for every non-power-of-2 type $&gt; 85$ bits (i96, i100, i193, etc.), leaving behind two slow multi-precision rotates in codegen. Computing the combined constant using uint64_t or APInt and truncating back to ShiftVT keeps the fold working correctly.
  2. Putting Bitsize directly into ShiftVT (BitsizeC = DAG.getConstant(Bitsize, dl, ShiftVT)) means for i256 with i8 shift amount, BitsizeC truncates to 0 and causes a divide by zero in FoldConstantArithmetic.
  3. We already have llvm/test/CodeGen/X86/combine-rotates.ll for testing this exact fold. It would be cleaner to add test cases there rather than adding a separate file.

I found these 3 while checking this issue and PR.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

⚠️ We detected that you are using a GitHub private e-mail address to contribute to the repo.
Please turn off Keep my email addresses private setting in your account.
See LLVM Developer Policy and LLVM Discourse for more information.

@robinbishtt

Copy link
Copy Markdown
Contributor Author

@GiTaDi-CrEaTe Thanks for the review. Squashed into a single commit and moved the tests into combine rotates.ll. Note that for i256 with an i8 shift amount, the new guard returns before BitsizeC is created so that case is not affected.

; CHECK-NEXT: retq
%s1 = shl i193 %a, 1
%r1 = lshr i193 %a, 192
%rot1 = or i193 %s1, %r1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use fshl/fshr intrinsics instead of expanded patterns

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review. I tried the fshl/fshr form first but those tests do not reach this fold so they would pass without the fix.

On an unfixed llc (20.1.2) nested llvm.fshl/llvm.fshr on i193 i100 and i255 with large amounts for example fshl by 1 then by 184 on i193 and fshl by 90 twice on i100 all give the correct result. The expanded shl/lshr/or form gives the wrong result for the same widths and amounts.

From reading DAGCombiner.cpp, fshl(x, x, c) only becomes a rotate when ROTL is legal or custom for the type (visitFunnelShift) and i193 is not. MatchRotate turns the expanded pattern into ROTL/ROTR before legalization which is the path that hits the overflow in visitRotate.

So I kept the expanded form, so the test fails without the fix. I can add fshl/fshr tests as extra coverage if you want. Is that OK?

Also, would you prefer that I keep the fold by doing the amount arithmetic in a wider type, instead of skipping it for non power of 2 widths as this patch does now?

@robinbishtt
robinbishtt requested a review from RKSimon November 9, 2026 15:05
@robinbishtt

Copy link
Copy Markdown
Contributor Author

Ping @cofibrant @boomanaiden154

return SDValue();
bool SameSide = (N->getOpcode() == NextOp);
unsigned CombineOp = SameSide ? ISD::ADD : ISD::SUB;
SDValue BitsizeC = DAG.getConstant(Bitsize, dl, ShiftVT);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why all these calculations are done on SDValues instead of APInt. Surely C1 and C2 and BitsizeC are all compile time constants here? Am I missing something?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

N1 and N0.getOperand(1) are checked with isConstantIntBuildVectorOrConstantInt, so they can be non-splat constant BUILD_VECTORs as well as scalar constants e.g. combine_vec_rot_rot in combine-rotates.ll has per lane amounts <1, 2, 3, 4> and <12, 13, 14, 15>. Using FoldConstantArithmetic on SDValues handles both scalars and per lane vector constants including undef lanes without manual lane iteration.

If you and @RKSimon prefer keeping the fold for non power of 2 types instead of bailing out we could either do the FoldConstantArithmetic sequence in a wider integer or vector VT and truncate back to ShiftVT or compute it per lane with APInt. Happy to implement whichever approach you prefer.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh. Thanks for explaining. It's a shame we don't have a better way of handling vector constants than calling FoldConstantArithmetic (which is allowed to fail) on SDValues.

@robinbishtt
robinbishtt requested a review from jayfoad November 11, 2026 05:26
@robinbishtt

Copy link
Copy Markdown
Contributor Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend:X86 llvm:SelectionDAG SelectionDAGISel as well

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[X86] Wrong code: rotate of rotate fold overflows the i8 shift amount on non power of 2 integers such as i100 and i193

4 participants