fix: Avoid coercing publishBatchHook to a boolean (#5934)

## Explanation

<!--
Thanks for your contribution! Take a moment to answer these questions so
that reviewers have the information they need to properly understand
your changes:

* What is the current state of things and why does it need to change?
* What is the solution your changes offer and how does it work?
* Are there any changes whose purpose might not obvious to those
unfamiliar with the domain?
* If your primary goal was to update one package but you found you had
to update another one along the way, why did you do so?
* If you had to upgrade a dependency, why did you do so?
-->

With the previous syntax, publishBatchHook would be coerced into a
boolean, which would halt execution further down the function when it is
awaited. This PR fixes normal function execution by avoiding coercion.

## References

<!--
Are there any issues that this pull request is tied to?
Are there other links that reviewers should consult to understand these
changes better?
Are there client or consumer pull requests to adopt any breaking
changes?

For example:

* Fixes #12345
* Related to #67890
-->

## Changelog

<!--
THIS SECTION IS NO LONGER NEEDED.

The process for updating changelogs has changed. Please consult the
"Updating changelogs" section of the Contributing doc for more.
-->

## Checklist

- [ ] I've updated the test suite for new or updated code as appropriate
- [ ] I've updated documentation (JSDoc, Markdown, etc.) for new or
updated code as appropriate
- [ ] I've communicated my changes to consumers by [updating changelogs
for packages I've
changed](https://github.com/MetaMask/core/tree/main/docs/contributing.md#updating-changelogs),
highlighting breaking changes as necessary
- [ ] I've prepared draft pull requests for clients and consumer
packages to resolve any breaking changes

Author: pedronfigueiredo

packages/transaction-controller/CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- Avoid coercing `publishBatchHook` to a boolean ([#5934](https://github.com/MetaMask/core/pull/5934))
+
 ## [57.1.0]
 
 ### Added

packages/transaction-controller/src/utils/batch.test.ts

@@ -1364,7 +1364,12 @@ describe('Batch Utils', () => {
             ...request,
             publishBatchHook: undefined,
             isSimulationEnabled: () => isSimulationSupportedMock(),
-            request: { ...request.request, useHook: true },
+            request: {
+              ...request.request,
+              disable7702: true,
+              disableHook: true,
+              disableSequential: false,
+            },
           }),
         ).rejects.toThrow(`Can't process batch`);
       });
@@ -1380,6 +1385,8 @@ describe('Batch Utils', () => {
             ...request.request,
             requireApproval: false,
             disable7702: true,
+            disableHook: true,
+            disableSequential: false,
           },
         }).catch(() => {
           // Intentionally empty
@@ -1403,7 +1410,12 @@ describe('Batch Utils', () => {
           addTransactionBatch({
             ...request,
             publishBatchHook: undefined,
-            request: { ...request.request, disable7702: true },
+            request: {
+              ...request.request,
+              disable7702: true,
+              disableHook: true,
+              disableSequential: false,
+            },
           }),
         ).rejects.toThrow('Test error');
 
@@ -1425,6 +1437,8 @@ describe('Batch Utils', () => {
             ...request.request,
             origin: ORIGIN_MOCK,
             disable7702: true,
+            disableHook: true,
+            disableSequential: false,
           },
         }).catch(() => {
           // Intentionally empty
@@ -1467,6 +1481,8 @@ describe('Batch Utils', () => {
             ...request.request,
             origin: ORIGIN_MOCK,
             disable7702: true,
+            disableHook: true,
+            disableSequential: false,
           },
         }).catch(() => {
           // Intentionally empty

packages/transaction-controller/src/utils/batch.ts

@@ -425,9 +425,13 @@ async function addTransactionBatchWithHook(
     disableSequential = true;
   }
 
-  const publishBatchHook =
-    (!disableHook && requestPublishBatchHook) ??
-    (!disableSequential && sequentialPublishBatchHook.getHook());
+  let publishBatchHook = null;
+  if (!disableHook) {
+    publishBatchHook = requestPublishBatchHook;
+  } else if (!disableSequential) {
+    publishBatchHook = sequentialPublishBatchHook.getHook();
+  }
+
   if (!publishBatchHook) {
     log(`No supported batch methods found`, {
       disable7702,