DOCSP-48557 Update Spark streaming write configuration to include all batch options #261

mballard-mdb · 2025-07-02T15:26:39Z

Pull Request Info

PR Reviewing Guidelines

JIRA - https://jira.mongodb.org/browse/DOCSP-48557

Staging Links

batch-mode/batch-write-config

streaming-mode/streaming-write-config

Self-Review Checklist

Is this free of any warnings or errors in the RST?
Did you run a spell-check?
Did you run a grammar-check?
Are all the links working?
Are the facets and meta keywords accurate?
Are the page titles greater than 20 characters long and SEO relevant?

netlify · 2025-07-02T15:29:06Z

✅ Deploy Preview for docs-spark-connector ready!

Name	Link
🔨 Latest commit	`e49a0a3`
🔍 Latest deploy log	https://app.netlify.com/projects/docs-spark-connector/deploys/686576858efa8c0008e1d2b9
😎 Deploy Preview	https://deploy-preview-261--docs-spark-connector.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

docs-builder-bot · 2025-07-02T15:30:35Z

🔄 Deploy Preview for docs-spark-connector processing

Item	Details
🔨 Latest Commit	c3b319db35f54b44697f16f2fc26dc3f7321319d
😎 Deploy Preview	https://deploy-preview-261--docs-spark-connector.netlify.app
🔍 Build Logs	View Logs

shuangela

great job on this! added some comments related to wording as well as a question about a default setting in one of the parameters.

shuangela · 2025-07-02T16:41:23Z

source/batch-mode/batch-write-config.txt

+         w Option </reference/write-concern/#w-option>` in the {+mdb-server+}
+         manual. 
+       | 
+       | **Default:** ``1``


Q: How did you get that this default is 1? the server manual states only that "If the write concern is missing the w field, MongoDB sets the w option to the default write concern," and then later on in the table it says "{ w: "majority" } is the default write concern for most MongoDB deployments". The implicit default write concern for most mongo deployments seems to be majority: https://www.mongodb.com/docs/manual/reference/write-concern/#std-label-wc-default-behavior

This question applies to the other writeConcern.w option as well

I just moved what was on the batch page to the stream page, so I didn't write any of the info. I assumed most of it was correct but I'll double check everything now.

I see that it can be 1 or majority, so I'm going to include both, with majority taking precedence as it is most cases like you said

I think it's a bit odd if this option has two potential defaults, so it would be good to double check with the technical reviewer in this case

Defaults to Acknowledged which is the default set by the server.

shuangela · 2025-07-02T16:43:47Z

source/streaming-mode/streaming-write-config.txt

@@ -56,13 +56,126 @@ You can configure the following properties when writing data to MongoDB in strea
         interface.
       |
       | **Default:** ``com.mongodb.spark.sql.connector.connection.DefaultMongoClientFactory``
+
+   * - ``convertJson``
+     - | Specifies whether the connector parses the string and converts extended JSON


S: "whether" seems a bit imprecise, as there are more than two options. Perhaps change wording to "specifies if and how", or something similar. applies to all instances of convertJson

shuangela · 2025-07-02T16:46:06Z

source/streaming-mode/streaming-write-config.txt

@@ -56,13 +56,126 @@ You can configure the following properties when writing data to MongoDB in strea
         interface.
       |
       | **Default:** ``com.mongodb.spark.sql.connector.connection.DefaultMongoClientFactory``
+
+   * - ``convertJson``
+     - | Specifies whether the connector parses the string and converts extended JSON


Suggested change

- | Specifies whether the connector parses the string and converts extended JSON

- | Specifies whether the connector parses string values and converts extended JSON

"the string" is a bit unclear to me as i'm unsure what the specific string we're referring to is. from my understanding this option applies to the whole value of the write operation so "string values" or "strings" would be more clear

applies to all

shuangela · 2025-07-02T16:46:47Z

source/streaming-mode/streaming-write-config.txt

+       | **Default:** ``false``
+
+   * - ``idFieldList``
+     - | Field or list of fields by which to split the collection data. To


S: to keep parallelism, "Specifies a field or list of fields.."

applies to all

shuangela · 2025-07-02T16:55:26Z

source/batch-mode/batch-write-config.txt

-       | **Default:** ``1``
+       | For more information on ``j`` values, see :manual:`WriteConcern j
+         Option </reference/write-concern/#j-option>` in the {+mdb-server+}
+         manual.

   * - ``writeConcern.wTimeoutMS``
     - | Specifies ``wTimeoutMS``, a write-concern option to return an error
         when a write operation exceeds the number of milliseconds. If you


Suggested change

when a write operation exceeds the number of milliseconds. If you

when a write operation exceeds the specified number of milliseconds. If you

applies to all

shuangela · 2025-07-02T16:56:40Z

source/batch-mode/batch-write-config.txt

+         manual. 
+       | 
+       | **Default:** ``1``
+
   * - ``writeConcern.journal``
     - | Specifies ``j``, a write-concern option to enable request for


Suggested change

- | Specifies ``j``, a write-concern option to enable request for

- | Specifies ``j``, a write-concern option requesting acknowledgment from MongoDB that the data has been written to the on-disk journal...

applies to all

shuangela · 2025-07-02T16:57:17Z

source/streaming-mode/streaming-write-config.txt

+       | **Default:** ``true``
+
+   * - ``upsertDocument``
+     - | When ``true``, replace and update operations will insert the data


Suggested change

- | When ``true``, replace and update operations will insert the data

- | When ``true``, replace and update operations insert the data

applies to all

shuangela · 2025-07-02T16:58:29Z

source/streaming-mode/streaming-write-config.txt

+       |
+       | For more information on ``wTimeoutMS`` values, see
+         :manual:`WriteConcern wtimeout </reference/write-concern/#wtimeout>` in
+         the {+mdb-server+} manual.

   * - ``checkpointLocation``
     - | The absolute file path of the directory to which the connector writes checkpoint


Suggested change

- | The absolute file path of the directory to which the connector writes checkpoint

- | The absolute file path of the directory where the connector writes checkpoint

shuangela · 2025-07-02T16:58:50Z

source/batch-mode/batch-write-config.txt

@@ -139,33 +139,35 @@ You can configure the following properties when writing data to MongoDB in batch
       |
       | **Default:** ``true``

+   * - ``writeConcern.w``
+     - | Specifies ``w``, a write-concern option to request acknowledgment that


Suggested change

- | Specifies ``w``, a write-concern option to request acknowledgment that

- | Specifies ``w``, a write-concern option requesting acknowledgment that

shuangela

lgtm with a note to ask tech reviewer for confirmation on something!

DOCSP-48557 spark options update

3266e52

mballard-mdb added 2 commits July 2, 2025 11:59

DOCSP-48557 links and reorder

a02ce80

DOCSP-48667 link fix + mdb-server consistency

c5ee8a3

shuangela requested changes Jul 2, 2025

View reviewed changes

mballard-mdb added 2 commits July 2, 2025 13:26

DOCSP-48557 wording

c3b319d

DOCSP-48557 remove ital

e49a0a3

mballard-mdb requested a review from shuangela July 2, 2025 18:12

shuangela approved these changes Jul 2, 2025

View reviewed changes

	- \| Specifies whether the connector parses the string and converts extended JSON
	- \| Specifies whether the connector parses string values and converts extended JSON

	when a write operation exceeds the number of milliseconds. If you
	when a write operation exceeds the specified number of milliseconds. If you

	- \| Specifies ``j``, a write-concern option to enable request for
	- \| Specifies ``j``, a write-concern option requesting acknowledgment from MongoDB that the data has been written to the on-disk journal...

	- \| When ``true``, replace and update operations will insert the data
	- \| When ``true``, replace and update operations insert the data

	- \| The absolute file path of the directory to which the connector writes checkpoint
	- \| The absolute file path of the directory where the connector writes checkpoint

	- \| Specifies ``w``, a write-concern option to request acknowledgment that
	- \| Specifies ``w``, a write-concern option requesting acknowledgment that

DOCSP-48557 Update Spark streaming write configuration to include all batch options #261

Are you sure you want to change the base?

DOCSP-48557 Update Spark streaming write configuration to include all batch options #261

Uh oh!

Conversation

mballard-mdb commented Jul 2, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Info

Staging Links

Self-Review Checklist

Uh oh!

netlify bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for docs-spark-connector ready!

Uh oh!

docs-builder-bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔄 Deploy Preview for docs-spark-connector processing

Uh oh!

shuangela left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shuangela left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mballard-mdb commented Jul 2, 2025 •

edited by github-actions bot

Loading

netlify bot commented Jul 2, 2025 •

edited

Loading

docs-builder-bot commented Jul 2, 2025 •

edited

Loading