Feature Request: Separate handling of _id and id fields (as in jenssegers/mongodb)

[szotyi41]

Hi,

In a previous version of our Laravel project, we used the jenssegers/mongodb package, which allowed us to define and use both id and _id fields independently. This was extremely useful for us, as our project was originally built on MySQL and later migrated to MongoDB.

Throughout our codebase (both frontend and backend), we consistently use the id field as a numeric identifier. The old package allowed this without overwriting the MongoDB-native _id field (BSON ObjectId). However, with the current behavior, assigning a value to the id field seems to overwrite the _id, which breaks our expectations and potentially a large part of the application.

We’d really appreciate support for handling the id and _id fields separately again — just like jenssegers/mongodb used to.

Do you have any suggestions or potential workarounds? Refactoring the entire project would be extremely risky and time-consuming.

Thanks in advance!

[alcaeus]

Hi,

there are two separate answers here: one for root documents, one for embedded documents.

For embedded documents, you can disable this behaviour through the rename_embedded_id_field connection option as shown in our documentation. This will allow you to continue having an id field in embedded documents without any rename logic.

For root documents, there is no option to this behaviour. The primary key in MongoDB is always _id by convention. Its type and value can be chosen when inserting a document, but if no value was provided then the driver will automatically create a BSON ObjectID.

I would like to understand your use case better, so allow me to ask some questions:

define and use both id and _id fields independently. [...] our project was originally built on MySQL and later migrated to MongoDB

Could you explain how having two identifiers with separate values is useful? When you migrated from MySQL to MongoDB, why did you not set the value of _id to that of id?

we consistently use the id field as a numeric identifier. The old package allowed this without overwriting the MongoDB-native _id field (BSON ObjectId).

Why do you not want to set the _id field to that numerical value if you are not using the object ID? As it is now, you're slowing down all of your operations, as your database is forced to store additional data (namely, 12 bytes for every unused ObjectId), and has to maintain an index for a field that is not used at all.

Refactoring the entire project would be extremely risky and time-consuming.

If you are only using numerical identifiers, then the solution would be to set the _id field to the value of id, then remove the id field entirely. I presume that since you're treating the id field as your primary key, your entire application will use id to access this field, which is exactly the behaviour we're now using: to your application, the identifier is stored in the id property of the model, whereas internally it is mapped to the _id field in the document as per MongoDB specifications. For example, an update command using the MongoDB shell to update the identifier of each document would look like this:

db.myColl.updateMany({}, [{$set: {_id: '$id'}}])

For existing documents, this will leave the original id field in place, although I would recommend removing it in favour of consistency.

---

While in your case migrating to ObjectId as identifier type might not be feasible (since you have pre-existing documents with numeric identifiers), for new documents/database I would always recommend using ObjectId. I'm leaving this explanation here for anybody that stumbles upon this in the future, as I've heard the argument of "we don't want to use ObjectId" multiple times.

While in MySQL you would typically use a numeric primary key with auto_increment, this has considerable downsides in distributed databases under high load in order to avoid primary key collisions. For that reasons, we designed the ObjectId specification in a way that minimises such collisions. An ObjectId consists of three parts:

4 byte timestamp    5 byte process unique   3 byte counter
|<----------------->|<---------------------->|<------------>|
[----|----|----|----|----|----|----|----|----|----|----|----]
0                   4                   8                   12

• The 3 byte counter starts at a random value and is incremented every time an ObjectId is generated
• The 5 byte unique value is initialised when the first ObjectId is generated and remains the same during the process' lifetime
• The 4 byte timestamp uses a monotonic clock

Assuming your system is capable of properly generating random values, there is almost no chance of collision between two ObjectIds generated on different systems. Also, each process can quickly generate up to 16 M unique object IDs every second without having to worry about any conflicts occurring.

Last but not least, having the timestamp in the highest order bytes serves two purposes: first of all, it quickly allows sorting documents by their age, with new documents being appended on the index. Second, it also makes any createdAt fields or anything else obsolete, as the ObjectId itself stores the creation time.

That isn't to say that there aren't downsides to ObjectIds. Unfortunately, Laravel assumes that any primary key will always map to a string or int in the database, and we haven't yet been able to convince the creators that this assertion is fundamentally wrong, not only in the context of MongoDB. This means there may be issues when referencing a document that is identified by an ObjectId, and in those cases you may still want to use a different identifier type or generate an ObjectId and use it in a different format. Speaking of formats, an ObjectId is essentially a 96-bit integer. Since most platforms will struggle with any int wider than 64 bit, we usually opt for a 24-character hex string when serialising an ObjectId. This is also what you would use in URLs for a document identifier by an ObjectId. We're currently working on improving the situation around references, as well as making it easier to retrieve documents identified by an ObjectId when only having a string. Fixing this would also allow you to use a BSON Binary UUID, which combines efficient storage with a well-known and also collision-avoiding identifier.

[masterbater]

Hi,

In a previous version of our Laravel project, we used the jenssegers/mongodb package, which allowed us to define and use both id and _id fields independently. This was extremely useful for us, as our project was originally built on MySQL and later migrated to MongoDB.

Throughout our codebase (both frontend and backend), we consistently use the id field as a numeric identifier. The old package allowed this without overwriting the MongoDB-native _id field (BSON ObjectId). However, with the current behavior, assigning a value to the id field seems to overwrite the _id, which breaks our expectations and potentially a large part of the application.

We’d really appreciate support for handling the id and _id fields separately again — just like jenssegers/mongodb used to.

Do you have any suggestions or potential workarounds? Refactoring the entire project would be extremely risky and time-consuming.

Thanks in advance!

I think v4 is still supported, in v5 _id and id will always be the same value.

One way to refactor is find and replace ->id or ['id'] to mysql_legacy_id there is a regex for that to find those.

Then do mass rename to field id

db.collection.updateMany(
  {},
  { $rename: { "id": "mysql_legacy_id" } }
)

[ctadlock]

I just created a bug related to rename_embedded_id_field: #3428