Using a pre-trained model from one scene to train another in Nerfstudio (Spectacular AI): camera parameter size mismatch?

[MengjiaoH]

Hi, I’m using the Spectacular AI app to capture videos of two different scenes.

Here's my workflow:

1. I recorded Scene 1 and used it to train a model. I saved the trained model as a checkpoint.
2. Then I recorded Scene 2 and tried to continue training from the checkpoint created from Scene 1.
   However, I'm encountering an error related to a mismatch in camera parameter sizes. It seems like the camera intrinsics or other parameters differ between the two scenes.

RuntimeError: Error(s) in loading state_dict for NerfactoModel:
size mismatch for field.embedding_appearance.embedding.weight: copying a param with shape torch.Size([40, 32]) from checkpoint, the shape in current model is torch.Size([39, 32]).
size mismatch for camera_optimizer.pose_adjustment: copying a param with shape torch.Size([40, 6]) from checkpoint, the shape in current model is torch.Size([39, 6]).

My questions are:

1. Is it possible to use a model pre-trained on one scene to train another scene in Nerfstudio?
2. If so, how can I correctly handle differences in camera parameters between scenes?
3. Or is this workflow not supported by Nerfstudio?

Any guidance or clarification would be appreciated. Thanks!

[gno1235]

1. This approach is highly unlikely to be valid. For an advanced example using pretraining, please refer to https://cat3d.github.io/.

2. Even if captured with the same camera, it is not possible to obtain exactly the same camera parameters in the COLMAP optimization process. For your attempt to succeed, you need to predefine the second camera using the first camera’s parameters and then optimize while keeping those parameters fixed.

3. It seems that the built-in models in Nerfstudio do not support this.