nn.Sequential cannot accept multiple inputs

[Shamik-07]

@rasbt
Firstly thank you very much for all the work that you are doing, even now, with the latest addition of qwen 3 0.6B. 🙏

I noticed that the following comment:

# ModuleList since Sequential can only accept one input, and we need `x, mask, cos, sin`

is added in this notebook standalone-llama32.ipynb, however, that might be incorrect as i ran this notebook converting-llama2-to-llama3.ipynb with the following code:

class Llama3Model(nn.Module):
        ....
        self.trf_blocks = nn.Sequential(
            *[TransformerBlock(cfg) for _ in range(cfg["n_layers"])])

Code Snippet below for nn.ModuleList in notebook standalone-llama32.ipynb

class Llama3Model(nn.Module):
        ....
        self.trf_blocks = nn.ModuleList(  # ModuleList since Sequential can only accept one input, and we need `x, mask, cos, sin`
            [TransformerBlock(cfg) for _ in range(cfg["n_layers"])]
        )

If you think that the comment is incorrect, I would be happy to create a PR and remove/amend the comment in the standalone-llama32.ipynb if you like.

[casinca]

Hello, I believe Sebastian left that comment from the old implementation. The comment is right when you (usually) use nn.Sequential as a direct forward call like: x = self.trf_blocks(x, mask, self.cos, self.sin) would give an error.

But here, he looped over the nn.Sequential container, the same way as you would with nn.ModuleList, which ends up being the same thing indeed.

Imo, you're right that it's PR worthy in any case: switching llama2-llama3 to use nn.ModuleList or removing the comment in standalone-llama.

[rasbt]

Thanks for the comments @Shamik-07 and @casinca . Like @casinca pointed out the comment is actually correct as we use:

        for block in self.trf_blocks:
            x = block(x, mask, self.cos, self.sin)

[Shamik-07]

Thank you for replying and i agree that there's an error with x = self.trf_blocks(x, mask, self.cos, self.sin).

Should i create a PR with nn.ModuleList, remove the comment, or leave it as is ?

[Shamik-07]

Am leaving it as then

Thanks for the comments @Shamik-07 and @casinca . Like @casinca pointed out the comment is actually correct as we use:

        for block in self.trf_blocks:
            x = block(x, mask, self.cos, self.sin)