How would we multiply a tensor with a dimension of three or more?

[kunal-1320]

For instance, when multiplying two matrices using matrix multiplication, the length of matrix 1's column and the length of matrix 2's row should match.

Does this hold true for dimension three tensors?
(In order to visualize a matrix in three dimensions, I may refer to dimension 2 or 3 or....

code:

for the dimension 2

matrixone = torch.tensor([[2,3,4],
                                             [5,6,7]])

matrixtwo = torch.tensor([[8,9],
                          [10,11],
                          [12,13]])

_this one is pretty easy_ 

for dimension 3

mat1= torch.tensor([[[5,4,3],
                     [6,4,8],
                     [4,7,1]],
                    [[7,2,3],
                     [3,5,2],
                     [4,1,9]]])

mat2= torch.tensor([[[3,4,3],
                     [7,4,8],
                     [4,7,1]],
                    [[2,2,5],
                     [3,0,2],
                     [6,1,9]]])

print(torch.matmul(mat1,mat2)

_**Can we say something about the rule I mentioned for dimension 2 matrices in this case, and does it also apply here?** If so, which parameter (i.e., the row, column, or depth) would be same? _

[IlanVinograd]

http://matrixmultiplication.xyz/

This is good site to understand multiplication with matrix try it.

A picture is worth a thousand words

[kunal-1320]

Thank you for your remark, hello.
The website is useless for anything, if the form will [2,3,2].
--(means solely for vectors and scalars, they can only be rank 1 tensors.)

[IlanVinograd]

Ow sorry I did'nt understand the question in the first time.

For multiplying matrices, it's essential that the number of columns in the first matrix matches the number of rows in the second matrix. This rule is also applicable to tensors of higher orders (such as three-dimensional tensors), but its interpretation becomes somewhat more complex due to the addition of extra dimensions.

In the case of three-dimensional tensors, which can be thought of as stacks of matrices, the matching size rule for multiplication is modified. When multiplying two three-dimensional tensors, seen as stacks of matrices, each matrix from the first tensor is multiplied by the corresponding matrix in the second tensor. The dimensions that need to match are the "depth" sizes of the tensors and the sizes corresponding to the columns of the first tensor and the rows of the second tensor in each of these matrices.

For your specific example with tensors, the multiplication task is as follows:

mat1 has dimensions (2, 3, 3), which can be interpreted as 2 stacks of 3 matrices of size 3x3.
mat2 has dimensions (2, 3, 3), similarly interpreted as 2 stacks of 3 matrices of size 3x3.
For multiplying three-dimensional tensors in PyTorch, as in your example, it's important that the dimensions related to the "depth" (in this case, the number of matrices in the stack) and the dimensions of columns of the first tensor and rows of the second tensor in each matrix are compatible. In your example, both tensors are compatible for multiplication since they have the same sizes.

However, it's important to note that the result of multiplying two three-dimensional tensors in PyTorch may not entirely align with the intuitive understanding of matrix multiplication due to the library's specifics and the peculiarities of tensor multiplication operation. The result of the multiplication will depend on the broadcasting rules and the specific implementation of the multiplication operation in PyTorch.

Now, let's look at the result of multiplying your tensors.

The result of multiplying your three-dimensional tensors in PyTorch is as follows:

tensor([[[ 55,  57,  50],
         [ 78,  96,  58],
         [ 65,  51,  69]],

        [[ 38,  17,  66],
         [ 33,   8,  43],
         [ 65,  17, 103]]])

This result represents a tensor of size (2, 3, 3), where each element is obtained as a result of multiplying the corresponding matrices from your original tensors mat1 and mat2.

Thus, the rule about matching sizes, applied to two-dimensional matrices, adapts for three-dimensional tensors in the sense that each "layered" matrix from the first tensor is multiplied by the corresponding "layered" matrix from the second tensor. It's crucial that the dimensions of the tensors allow for such multiplication, which was ensured in your case.

The formula for multiplying three-dimensional matrices (tensors) extends the concept of matrix multiplication into higher dimensions. For two three-dimensional tensors, A and B, where A has dimensions (i, j, k) and B has dimensions (i, k, l), the resulting tensor, C, will have dimensions (i, j, l). Here, i represents the depth (or the number of matrices in the stack), j and k represent the rows and columns of matrices in A, and k and l represent the rows and columns of matrices in B, respectively.

The multiplication is conducted over the last two dimensions of each tensor (similar to standard matrix multiplication) and is broadcasted across the first dimension. The element-wise formula for the multiplication to obtain an element c_{d,e,f} in the resulting tensor C is as follows:

c_{d,e,f} = sum_{r=1}^{k} a_{d,e,r} * b_{d,r,f}

Where:

• d is the index along the first dimension (depth),
• e is the index along the second dimension (rows for A and C),
• f is the index along the third dimension (columns for B and C),
• r runs through the common dimension size k of A and B,
• a_{d,e,r} is an element from tensor A,
• b_{d,r,f} is an element from tensor B.

This formula indicates that for each position (d,e,f) in the resulting tensor C, you sum over the products of corresponding elements in the matrices at depth d from both A and B. This process is repeated for each matrix in the stack (i.e., across the first dimension), effectively performing matrix multiplication for each pair of matrices in the corresponding positions of the two input tensors.

[kunal-1320]

Thanks 👍