Subsequent bitfields of different types being decoded in a single byte in cyclic manner

[Indra5196]

I have a scenario where I have two bitfields of different size types subsequently

class BitFieldStruct { std::uint8_t a : 3; std::uint16_t b: 7; };

Note that the types of both bitfields are different. and the size of this class must be 2 bytes. The kaitai generated for such a class is as follows.

meta:
  id: bitfieldstruct
  endian: le
  bit-endian: le
  imports:
    - /default/memory_chunk

seq:
  - id: num_chunks
    contents: [1]
  - id: chunk_1
    type: base_types__memory_chunk
 
instances:
  payload:
    io: _root.chunk_1.data._io
    pos: 0
    type: bitfieldstruct
    -dynamic-type: "struct"
    -cpp-datatype: "BitFieldStruct"
 
types:
  bitfieldstruct:
    seq:
      - id: members
        size: 2
        type: bitfieldstruct_members
 
  bitfieldstruct_members:
    instances:
      a:
        pos: 0
        type: b3
        -dynamic-type: "primitive"
        -cpp-datatype: "unsigned char"
        -cpp-name: "a"
 
      b:
        pos: 0
        type: b7
        -dynamic-type: "primitive"
        -cpp-datatype: "short unsigned int"
        -cpp-name: "b"

Using this kaitai, we generated code using kaitai_struct_compiler for C++ (compiled using gcc 13.3.0). The bytes which we are trying to decode are 0x1D, 0x00 using the generated API's. Now, looking at the binary representation 00011101 00000000, I expect decoded values as a = 5 and b = 3. But the values I am getting is a = 5 and b = 35. Upon close look, I see that the binary value of b as 0100011. It seems that for reading the last 7 bits, the read is happening at the same byte in a cyclic fashion, and the read is NOT moving over to the next byte as expected.

Am I missing something here or is this a known issue..??

Note that if both the bitfield types are same (Lets say std::uint8_t), the decode is happening as expected from the generated API's

[generalmimon]

There's no reason to use instances in bitfieldstruct_members (I don't know how you came up with that idea - https://doc.kaitai.io/user_guide.html#_bit_sized_integers does not suggest anything like that). The correct way is to just use seq:

  bitfieldstruct_members:
    seq:
      - id: a
        type: b3
      - id: b
        type: b7

According to your example:

Now, looking at the binary representation 00011101 00000000, I expect decoded values as a = 5 and b = 3.

... this will work as you expect - the b field will continue from where a ended, as is characteristic of the seq section.

But the values I am getting is a = 5 and b = 35. Upon close look, I see that the binary value of b as 0100011. It seems that for reading the last 7 bits, the read is happening at the same byte in a cyclic fashion, and the read is NOT moving over to the next byte as expected.

Am I missing something here or is this a known issue..??

This is a known issue, see #564. In short, instances only save and restore byte-level stream position, not bit-level position, which is thus changed permanently if you use bit-sized integers in them. This also implies that if you invoke the b instance before a, you will get different results, which suggests that something is broken - the order in which instances are invoked should never affect the results.

Therefore, for the time being, you should never use any bit-sized integers (type: bX) via instances, otherwise you'll run into undefined behavior. And even when #564 is fixed, based on the expected results you provided, it's clear that you never intended to use instances - seq is what you need.

[Indra5196]

In our observation, for simple sequences like above, this works fine. But not for structs which have both bitfields and non-bitfield members, or the structs with improper alignments. For example:

struct abc {
    uint8_t a;
    int b;
}

[generalmimon]

@Indra5196:

In our observation, for simple sequences like above, this works fine.

If it looks working to you, it's just a coincidence. Bottom line, bit-sized integers are only supported in seq at the moment. If you use them in instances, don't be surprised if you get unexpected results, and I told you so.

[generalmimon]

@Indra5196:

In our observation, for simple sequences like above, this works fine. But not for structs which have both bitfields and non-bitfield members, or the structs with improper alignments. For example:

struct abc {
    uint8_t a;
    int b;
}

On a second look, what do you mean by this in "this works fine"? Based on what you write here (especially the part "not for structs which have both bitfields and non-bitfield members"), you might be encountering #1070, which however doesn't seem to relate to your original post in #1238 (comment) - it's a completely different issue.

Please provide .ksy snippets instead of (or in addition to) the C/C++ code if possible - I have no idea what you wanted to show with this code snippet:

struct abc {
    uint8_t a;
    int b;
}

For example, I don't see any connection with bit-sized integers here.

[Indra5196]

@generalmimon

Here is the ksy for the above struct:

meta:
  id: abc
  endian: le
  bit-endian: le
  imports:
    - /default/memory_chunk

seq:
  - id: num_chunks
    contents: [1]
  - id: chunk_1
    type: base_types__memory_chunk

instances:
  payload:
    io: _root.chunk_1.data._io
    pos: 0
    type: abc
    -dynamic-type: "struct"
    -cpp-datatype: "abc"

types:
  abc:
    seq:
      - id: members
        size: 8
        type: abc_members

  abc_members:
    seq:
      - id: a
        type: u1

      - id: b
        type: s4

Since there will be a padding of 3 bytes after the uint8_t member, seq does not seem to work in this case. As the decoder functions generated from such a ksy (maybe) expects the next member to be present right next to uint8_t member

[generalmimon]

@Indra5196:

Since there will be a padding of 3 bytes after the uint8_t member, seq does not seem to work in this case. As the decoder functions generated from such a ksy (maybe) expects the next member to be present right next to uint8_t member

Yes. Kaitai Struct does not implicitly add any padding like C does. In seq, each field starts exactly where the previous field ended, without any gaps (the only exception is when you have a bit-sized integer followed by any byte-aligned type). If you want padding like in a C struct, you have to specify it explicitly:

  abc_members:
    seq:
      - id: a
        type: u1
      - id: padding
        size: 3
      - id: b
        type: s4

Or if you want a slightly more flexible solution:

  abc_members:
    seq:
      - id: a
        type: u1
      - id: padding
        size: -_io.pos % 4
      - id: b
        type: s4

Assuming that abc_members starts at a 4-byte aligned offset, this is equivalent to size: 3. The % operator performs the modulo operation, and -1 % 4 == 3.

There are some proposals to support specifying alignment in #12, but none of them have been implemented yet.