Burp is able to decrypt certain protobufs but not the built-in APIs

[Shra1V32]

Hello,

I've noticed that certain gRPC-encoded messages (likely multi-message protobuf) are failing to decode or encode using .decode_message and .encode_message. However, the Burp extension can handle them successfully.

I understand this information might not be sufficient, but I have specific payloads that I can share via email to assist with thorough testing.

Thanks!

[rwinkelmaier]

Hi!

While the main library functions don't do automatic gRPC decoding, you can use the functions in blackboxprotobuf.lib.payloads module to do the same decoding.

The main payloads modules (https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/blackboxprotobuf/lib/payloads/__init__.py) has some functions for automatically finding the right decoder, but you should be able to use functions blackboxprotobuf.lib.payloads.grpc (https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/blackboxprotobuf/lib/payloads/grpc.py). IIRC those functions do handle multiple protobuf messages in a single payload as well.

[Shra1V32]

Yeah, I've observed looking into the codes, but it's better if implemented inside the APIs. Any plans to implement that sooner?

[rwinkelmaier]

No plans at the moment. I don't remember the exact reasons it wasn't included, but likely did not want to over complicate the API or introduce assumptions that might fail. I might revisit that later, or maybe could add some new functions to api.py to handle gRPC easier, but would probably not be for a little while.

[Shra1V32]

Okay then, I might probably contribute to fixing this issue over the next week - if I correctly understand the entire codebase.

[rwinkelmaier]

Could you clarify on what use case you are looking for? Do you want to have an all-in-one function which decodes gRPC messages, or something that mimics the burp extensions and tries to autodetect extra encodings?

In either case, the code is already there, but I want to be mindful about how the API is designed. I've been avoiding heavily overloading the encode_message and decode_message functions.

When the gRPC logic was moved from the burp code to the library code, the intention was that callers could use blackboxprotobuf.lib.payloads directly without too much hassle. For example, gRPC can be done with something like:

import blackboxprotobuf
from blackboxprotobuf.lib.payloads import grpc
...
payloads, encoding = grpc.decode_grpc(buffer)
message, typedef = blackboxprotobuf.decode_message(payloads[0])
...
modified_payload= blackboxprotobuf.encode_message(modified_message, typedef)
modified_buffer = grpc.encode_grpc([modified_payload], encoding)

But it looks like integrating the trial and error system into a caller might not be as simple, so maybe room to wrap that all up into a new API function.

[rwinkelmaier]

I put together a couple functions in a PR: #47. I can't merge into the main branch or push new versions to pypi at the moment, but let me know if that's about what you're looking for.

I'll try to add json versions to the PR as well.

[Shra1V32]

Hi @rwinkelmaier,

Thank you for your assistance. The implementation works perfectly, and I believe we can merge it into the main branch.

I did notice a couple of points worth addressing:

The .encode_wrapped_message method returns a bytearray instead of bytes, which differs from the return type of .encode_message.

There is a potential exception when raising BlackboxProtobufException within _resolve_typedef.

Please let me know if you would like me to add commits to the draft PR to address these issues.

Additionally, I observed that the files in the codebase reference the global blackboxprotobuf module. Would it be preferable to use relative imports instead?

[Shra1V32]

Also, Is it really "impossible" to make the compressed gRPC work?

[rwinkelmaier]

Thanks for taking a look.

Good catch on the bugs. I was expecting mypy to catch an incorrect return type, but apparently mypy allows them to be switched "for convenience". With that feature disabled, it finds a few other cases, but I might need to double check that it doesn't impact performance.

For the imports, relative imports are probably more readable, but I would have to check it works with Jython for Burp.

Regarding gRPC compression, I don't think it's impossible, but not straightforward and wasn't sure how often it was needed. From my understanding, there are a few potential compression algorithms and they're defined in an HTTP2 header rather than the gRPC data, so have to find a way to either get the algorithm into BBPB from the caller, or add another level of trial and error to guess the algorithm.

Would also need to be wary of adding dependencies to support compression algorithms that aren't built into python, but maybe gzip/zlib would be sufficient for most cases.