zimdump: new function to analyze ZIM content size with hints about compression factor

[benoit74]

For openzim/mwoffliner#2180 I had to analyze the ZIM content.

I did it with python-libzim binding because I'm way more comfortable with it.

The struggle I had (which luckily was not blocker) is that while it is possible to have access to an Item size (uncompressed AFAIK), I did not found any way to get its compressed size. It was hence hard to be 100% sure where the increased ZIM size went from.

Is that mostly normal since there is no such compressed size, because we only compress the cluster, not individual items? Or is it just something which is missing in the binding(s)? Should I have used another tool / zimtool to do this analysis?

At least having a rough estimation of compression factor for every item would help to analyze a bit deeper such situations. Maybe simply exposing clusters, and which cluster is used by which item, and every cluster compression factor (compressed and uncompressed size for instance) would be sufficient.

[veloman-yunkan]

Is that mostly normal since there is no such compressed size, because we only compress the cluster, not individual items?

Exactly.

[veloman-yunkan]

At least having a rough estimation of compression factor for every item would help to analyze a bit deeper such situations. Maybe simply exposing clusters, and which cluster is used by which item, and every cluster compression factor (compressed and uncompressed size for instance) would be sufficient.

Another option is to output proper statistics during/after ZIM creation.

[benoit74]

Another option is to output proper statistics during/after ZIM creation.

Or with a dedicated tool (here I want to analyze a ZIM right after creation, but I can imagine other scenarii where it might have been a while and creation logs are not here anymore).

Maybe a feature for zimdump?

[kelson42]

Maybe a feature for zimdump?

This is what I though as I first read your issue.