Reduce resources to load language models

[pemistahl]

Currently, the language models are parsed from json files and loaded into simple maps at runtime. Even though accessing the maps is pretty fast, they consume a significant amount of memory. The goal is to investigate whether there are more suitable data structures available that require less storage space in memory, something like NumPy for Python.

One promising candidate could be ndarray.

[ghost]

Which files ? if you require the processing in Python or in JavaScript(Node) I can work on a Google proto buffer format; quite sure the persisted model would be way lighter, maybe the processing would be fast, I do not know.
Any way, I'm glad to help.
I'm happy that you provide a JS binding as well, I'm looking for a fast language detection runnable on Node.
Thanks

[ghost]

I know this is half road, as you were asking for a better structure to gain processing time. But for big model on memory here is a solution:

I changed the format a little bit from regular Map<string: string> to Map<number[]: string[]>. I guess you treat as so anyway, so hopefully not a problem.

Here is a working example in JavaScript/Node: https://github.com/bacloud23/lingua-rs-bigrams

So here how it goes:

• You encode/persist JSON once back and forth into a lightweight binary file.
• With proto buffers, you definitely can load the binary file with the defined format, and decode the entire object (it would be way lighter).
• But If you work with the values of the ngrams key iteratively and not cumulatively (I guess so), you can (I guess) load one Pair at a time inside a loop. I think it comes with a processing cost though (again if even possible).
• You can do the same in Rust or in Python with the same proto model and the new encoding.

Drawback: new protobufjs dependency.

[getreu]

@ghost: By how much your solution reduces the binary size?

[adamreichold]

I am not sure ndarray (or any n-dimensional array data structure for that matter) can help with associative look-ups of ngram keys. If I understand things correctly, then the main issue is the memory required to store the keys, i.e. the ngrams.

If so, I think one promising candidate would be finite state transducers (FST), c.f. https://burntsushi.net/transducers/, via the fst crate. It would try to compress the key space and also allow being memory-mapped directly from disk. The downside is that values are limited to u64 but I think this would actually suffice to store frequencies as u32 and probabilities as fractions of two u32.

I think ideally, all models for a given language would also be serialized and thereby memory-mapped as a single FST by encoding model type and ngram name into the key itself, e.g. most common fivegram "nicht" could be encoded as the byte string mapping

[1 /* = most common */, 5 /* fivegrams */, b'n', b'i', b'c', b'h', b't'] => 0

(where the value does not really matter if we only need set membership) or the bigram probability 338628/8845267 of "ni" could be encoded as the byte string mapping

[2 /* = probabilities */, 2, /* bigrams */, b'n', b'i'] => (338628 << 32) | 8845267

The main caveat I see is that FST are meant to store longer tokens as there is by design little sequential redundancy in bigrams, so the compression effect could be quite limited. The upside of memory-mapping a single look-up-efficient data structure per language should still be an improvement though.

[adamreichold]

Thinking about it a bit, ngram length does not need to be encoded at all as it is self-describing in that approach. Also it might be preferable to store the model type at the end to simplify exploiting common suffixes in fst's approximate construction algorithm.