'Fastest way to dump nested dict to the hard drive
I have a big (several Gigs) nested dictionary of this structure:
{
string1: {string1_1: int1_1, string1_2: int1_2, ...},
string2: {string2_1: int2_1, string2_2: int2_2, ...},
...
}
Its a kind of word co-occurences counts in a big text corpus, so the amount of keys in inner dicts varies.
I am trying to find the fastest way to save this structure to the hard drive for reusing. Pickle/cpickle.dump is impossibly slow. Msgpack.pack is better but also it is faster for me to recalculate the whole dict from raw data than to dump-load it.
Does anybody have any experience of serializing such huge dicts? Any tips/tricks and libraries are appreciated.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|