'Fastest way to dump nested dict to the hard drive

I have a big (several Gigs) nested dictionary of this structure:

{ 
  string1: {string1_1: int1_1, string1_2: int1_2, ...},
  string2: {string2_1: int2_1, string2_2: int2_2, ...},
...
}

Its a kind of word co-occurences counts in a big text corpus, so the amount of keys in inner dicts varies.

I am trying to find the fastest way to save this structure to the hard drive for reusing. Pickle/cpickle.dump is impossibly slow. Msgpack.pack is better but also it is faster for me to recalculate the whole dict from raw data than to dump-load it.

Does anybody have any experience of serializing such huge dicts? Any tips/tricks and libraries are appreciated.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source