I am trying to dump a dictionary into pickle format, using 'dump' command provided in python. The file size of the dictionary is around 150 mb, but an exception occurs when only 115 mb of the file is dumped. The exception is:

Traceback (most recent call last): File "C:\Python27\generate_traffic_pattern.py", line 32, in <module> b.dump_data(way_id_data,'way_id_data.pickle') File "C:\Python27\class_dump_load_data.py", line 8, in dump_data pickle.dump(data,saved_file) File "C:\Python27\lib\pickle.py", line 1370, in dump Pickler(file, protocol).dump(obj) File "C:\Python27\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 663, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 600, in save_list self._batch_appends(iter(obj)) File "C:\Python27\lib\pickle.py", line 615, in _batch_appends save(x) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 599, in save_list self.memoize(obj) File "C:\Python27\lib\pickle.py", line 247, in memoize self.memo[id(obj)] = memo_len, obj 
MemoryError

I am really confused, since my same code was working fine earlier.

Are you dumping just that one object, and that's all?

If you are calling dump many times, then calling Pickler.clear_memo() between dumps will flush the internally stored backreferences (causing the 'leak'). And your code should just work fine...