I am trying to build a offline wiktionary using the wikimedia dump files (.xml.bz2) using Python. I started with this article as the guide. It involves a number of languages, I wanted to combine all the steps as a single python project. I have found almost all the libraries required for the process. The only hump now is to effectively split the large .xml.bz2 file into number of smaller files for quicker parsing during search operations.

I know that bz2 library exists in python, but it provides only compress and decompress operations. But I need something that could do something like bz2recover does from the command line, which splits large files into a number of smaller junks.

One more important point is the splitting shouldn't split the page contents which start with <page> and ends </page> in the xml document that has been compressed.

Is there a library previously available which could handle this situation or the code has to be written from scratch?(Any outline/pseudo-code would be greatly helpful).

Note: I would like to make the resulting package cross-platform compatible, hence couldn't use OS specific commands.

At last I have written a Python Script myself:

import os
import bz2def split_xml(filename):''' The function gets the filename of wiktionary.xml.bz2 file as  input and createssmallers chunks of it in a the diretory chunks'''# Check and create chunk diretoryif not os.path.exists("chunks"):os.mkdir("chunks")# Counterspagecount = 0filecount = 1#open chunkfile in write modechunkname = lambda filecount: os.path.join("chunks","chunk-"+str(filecount)+".xml.bz2")chunkfile = bz2.BZ2File(chunkname(filecount), 'w')# Read line by linebzfile = bz2.BZ2File(filename)for line in bzfile:chunkfile.write(line)# the </page> determines new wiki pageif '</page>' in line:pagecount += 1if pagecount > 1999:#print chunkname() # For Debuggingchunkfile.close()pagecount = 0 # RESET pagecountfilecount += 1 # increment filename           chunkfile = bz2.BZ2File(chunkname(filecount), 'w')try:chunkfile.close()except:print 'Files already close'if __name__ == '__main__':# When the script is self runsplit_xml('wiki-files/tawiktionary-20110518-pages-articles.xml.bz2')