I have a project where I am given a file and i need to extract the strings from the file. Basically think of the "strings" command in linux but i'm doing this in python. The next condition is that the file is given to me as a stream (e.g. string) so the obvious answer of using one of the subprocess functions to run strings isn't an option either.

I wrote this code:

def isStringChar(ch):if ord(ch) >= ord('a') and ord(ch) <= ord('z'): return Trueif ord(ch) >= ord('A') and ord(ch) <= ord('Z'): return Trueif ord(ch) >= ord('0') and ord(ch) <= ord('9'): return Trueif ch in ['/', '-', ':', '.', ',', '_', '$', '%', '\'', '(', ')', '[', ']', '<', '>', ' ']: return True# default out
return Falsedef process(stream):
dwStreamLen = len(stream)
if dwStreamLen < 4: return NonedwIndex = 0;
strString = ''
for ch in stream:if isStringChar(ch) == False:if len(strString) > 4:#print strStringstrString = ''else:strString += ch

This technically works but is WAY slow. For instance, I was able to use the strings command on a 500Meg executable and it produced 300k worth of strings in less than 1 second. I ran the same file through the above code and it took 16 minutes.

Is there a library out there that will let me do this without the burden of python's latency?

Thanks!

Of similar speed to David Wolever's, using re, Python's regular expression library. The short story of optimisation is that the less code you write, the faster it is. A library function that loops is often implemented in C and will be faster than you can hope to be. Same goes for the char in set() being faster than checking yourself. Python is the opposite of C in that respect.

import sys
import rechars = r"A-Za-z0-9/\-:.,_$%'()[\]<> "
shortest_run = 4regexp = '[%s]{%d,}' % (chars, shortest_run)
pattern = re.compile(regexp)def process(stream):data = stream.read()return pattern.findall(data)if __name__ == "__main__":for found_str in process(sys.stdin):print found_str

Working in 4k chunks would be clever, but is a bit trickier on edge-cases with re. (where two characters are on the end of the 4k block and the next 2 are at the start of the next block)