StringIO portability between python2 and python3 when capturing stdout

2024/10/4 23:33:38

I have written a python package which I have managed to make fully compatible with both python 2.7 and python 3.4, with one exception that is stumping me so far. The package includes a command line script, and in my unit tests I use this code to run the script's main routine while overriding sys.argv to pass command line arguments for argparse, and capturing the script's stdout for comparison:

@contextlib.contextmanager
def runmain(mainfunction, arglist):"""Run mainfunction with arglist in sys.srgv, and capture stdout."""origargv, sys.argv   = sys.argv,   arglistorigout,  sys.stdout = sys.stdout, io.StringIO()rtn = mainfunction()sys.stdout.seek(0)yield (rtn, sys.stdout.read())sys.stdout = origoutsys.argv   = origargvclass test_imdutil_main(unittest.TestCase):def test_help(self):"""Test -h option."""with runmain(imdutil_main, ['imdutil.py', '-h']) as (rtn, capture):# do stuff with rtn and capture...

This works well in python 3.4, but in python 2.7 it generates an error:

TypeError: unicode argument expected, got 'str'

I haven't managed to figure out a way to capture stdout from arbitrary functions which is portable between python 2.7 and python 3.4.

As an aside, I have to admit that I don't understand decorations, context managers or the "yield" keyword very well at all. The inspiration for my runmain() function came from:

http://schinckel.net/2013/04/15/capture-and-test-sys.stdout-sys.stderr-in-unittest.testcase/

Incidentally, my complete package where this code comes from is here:

https://github.com/NF6X/pyImageDisk

At the moment, its unit tests are partially broken under python 2.7 because of this issue. Can anybody help me figure out how to solve this stdout redirection problem in a portable, pythonic manner, preferably without adding any more external dependencies?

Answer

You replaced the Python 2 bytes-only sys.stdout with one that only takes Unicode. You'll have to adjust your strategy on the Python version here, and use a different object:

try:# Python 2from cStringIO import StringIO
except ImportError:# Python 3from io import StringIO

and remove the io. prefix in your context manager:

origout,  sys.stdout = sys.stdout, StringIO()

The cStringIO.StringIO object is the Python 2 equivalent of io.BytesIO; it requires that you write plain bytestrings, not aunicode objects.

You can also use io.BytesIO in Python 2, but then you want to test if sys.stdout is a io.TextIOBase subclass; if it is not, replace the object with a binary BytesIO, object, otherwise use a StringIO object:

import ioif isinstance(sys.stdout, io.TextIOBase):# Python 3origout, sys.stdout = sys.stdout, io.StringIO()
else:# Python 2 or an unorthodox binary stdout setuporigout, sys.stdout = sys.stdout, io.BytesIO()
https://en.xdnf.cn/q/70554.html

Related Q&A

How to redirect data to a getpass like password input?

Im wring a python script for running some command. Some of those commands require user to input password, I did try to input data in their stdin, but it doesnt work, here is two simple python program…

How to grab one random item from a database in Django/postgreSQL?

So i got the database.objects.all() and database.objects.get(name) but how would i got about getting one random item from the database. Im having trouble trying to figure out how to get it ot select on…

Pyspark Dataframe pivot and groupby count

I am working on a pyspark dataframe which looks like belowid category1 A1 A1 B2 B2 A3 B3 B3 BI want to unstack the category column and count their occurrences. So, the result I want is shown belowid A …

Create an excel file from BytesIO using python

I am using pandas library to store excel into bytesIO memory. Later, I am storing this bytesIO object into SQL Server as below-df = pandas.DataFrame(data1, columns=[col1, col2, col3])output = BytesIO()…

python send csv data to spark streaming

I would like to try and load a csv data in python and stream each row spark via SPark Streaming.Im pretty new to network stuff. Im not exactly if Im supposed to create a server python script that once …

Python string representation of binary data

Im trying to understand the way Python displays strings representing binary data.Heres an example using os.urandomIn [1]: random_bytes = os.urandom(4)In [2]: random_bytes Out[2]: \xfd\xa9\xbe\x87In [3]…

Combining Spark Streaming + MLlib

Ive tried to use a Random Forest model in order to predict a stream of examples, but it appears that I cannot use that model to classify the examples. Here is the code used in pyspark:sc = SparkContext…

How to select dataframe rows according to multi-(other column)-condition on columnar groups?

Copy the following dataframe to your clipboard:textId score textInfo 0 name1 1.0 text_stuff 1 name1 2.0 different_text_stuff 2 name1 2.0 text_stuff …

Python Recursive Search of Dict with Nested Keys

I recently had to solve a problem in a real data system with a nested dict/list combination. I worked on this for quite a while and came up with a solution, but I am very unsatisfied. I had to resort t…

Scrapy: how to catch download error and try download it again

During my crawling, some pages failed due to unexpected redirection and no response returned. How can I catch this kind of error and re-schedule a request with original url, not with the redirected url…