I want to write data to files where a row from a CSV should look like this list (directly from the Python console):
row = ['\xef\xbb\xbft_11651497', 'http://kozbeszerzes.ceu.hu/entity/t/11651497.xml', "Szabolcs Mag '98 Kft.", 'ny\xc3\xadregyh\xc3\xa1za', 'ny\xc3\xadregyh\xc3\xa1za', '4400', 't\xc3\xbcnde utca 20.', 47.935175, 21.744975, u'Ny\xedregyh\xe1za', u'Borb\xe1nya', u'Szabolcs-Szatm\xe1r-Bereg', u'Ny\xedregyh\xe1zai', u'20', u'T\xfcnde utca', u'Magyarorsz\xe1g', u'4405']
Py2k does not do Unicode, but I had a UnicodeWriter wrapper:
import cStringIO, codecs
class UnicodeWriter:"""A CSV writer which will write rows to CSV file "f",which is encoded in the given encoding."""def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):# Redirect output to a queueself.queue = cStringIO.StringIO()self.writer = csv.writer(self.queue, dialect=dialect, **kwds)self.stream = fself.encoder = codecs.getincrementalencoder(encoding)()def writerow(self, row):self.writer.writerow([unicode(s).encode("utf-8") for s in row])# Fetch UTF-8 output from the queue ...data = self.queue.getvalue()data = data.decode("utf-8")# ... and reencode it into the target encodingdata = self.encoder.encode(data)# write to the target streamself.stream.write(data)# empty queueself.queue.truncate(0)def writerows(self, rows):for row in rows:self.writerow(row)
However, these lines still produce the dreaded encoding error message below:
f.write(codecs.BOM_UTF8)
writer = UnicodeWriter(f)
writer.writerow(row)UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 9: ordinal not in range(128)
What is there to do? Thanks!