Prevent encoding errors in Python

2024/10/14 4:27:30

I have scripts which print out messages by the logging system or sometimes print commands. On the Windows console I get error messages like

Traceback (most recent call last):File "C:\Python32\lib\logging\__init__.py", line 939, in emitstream.write(msg)File "C:\Python32\lib\encodings\cp850.py", line 19, in encodereturn codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2019' in position 4537:character maps to <undefined>

Is there a general way to make all encodings in the logging system, print commands, etc. fail-safe (ignore errors)?

Answer

The problem is that your terminal/shell (cmd as your are on Windows) cannot print every Unicode character.

You can fail-safe encode your strings with the errors argument of the str.encode method. For example you can replace not supported chars with ? by setting errors='replace'.

>>> s = u'\u2019'
>>> print s
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "C:\Python27\lib\encodings\cp850.py", line 12, in encodereturn codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can\'t encode character u'\u2019' in position0: character maps to <undefined>
>>> print s.encode('cp850', errors='replace')
?

See the documentation for other options.

Edit If you want a general solution for the logging, you can subclass StreamHandler:

class CustomStreamHandler(logging.StreamHandler):def emit(self, record):record = record.encode('cp850', errors='replace')logging.StreamHandler.emit(self, record)
https://en.xdnf.cn/q/69454.html

Related Q&A

How do I get the operating system name in a friendly manner using Python 2.5?

I tried:print os.nameAnd the output I got was::ntHowever, I want output more like "Windows 98", or "Linux".After suggestions in this question, I also tried:import os print os.name i…

Extend dataclass __repr__ programmatically

Suppose I have a dataclass with a set method. How do I extend the repr method so that it also updates whenever the set method is called: from dataclasses import dataclass @dataclass class State:A: int …

find least common denominator for list of fractions in python

I have a list of fractionsfrom fractions import Fractionfractions_list=[Fraction(3,14),Fraction(1,7),Fraction(9,14)]The output should be a list with the numerators for each fraction, then the denominat…

How to configure uwsgi to encode logging as json except app output

Im running uwsgi around a Python Flask webapp with these options (among others) to get JSON-encoded log records on stdout:fmt=${"timestamp": "${strftime:%FT%TZ}", "level":…

Testing aiohttp client with unittest.mock.patch

Ive written a simple HTTP client using aiohttp and Im trying to test it by patching aiohttp.ClientSession and aiohttp.ClientResponse. However, it appears as though the unittest.mock.patch decorator is …

GridsearchCV: cant pickle function error when trying to pass lambda in parameter

I have looked quite extensively on stackoverflow and elsewhere and I cant seem to find an answer to the problem below. I am trying to modify a parameter of a function that is itself a parameter inside …

How to insert a carriage return in a ReportLab paragraph?

Is there a way to insert a carriage return in a Paragraph in ReportLab? I am trying to concatenate a "\n" to my paragraph string but this isnt working. Title = Paragraph("Title" + …

How to get predictions and calculate accuracy for a given test set in fast ai?

Im trying to load a learner which was exported by learn.export() and I want to run it against a test set. I want my test set have labels so that I can measure its accuracy. This is my code: test_src = …

Splitting the legend in matploblib

Is it possible to split up a single big legend into multiple (usually 2) smaller ones.from pylab import *t = arange(0.0, 2.0, 0.01) s = sin(2*pi*t) plot(t, s, linewidth=1.0, label="Graph1") g…

Python 3.x - iloc throws error - single positional indexer is out-of-bounds

I am scraping election data from a website and trying to store it in a dataframe import pandas as pd import bs4 import requestscolumns = [Candidate,Party,Criminal Cases,Education,Age,Total Assets,Liabi…