UTF-8 error with Python and gettext

2024/4/15 2:16:13

I use UTF-8 in my editor, so all strings displayed here are UTF-8 in file.

I have a python script like this:

# -*- coding: utf-8 -*-
parser = optparse.OptionParser(description=_('automates the dice rolling in the classic game "risk"'), usage=_("usage: %prog attacking defending"))

Then I used xgettext to get everything out and got a .pot file which can be boiled down to:

"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: 8bit\n"#: auto_dice.py:16
msgid "automates the dice rolling in the classic game \"risk\""
msgstr ""

After that, I used msginit to get a de.po which I filled in like this:

"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"#: auto_dice.py:16
msgid "automates the dice rolling in the classic game \"risk\""
msgstr "automatisiert das Würfeln bei \"Risiko\""

Running the script, I get the following error:

  File "/usr/lib/python2.6/optparse.py", line 1664, in print_helpfile.write(self.format_help().encode(encoding, "replace"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 60: ordinal not in range(128)

How can I fix that?


That error means you've called encode on a bytestring, so it tries to decode it to Unicode using the system default encoding (ascii on Python 2), then re-encode it with whatever you've specified.

Generally, the way to resolve it is to call s.decode('utf-8') (or whatever encoding the strings are in) before trying to use the strings. It might also work if you just use unicode literals: u'automates...' (that depends on how strings are substituted from .po files, which I don't know about).

This sort of confusing behaviour is improved in Python 3, which won't try to convert bytes to unicode unless you specifically tell it to.


Related Q&A

Add build information in Jenkins using REST

Does anyone know how to add build information to an existing Jenkins build? What Im trying to do is replace the #1 build number with the actual full version number that the build represents. I can do …

Combining element-wise and matrix multiplication with multi-dimensional arrays in NumPy

I have two multidimensional NumPy arrays, A and B, with A.shape = (K, d, N) and B.shape = (K, N, d). I would like to perform an element-wise operation over axis 0 (K), with that operation being matrix …

Target array shape different to expected output using Tensorflow

Im trying to make a CNN (still a beginner). When trying to fit the model I am getting this error:ValueError: A target array with shape (10000, 10) was passed for output of shape (None, 6, 6, 10) while …

Using openpyxl to refresh pivot tables in Excle

I have a file that has several tabs that have pivot tables that are based on one data tab. I am able to write the data to the data tab without issue, but I cant figure out how to get all of the tabs wi…

python: is there a library function for chunking an input stream?

I want to chunk an input stream for batch processing. Given an input list or generator,x_in = [1, 2, 3, 4, 5, 6 ...]I want a function that will return chunks of that input. Say, if chunk_size=4, then,x…

jinja2: How to make it fail Silently like djangotemplate

Well i dont find the answer Im sure that its very simple, but i just dont find out how to make it work like Django when it doesnt find a variablei tried to use Undefined and create my own undefined but…

ImportError when from transformers import BertTokenizer

My code is: import torch from transformers import BertTokenizer from IPython.display import clear_outputI got error in line from transformers import BertTokenizer: ImportError: /lib/x86_64-linux-gnu/li…

How to get feature names of shap_values from TreeExplainer?

I am doing a shap tutorial, and attempting to get the shap values for each person in a dataset from sklearn.model_selection import train_test_split import xgboost import shap import numpy as np import …

How can I clear a line in console after using \r and printing some text?

For my current project, there are some pieces of code that are slow and which I cant make faster. To get some feedback how much was done / has to be done, Ive created a progress snippet which you can s…

installing pyaudio to docker container

I am trying to install pyaudio to my docker container and I was wondering if anyone had any solution for Windows. I have tried two methods: Method 1: Using pipwin - Error Code: => [3/7] RUN pip inst…