Python 2.7.6 + unicode_literals - UnicodeDecodeError: ascii codec cant decode byte

2024/10/12 4:24:45

I'm trying to print the following unicode string but I'm receiving a UnicodeDecodeError: 'ascii' codec can't decode byte error. Can you please help form this query so it can print the unicode string properly?

>>> from __future__ import unicode_literals
>>> ts='now'
>>> free_form_request='[EXID(이엑스아이디)] 위아래 (UP&DOWN) MV'
>>> nick='me'>>> print('{ts}: free form request {free_form_request} requested from {nick}'.format(ts=ts,free_form_request=free_form_request.encode('utf-8'),nick=nick))Traceback (most recent call last):File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xec in position 6: ordinal not in range(128)

Thank you very much in advance!

Answer

Here's what happen when you construct this string:

'{ts}: free form request {free_form_request} requested from {nick}'.format(ts=ts,free_form_request=free_form_request.encode('utf-8'),nick=nick)
  1. free_form_request is encode-d into a byte string using utf-8 as the encoding. This works because utf-8 can represent [EXID(이엑스아이디)] 위아래 (UP&DOWN) MV.
  2. However, the format string ('{ts}: free form request {free_form_request} requested from {nick}') is a unicode string (because of imported from __future__ import unicode_literals).
  3. You can't use byte strings as format arguments for a unicode string, so Python attempts to decode the byte string created in 1. to create a unicode string (which would be valid as an format argument).
  4. Python attempts the decode-ing using the default encoding, which is ascii, and fails, because the byte string is a utf-8 byte string that includes byte values that don't make sense in ascii.
  5. Python throws a UnicodeDecodeError.

Note that while the code is obviously doing something here, this would actually not throw an exception on Python 3, which would instead substitute the repr of the byte string (the repr being a unicode string).


To fix your issue, just pass unicode strings to format.

That is, don't do step 1. where you encoded free_form_request as a byte string: keep it as a unicode string by removing .encode(...):

'{ts}: free form request {free_form_request} requested from {nick}'.format(ts=ts, free_form_request=free_form_request, nick=nick)

Note Padraic Cunningham's answer in the comments as well.

https://en.xdnf.cn/q/118236.html

Related Q&A

Retrieving data from Quandl with Python

How can I get the latest prices from a Quandl dataset with the Python API (https://www.quandl.com/help/python)? On https://www.quandl.com/help/api, it says "You can use rows=n to get only the fir…

Django: Using same object how to show 2 different results in django template?

Using the same object how to SHOW 2 different results using django template ?In one page there are two divs, it should show different information using the same object.INPUTobject data has follows[{&q…

Override attribute access precedence having a data descriptor

I have a bunch of instances of a MongoEngine model. And the profiler shows that a lot of time is spent in __get__ method of MongoEngine model fields:ncalls tottime percall cumtime percall filename:…

Understanding pythons reverse slice ( [::-1] )

I always thought that omitting arguments in the python slice operation would result into:start = 0 end = len(lst) step = 1That holds true if the step is positive, but as soon as the step is negative, l…

How to print list elements (which are also lists) in separated lines in Python

Ive checked the post and answers on the SO post Printing list elements on separated lines in Python, while I think my problem is a different one.What I want is to transform:lsts = [[1], [1, 1], [1, 2, …

Python3 threading, trying to ping multiple IPs/test port simultaineously

Full (non-working) code belowFull (working, w/o threading) code here: http://pastebin.com/KUYzNtT2Ive written a small script that does the following:Pull network information from a database Ping each I…

How to print a list of numbers without square brackets?

Im generating a list of random digits, but Im struggling to figure out how to output the digits in a single row without the square brackets?import random def generateAnswer(answerList):listofDigits =…

How to save plotly offline by running my script

I am using below code in my jupyter notebook.import pandas as pd import numpy as np %matplotlib inlinefrom plotly import __version__ from plotly.offline import download_plotlyjs, init_notebook_mode, pl…

Same output of the Keras model

I have a Keras model for predicting moves in the game. I have an input shape of (160,120 ,1). I have the following model with an output of 9 nodes:from keras.models import Sequential from keras.layers.…

Extract common element from 2 tuples python [duplicate]

This question already has answers here:Find intersection of two nested lists?(21 answers)Is there a way to get the difference and intersection of tuples or lists in Python? [duplicate](1 answer)Close…