Crash when folder contains filenames with Windows encoding

**ocean** · 05-09-22, 20:06

Crash happens in elistboxcontent.cpp, line 920

Code:

const char *string = (PyUnicode_Check(pstring)) ? PyUnicode_AsUTF8(pstring) : "<not-a-string>";

"pstring" contains escaped characters and PyUnicode_AsUTF8(pstring) fails. Changed it to this bellow and it doesn't crash anymore. (Note: there are atleast 5 similar checks in this file, possibly they all should be fixed same way)

Code:

const char *string = (PyUnicode_Check(pstring)) ? PyBytes_AsString(PyUnicode_AsEncodedString(pstring, "utf-8", "surrogateescape")) : "<not-a-string>";

But that only fixes the crash. GUI expects UTF-8. Movielist change "decode(encoding)" is still needed to display latin-1 characters correctly.

**Huevos** · 05-09-22, 20:38

How do you trigger this crash?

Are the current changes in enigma repo of service.cpp working for you?

**twol** · 05-09-22, 20:45

Originally Posted by ocean

Crash happens in elistboxcontent.cpp, line 920

Code:

const char *string = (PyUnicode_Check(pstring)) ? PyUnicode_AsUTF8(pstring) : "<not-a-string>";

"pstring" contains escaped characters and PyUnicode_AsUTF8(pstring) fails. Changed it to this bellow and it doesn't crash anymore. (Note: there are atleast 5 similar checks in this file, possibly they all should be fixed same way)

Code:

const char *string = (PyUnicode_Check(pstring)) ? PyBytes_AsString(PyUnicode_AsEncodedString(pstring, "utf-8", "surrogateescape")) : "<not-a-string>";

But that only fixes the crash. GUI expects UTF-8. Movielist change "decode(encoding)" is still needed to display latin-1 characters correctly.

Is this crash still from movielist or another location?
if another issue how did you get there

**ocean** · 05-09-22, 21:50

Originally Posted by Huevos

How do you trigger this crash?

That is the original crash without any changes to movielist. (You asked to test with old movielist)

Originally Posted by Huevos

Are the current changes in enigma repo of service.cpp working for you?

Yes, service.cpp changes are working fine.

**Huevos** · 06-09-22, 01:07

@ocean, what are you using to FTP the file to the STB?

Is it possible that FTP is interfering with the filename. Windows is supposed to save filenames as UTF16.

**ocean** · 06-09-22, 05:17

Originally Posted by Huevos

Is it possible that FTP is interfering with the filename. Windows is supposed to save filenames as UTF16.

No, it's not only FTP issue. If files are saved to USB drive in Windows. When you plug that USB drive to box, it has same issues.

But with current changes in repo it's already working. Just include latest movielist changes I posted earlier, then it doesn't affect anything in movieselection.py

**birdman** · 06-09-22, 11:10

Originally Posted by ocean

But that only fixes the crash. GUI expects UTF-8. Movielist change "decode(encoding)" is still needed to display latin-1 characters correctly.

Linux filenames have no concept of Latin-1 (or of iso8859-1, Windows-1252, utf8 etc.) They are just an array of bytes.
The filename has to be kept (and passed around) as bytes while any display has to use a utf8 representation of this.

**Huevos** · 06-09-22, 11:21

Originally Posted by birdman

Linux filenames have no concept of Latin-1 (or of iso8859-1, Windows-1252, utf8 etc.) They are just an array of bytes.
The filename has to be kept (and passed around) as bytes while any display has to use a utf8 representation of this.

Gordon, we are talking about GUI display. We have to feed something to the GUI that it understands.

**birdman** · 07-09-22, 01:22

Originally Posted by Huevos

Gordon, we are talking about GUI display. We have to feed something to the GUI that it understands.

Yes. But you also need to retain the original name in bytes in order to pass it to the filesystem.
If you start with a filename you should only ever need to decode it (in some lossless way) to produce a display value.
If you start with a display string (such as setting up a timer/Autotimer) then you only ever need to encode it to get a filename.

So if you ever find yourself encoding and decoding the same value then something has gone wrong in the logic.

**Huevos** · 07-09-22, 12:49

The current code works so far but open to any improvements you suggest.

**ocean** · 07-09-22, 13:13

@Huevos Could you still add some fix for RedirectOutput.py? There are debug prints, one in navigation.py, line 110, print("[Navigation] playing ref", ref and ref.toString())

If you don't like try/except, maybe change this:

Code:

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")

To this in RedirectOutput.py?

Code:

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")
		else:
			data = data.encode("UTF-8", "ignore").decode()

Everything else seems working for me.

**Huevos** · 08-09-22, 15:42

Originally Posted by ocean

@Huevos Could you still add some fix for RedirectOutput.py? There are debug prints, one in navigation.py, line 110, print("[Navigation] playing ref", ref and ref.toString())

If you don't like try/except, maybe change this:

Code:

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")

To this in RedirectOutput.py?

Code:

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")
		else:
			data = data.encode("UTF-8", "ignore").decode()

Everything else seems working for me.

That crash was caused by something nasty in RecordTimer.py...

Code:

< 4037.8571> 17:38:38.4681 Traceback (most recent call last):
< 4037.8572> 17:38:38.4681 File "/usr/lib/enigma2/python/StartEnigma.py", line 224, in processDelay
< 4037.8581> 17:38:38.4690 callback(*retval)
< 4037.8581> 17:38:38.4691 File "/usr/lib/enigma2/python/Screens/InfoBarGenerics.py", line 3333, in recordQuestionCallback
< 4037.8585> 17:38:38.4694 File "/usr/lib/enigma2/python/Screens/InfoBarGenerics.py", line 3253, in startInstantRecording
< 4037.8588> 17:38:38.4697 File "/usr/lib/enigma2/python/RecordTimer.py", line 1275, in record
< 4037.8591> 17:38:38.4701 File "/usr/lib/enigma2/python/timer.py", line 229, in addTimerEntry
< 4037.8594> 17:38:38.4704 File "/usr/lib/enigma2/python/timer.py", line 268, in calcNextActivation
< 4037.8597> 17:38:38.4706 File "/usr/lib/enigma2/python/timer.py", line 363, in processActivation
< 4037.8601> 17:38:38.4710 File "/usr/lib/enigma2/python/RecordTimer.py", line 1009, in doActivate
< 4037.8607> 17:38:38.4717 File "/usr/lib/enigma2/python/RecordTimer.py", line 676, in activate
< 4037.8611> 17:38:38.4720 File "/usr/lib/enigma2/python/RecordTimer.py", line 485, in log_tuner
< 4037.8614> 17:38:38.4723 File "/usr/lib/enigma2/python/RecordTimer.py", line 284, in log
< 4037.8617> 17:38:38.4726 File "/usr/lib/enigma2/python/Tools/RedirectOutput.py", line 16, in write
< 4037.8620> 17:38:38.4729 TypeError: in method 'ePythonOutput', argument 1 of type 'char const *

What exactly is that bad output?

**Huevos** · 08-09-22, 15:54

Python:

Code:

ePythonOutput(self.line, self.level)

C++:

Code:

void ePythonOutput(const char *, int lvl = lvlDebug);

const char * ... shouldn't we be feeding that bytes?

**twol** · 08-09-22, 17:42

Originally Posted by Huevos

Python:

Code:

ePythonOutput(self.line, self.level)

C++:

Code:

void ePythonOutput(const char *, int lvl = lvlDebug);

const char * ... shouldn't we be feeding that bytes?

Thats in eerror.h but its also in eerror.cpp

Code:

 void ePythonOutput(const char *string, int lvl)

Which confuses me - why in both

**ocean** · 08-09-22, 17:54

Originally Posted by Huevos

... shouldn't we be feeding that bytes?

It's possible to change that to take bytes. But do you want log files to be other than UTF-8? In receiver menu there is option to view log, it most likely crashes if log is not UTF-8

View Entry Info: Crash when folder contains filenames with Windows encoding

Thread: Crash when folder contains filenames with Windows encoding

Thread Tools

Display

The Following User Says Thank You to Huevos For This Useful Post:

Posting Permissions

Options

About

Site Links

Social Media