Navigating movielist repeatable crash

[alphacabbage1]

Hi,

The box repeatedly crashes when navigating the movielist via the remote - e.g., holding down the up or down keys will scroll about 20 items before a crash & reboot. Similar with left and right key navigation.


< 26347.7467> 18:17:23.6115 [InfoBarGenerics] Key: 108 (Repeat) KeyID='KEY_DOWN' Binding='('DOWN',)'.
< 26347.7478> 18:17:23.6126 [ActionMap] Keymap 'DirectionActions' -> Action = 'down'.
< 26347.7609> 18:17:23.6257 PC: 7626aa78
< 26347.7610> 18:17:23.6258 00000000 76223afc 00000000 75dca010
< 26347.7611> 18:17:23.6259 00004d61 00004d61 00000006 00000000
< 26347.7612> 18:17:23.6260 b8b1aabc 00028934 00000001 ffffffff
< 26347.7612> 18:17:23.6260 764660f4 763c4c6c 00000000 763ac148
< 26347.7612> 18:17:23.6260 00004d61 76386000 76386000 7fd1dc68
< 26347.7613> 18:17:23.6261 00000000 00000018 0000005d 7fd1dd80
< 26347.7613> 18:17:23.6261 00000000 762a7570 00000000 00000000
< 26347.7613> 18:17:23.6261 76388570 7fd1d938 00000000 7626aa64
< 26347.7617> 18:17:23.6265 Backtrace:
< 26347.7623> 18:17:23.6271 /usr/bin/enigma2(_Z17handleFatalSignaliP9siginfo_tPv) [0x46B618]
< 26347.7628> 18:17:23.6276 /lib/libc.so.6(n/a) [0x7626AA78]
< 26347.7631> 18:17:23.6280 /lib/libc.so.6(raise) [0x76223AFC]
< 26347.7634> 18:17:23.6282 /lib/libc.so.6(abort) [0x7620E558]
< 26347.7635> 18:17:23.6283 -------FATAL SIGNAL (6)


Hope this is useful but I probably wont be able to test for a day or two...

Cheers,
Martin

[ccs]

Looks ok on 6.0.004.001 (Dev), there have been movielist commits recently, maybe try 6.0.003 which has just been released.

[Huevos]

Yes, I can't reproduce this either.

[adm]

Yes, I can't reproduce this either.

Nor can I reproduce it on 6.0.003

[twol]

This is a C++/driver error ————- and I cannot reproduce it on my Giga4K.
Would be interested if anybody else can reproduce it

[alphacabbage1]

Thanks for the replies. I just ran a quick test -- it's the same repeatable behaviour under 6.0.003 but it looks directory specific. The directory that always crashes is probably the largest with 18 subdirectories and 1,408 files. Trying a different directory with 20 folders + 467 files I can hold the down-arrow and it scrolls forever without crashing. (Scrolling down from the top of the problem directory shows four screens (the 18 directories and 41 recordings) before the lock-up).

[alphacabbage1]

It's looking more file than directory specific now. So far, I've only found two directories that show the problem:

/media/hdd/TV/Film
/media/hdd/TV/Film/_World/European/FR

The latter isn't particularly heavily populated so it's not looking like a size thing. In both cases the crash happens at a repeatable point in the list -- whether approaching from above (via down-arrow) or below (up-arrow). When there's more time, I'll try copying the smaller directory to a temp location and knocking out files/recordings one-by-one until the crash doesn't happen.

Meanwhile, OpenWebif sits and spins indefinitely when trying to parse /media/hdd/TV/Film and the debug logs shows:



< 12189.1465> 15:57:11.4852 [eEPGChannelData] schedule finished(1637683031)
< 12203.1056> 15:57:25.4442 [gRC] main thread is non-idle! display spinner!
< 12210.4475> 15:57:32.7861
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 257, in render
request.write(six.ensure_binary(out))
File "/usr/lib/python3.9/site-packages/six.py", line 912, in ensure_binary
return s.encode(encoding, errors)
builtins.UnicodeEncodeError: 'utf-8' codec can't encode character 'udcc8' in position 76222: surrogates not allowed

< 12223.3524> 15:57:45.6910 [eBackgroundFileEraser] deleting '/media/hdd/timeshift/pts_livebuffer_3.sc.del'


I'll keep plugging away but it may be a while.

[twol]

It's looking more file than directory specific now. So far, I've only found two directories that show the problem:

/media/hdd/TV/Film
/media/hdd/TV/Film/_World/European/FR

The latter isn't particularly heavily populated so it's not looking like a size thing. In both cases the crash happens at a repeatable point in the list -- whether approaching from above (via down-arrow) or below (up-arrow). When there's more time, I'll try copying the smaller directory to a temp location and knocking out files/recordings one-by-one until the crash doesn't happen.

Meanwhile, OpenWebif sits and spins indefinitely when trying to parse /media/hdd/TV/Film and the debug logs shows:



< 12189.1465> 15:57:11.4852 [eEPGChannelData] schedule finished(1637683031)
< 12203.1056> 15:57:25.4442 [gRC] main thread is non-idle! display spinner!
< 12210.4475> 15:57:32.7861
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 257, in render
request.write(six.ensure_binary(out))
File "/usr/lib/python3.9/site-packages/six.py", line 912, in ensure_binary
return s.encode(encoding, errors)
builtins.UnicodeEncodeError: 'utf-8' codec can't encode character 'udcc8' in position 76222: surrogates not allowed

< 12223.3524> 15:57:45.6910 [eBackgroundFileEraser] deleting '/media/hdd/timeshift/pts_livebuffer_3.sc.del'


I'll keep plugging away but it may be a while.
I am not at my PC, so if you want/can … try this
Use filezilla, copy module below to PC and change line 257 to request.write(out) and copy back & reboot ! if not I will do it tomorrow
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 257, in render
request.write(six.ensure_binary(out))

[alphacabbage1]

change line 257 to request.write(out)
Thanks for the pointer. I've given it a spin (being careful with whitespace/indentation) and OpenWebif returns "Processing Failed"...


Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 257, in render
request.write(out)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 269, in write
http.Request.write(self, data)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1229, in write
self.channel.writeSequence(toChunk(data))
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2580, in writeSequence
self.transport.writeSequence(iovec)
File "/usr/lib/python3.9/site-packages/twisted/internet/_newtls.py", line 192, in writeSequence
FileDescriptor.writeSequence(self, iovec)
File "/usr/lib/python3.9/site-packages/twisted/internet/abstract.py", line 381, in writeSequence
_dataMustBeBytes(i)
File "/usr/lib/python3.9/site-packages/twisted/internet/abstract.py", line 23, in _dataMustBeBytes
raise TypeError("Data must be bytes")
builtins.TypeError: Data must be bytes

[twol]

Thanks for the pointer. I've given it a spin (being careful with whitespace/indentation) and OpenWebif returns "Processing Failed"...


Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 257, in render
request.write(out)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 269, in write
http.Request.write(self, data)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1229, in write
self.channel.writeSequence(toChunk(data))
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2580, in writeSequence
self.transport.writeSequence(iovec)
File "/usr/lib/python3.9/site-packages/twisted/internet/_newtls.py", line 192, in writeSequence
FileDescriptor.writeSequence(self, iovec)
File "/usr/lib/python3.9/site-packages/twisted/internet/abstract.py", line 381, in writeSequence
_dataMustBeBytes(i)
File "/usr/lib/python3.9/site-packages/twisted/internet/abstract.py", line 23, in _dataMustBeBytes
raise TypeError("Data must be bytes")
builtins.TypeError: Data must be bytes

OK makes sense - python3 at its best! probably latin1 and an umlaut .. or such symbol
can you try: request.write(out.encode('utf-16', 'surrogatepass').decode('utf-16').encode('utf-8'))

[alphacabbage1]

OK makes sense - python3 at its best! probably latin1 and an umlaut .. or such symbol
can you try: request.write(out.encode('utf-16', 'surrogatepass').decode('utf-16').encode('utf-8'))
Getting closer! The movielist root now renders but trying to get to the first problem directory sets off the spinner with...


Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 257, in render
request.write(out.encode('utf-16', 'surrogatepass').decode('utf-16').encode('utf-8'))
builtins.UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 152446-152447: illegal encoding

[twol]

Getting closer! The movielist root now renders but trying to get to the first problem directory sets off the spinner with...


Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 257, in render
request.write(out.encode('utf-16', 'surrogatepass').decode('utf-16').encode('utf-8'))
builtins.UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 152446-152447: illegal encoding
Need to think about it - really doesn't like the text - will have to import the module, change it and then post back to you - not straight forward
So try this...

[alphacabbage1]

So try this...
Thanks again! I removed the bracket typo on 260 but it's back to "Processing Failed".


Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 258, in render
encoder = chardet.detect(out)["encoding"]
File "/usr/lib/python3.9/site-packages/chardet/__init__.py", line 36, in detect
raise TypeError('Expected object of type bytes or bytearray, got: '
builtins.TypeError: Expected object of type bytes or bytearray, got: <class 'str'>


I think I've found the culprit: /mnt/hdd/TV/Film/_World/European/FR/Cafe_de_Flore_140824.meta


-1:41:0:0:0:0:0:0:0:0:
CafÂe de Flore
Romantic drama connecting a single mother in 1960s Paris and a contemporary DJ in Montreal. In French with English subtitles. Contains strong language and some sexual content. [2011] [S]
1408914300

663914253
2438639616
f:0
188
0
This was one of many imports from a Topfield box and it's the only .meta showing ansi rather than utf-8 encoding (under notepad++). Only tested via OpenWebif remote but cycling through the folder without it works Ok. Entering a directory where it's the sole occupant triggers the crash...



< 7033.0016> 16:42:53.3272 [InfoBarGenerics] Key: 352 (Break) KeyID='KEY_OK' Binding='('OK',)'.
< 7033.0771> 16:42:53.4027 [Trashcan] Debug path /media/hdd/TV/Film/_World/European/FR/TEST/ => /media/hdd/TV/.Trash
< 7033.1023> 16:42:53.4279 [Pixmap] setPixmapNum(1) failed! defined pixmaps: []
< 7033.1033> 16:42:53.4289 [Pixmap] setPixmapNum(7) failed! defined pixmaps: []
< 7033.1326> 16:42:53.4582 PC: 75f63a78
< 7033.1327> 16:42:53.4583 00000000 75f1cafc 00000000 75ac3010
< 7033.1328> 16:42:53.4583 0000039b 0000039b 00000006 00000000
< 7033.1328> 16:42:53.4584 b8b1aabc 00028934 00000001 ffffffff
< 7033.1329> 16:42:53.4585 7615f0f4 760bdc6c 00000000 760a5148
< 7033.1329> 16:42:53.4585 0000039b 7607f000 7607f000 7fdf9fd8
< 7033.1330> 16:42:53.4586 00000000 00000018 0000001f 7fdfa0f0
< 7033.1331> 16:42:53.4587 00000000 75fa0570 00000000 00000000
< 7033.1331> 16:42:53.4587 76081570 7fdf9ca8 00000000 75f63a64
< 7033.1343> 16:42:53.4599 Backtrace:
< 7033.1349> 16:42:53.4606 /usr/bin/enigma2(_Z17handleFatalSignaliP9siginfo_tPv) [0x46B618]
< 7033.1363> 16:42:53.4619 /lib/libc.so.6(n/a) [0x75F63A78]
< 7033.1366> 16:42:53.4622 /lib/libc.so.6(raise) [0x75F1CAFC]
< 7033.1369> 16:42:53.4625 /lib/libc.so.6(abort) [0x75F07558]
< 7033.1370> 16:42:53.4626 -------FATAL SIGNAL (6)


I'm trying to find something equivalent under /mnt/hdd/TV/Film/ but it's not showing itself, yet...

[twol]

Thanks again! I removed the bracket typo on 260 but it's back to "Processing Failed".


Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/twisted/protocols/basic.py", line 548, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2243, in lineReceived
self.allContentReceived()
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 2369, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3.9/site-packages/twisted/web/http.py", line 1003, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 229, in process
self.render(resrc)
File "/usr/lib/python3.9/site-packages/twisted/web/server.py", line 294, in render
body = resrc.render(self)
File "/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/base.py", line 258, in render
encoder = chardet.detect(out)["encoding"]
File "/usr/lib/python3.9/site-packages/chardet/__init__.py", line 36, in detect
raise TypeError('Expected object of type bytes or bytearray, got: '
builtins.TypeError: Expected object of type bytes or bytearray, got: <class 'str'>


I think I've found the culprit: /mnt/hdd/TV/Film/_World/European/FR/Cafe_de_Flore_140824.meta


-1:41:0:0:0:0:0:0:0:0:
CafÂe de Flore
Romantic drama connecting a single mother in 1960s Paris and a contemporary DJ in Montreal. In French with English subtitles. Contains strong language and some sexual content. [2011] [S]
1408914300

663914253
2438639616
f:0
188
0
This was one of many imports from a Topfield box and it's the only .meta showing ansi rather than utf-8 encoding (under notepad++). Only tested via OpenWebif remote but cycling through the folder without it works Ok. Entering a directory where it's the sole occupant triggers the crash...



< 7033.0016> 16:42:53.3272 [InfoBarGenerics] Key: 352 (Break) KeyID='KEY_OK' Binding='('OK',)'.
< 7033.0771> 16:42:53.4027 [Trashcan] Debug path /media/hdd/TV/Film/_World/European/FR/TEST/ => /media/hdd/TV/.Trash
< 7033.1023> 16:42:53.4279 [Pixmap] setPixmapNum(1) failed! defined pixmaps: []
< 7033.1033> 16:42:53.4289 [Pixmap] setPixmapNum(7) failed! defined pixmaps: []
< 7033.1326> 16:42:53.4582 PC: 75f63a78
< 7033.1327> 16:42:53.4583 00000000 75f1cafc 00000000 75ac3010
< 7033.1328> 16:42:53.4583 0000039b 0000039b 00000006 00000000
< 7033.1328> 16:42:53.4584 b8b1aabc 00028934 00000001 ffffffff
< 7033.1329> 16:42:53.4585 7615f0f4 760bdc6c 00000000 760a5148
< 7033.1329> 16:42:53.4585 0000039b 7607f000 7607f000 7fdf9fd8
< 7033.1330> 16:42:53.4586 00000000 00000018 0000001f 7fdfa0f0
< 7033.1331> 16:42:53.4587 00000000 75fa0570 00000000 00000000
< 7033.1331> 16:42:53.4587 76081570 7fdf9ca8 00000000 75f63a64
< 7033.1343> 16:42:53.4599 Backtrace:
< 7033.1349> 16:42:53.4606 /usr/bin/enigma2(_Z17handleFatalSignaliP9siginfo_tPv) [0x46B618]
< 7033.1363> 16:42:53.4619 /lib/libc.so.6(n/a) [0x75F63A78]
< 7033.1366> 16:42:53.4622 /lib/libc.so.6(raise) [0x75F1CAFC]
< 7033.1369> 16:42:53.4625 /lib/libc.so.6(abort) [0x75F07558]
< 7033.1370> 16:42:53.4626 -------FATAL SIGNAL (6)


I'm trying to find something equivalent under /mnt/hdd/TV/Film/ but it's not showing itself, yet...

Far easier to fix the film!

[alphacabbage1]

Far easier to fix the film!
:) I think I've sorted everything under the Films directory - around 15 .meta with ansi encoding, all Topfield imports. I'm not sure why these few and not others were ansi-encoded but they're now converted and character-corrected. There's bound to be more under Drama/* but I'll search some other time.

So, I guess python3 is stricter about the encoding and falls over in protest where ye olde python2 marched on.

Anyway, thanks again for the help.

[birdman]

So, I guess python3 is stricter about the encoding and falls over in protest where ye olde python2 marched on.Whereas that is true, you weren't failing in Py3 - you were failing somewhere in C++ code.

[alphacabbage1]

you weren't failing in Py3 - you were failing somewhere in C++ code.
I feel I'm moving from the unknown to the unknowable. ;) I've never really understood the ecosystem; where OpenVix begins and ends, what it's made of. It's seems amazing that it hangs together so well - or works at all. :) I know there's a huge effort behind it. ThX all.