AutoTimer error when it runs.

[birdman]

I agree with ccs. Something is wrong with the encoding.

So, going back to my previous point, it would make sense(?) if everything putting text into the epg cache put it in as utf-8 (ignoring errors on encoding?).
That way everything retrieving data would always get utf-8 and always work.

[Migu]

I agree with ccs. Something is wrong with the encoding. It is a pity you didn't show us a screenshot from the 6.0 image of the same problem.

Unfortunately I cannot take a screenshot from the 6.0 image of the same problem right now, because currently there is no problematic events in any of channels.

The description for it was just empty in the screen. It was like in the below screenshot, which I have edited manually:
screen_snapshot_fw_6_0_006_edited.jpg
Could it be so that truncating UTF-8 encoded Unicode text in the end may cut in the middle of a UTF-8 encoded code-point and cause corruption?

[Huevos]

So, going back to my previous point, it would make sense(?) if everything putting text into the epg cache put it in as utf-8 (ignoring errors on encoding?).
That way everything retrieving data would always get utf-8 and always work.

If you start with an incorrectly encoded string and convert it to UTF8 the output will be wrong. That is what encoding.conf is for. And even then it still relies on the input maintaining the encoding correctly.

[ccs]

I was seeing, for example, Benalmádena until I changed the encoding table to ISO6937, when the correct text Benalmádena was stored.

@Joe_90 worked out what the encoding should have been, although I don't know how.

[Joe_90]

I was seeing, for example, Benalmádena until I changed the encoding table to ISO6937, when the correct text Benalmádena was stored.

@Joe_90 worked out what the encoding should have been, although I don't know how.

IIRC - because ISO6937 was specifically mentioned in the Freeview/Freesat D Book as one of the acceptable encodings.


EDIT - here's the reference in "D" book:

F.1 Introduction
Table F-1 lists the displayable characters that shall be implemented by all
decoders conforming to this specification.
Note that the UK MHEG-5 profile specifies several characters that are not
defined within ISO/IEC 6937 [42] on which the DVB SI specifications are
based. These characters may be broadcast in, and shall be displayed
correctly within MHEG-5 applications; there is no guarantee that they will be
displayed correctly within SI delivered text. The characters in the shaded
cells are shown only to emphasise gaps.

It goes on to list UCS-2, UTF-8 and ISO 6937 encodings and the name for each character.

[ccs]

The description is long, but not the longest one. The problematic event description, which does not work:

Naisleijonien kesäleiri 2020 oli Anni Keisalalle isku vasten kasvoja - ja tarpeellinen sellainen. Nuorena hän pärjäsi lahjoillaan, mutta enää se ei riittänyt. Hänen oli otettava itseään niskasta kiinni. Keisala teki käänteentekevän pä
Char count: 234
Byte count: 248

This is what the description probably should look like (6 extra characters at the end).....

Naisleijonien kesäleiri 2020 oli Anni Keisalalle isku vasten kasvoja - ja tarpeellinen sellainen. Nuorena hän pärjäsi lahjoillaan, mutta enää se ei riittänyt. Hänen oli otettava itseään niskasta kiinni. Keisala teki käänteentekevän päätökse

[birdman]

If you start with an incorrectly encoded string and convert it to UTF8 the output will be wrong.

That's not what I'm trying to say.
What I meant was that everything in the EPG cache should be utf-8 (whether it's the "correct" text is another matter).
That way you won't get an error when searching/extracting from it.

So anything that updates the EPG cache should only put valid utf-8 into it, which appears to not be the case currently.

[ccs]

root@vuultimo4k:~# echo Naisleijonien kesäleiri 2020 oli Anni Keisalalle isku vasten kasvoja - ja tarpeellinen sellainen. Nuorena hän pärjäsi lahjoillaan, mutta enää se ei riittänyt. Hänen oli otettava itseään niskasta kiinni. Keisala teki käänteentekevän päätökse|od -h
0000000     614e    7369    656c    6a69    6e6f    6569    206e    656b
0000020     c373    6ca4    6965    6972    3220    3230    2030    6c6f
0000040     2069    6e41    696e    4b20    6965    6173    616c    6c6c
0000060     2065    7369    756b    7620    7361    6574    206e    616b
0000100     7673    6a6f    2061    202d    616a    7420    7261    6570
0000120     6c65    696c    656e    206e    6573    6c6c    6961    656e
0000140     2e6e    4e20    6f75    6572    616e    6820    a4c3    206e
0000160     c370    72a4    c36a    73a4    2069    616c    6a68    696f
0000200     6c6c    6161    2c6e    6d20    7475    6174    6520    c36e
0000220     c3a4    20a4    6573    6520    2069    6972    7469    c374
0000240     6ea4    7479    202e    c348    6ea4    6e65    6f20    696c
0000260     6f20    6574    7474    7661    2061    7469    6573    a4c3
0000300     a4c3    206e    696e    6b73    7361    6174    6b20    6969
0000320     6e6e    2e69    4b20    6965    6173    616c    7420    6b65
0000340     2069    c36b    c3a4    6ea4    6574    6e65    6574    656b
0000360     c376    6ea4    7020    a4c3    a4c3    c374    6bb6    6573
0000400     000a
0000401
root@vuultimo4k:~#


root@vuultimo4k:~# echo Naisleijonien kesäleiri 2020 oli Anni Keisalalle isku vasten kasvoja - ja tarpeellinen sellainen. Nuorena hän pärjäsi lahjoillaan, mutta enää se ei riittänyt. Hänen oli otettava itseään niskasta kiinni. Keisala teki käänteentekevän päätökse|od -c
0000000    N   a   i   s   l   e   i   j   o   n   i   e   n       k   e
0000020    s 303 244   l   e   i   r   i       2   0   2   0       o   l
0000040    i       A   n   n   i       K   e   i   s   a   l   a   l   l
0000060    e       i   s   k   u       v   a   s   t   e   n       k   a
0000100    s   v   o   j   a       -       j   a       t   a   r   p   e
0000120    e   l   l   i   n   e   n       s   e   l   l   a   i   n   e
0000140    n   .       N   u   o   r   e   n   a       h 303 244   n
0000160    p 303 244   r   j 303 244   s   i       l   a   h   j   o   i
0000200    l   l   a   a   n   ,       m   u   t   t   a       e   n 303
0000220  244 303 244       s   e       e   i       r   i   i   t   t 303
0000240  244   n   y   t   .       H 303 244   n   e   n       o   l   i
0000260        o   t   e   t   t   a   v   a       i   t   s   e 303 244
0000300  303 244   n       n   i   s   k   a   s   t   a       k   i   i
0000320    n   n   i   .       K   e   i   s   a   l   a       t   e   k
0000340    i       k 303 244 303 244   n   t   e   e   n   t   e   k   e
0000360    v 303 244   n       p 303 244 303 244   t 303 266   k   s   e
0000400   \n
0000401
root@vuultimo4k:~#

root@vuultimo4k:~# echo Naisleijonien kesäleiri 2020 oli Anni Keisalalle isku vasten kasvoja - ja tarpeellinen sellainen. Nuorena hän pärjäsi lahjoillaan, mutta enää se ei riittänyt. Hänen oli otettava itseään niskasta kiinni. Keisala teki käänteentekevän päätökse|wc
        1        32       257

[Mikou]

I have noticed now this same issue with some Finnish channels. For some reason there is something wrong at EPG descriptions and autotimer fails. But there is also an issue with recording; if I manually add timer, the system will keep rebooting during while recording the program. Do You have any known fixes or workarounds for this issue?

[twol]

I have noticed now this same issue with some Finnish channels. For some reason there is something wrong at EPG descriptions and autotimer fails. But there is also an issue with recording; if I manually add timer, the system will keep rebooting during while recording the program. Do You have any known fixes or workarounds for this issue?

if it keeps rebooting you should have crash logs.
If you haven't done so, then turn on debug logs and save to other than flash(menu/setup/system/logs/settings)
post crash and debug logs

[Migu]

I am typically using AutoTimer to search some text in a title. Log is displaying for example "[eEPGCache] lookup events with 'Simpsonit' in title (ignore case)". I am just wondering why an illegal description is causing the issue, although the code should not care about the description text at all? Would it be possible to fix at least the title search case?

[twol]

I am typically using AutoTimer to search some text in a title. Log is displaying for example "[eEPGCache] lookup events with 'Simpsonit' in title (ignore case)". I am just wondering why an illegal description is causing the issue, although the code should not care about the description text at all? Would it be possible to fix at least the title search case?

As requested ………..please post debug logs

[Migu]

As requested ………..please post debug logs

The debug log attached (SW 6.2.011). The exception is the same as in the first post.

<   196.0866> 18:49:10.1041 [eEPGCache] lookup events with 'Eräelämää' in title (ignore case)<   196.1279> 18:49:10.1454 [Task] >>> Error: [<Components.Task.FailedPostcondition object at 0x6917d418>]
<   196.1295> 18:49:10.1471 [Task] job Components.Task.Job name=Automaattiajastus #tasks=27 completed with [<Components.Task.FailedPostcondition object at 0x6917d418>] in Components.Task.Task name=Eräelämää_4
<   196.1317> 18:49:10.1493 [Task] unrecoverable task failure
<   196.1322> 18:49:10.1498  Automaattiajastus
Virhe: [Failure instance: Traceback: <class 'SystemError'>: <built-in function eEPGCache_search> returned a result with an exception set
/usr/lib/python3.10/threading.py:1009:_bootstrap_inner
/usr/lib/python3.10/threading.py:946:run
/usr/lib/python3.10/site-packages/twisted/_threads/_threadworker.py:47:work
/usr/lib/python3.10/site-packages/twisted/_threads/_team.py:182:doWork
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:244:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:260:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
/usr/lib/enigma2/python/Plugins/Extensions/AutoTimer/AutoTimer.py:344:JobStart
/usr/lib/enigma2/python/Plugins/Extensions/AutoTimer/AutoTimer.py:450:parseTimer

[Huevos]

So you need to go through you channels and find which one has junk in the epg.

[Migu]

Yes. I have a python script, which goes through all channels, and try to get epg data by requesting "http://<ip>/api/epgservice?sRef=<sref>&endTime=<endtime>". In case of failure, it tries to find the longest possible request without errors. And yes, in that way I can find the program, which one has junk in the epg, and I can create an autotimer for it to reproduce the issue.