Weird EPG errors. EPG Refresh. Recent versions of OpenViX e.g. 5.4.008.

[Joe_90]

Precisely.
But the original data provider will not be providing the time in Mjd format, so some other entity will be doing a conversion somewhere.
That conversion code may be doing odd things with timezone manipulation. We don't know, but it could explain what is being seen.

I'm not following your train of thought. What other entities/conversions are involved? The raw DVB table information would be created by Redbee (Freesat) or whoever is acting as the media company for Sky and is placed in the satellite or terrestrial datastream. If the data on the stream was in error, then Freesat receivers or Sky boxes would have the same problems of ghost data, perhaps?

[birdman]

That's my point. If the raw data is fine the bug is before the point where the value of startseconds exceeding the number of seconds in a day is being used to establish the start time.
The bug fix above is at a point where the incorrect (startseconds) data is being used and not where it is being incorrectly assigned.

So you are suggesting that the problem is in the data being written out to epg.dat since it goes away when that data is first deleted?

But if that were the case why does it only affect data from OpenTV and why does fixing the OpenTV-specific code fix the problem?

Does anyone actually have the raw data to confirm that it is fine?

EDIT: Hmmmm, I see what you mean.
#
The DVB standard note above mentions:

93/10/13 12:45:00 is coded as "0xC079124500".

but this section of code is getting that 6-BCD digit field as a 16-bit field of "2s units".
So something else has been doing that.

[LraiZer]

OpenTV is not DVB standard.

Test channel: Gemporia Craft
Test program: Sewing Street / Yarn Lane REPEATS

Without the patch this channel still continues to have bogus events for this program after deleting epg.dat and without performing any saving/loading of the epg.dat file

With the patch this frequently bogus program is now fixed, I've added a widget to the skin to also show the event ID's so you can match the corrections made in the screenshots.

These are not old programs that can fixed by deleting epg.dat and clearing cache as some suggest, these are current/future programs that appear wrongly offset by 2^17. I've seen up to 6 future bogus events for this program on this channel over last few days.

1_0_1_D3D6_829_2_11A0000_0_0_0_20210314004831.jpg1_0_1_D360_830_2_11A0000_0_0_0_20210314003734.jpg

[Huevos]

Gordon, your right the code makes no logical sense.

startSecond having a value greater than 86400 is just an indicator that the data is broken.

And the acquired knowledge is when the data is broken it is always 0x20000 seconds too far in the future so the fix is to just subtract that.

The data in startMjd must be broken too, because, the maximum value startSecond could ever have is 0x20000 and this is always the value that needs subtracting irespective of how much over 86400 the value is.

[adm]

So you are suggesting that the problem is in the data being written out to epg.dat since it goes away when that data is first deleted?
.

No I'm suggesting possibly more than one problem

Consider this scenario.

I start with no EPG data having stopped Enigma and deleted the epg.dat file and the RAM cache.

The box now gets its EPG data via IEPG/OpenTV

The box has EPG data for days A, B, C, D, E, F and G

Because of the “bug” some information from day A gets inserted into day B or day C, 36 hours later – but I don’t notice this at the time. The bug would also created holes in the EPG for day A. The same source data cannot establish two different time slots 36 hours apart. I haven’t noticed holes in the EPG for the current day, especially as they would be very noticeable with so may ghost/double entries in the future which should have created the equivalent number of holes.

On day B I turn on my box and notice the ghost/duplicate entries in the EPG. I now decide to update the source data from IEPG/OpenTV.

I’m now fetching data for days B, C, D, E, F, G and H – day A is already history so buggy data from day A cannot create ghost/duplicate data in the EPG for day B. [Edit: or can it]

However, the ghost/duplicate entries in day B remains! The EPG is not being overwritten with anything new for day B

Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correct. The content of this forced EPG data must be identical to that of the forced EPG request issued before the epg.dat and epg cache deletion but then it didn’t result in the EPG data being corrected.

Doesn’t this suggest that not only is there a bug in the way the start time is (was) being calculated in that the a ghost time is generated 36 hours in the future but also a bug that prevents more up to date (and correct) source data from the broadcaster also updating a previously corrupted EPG cache?

As has been pointed out if branded sky and freesat boxes don’t have this problem, nor does fetching IEPG in a different way, then the broadcasters are not sending the equivalent of a startsecond value greater than the number of seconds in a day. Startseconds being such a large value is the bug and the fix should be to find out why rather than compensating for the problem at a later stage in processing. Bugs fixed in the wrong place have a nasty habit of returning with other symptoms.

[birdman]

Gordon, your right the code makes no logical sense.

Hence the need for a comment associated with the fix.

And the raw data is not fine. It's being sent over the air in this "perverted" format by OpenTV.

[adm]

These are not old programs that can fixed by deleting epg.dat and clearing cache as some suggest, these are current/future programs that appear wrongly offset by 2^17. I've seen up to 6 future bogus events for this program on this channel over last few days.

If it is a current or future program incorrectly assigned to a future timeslot offset by 36 hours do you also see a hole in the EPG where it should have occurred?

[birdman]

Because of the “bug” some information from day A gets inserted into day B or day C, 36 hours later – but I don’t notice this at the time. The bug would also created holes in the EPG for day A. The same source data cannot establish two different time slots 36 hours apart. I haven’t noticed holes in the EPG for the current day, especially as they would be very noticeable with so may ghost/double entries in the future which should have created the equivalent number of holes.

However, let's assume that when the data is first broadcast by OpenTV it is correct.
It ends up in your EPG.
At some future point (while the programme is still running?) this field becomes corrupted.
Now you have a "new" bogus future event.

On day B I turn on my box and notice the ghost/duplicate entries in the EPG. I now decide to update the source data from IEPG/OpenTV.

I’m now fetching data for days B, C, D, E, F, G and H – day A is already history so buggy data from day A cannot create ghost/duplicate data in the EPG for day B.

It can't newly create them. But the refetch will not delete items - it (might) change events with the same eventID, but the bogus programme has now finished, so you will not get any update for it. Hence it remains in place.

Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correct

Because now you have removed the bogus entry and there is now no data to recreate it.

[BrokenUnusableAccount]

If it is a current or future program incorrectly assigned to a future timeslot offset by 36 hours do you also see a hole in the EPG where it should have occurred?

In all the cases I noticed it should have occurred in the past so no hole seen.

[adm]

In all the cases I noticed it should have occurred in the past so no hole seen.

I haven't noticed any holes either and I mostly use the grid view of the EPG where holes would be very noticeable. At some time some correct information must have populated these potential holes.

I don’t fully understand how the EPG data is managed on the boxes but my understanding is that the over the air Freesat EPG comes directly from the broadcast stream when tuned to, say, BBC1 as does the Sky FTA EPG when tuned into the IEPG channel.

The over the air data received by an Enigma box over air must be correct or else the same problem would be seen on Freesat and Sky branded boxes. It must be some software running on the Enigma box that is wrongly interpreting the broadcast EPG data and creating values of startseconds that are not sensible.

[Huevos]

In all the cases I noticed it should have occurred in the past so no hole seen.

Gemporia Craft has holes.

[Huevos]

It can't newly create them. But the refetch will not delete items - it (might) change events with the same eventID, but the bogus programme has now finished, so you will not get any update for it. Hence it remains in place.

As explained by LraiZer modified entries can't be removed because eventData type is only 8 bit so impossible for it to hold the identifier of OpenTV (0x4000).

https://www.world-of-satellite.com/s...l=1#post511282

[LraiZer]

Without the patch being applied first, the fix for updating old events was just removing more correct events that were overlapped by the bogus events, resulting in the event update fix making the first issue 10 times worse.

128 is an old removed free source type so i guess you could temp just change OPENTV=16384 to that as a bypass hack until update patch is fully tested and working?

[BrokenUnusableAccount]

Gordon, you're right the code makes no logical sense.

As I tried to explain in #103, #105 (note: in #105 I wrote 249 when I should have written 239)

They are kind of treating the field I've called startSecond as a signed time.
However it doesn't roll from meaning a +ve number to meaning a -ve number in the usual place (when the MSB flips to a 1).
Instead it seems it should be treated as a signed negative number if it's greater than or equal to 86400.

Then just as when you interpret a normal negative signed 17 bit number as unsigned the result is 0x20000 too high.

[LraiZer]

Just for the curious, here is a debug data dump log of the current bogus events being fixed.
Data cycle appears to currently be running at 139 seconds to populate the epg