View Full Version : [Zgemma H7] Weird EPG errors. EPG Refresh. Recent versions of OpenViX e.g. 5.4.008.
BrokenUnusableAccount
04-03-21, 06:11
I'm trying to get the most over the air EPG I can by using EPG Refresh (as much as possible for both Freeview and 28.2°E satellite both radio and television).
I have EPG Refresh set to spend 150 seconds on each of 11426H, 11778V and each of the Freeview multiplexes in the small hours every morning.
I keep getting weird extra programs in the satellite TV part of the EPG from the previous week but not at the same time.
For instance on 5th March 2021 on satellite BBC 2 both SD and HD I got program "Chris Packham: Asperger's and Me" 11:54-12:54 (actually shown on, as far as I can see, 3rd March 23:30-00:30, before EPG Refresh ran).
It's kind of showing over the middle of "Athletics..." 09:00-13:00.
There are numerous similar examples all through the EPG for around this time on 5th March.
At the moment I'm thinking it's something to do with OpenTV EPG going over the top of existing Freesat EPG information but it seems like it must be some weird kind of bug that makes it manifest itself in this way.
Not really expecting an quick fix, just thought it's so weird I must document it.
What sources have you enabled? With multiple sources in use the data is different from each source. Its recommended not to enable the freesat epg when using opentv data.
What sources have you enabled? With multiple sources in use the data is different from each source. Its recommended not to enable the freesat epg when using opentv data.
I'm starting to believe that there is another bug in the EPG collection from 28.2E satellite and opentv irrespective of how the IEPG is accessed/obtained. A few weeks back I thought the problem with two programs being displayed or overwritten in a EPG time slot may just be that the 7 day EPG obtained with the OpenTV EPG downloader being overwritten by the now/next data EIT when program times changed.
I now believe that I'm seeing something similar to BefuddledBrian in that the overwriting of the current/future EPG is with a program broadcast in the past – and with data that may/will have been in the EPG previously. My last experience of this the overwrite the programs in the EPG was from the previous evening at the time when I switched from watching live TV to watching a delayed version of a program. This was programs from, say, 8pm on a Tuesday being overwritten into the EPG data for 10am on a following Wednesday. The highlighting in the (grid) EPG goes wrong at this point as though both program names are present.. It’s not every channel – usually more than one but less than six. Unfortunately although I regularly see this overwriting/double EPG entries I cannot establish a firm pattern for what is triggering it.
Other things that may be relevant:
I have both opentv EPG downloader and EIT enabled because I want the over the air EPG for both FTA Sky and UK terrestrial Freeview. Freesat EPG enable = no
I put my box into deep standby every night by first switching off with the remote off button which puts the box into standby. A power timer than kicks in which then puts the box into deep standby 10 minutes later. This means that the EPG data in RAM must be transferred to/from the hard disk on shutdown and subsequent power-up.
In 5.4.007 and previous versions I often see a missing 4 hours of EPG on satellite and for all channels. The 7 day EPG data is available prior AND after to the missing 4 hours. A manual forced download using OpenTV EPG downloader does not correct the problem. The only way seems to use the menu to delete the EPG cache and then force a opentv download. One thing to note that during this "blank" 4 hours in the EPG the program description on all channels is identical and it is a partial description of a program previously broadcast - in my experience it only consists of a few words but can be identified as a real program.
I have a habit of watching a recording and than switching back to live TV without pressing the recording stop button which means the recording may carry on for perhaps 10 minutes in the background before automatically reaching the end. I have menu → setup → epg → grid EPG → channel selection = selecet current service which means that when I go back to the (grid) epg the program that I’m watching on live TV is highlighted UNLESS I have previously watched a recording and not terminated it with a record stop button press. Afterwards the highlighted channel in the (grid) EPG is always the channel I was watching prior to watching the recording – and it never changes irrespective of changing channels for live TV.
The missing 4 hours may be down to not letting it download data long enough. It has been reported a few times to use 180 seconds instead of 150 (I have mine set to 180s). There is no harm trying a longer download duration. I haven't noticed anything myself on my H7 and I do have EIT epg enabled (for saorview) which does download UK satellite data too as well as using opentvzapper. I also use epgimport for 30W. Also its so harm to delete the epg cache and refresh the data as the database may be corrupted.
The missing 4 hours may be down to not letting it download data long enough. It has been reported a few times to use 180 seconds instead of 150
Its nothing to do with time. 150/180 seconds is EPG refresh and not OpenTV EPG download which I'm currently using. In addition, in the past I've actually selected IEPG channel and left it for 10+ minutes and it still hasn't filled the 4 hour hole in the epg.
I haven't noticed anything myself on my H7 and I do have EIT epg enabled (for saorview) which does download UK satellite data too as well as using opentvzapper. I also use epgimport for 30W. Also its so harm to delete the epg cache and refresh the data as the database may be corrupted.
The problem is that the EPG cache may get corrupted but may not be noticed. Autotimers etc. are trying to set timers based on this data. The 4 hour hole in EPG data may only be seen if you scroll through 3 or 4 days of data and then you may find it in a random place, something that I don't often do. A reboot doesn't clear this 4 hour hole nor does a manual force to refetch EPG data. It is if the EPG cache has a section protected by a read only flag prevting an overwrite. When the problem does occur it is always a 4 (ish) hour hole.
It has been hinted that in the EPG settings menu that the periodic Automatic save and Automatic reload functions are buggy. I have both options set to "no" but maybe the same mechanism is used to save and reload the EPG data when the box is placed into deep standby and possibly the same problems may not be seen by those who only put the box into standby.
BTW, OpenTV EPG download has no option to first delete the cache
I regularly look ahead through the epg for programs of interest and have not noticed this behaviour. The only times I have noticed gaps are when the data download has been interrupted. It was Brian that said he was using 150 secs to download his data. By the looks of his description it seems his data is corrupted (my data for 5th March on BBC2 is correct as I have been looking at it all week for the athletics). I don't clear my cache unless needed but it does need it now and again.
I have had similar issues with EPG "corruption" recently. I'm using OpenTVzapper (with Freesat EPG disabled, but with EIT reading enabled for my terrestrial channels). It's set on the default options (run every 6 hours).
On Channel 4 I noticed that the EPG entry for 09.30 contained the details for "Naked Attraction" which had been broadcast the previous evening. No number of forced re-runs of OpenTVzapper would clear the entry. I had to delete the EPG cache and then re-run. Like @adm, I have PowerTimers which shut the box down if more than 20 minutes in standby. I would also manually put the box in Deep Standby when I'm finished using it. I can't figure out what is triggering the issue.
Can you guys please provide screengrabs.
Can you guys please provide screengrabs.
Blank epg example ! posted a month ago
https://www.world-of-satellite.com/showthread.php?63961-OpenTV-EPG-downloader-oddity
Double entry/overwrite example I posted 4 months ago
https://www.world-of-satellite.com/showthread.php?63384-Grid-EPG-highlight-bug
I initially posted that it may be a highlighting bug but have since come to the conclusion that this occurs when the EPG has been overwritten with data from the past and/or data from the past has been overwriten with new epg data but the previous data not deleted.
I've seen the same problems on 28.2E sky FTA channels when using EPG Refresh, OpenTV EPG download and a bespoke method where I set a timer to tune into IEPG every day at 6:30am and record 15 minutes (the recording was routed to trash but it populated the EPG). From this I don't believe it is necessarily related to the mechanism that sets the tuning into IEPG. With reference to double entries/overwritting of the EPG I've seen examples for a long time so not just 5.4.xxx. In the past week I've seen the blank 4 hours once and double/overwritten EPG a couple of times.
The double entries/overwritting of the EPG has been seen in the past when using Freesat epg and OpwenTv disabled.
BrokenUnusableAccount
04-03-21, 21:30
Thank you @ronand, @adm, and @fat-tony.
When I wrote my original message I had these EPG settings:
EPG button action = Grid EPG
INFO button action = Event Info
EPG location = /media/hdd
EPG filename = epg
Automatic save = Yes
Save every (in hours) = 1
Automatic reload = No
Show EIT now/next in infobar = No
Enable EIT EPG = Yes
Enable MHW EPG = No
Enable FreeSat EPG = Yes
Enable ViaSat EPG = No
Enable Netmed EPG = No
Enable Virgin EPG = No
Enable OpenTV EPG = Yes
Maintain old EPG data for = 45
Include EIT in http streams = Yes
Include AIT in http streams = Yes
Country for EPG event rating information = Auto Detect
Country for EPG event genre information = Auto Detect
and these EPG Refresh settings:
Refresh EPG automatically = Yes
Duration to stay on service (seconds) = 150
EPG refresh auto-start earliest (hh:mm) = 03:37
EPG refresh auto-start latest (hh:mm) = 08:27
Delay if not in standby (minutes) = 3
Refresh EPG using = Fake recording
Show Advanced Options = Yes
Skip protected Services = Background only
Show Setup in plugins = Yes
Show Setup in extension menu = Yes
Show 'EPGRefresh Start now' in extension menu. = Yes
Show popup when refresh starts and ends. = Yes
Wake up from standby for EPG refresh. = Yes
Force scan even if receiver is in use. = Yes
Shutdown after EPG refresh. = No
Flush EPG before refresh = Yes
Inherit Services from AutoTimer = No
Run AutoTimer after refresh = Yes
and EPG Refresh was zapping to 11426H (Freesat), 11778V (OpenTV) and each of the Freeview multiplexes in that order.
In particular notice that I have EPG Refresh set to flush the EPG cache before it begins zapping.
The spurious old information still appeared on the EPG.
In fact yesterday I'd also tried manually flushing the EPG, rebooting and manually flushing it again before starting EPG Refresh.
No difference. The spurious old EPG stuff still reappeared from somewhere!
With multiple sources in use the data is different from each source. Its recommended not to enable the freesat epg when using opentv data.
I'm trying to get some EPG for satellite radio. But yes - point taken, I may yet have to give up and accept that I can't have EPG for satellite radio.
Tonight I've fiddled with EPG Refresh and I now have it doing OpenTV first, then Freesat and then Freeview, and I'm going to see if that makes any difference.
So far I think it does seem better.
It's not that straightforward getting EPG Refresh to zap in a particular order. It tends to sort things into an order of it's own preference but since it's only the transponder (or multiplex) that matters you can try choosing different things on the same transponders and checking what order they end up being in.
Blank epg example ! posted a month ago
https://www.world-of-satellite.com/showthread.php?63961-OpenTV-EPG-downloader-oddity
Double entry/overwrite example I posted 4 months ago
https://www.world-of-satellite.com/showthread.php?63384-Grid-EPG-highlight-bug
I initially posted that it may be a highlighting bug but have since come to the conclusion that this occurs when the EPG has been overwritten with data from the past and/or data from the past has been overwriten with new epg data but the previous data not deleted.
I've seen the same problems on 28.2E sky FTA channels when using EPG Refresh, OpenTV EPG download and a bespoke method where I set a timer to tune into IEPG every day at 6:30am and record 15 minutes (the recording was routed to trash but it populated the EPG). From this I don't believe it is necessarily related to the mechanism that sets the tuning into IEPG. With reference to double entries/overwritting of the EPG I've seen examples for a long time so not just 5.4.xxx. In the past week I've seen the blank 4 hours once and double/overwritten EPG a couple of times.
The double entries/overwritting of the EPG has been seen in the past when using Freesat epg and OpwenTv disabled.The most recent commit was 19 days ago so we need screengrabs from a build more recent than that. I can't find any holes in my EPG data at the moment.
Also, although it is possible to discard all epg data before running OpentvZapper, it is a hack. epgcache.cpp should be in charge of deleting items as and when necessary. So if anything needs fixing it is in there.
BrokenUnusableAccount
05-03-21, 01:00
Can you guys please provide screengrabs.
I'll do screenshots next time I get examples of this problem.
BrokenUnusableAccount
05-03-21, 02:38
Can you guys please provide screengrabs.
I put EPG Refresh back to zapping to 11426H (Freesat), 11778V (OpenTV) and each of the Freeview multiplexes in that order and manually started it at about 00:15 on 5th March 2021.
(At this stage I don't know if that "helped" it go wrong or not)
I think, but I'm not sure, that it seemed like it wouldn't go wrong if I ran EPG Refresh before midnight.
Here is screenshot showing
Match of the Day 12:09-12:44 over the middle of Football Focus 12:00-13:00 on BBC One HD and BBC One South.
the end of Being Bridget Jones 11:54-12:54 over the end of Athletics... 08:45-12:20
(then 60s Icons:... 12:20-13:00 over then end of that) on BBC Two HD.
Made in Britain 12:09-12:59 over the end of New: Simply Raymond Blac 11:40-12:40
(then ITV Lunchtime News 12:40-12:55 over the end of that) on ITV Meridian HD.
Mock the Week 12:04-12:44 over the top of American Picker 12:00-13:00 on Dave.
Around the World in 80 Treasures 12:09-13:09 over the middle of ..programmes start at 7:00pm 11:30-17:30 on BBC Four HD.
61655
something_fishy
05-03-21, 09:27
I think that I am seeing the same thing. It seems most visible as a random program appearing late morning on top of BBC2's long rolling news coverage. Or in the following example sports coverage:
61656
This is using the built in openTV downloader with all other epg sources turned off:
61657
61658
This is with the most recent version of Openvix:
61659
To confirm this effect I cleared the epg cache and did a OpenTV forced download from the extension menu. This populated the epg without this problem. This is the day after following a few automatic updates.
Before this I tried the freesat epg which I also don't think demonstrated this issue.
Thanks
Yes Bridget Jones has crept into my BBCHD epg listing for tomorrow
Being Bridget Jones actually broadcast on bbc2 hd on Thursday (4th) at 23:30 for 1 hour.
Also to be broadcast next Friday (12th) at 02:00.
abu baniaz
05-03-21, 10:42
Maybe it was in the Diary?
Being Bridget Jones actually broadcast on bbc2 hd on Thursday (4th) at 23:30 for 1 hour.
Also to be broadcast next Friday (12th) at 02:00.
I was actually watching some of that programme last night. I checked my own EPG just now and BBC2HD does not have that entry overwriting the athletics tomorrow. The fact that it has happened in the same place for two users would indicate an issue with the actual processing of the EPG data, rather than any setting problem, particularly as EIT and Freesat data is turned off in one configuration. It's very similar to what I've seen myself, where programmes scheduled for late evening (23:00ish) will overwrite events prior to 12:00 on a subsequent day.
I think we can now rule out corrupt local epg files. I have EIT epg enabled (for saorview) and opentvzapper for 28e. But of course the EIT epg on 28e get absorbed too.
I have the same config settings as I use Saorview also. This is not a recent phenomenon, though. I've seen it in the past, but it was usually dismissed as a settings issue/clash between OpenTV and Freesat sources, which doesn't seem to be the case now.
I deleted my epg cache and did a forced opentv download and the data was badly corrupted. Most of the descriptions were missing and some short descriptions were just random characters.
6166161660
BrokenUnusableAccount
05-03-21, 15:24
I deleted my epg cache and did a forced opentv download and the data was badly corrupted. Most of the descriptions were missing and some short descriptions were just random characters.
I get that occasionally. Many missing program titles, just the times.
I have the impression that it's caused by some kind of data corruption that is hard to get rid of. I typically repeat two or three times deleteing the EPG cache and rebooting and that fixes it.
something_fishy
05-03-21, 17:07
I get that occasionally. Many missing program titles, just the times.
I have the impression that it's caused by some kind of data corruption that is hard to get rid of. I typically repeat two or three times deleteing the EPG cache and rebooting and that fixes it.
I've seen that too, particularly when I'm testing (doesn't seem an issue if I just let the OpenTV scheduler do its stuff). Sometimes I've had to create a epg file to break the cycle.
Example 6th March 2021, running 5.4.088, Xtrend ET10k
Grid EPG where the double entry is on CH4 HD
61663
Alternative EPG view showing two programms in the same time slot
61664
The embedded program (big fat quiz of everything) was broadcast the previous evening although it seems to be repeated often.
I have both CH4 and CH4 HD in my 28.2E Bouquet and both CH4 and CH4 HD EPG have the same problem
Ch4 EPG
61665
I also have tuners set for terrestrial UK Freview and the the EPG for CH4 from Freeview (EIT) does NOT show the same problem.
Have you got Freesat and Opentv enabled at the same time?
Could this be cache issues? If so, any value to adding option to delete cache before it downloads epg data? Happy to try code it in as a PoC
Quick look at CrossEPG and it looks like it deals with epg cache via new cache instance!
Could this be cache issues? If so, any value to adding option to delete cache before it downloads epg data? Happy to try code it in as a PoC
Quick look at CrossEPG and it looks like it deals with epg cache via new cache instance!You would need to only delete the channels being downloaded, and also would have to occur for all types including eit.
Step one would be log everything that is ever downloaded to find where the conflicting data is coming from.
Have you got Freesat and Opentv enabled at the same time?
No,
61666
What about epg.dat after booting from deep standby? It will hold out of date data which will need flushing out.
Works ok on freeview eit though, I always use deep standby overnight.
These strange overlaps have always existed, and must be in PLi too because the code is identical.
If we want to go anywhere with this the only way to make progress is to do what I said above and log absolutely everything that enters the epgcache.
BrokenUnusableAccount
06-03-21, 16:18
I've fiddled with EPG Refresh and I now have it doing OpenTV first, then Freesat and then Freeview, and I'm going to see if that makes any difference.
So far I think it does seem better.
No doesn't seem to make much difference.
I will try with Freesat EPG switched off next.
No doesn't seem to make much difference.
I will try with Freesat EPG switched off next.
I've seen the problem on 28.2E with all combinations of EIT, Freesat and OpenTV and with using OpenTV EPG downloader, EPG Refresh and with just switching to IEPG at 6:30am for 10 minutes.
The main problem with fault finding is that the problem is somewhat intermittent. Some days no problems are seen, some days only one channel has the problem and on other days it may be six channels with double entries.
I cannot establish a pattern but its often the first few timeslots (in the first 2 to 3 hours) in the EPG referenced to the time I manually turned on my box from deep standby but having said that I have many autotimers set-up that set timers at various times during the night/day that will switch the box on for recording and presumably re-trigger some of the EPG fetching mechanisms.
BrokenUnusableAccount
07-03-21, 00:29
My box is set up to never go in to deep standby - I theorise, rightly or wrongly, that it's likely that regularly starting up again from deep standby would stress it more than leaving it in normal standby does.
Plus occasionally I stream from it to my laptop and it's nice not to have to start it up first.
I think Running EPG Refresh soon after midnight gives the most weird extra EPG entries and soon before midnight the least, or maybe none. :confused:
BrokenUnusableAccount
07-03-21, 04:50
I will try with Freesat EPG switched off next.
Not much different with Freesat and EIT EPGs switched off.
I noticed odd entries like this while back when EPG Refresh was in default Vix image (before days of OpenTV zapper). I then went back to CrossEPG and dont notice anything anymore except actual odd last minute changes that are picked up by EIT (and to be expected).
I have ABM scan 6am daily with CrossEPG 6.05am daily.
I think Running EPG Refresh soon after midnight gives the most weird extra EPG entries and soon before midnight the least, or maybe none. :confused:
That's the impression I get - old data getting flushed out from "yesterday" while it's running perhaps?
This morning at approx 8:30am when I switched on (and for at least 15 minutes afterwards) the entry for "Live Cricket, England v England" on 28.2E Ch4, 28.2E Ch4 HD and Freeview Ch4 was missing and on the grid EPG the highlighting of that time slot had the same problems as seen when there is a double entry. Two hours later when I checked (box left on) the missing entry had been populated on both 28.2E and UK terrestrial Freeview.
Live cricket ended yesterday :), so that was probably a change of schedule.
I'm seeing ""Sunday Brunch" on CH4 freeview (originally scheduled to start at 11:45).
I wonder when the change of schedule actually happended?
I've got a copy of epg.dat from 22:09 last night when the box shutdown, I'll see if it's in there.
There's one entry for "Live Cricket: India v England:", there should have been 2 (today and tomorrow) if the game hadn't ended early,
so the schedules on freeview partly changed yesterday.
I know freeview is ok, but here goes.
I've booted up my et10k from deep standby (closed down last night) with no tuners connected.
epg.dat from last night shows a change of schedule for the cricket today and tomorrow (and also has one entry for "Live Cricket: India v England:" using notepad++).
Live cricket ended yesterday :), so that was probably a change of schedule.
I'm seeing ""Sunday Brunch" on CH4 freeview (originally scheduled to start at 11:45).
I wonder when the change of schedule actually happended?
So at 8:30am my EPG entry was blank for that Ch4 and Ch4 HD time slot
Before posting this (and before trying a new forced update of EPG data) it was showing ....
61667
But as you indicate the Cricket has finished (I have zero interest in Cricket so I haven't actully been watching Ch4 during those times). Live:Sunday Bruch is still scheduled for 11:45 to 14:45
Pressing blue and forcing a openTV EPG download DID NOT update the entry for this current listing nor the next time slot which must have also changed.
I now seem to have problems with 28.2E EPG updating even after a Cache delete and a couple of reboots
Edit Update
After a EPG Cache delete and a reboot the 28.2E data didn’t update after a press blue and selecting a forced opentv download
After a EPG Cache delete and a mains power shutdown the 28.2E data didn’t update after a press blue and selecting a forced opentv download
After a EPG Cache delete and a mains power shutdown the 28.2E data didn’t update just leaving the box for 10 minutes.
In the end I reloaded a delete EPG plugin and used that to stop Enigma2 to delete EPG files. After a reboot the 28.2E EPG repopulated within 10 minutes.
https://www.world-of-satellite.com/showthread.php?52890-Remove-EPG
I've had intermittent results with EPG delete via menu also tbh.
@adm, you put box to deep sleep don't you (presumably timed)? As a test, is it worth seeing if EPG can be deleted via that script to fully ensure it's deleted?
Then when box is woken up, it will have (in theory) no epg.dat then EPG download can be done.
something_fishy
08-03-21, 11:44
A feast of spurious mid-morning programs today:
BBC1HD: Margin call 11:44-13:24 interjecting on Crimewatch Live 11:00-11:45 and Defenders UK 11:45-12:14
BBC2HD: Mindhorn 11:49-13:09 over BBC news 10:00-12:15 and Politics Live 12:15-13:00
ITV Anglia HD: Die Another Day 11:29-13:54 over This Morning 10:00-12:30 and Loose Women 12:30-113:30
C4HD: Knight and Day 10:24-12:29 over Frasier
Ch5HD: When Cruises go horribly wrong 11:49-13:14 over George Clarkes New Build... 11:15-12:15
BBC4HD: TOTP 12:04-12:34 over programmes start ... 11:30-17:30
BBC1 East E: (as BBC1HD)
I've only noticed them on BBC2HD before, but that might be because they're easier to spot there rather than sign of a real change.
My box never goes to sleep BTW and openTV is updating every three hours.
A feast of spurious mid-morning programs today:
BBC1HD: Margin call 11:44-13:24 interjecting on Crimewatch Live 11:00-11:45 and Defenders UK 11:45-12:14
BBC2HD: Mindhorn 11:49-13:09 over BBC news 10:00-12:15 and Politics Live 12:15-13:00
ITV Anglia HD: Die Another Day 11:29-13:54 over This Morning 10:00-12:30 and Loose Women 12:30-113:30
C4HD: Knight and Day 10:24-12:29 over Frasier
Ch5HD: When Cruises go horribly wrong 11:49-13:14 over George Clarkes New Build... 11:15-12:15
BBC4HD: TOTP 12:04-12:34 over programmes start ... 11:30-17:30
BBC1 East E: (as BBC1HD)
I've only noticed them on BBC2HD before, but that might be because they're easier to spot there rather than sign of a real change.
My box never goes to sleep BTW and openTV is updating every three hours.
Just to report that I see none of those problems today. This may rule out the actual information from the source and it's more to do with something that is happening on the box itself and possibly at the time the box updates the information and/or when the box is switched on. Timing is likely to be very different for every box owner.
I see all those bad entries. My box was in standby (I rarely use deep standby) and opentvzapper is set to refresh every 3 hours
I use CrossEPG as I've mentioned earlier and dont have any of these issues and I have most EPG sources on (like EiT, OpenTV, FreeSat, etc).
Anyhow, I noticed a channel had a schedule change that must have been after 6.05am EPG update I run. EPG showed old schedule until the time came so assumed EiT changed this. I noticed it kept going back to old data and only would update to latest schedule when I was on it.
I got around this by saving EPG from within menu and it then showed new correct data.
I actually have EPG set to automatically save every 12 hours so have now increased this to 6. I have never noticed any issues with it saving every 12 hours and only increased how often it save so it saves last min schedule changes bit more often.
So what i'm getting at is when someone has these ghost entries, does doing a EPG save from menu and going back into EPG mitigate this?
something_fishy
08-03-21, 12:26
Also on S4C, 4Seven, Ch4 (same as 4HD), 5USA, BBC red button, Film 4+1, ROK (I don't know what this channel is)
And most of the movie channels.
So what i'm getting at is when someone has these ghost entries, does doing a EPG save from menu and going back into EPG mitigate this?
I'd be surprised if it made any difference as it (should) only copy the epg cache to epg.dat. Worth a try though if anyone still has the overlaps showing.
something_fishy
08-03-21, 12:31
@bbbuk
Saving the EPG cache makes no difference to me. Nor does saving the cache and then (re)loading the cache.
A feast of spurious mid-morning programs today:
ITV Anglia HD: Die Another Day 11:29-13:54 over This Morning 10:00-12:30 and Loose Women 12:30-113:30
As far as I can tell, this was broadcast on Saturday 6th (2 days ago) on ITV at 23:05 'til 01:30.
something_fishy
08-03-21, 13:10
As far as I can see they were all broadcast late Saturday crossing midnight:
Margin call: 23:20-01:00
Mindhorn: 23:25-00:45
Die another day: 23:05-01:30
Knight and Day: 22:00-00:05
Cruises...: 23:25-00:50
TOTP (20/09/1990): 23:40-00:10
So the program durations are correct but they're offset by 36 hours 24 minutes.
This commit makes me wish I understood a lot more.....
[epgcache] OpenTV: Use event_id from SI tables
Having the event_id available means if the event start time changes the stale data will be removed from the cache rather than co-existing alongside the new fresh data.
https://github.com/OpenViX/enigma2/commit/0731baaaa16d7985cb42b003d3ec50dd5631b871
https://github.com/OpenViX/enigma2/commits/master/lib/dvb/epgcache.cpp
BrokenUnusableAccount
08-03-21, 16:07
This commit makes me wish I understood a lot more.....
https://github.com/OpenViX/enigma2/commit/0731baaaa16d7985cb42b003d3ec50dd5631b871
I think that one is a result of this discussion about making the VPS plugin work again:
https://www.world-of-satellite.com/showthread.php?62706-EPG-Refresh-iEPG-and-VPS
I think that one is a result of this discussion about making the VPS plugin work again:
https://www.world-of-satellite.com/showthread.php?62706-EPG-Refresh-iEPG-and-VPSNo, it's not connected.
something_fishy
08-03-21, 18:14
Programs that aired crossing midnight this morning are appearing in the schedule for Tuesday AM (again lagged by 36 hrs 24 minutes). Look for The Womens Football Show (BBC1), Festival of Funny (BBC2), Trauma (ITV), Amsterdam Vice (C4), Family Guy (ITV2).
I so far haven't found one if the original air date doesn't cross midnight, but I have found programs that cross midnight that don't appear repeated (eg Space Cowboys ITV4).
spanner123
08-03-21, 18:20
Why not use CrossEPG, it's works fine 99% of the time!
something_fishy
08-03-21, 18:29
Why not use CrossEPG, it's works fine 99% of the time!
Firstly because the developers might want to fix it, secondly because I like the idea of pulling the data direct from the satellite feed as it opens the door for real time (or at least intraday) updates. I'd hate to miss my Chris Whitty fix just because I thought it was Pointless.
Why not use CrossEPG, it's works fine 99% of the time!
As long as the person/organisation supporting the method still has an interest in it. So many plugins have gone obsolete or “private” once the author has lost interest in it or its starts costing money when the bandwidth by people accessing the information becomes excessive.
BrokenUnusableAccount
08-03-21, 18:44
Why not use CrossEPG, it's works fine 99% of the time!
I've always been put off by it's buggy seemingly unfinished user interface.
For instance just selecting it from the plugins menu after installing it crashes the box or if you do it again makes the screen flash a few times in a weird way.
Also presumably it can't to Freesat for Radio EPG or the Freeview EPG, and it can't then automatically run my autotimer search.
abu baniaz
08-03-21, 18:51
Firstly because the developers might want to fix it, secondly because I like the idea of pulling the data direct from the satellite feed as it opens the door for real time (or at least intraday) updates. I'd hate to miss my Chris Whitty fix just because I thought it was Pointless.
CrossEPG has two methods of EPG acquisition. The most commonly used one by our image users is the one that "pulls data direct from the satellite feed".
CrossEPG is too convuluted to update.
Use the method/s that work best for your setup.
spanner123
08-03-21, 19:25
I've always been put off by it's buggy seemingly unfinished user interface.
For instance just selecting it from the plugins menu after installing it crashes the box or if you do it again makes the screen flash a few times in a weird way.
Also presumably it can't to Freesat for Radio EPG or the Freeview EPG, and it can't then automatically run my autotimer search.
Never had any of that in all the years I have been using it. Most radio stations have full EPG which I think only works if you have Freesat EPG enabled.
If anyone is looking for actual programme schedule changes, New: Unforgotten has moved from 21:00 today to 21:00 tomorrow. (Already showing on freeview)
BrokenUnusableAccount
08-03-21, 21:09
I'm really struggling to understand some of the stuff in opentv.cpp and epgcache.cpp.
For instance the end of line 160 in opentv.cpp.
startTimeBcd = (((startMjd - 40587) * 86400) + (UINT16(&buffer[2]) << 1));
What is function UINT16?
I've written C before but what does "&buffer[2]" do?
& is the "address off" function but the result is treated as if it was a number not an address (it's shifted left)!?
Can't tell you about the UINT stuff, but the calculation before the "+" seems to be the number of seconds in the integer days elapsed since 1/1/1970. (startTimeBcd is the startMjd (current day number since 17 November 1858) less the number of days to the unix epoch (1/1/1970) multiplied by the number of seconds in a day. Presumably the &buffer[2] contains a number of seconds to establish the start time in BCD?
It's odd that the time errors in the EPG times are offset 36 hours and 24 minutes ahead, i.e. 2184 minutes which is 0x888 or binary 100010001000. Seems like some error in data manipulation, maybe?
I'm really struggling to understand some of the stuff in opentv.cpp and epgcache.cpp.
For instance the end of line 160 in opentv.cpp.
startTimeBcd = (((startMjd - 40587) * 86400) + (UINT16(&buffer[2]) << 1));
What is function UINT16?
I've written C before but what does "&buffer[2]" do?
& is the "address off" function but the result is treated as if it was a number not an address (it's shifted left)!?
I come from a hardware language background (VHDL) so I may be mistaken in my observation but ….
I would be more worried that a 16 bit signed 2s complement value (startMJd) multiplied by a 17 bit number (86400 or hexadecimal 1_5180) would result in a 33 bit number but startTimeBcd appears to be defined only as signed 2s complement 32 bit number. There doesn’t seem to be any sanity checking for under/over flows in the number ranges which may make a large positive number plus 1 = a negative number etc.
Also where have those arbitrary values of 40587 and 86400 come from?
However, this thread is possibly the wrong place to disscuss code in what may be an unrelated module to that causing the double entries
Edit
Tony has probably answered the 40587 and 86400 question.
Does the BCD and the end of the startTimeBcd indicate that the result needs to be in a binary coded decimal format?
What is function UINT16?At a guess, a cast to an unsigned 16-bit integer.
I've written C before but what does "&buffer[2]" do?
& is the "address off" function but the result is treated as if it was a number not an address (it's shifted left)!?&buffer[2] is the location of the 3rd element of buffer.
It's the same as buffer +2.
It's possible that UINT16 takes an address and uses it as a pointer to an unsigned 16-bit integer, whence shifting would be OK.
So the secret is to determine what UNT16 actually is.
Also where have those arbitrary values of 40587 and 86400 come from?86400 isn't arbitrary - it's the number of seconds in a day.
The 40587 is probably something to do with a different base date. Such as something that started in ~1859, which rings a bell somewhere as something Microsoft/DEC used...
But using usefully-tagged parameters would help
EDIT: Tracked down the bell...
Modified Julian Date (MJD) A continuous measure in days since midnight at the start of 17 November 1858. Based on UTC.
And a bit more from:
https://www.secret-bases.co.uk/wiki/Julian_day_number
The Modified Julian Date (MJD) was introduced by the Smithsonian Astrophysical Observatory in 1957 to record the orbit of Sputnik (https://www.secret-bases.co.uk/wiki/Sputnik_1) via an IBM 704 (https://www.secret-bases.co.uk/wiki/IBM_704) (36-bit machine) and using only 18 bits until August 7, 2576. MJD is the epoch of VAX/VMS and its successor OpenVMS (https://www.secret-bases.co.uk/wiki/OpenVMS#Timekeeping), using 63-bit date/time, which allows times to be stored up to July 31, 31086, 02:48:05.47. The MJD has a starting point of midnight on November 17, 1858 and is computed by MJD = JD - 2400000.5
As this doesn't seem to happen with CrossEPG (that I have noticed), what is difference between using EPG sources from CrossEPG and OpenTV/EPGRefresh ?
I'm aware CrossEPG does something with cache unlike OpenTV/EPGRefresh but fundamentality is there any other difference?
Also, for comparison reasons are many of the FTA channels exactly same between FreeSat and Sky? ie is Channel 4 HD via FreeSat exactly same frequency as Sky UK?
The FTA channels are exactly the same between Freesat and Sky. There is no CH4 HD on Freesat's EPG, but it's FTA on Sky's EPG, so you have to pull the info from there.
Has anyone tried viewing the OpenWebIf version of the epg when the overlaps are showing via the gui?
I don't think it'll prove anything, but it might come up with a clue.
Also, does epg search find the phantom programmes?
Can't tell you about the UINT stuff, but the calculation before the "+" seems to be the number of seconds in the integer days elapsed since 1/1/1970. (startTimeBcd is the startMjd (current day number since 17 November 1858) less the number of days to the unix epoch (1/1/1970) multiplied by the number of seconds in a day. Presumably the &buffer[2] contains a number of seconds to establish the start time in BCD?
It's odd that the time errors in the EPG times are offset 36 hours and 24 minutes ahead, i.e. 2184 minutes which is 0x888 or binary 100010001000. Seems like some error in data manipulation, maybe?
Maybe one of the devs could look at the date manipulation in the OpenTVreader code which is triggered by OpenTVzapper? I looked through the EPG this morning and found examples of events from before midnight Sunday being transposed into the EPG before midday today. It happened on about 10 channels that I looked at quickly from BBC2 onwards. Not all channels affected but not limited to BBC/ITV/C4. The offset seems to be consistent at 2184 minutes which @something_fishy identified. 2184 decimal is 0x888. Looks like a bit mask error.
Here is some further information on the structure of the start_time field in the DVB EIT definition. I don't know if the proprietary data in the Sky EPG information conforms to this standard but I would guess that it would:
start_time: This 40-bit field contains the start time of the event in Universal Time, Co-ordinated (UTC) and Modified
Julian Date (MJD) (see annex C). This field is coded as 16 bits giving the 16 LSBs of MJD followed by 24 bits coded as
6 digits in 4-bit Binary Coded Decimal (BCD). If the start time is undefined (e.g. for an event in a NVOD reference
service) all bits of the field are set to "1".
EXAMPLE 1:
93/10/13 12:45:00 is coded as "0xC079124500".
duration: A 24-bit field containing the duration of the event in hours, minutes, seconds. format: 6 digits,
4-bit BCD = 24 bit.
EXAMPLE 2:
01:45:30 is coded as "0x014530".
Has anyone tried viewing the OpenWebIf version of the epg when the overlaps are showing via the gui?It's possible to get a raw dump of the data using the OpenWebIF API.
https://dream.reichholf.net/e2web/
something_fishy
09-03-21, 13:39
Has anyone tried viewing the OpenWebIf version of the epg when the overlaps are showing via the gui?
At least one does (I don't know about the others as the phantoms are now in the past in many cases)
Also, does epg search find the phantom programmes?
A search on the phantom finds other instances on other channels.
Bates Motel (BBC1HD @23:25-00:10) should have a phantom on Thursday ay 11:49. There is nothing in the guide yet, lets see if this changes tomorrow when the original has moved into the past.
BrokenUnusableAccount
09-03-21, 14:08
I don't think I'm getting any "phantoms" if I run EPG Refresh daily at 08:37 instead of my usual 03:37.
But I will continue to experiment running it manually after midnight.
BrokenUnusableAccount
09-03-21, 17:53
At a guess, a cast to an unsigned 16-bit integer.
&buffer[2] is the location of the 3rd element of buffer.
It's the same as buffer +2.
It's possible that UINT16 takes an address and uses it as a pointer to an unsigned 16-bit integer, whence shifting would be OK.
So the secret is to determine what UNT16 actually is.
Following through the chain of .h files including other .h files it seems UINT16 is as you say, taking an address and returning the 16 bit value pointed to by it.
But it swaps the byte order if necessary depending on the architecture of the targeted system.
But it swaps the byte order if necessary depending on the architecture of the targeted system.A lot of network code does that (through macros).
Usually what comes down the wire is Big Endian (as it was probably developed on a Sun Microsystems box), but it has to be manipulated locally in native mode.
True of any binary transfer of data, really.
BrokenUnusableAccount
10-03-21, 02:36
Do any legitimate EPG entries ever start at xx:x4 or xx:x9 ?
I don't think I've ever seen any.
Do any legitimate EPG entries ever start at xx:x4 or xx:x9 ?
I don't think I've ever seen any.
Sky ITV London Weather is in some schedules starting at 15:59 today - I haven't sky so don't see it on ViX, but there are others on freeview ...
something_fishy
10-03-21, 11:46
Bates Motel (BBC1HD @23:25-00:10) should have a phantom on Thursday ay 11:49. There is nothing in the guide yet, lets see if this changes tomorrow when the original has moved into the past.
There is now a phantom entry for Bates Motel at 11:49-12:34 on Thursday on BBC1 (amongst others).
As far as I can see the phantoms are only created when the program crosses midnight and are in the past. Starts or ends on midnight don't trigger it (so news channels are unaffected) and it appears insensitive to start time:
CourtTV has a phantom "Death of George Floyd Murder Trial at 09:24-13:24 today (repeated from Monday night/Tuesday morning) and on Thursday (repeated from Tuesday night/Wednesday morning).
I don't think I'm getting any "phantoms" if I run EPG Refresh daily at 08:37 instead of my usual 03:37.
But I will continue to experiment running it manually after midnight.
A one off test at a set time for a day or two isn't necessarily going to provide any meaningful results.
Without changing any settings, in the past 3 days I’ve only seen one occurrence of the problem but this doesn’t mean that tomorrow, or within a week, I will access the EPG and see duplicates on half a dozen channels.
BrokenUnusableAccount
10-03-21, 14:30
A one off test at a set time for a day or two isn't necessarily going to provide any meaningful results.
Without changing any settings, in the past 3 days I’ve only seen one occurrence of the problem but this doesn’t mean that tomorrow, or within a week, I will access the EPG and see duplicates on half a dozen channels.
The dependence on time of day I scan is not a one off thing I've noticed. I mentioned it earlier in this thread.
In particular results just before and just after midnight are very different.
BrokenUnusableAccount
11-03-21, 03:06
I have a modified version of opentv.cpp logging tons of stuff as IEPG data 1 is scanned now.
Definitely beginning to understand how it works, but not yet where it goes wrong.
BrokenUnusableAccount
11-03-21, 13:43
To save me searching can anyone point me to the location of the code that limits the size and number of debug logs?
Menu/Setup/System/Logs/Settings
Has anyone checked the stream data to see if bogus events are being sent by the provider? I will be busy decorating for next two weeks, but will try to look if i have some free time.
TODO: fix source type issue, update epg.dat from V7 to V8.
I think you may also be seeing an issue in epgcache code that prevents many of the epg readers inc. OpenTV from updating/removing previously cached events, which would result in you seeing overlapping events from events that were not removed/updated as no longer needed. Fixing this issue would also require updating the epg.dat save/load functions and the file structure of the epg.dat file so the epg Version would also need incrementing from V7 to V8. We need to adjust the eventData type(0xFF) size of the epg source from uint8_t to eventData type(0xFFFF) uint16_t. OpenTV source type(0x4000) and other many of the other readers with a greater source type ID than 255 are currently wrongly being set/saved/loaded as PRIVATE(0X00) source type. Some of the epgcache code checks for PRIVATE or 0 source type and these checks are blocking the wrongly set OpenTV source type (0x00) from ever updating or removing events, it is only allowing new events to be added.
When i next get some free time, i can make patch to fix this issue to test.
BrokenUnusableAccount
11-03-21, 16:32
Has anyone checked the stream data to see if bogus events are being sent by the provider?
That's what I'm trying to do first. But it's taking me quite a long time to figure out everything I need.
I'm working in
https://github.com/BrianG61UK/enigma2/tree/Dev2 but there's hardly anything there yet.
BrokenUnusableAccount
11-03-21, 17:23
Menu/Setup/System/Logs/Settings
Those are already set to maximum. I'm hoping to go higher/larger if I can.
You'll probably have to change them in /etc/enigma2/settings......
config.crash.daysloglimit=8
config.crash.debug_path=/media/hdd/logs/
config.crash.debugloglimit=4
config.crash.enabledebug=True
config.crash.logtimeformat=2
config.crash.sizeloglimit=10
Has anyone checked the stream data to see if bogus events are being sent by the provider?
If this was happening wouldn't everyone see the same double time slot entry? Even from the observations in this thread some people are seeing the problem on days where i see nothing and visa-versa.
BrokenUnusableAccount
11-03-21, 18:55
If this was happening wouldn't everyone see the same double time slot entry? Even from the observations in this thread some people are seeing the problem on days where i see nothing and visa-versa.
Note that I think I only see it if I scan IEPG data 1 after midnight and before some unknown time approximately between about 04:00 and 08:00, plus it's possibly worse the sooner after midnight, but not at all sure about that last bit.
I also wonder if it could be due to a weird compiler bug which might mean only some boxes have the problem.
If this was happening wouldn't everyone see the same double time slot entry? Even from the observations in this thread some people are seeing the problem on days where i see nothing and visa-versa.
I saw lots of doubles this morning on tomorrow's (Friday 12th) EPG slots just before midday, containing entries from Wednesday 10 just before midnight. All have the same 36 hour 24 minute (2184 minutes) offset from the original broadcast time. I know I keep banging on about this value being 0x888 or binary 100010001000 which I think is being caused by a bug in the Julian date handling for events starting before midnight and straddling the midnight hour, but I think the answer lies there.
To save me searching can anyone point me to the location of the code that limits the size and number of debug logs?
Use PuTTY and save log on your PC!
Telnet to box with PuTTY
stop enigma2
init 4
start enigma2 with debug level 4 logging
ENIGMA_DEBUG_LVL=4 enigma2
end debug logging
CTRL+C
start enigma2 in normal mode again
init 3
I saw lots of doubles this morning on tomorrow's (Friday 12th) EPG slots just before midday, containing entries from Wednesday 10 just before midnight. All have the same 36 hour 24 minute (2184 minutes) offset from the original broadcast time. I know I keep banging on about this value being 0x888 or binary 100010001000 which I think is being caused by a bug in the Julian date handling for events starting before midnight and straddling the midnight hour, but I think the answer lies there.
I think all agree that this should be the first place for Brian to look to debug and fix this issue ;)
...TODO: fix source type issue, update epg.dat from V7 to V8.
I think you may also be seeing an issue in epgcache code that prevents many of the epg readers inc. OpenTV from updating/removing previously cached events, which would result in you seeing overlapping events from events that were not removed/updated as no longer needed. Fixing this issue would also require updating the epg.dat save/load functions and the file structure of the epg.dat file so the epg Version would also need incrementing from V7 to V8. We need to adjust the eventData type(0xFF) size of the epg source from uint8_t to eventData type(0xFFFF) uint16_t. OpenTV source type(0x4000) and other many of the other readers with a greater source type ID than 255 are currently wrongly being set/saved/loaded as PRIVATE(0X00) source type. Some of the epgcache code checks for PRIVATE or 0 source type and these checks are blocking the wrongly set OpenTV source type (0x00) from ever updating or removing events, it is only allowing new events to be added.
When i next get some free time, i can make patch to fix this issue to test.Would this explain why these phantom entries arent noticable (that I have observed) when using CrossEPG as i've noticed its code does something with epgcache (creates new instance I think)?
Would this explain why these phantom entries arent noticable (that I have observed) when using CrossEPG as i've noticed its code does something with epgcache (creates new instance I think)?
As far as I am aware, CrossEPG reads the Sky EPG transponder data and stores it in its own database and then populates the EPG cache afterwards. I don't know whether it shares or uses the same code used by the OpenTV reader that EPGRefresh or OpenTVzapper calls. So, it may not encounter the same corruption issue.
If you install RadiotimesXmltvEmulator from the feed, as it is based on CrossEPG, you should also see these bogus events in the xmltv file that it generated in /tmp
I never noticed before... No event_id in that format.
<programme start="20210316110000 +0100" stop="20210316120000 +0100" channel="282_2_6155">
<title lang="en">Lorraine</title>
<sub-title lang="en">Entertainment - Magazine</sub-title>
<desc lang="en">Lorraine Kelly presents a topical mix of entertainment, discussion and showbiz glamour as well as the latest fashion, food and celebrity gossip.</desc>
</programme>
Any sign of the ghost events in that format?
"tvheadend" has some interesting (and very old) comments about event_id's. eg...
https://github.com/tvheadend/tvheadend/commit/be1a42d9092550ffa892e39ef91026e1405192d2
Try to detect duplicate EPG entries from the DVB feed and adjust
EPG accordingly. The EPG code will search for events with the same
DVB event ID +- 2 events from the current one. If the event id is
equal, the prvious (old) entry will be removed in favor of the new one.
Reason for not blindingly trusting the event id is that some networks
seem to (incorrectly) reuse IDs.
BrokenUnusableAccount
13-03-21, 02:03
Sorry - I couldn't work on this last night - I had to get up early this morning for something.
Tonight however I have:
This is the section of opentv.cpp I changed to log stuff, the lines I have added/changed are marked with //BB
OpenTvTitle::OpenTvTitle(const uint8_t * const buffer, uint16_t startMjd)
{
uint8_t descriptor_tag = buffer[0];
if (descriptor_tag == OPENTV_EVENT_TITLE_DESCRIPTOR)
{
uint8_t descriptor_length = buffer[1];
uint8_t titleLength = descriptor_length > 7 ? descriptor_length-7 : 0;
uint32_t startSecond = (UINT16(&buffer[2]) << 1); //BB
startTimeBcd = ((startMjd - 40587) * 86400) + startSecond; //BB
duration = UINT16(&buffer[4]) << 1;
//genre content
//uint8_t flag1 = buffer[6];
//uint8_t flag2 = buffer[7];
//uint8_t flag3 = buffer[8];
char tmp[OPENTV_EVENT_TITLE_LENGTH];
memset(tmp, '\0', OPENTV_EVENT_TITLE_LENGTH);
if (!huffman_decode (buffer + 9, titleLength, tmp, OPENTV_EVENT_TITLE_LENGTH * 2, false))
tmp[0] = '\0';
title = convertDVBUTF8(tmp, sizeof(tmp), 5);
eDebug("[OpenTV] +titl:\"%s\" SMJD:%i Ss:%i SBD:%i len:%i", tmp, startMjd, startSecond, startTimeBcd, duration); //BB
/* storing all the crc unique titles in the title reader phase,
would give us a titles reduction of 155,000 down to 13,000!
we currently add/delete as we go with reduction size ~5000 */
uint8_t *otvt = new uint8_t[titleLength];
memcpy(otvt, buffer + 9, titleLength);
crc32 = opentv_crc(otvt, titleLength);
delete [] otvt;
}
}
Just after midnight I see the "ghost events" showing with startMjd corresponding to today (the day that started just some minutes ago) and startSecond corresponding to start times of over 24:00.
The start times seem to be the old original start time plus 2^17 seconds which is 131072 (or 0x20000) seconds which is 2184 (or 0x888) minutes and 32 seconds.
:confused: but thinking hard.
https://www.brian-gregory.me.uk/GDL/Putty%20zgemmah7.lan%2020210312-124826.log
https://www.brian-gregory.me.uk/GDL/Putty%20zgemmah7.lan%2020210312-233623.log
https://www.brian-gregory.me.uk/GDL/Putty%20zgemmah7.lan%2020210313-000145.log
Just after midnight I see the "ghost events" showing with startMjd corresponding to today (the day that started just some minutes ago) and startSecond corresponding to start times of over 24:00.
The start times seem to be the old original start time plus 2^17 seconds which is 131072 (or 0x20000) seconds which is 2184 (or 0x888) minutes and 32 seconds.
:confused: but thinking hard.Given that there are only 86400 seconds in a day (0x15180) that looks as though 0x20000 is a flag (meaning unknown) that needs to be masked off.
Or it's a bit that's been shifted down that shouldn't be there....(signed vs unsigned right-shift)?
BrokenUnusableAccount
13-03-21, 02:33
Given that there are only 86400 seconds in a day (0x15180) that looks as though 0x20000 is a flag (meaning unknown) that needs to be masked off.
Or it's a bit that's been shifted down that shouldn't be there....(signed vs unsigned right-shift)?
I think it is as if it's a signed number but where as a normal x bit signed number when incremented the meaning goes from +(2^x)-1 to -(2^x) here the meaning changes from +ve to -ve at a different point, probably when it's over 86400 seconds.
The 0x20000 error is because the program currently just always interprets it as a positive unsigned value.
...and startSecond corresponding to start times of over 24:00.
The start times seem to be the old original start time plus 2^17 seconds which is 131072 (or 0x20000) seconds which is 2184 (or 0x888) minutes and 32 seconds.Wait!! That should not happen!!!
The max value for UINT16 is 0xffff. Shifted left by 1 that becomes 0x1fffe. There is no way that the 0x20000 bit should ever be set.
Is there a bug in the UINT16 definition???
BrokenUnusableAccount
13-03-21, 02:47
Wait!! That should not happen!!!
The max value for UINT16 is 0xffff. Shifted left by 1 that becomes 0x1fffe. There is no way that the 0x20000 bit should ever be set.
Is there a bug in the UINT16 definition???
I think I must have explained it badly. Did my second message make it clearer?
They are in a funny format.
Say something yesterday really started at 23:50 on 12th March.
I think they're trying to send it as 13th March starting at minus 00:10 but the software doesn't currently realise sky think negative start times are a thing. :eek:
Or maybe something earlier in the software is messing it up, but I don't think that's likely.
I think I must have explained it badly. Did my second message make it clearer?No.
The relevant thing is that (assuming UINT16 returns a 16-bit unsigned int)
(UINT16(&buffer[2]) << 1);
should never be able to set the 0x20000 bit. Regardless of who intends what.
The problem is that I cannot find a definition of UINT16 anywhere.
EDIT: Found it. It's in the dvbsi++ bytestream.h header file.
// deprecated
#define UINT16(p) r16(p)
#if __BYTE_ORDER == __BIG_ENDIAN
#define r16(p) (*(const uint16_t * const)(p))
...
#else
#define r16(p) bswap_16(*(const uint16_t * const)p)
BrokenUnusableAccount
13-03-21, 03:04
No.
The relevant thing is that (assuming UINT16 returns a 16-bit unsigned int)
(UINT16(&buffer[2]) << 1);
should never be able to set the 0x20000 bit. Regardless of who intends what.
It isn't.
It's like the error if a signed byte -17 is interpreted as unsigned giving incorrect value 249.
That's an error due to interpreting it wrongly of 256 or 0x100, even though the byte has no 0x100 bit in it.
It's like the error if a signed byte -17 is interpreted as unsigned giving incorrect value 249.
That's an error due to interpreting it wrongly of 256 or 0x100, even though the byte has no 0x100 bit in it.No. The compiler has been told it is 16-bit unsigned. Shifting it left 1 into a 32-bit value should never result in bit 18 being set.
[parent]: cat x.c
#include <stdio.h>
#include <stdint.h>
#include <byteswap.h>
static char buffer[4] = {
0x00, 0x00, 0xff, 0xff
};
#define UINT16(p) r16(p)
#if __BYTE_ORDER == __BIG_ENDIAN
#define r16(p) (*(const uint16_t * const)(p))
#else
#define r16(p) bswap_16(*(const uint16_t * const)p)
#endif
int main(int argc, char *argv[]) {
uint32_t startSecond = (UINT16(&buffer[2]) << 1);
printf("res: %08x, %d\n", startSecond, startSecond);
return 0;
}
[parent]: ./x
res: 0001fffe, 131070
Is correct. So why is that not what you are seeing??
BrokenUnusableAccount
13-03-21, 03:40
No. The compiler has been told it is 16-bit unsigned. Shifting it left 1 into a 32-bit value should never result in bit 18 being set.
It's not set.
I told you I explained it badly.
Example:
StartMjd=59286 (13th March 2021) at startSecond=130772 (more than 86400! and currently interpreted as 36:19:32) should be interpreted as minus 00:05 or 23:55 on previous day.
But it's not just a 17 bit signed number of seconds because 86399 has be be positive, probably anything over 86400 should be interpreted as negative, but isn't.
eDebug("[OpenTV] +titl:\"%s\" SMJD:%i Ss:%i SBD:%i len:%i", tmp, startMjd, startSecond, startTimeBcd, duration); //BB
You are printing unsigned values as signed there, which may be throwing us.
startMjd and duration are uint16_t
startSecond and startTimeBcd are uint32_t
You need %u not %i.
BrokenUnusableAccount
13-03-21, 03:57
You are printing unsigned values as signed there, which may be throwing us.
Oops.
Nothing printed out as negative, so I think I got away with it due to the 32 bit architecture.
It's not set.
I told you I explained it badly.
Example:
StartMjd=59286 (13th March 2021) at startSecond=130772 (more than 86400! and currently interpreted as 36:19:32) should be interpreted as minus 00:05 or 23:55 on previous day.
But it's not just a 17 bit signed number of seconds because 86399 has be be positive, probably anything over 86400 should be interpreted as negative, but isn't.
Ignoring the current definition of values for the moment ...
What happens to the value of StartMJD at mid-day? Does it go to 59287 at midday (0.5 rounded up) in which case could the value in buffer[2] actually represent a negative value after mid-day, and before midnight?
Why does the code multiply buffer[2] by 2? (left shift by 1). Does buffer[2] actually contain a time in seconds or the time in a resolution of 2 seconds?
However, having written the above questions the code seems to work for 99.99% of program times.
It's not set.
I told you I explained it badly.
Example:
StartMjd=59286 (13th March 2021) at startSecond=130772 (more than 86400! and currently interpreted as 36:19:32) should be interpreted as minus 00:05 or 23:55 on previous day.
But it's not just a 17 bit signed number of seconds because 86399 has be be positive, probably anything over 86400 should be interpreted as negative, but isn't.
Speculation
In your example of startSecond=130772, this is after buffer[2] has been shifted left by 1
So buffer[2] has a value of 0xFF6A
Ignoring the current definitions
Depending on types, casting this to a 32 bit value could result in either 0x0000_FF6A OR 0xFFFF_FF6A
Now shifting left by 1 gives either 0x0001_FED4 OR 0xFFFF_FED4
One value is more than the number of seconds in a day and the other could be a small negative number representing 1 hour 3 minutes
If MJD does increment by 1 at mid-day the latter could represent midnight on the 13/14th March boundary minus 1 hour 3 minutes rather than what you are assuming of midnight on the 12/13th March boundary plus 130772 seconds (approx 36.3 hours)
BUT again, why do 99.99% of program times appear to be correct?
BrokenUnusableAccount
13-03-21, 12:27
I'm sorry everyone, I must have explained it really badly.
BUT again, why do 99.99% of program times appear to be correct?
99.99% of the start times (startSecond) are less than 86400 and the software interprets them correctly.
I think what's happening is that after midnight programs that started before midnight but are still running get sent with these startSecond values over 86400 and the current MJD value (even though they really started when the MJD value was one less), and you can work out the start times in the normal format like this:
if (startSecond >= 86400) //BB...
{
startSecond -= (0x20000 - 86400);
--startMjd; // Interpret weird time beyond 86400 seconds as previous day
} //...BB
I'm sorry everyone, I must have explained it really badly.
99.99% of the start times (startSecond) are less than 86400 and the software interprets them correctly.
I think what's happening is that after midnight programs that started before midnight but are still running get sent with these startSecond values over 86400 and the current MJD value (even though they really started when the MJD value was one less), and you can work out the start times in the normal format like this:
if (startSecond >= 86400) //BB...
{
startSecond -= (0x20000 - 86400);
--startMjd; // Interpret weird time beyond 86400 seconds as previous day
} //...BB
From what I have seen most the ghost/duplicate programms were not running at the midnight boundary - they had finished long before that.
Or have I misunderstood in that what you are seeing is correct for programs spaning the boundary and not the cause of the duplicate listing.
<programme start="20210316110000 +0100" stop="20210316120000 +0100" channel="282_2_6155">
<title lang="en">Lorraine</title>
<sub-title lang="en">Entertainment - Magazine</sub-title>
<desc lang="en">Lorraine Kelly presents a topical mix of entertainment, discussion and showbiz glamour as well as the latest fashion, food and celebrity gossip.</desc>
</programme>
It may not be programs spanning midnight being the problem but 11pm - note the +1hr
something_fishy
13-03-21, 13:14
From what I have seen most the ghost/duplicate programms were not running at the midnight boundary - they had finished long before that.
That is not true in my case. I think that for all the phantoms that I have seen the original crossed midnight 36 hours, 24 minutes earlier.
BrokenUnusableAccount
13-03-21, 13:14
From what I have seen most the ghost/duplicate programms were not running at the midnight boundary - they had finished long before that.
Or have I misunderstood in that what you are seeing is correct for programs spaning the boundary and not the cause of the duplicate listing.
I think the code is correct for all programs where startSecond >= 86400.
Maybe they continue transmitting information in this "different" way for programs that have finished too.
Why does the code multiply buffer[2] by 2? (left shift by 1). Does buffer[2] actually contain a time in seconds or the time in a resolution of 2 seconds?Yes. It's a 16-bit field so can only hold a days-worth of seconds if the resolution is 2s.
I'm sorry everyone, I must have explained it really badly.I now see what you are suggesting.
But:
99.99% of the start times (startSecond) are less than 86400 and the software interprets them correctly.Why would that be the case?
If start times are "evenly" distributed throughout the day then you'd expect this to be reflected in this value, which would mean ~30% should have this property.
I think what's happening is that after midnight programs that started before midnight but are still running get sent with these startSecond values over 86400 and the current MJD value (even though they really started when the MJD value was one less), and you can work out the start times in the normal format like this:
if (startSecond >= 86400) //BB...
{
startSecond -= (0x20000 - 86400);
--startMjd; // Interpret weird time beyond 86400 seconds as previous day
} //...BB
Try this out and see how it fares.
BrokenUnusableAccount
13-03-21, 13:31
That is not true in my case. I think that for all the phantoms that I have seen the original crossed midnight 36 hours, 24 minutes earlier.
Yes. Actually 36 hours, 24 minutes and 32 seconds.
BrokenUnusableAccount
13-03-21, 13:32
Try this out and see how it fares.
I will, unless I realise it's wrong before tonight.
Everyone: I made the debug log files available at the bottom of message #99 (https://www.world-of-satellite.com/showthread.php?64109&p=511379&viewfull=1#post511379).
Since I seem to be incapable of explaining what I think is going on without confusing myself and getting it wrong it might be best to look at them and try and understand that way if you want to understand.
They're logs from starting with an empty EPG and manually zapping to IEPG data 1 for 3+ minutes.
Quick initial thoughts on this? Thanks for debugging this Brian, are you saying we may only need to decrement the mjd date if start seconds are sent greater than 1 day and the bugus times will then be correct with no adjustment required to the sent start seconds?
// if start seconds are sent greater than 1 day (86400)
// we need to decrement startTimeBcd by 2 days (172800)
if (startSeconds > 86400)
startTimeBcd -= 172800;
BrokenUnusableAccount
13-03-21, 16:02
Quick initial thoughts on this? Thanks for debugging this Brian, are you saying we may only need to decrement the mjd date if start seconds are sent greater than 1 day and the bugus times will then be correct with no adjustment required to the sent start seconds?
// if start seconds are sent greater than 1 day (86400)
// we need to decrement startTimeBcd by 2 days (172800)
if (startSeconds > 86400)
startTimeBcd -= 172800;
No.
Some of what I posted late last night, well I was living up to my handle. I was confused.
Today I'm saying as far as I can see
if (startSecond >= 86400)
{
startSecond -= (0x20000 - 86400);
--startMjd; // Interpret weird time beyond 86400 seconds as previous day
}
should always get you back to sanity.
I guess maybe this would do it too:
if (startSeconds >= 86400)
startTimeBcd -= 0x20000;
To be more cautious you could instead just throw away any record where startSecond >= 86400.
You can maintain old EPG data for as long as 12 hours in epg/settings.
It might at least help seeing where the phantoms are coming from without staying up too late. :)
It may not be programs spanning midnight being the problem but 11pm - note the +1hrThis, of course, means that in one timezone they are in one day whilst in a different timezone they are in a different day.
So the sent EPG contains a "standard" Mjd, requiring a -ve seconds offset?
The start_time in the schedule would/should always be coded as UTC. Any timezone (or DST) offset should be coded in the TOT in order to allow the software calculate and present the data correctly in the local time in the EPG viewed by the user.
The start_time in the schedule would/should always be coded as UTC. Any timezone (or DST) offset should be coded in the TOT in order to allow the software calculate and present the data correctly in the local time in the EPG viewed by the user.That's what I think too. But I also suspect that someone provides the data and someone else converts this into what gets transmitted, and it's not impossible that somewhere in between there is a mismatch of intent. Particularly as I doubt the original data uses Mjd format.
I guess maybe this would do it too:
if (startSeconds >= 86400)
startTimeBcd -= 0x20000;
This correctly fixed the many bogus events i was seeing on the Gemporia Craft channel. Looks good to me, thanks! Maybe you should submit a patch with your fix?
BrokenUnusableAccount
13-03-21, 19:12
This correctly fixed the many bogus events i was seeing on the Gemporia Craft channel. Looks good to me, thanks! Maybe you should submit a patch with your fix?
Okay. I will.
abu baniaz
13-03-21, 21:29
Thanks. Added here
https://github.com/OpenViX/enigma2/commit/2045f5ba782ae4c862f4fffa29539180660df1a6
Pull request submitted to PLi
https://github.com/OpenPLi/enigma2/pull/2889
That's what I think too. But I also suspect that someone provides the data and someone else converts this into what gets transmitted, and it's not impossible that somewhere in between there is a mismatch of intent. Particularly as I doubt the original data uses Mjd format.
The DVB Standard requires the MJD format. I extracted that info and posted it earlier in the thread:
start_time: This 40-bit field contains the start time of the event in Universal Time, Co-ordinated (UTC) and Modified
Julian Date (MJD) (see annex C). This field is coded as 16 bits giving the 16 LSBs of MJD followed by 24 bits coded as
6 digits in 4-bit Binary Coded Decimal (BCD). If the start time is undefined (e.g. for an event in a NVOD reference
service) all bits of the field are set to "1".
EXAMPLE 1:
93/10/13 12:45:00 is coded as "0xC079124500".
duration: A 24-bit field containing the duration of the event in hours, minutes, seconds. format: 6 digits,
4-bit BCD = 24 bit.
EXAMPLE 2:
01:45:30 is coded as "0x014530".
The DVB Standard requires the MJD format. I extracted that info and posted it earlier in the thread:Precisely.
But the original data provider will not be providing the time in Mjd format, so some other entity will be doing a conversion somewhere.
That conversion code may be doing odd things with timezone manipulation. We don't know, but it could explain what is being seen.
Precisely.
But the original data provider will not be providing the time in Mjd format, so some other entity will be doing a conversion somewhere.
That conversion code may be doing odd things with timezone manipulation. We don't know, but it could explain what is being seen.
Is this the correct fix? Where does the incorrect start seconds data come from?
As far as I’m aware the ghost/duplicate entries are for programs that have been previously broadcast and on the day they were broadcast must have had the correct EPG data suggesting the correct startMJD and correct startseconds. Where has the incorrect startseconds now come from? Probably not from the broadcaster as others have reported that other means for fetching the IEPG data don’t show these types of problem.
Also, in my experience, when the problem occurs a re-fetch of the IEPG doesn’t correct the problem UNLESS the EPG cache (and epg.dat file) are first deleted, and often by stopping Enigma to accomplish the deletion. If the EPG cache and epg.dat are deleted and the EPG data then populates from the IEPG then the EPG appears to be problem free suggesting that raw IEPG data wasn’t the cause of an incorrect startseconds value.
The raw data is fine. It's just some entries got incorrectly processed by the reader to 36 hours later than they should. By the time you notice the incorrect entries it looks they come from past programs.
The raw data is fine. It's just some entries got incorrectly processed by the reader to 36 hours later than they should.
That's my point. If the raw data is fine the bug is before the point where the value of startseconds exceeding the number of seconds in a day is being used to establish the start time.
The bug fix above is at a point where the incorrect (startseconds) data is being used and not where it is being incorrectly assigned.
Precisely.
But the original data provider will not be providing the time in Mjd format, so some other entity will be doing a conversion somewhere.
That conversion code may be doing odd things with timezone manipulation. We don't know, but it could explain what is being seen.
I'm not following your train of thought. What other entities/conversions are involved? The raw DVB table information would be created by Redbee (Freesat) or whoever is acting as the media company for Sky and is placed in the satellite or terrestrial datastream. If the data on the stream was in error, then Freesat receivers or Sky boxes would have the same problems of ghost data, perhaps?
That's my point. If the raw data is fine the bug is before the point where the value of startseconds exceeding the number of seconds in a day is being used to establish the start time.
The bug fix above is at a point where the incorrect (startseconds) data is being used and not where it is being incorrectly assigned.So you are suggesting that the problem is in the data being written out to epg.dat since it goes away when that data is first deleted?
But if that were the case why does it only affect data from OpenTV and why does fixing the OpenTV-specific code fix the problem?
Does anyone actually have the raw data to confirm that it is fine?
EDIT: Hmmmm, I see what you mean.
#
The DVB standard note above mentions:
93/10/13 12:45:00 is coded as "0xC079124500".
but this section of code is getting that 6-BCD digit field as a 16-bit field of "2s units".
So something else has been doing that.
OpenTV is not DVB standard.
Test channel: Gemporia Craft
Test program: Sewing Street / Yarn Lane REPEATS
Without the patch this channel still continues to have bogus events for this program after deleting epg.dat and without performing any saving/loading of the epg.dat file
With the patch this frequently bogus program is now fixed, I've added a widget to the skin to also show the event ID's so you can match the corrections made in the screenshots.
These are not old programs that can fixed by deleting epg.dat and clearing cache as some suggest, these are current/future programs that appear wrongly offset by 2^17. I've seen up to 6 future bogus events for this program on this channel over last few days.
6169861699
Gordon, your right the code makes no logical sense.
startSecond having a value greater than 86400 is just an indicator that the data is broken.
And the acquired knowledge is when the data is broken it is always 0x20000 seconds too far in the future so the fix is to just subtract that.
The data in startMjd must be broken too, because, the maximum value startSecond could ever have is 0x20000 and this is always the value that needs subtracting irespective of how much over 86400 the value is.
So you are suggesting that the problem is in the data being written out to epg.dat since it goes away when that data is first deleted?
.
No I'm suggesting possibly more than one problem
Consider this scenario.
I start with no EPG data having stopped Enigma and deleted the epg.dat file and the RAM cache.
The box now gets its EPG data via IEPG/OpenTV
The box has EPG data for days A, B, C, D, E, F and G
Because of the “bug” some information from day A gets inserted into day B or day C, 36 hours later – but I don’t notice this at the time. The bug would also created holes in the EPG for day A. The same source data cannot establish two different time slots 36 hours apart. I haven’t noticed holes in the EPG for the current day, especially as they would be very noticeable with so may ghost/double entries in the future which should have created the equivalent number of holes.
On day B I turn on my box and notice the ghost/duplicate entries in the EPG. I now decide to update the source data from IEPG/OpenTV.
I’m now fetching data for days B, C, D, E, F, G and H – day A is already history so buggy data from day A cannot create ghost/duplicate data in the EPG for day B. [Edit: or can it]
However, the ghost/duplicate entries in day B remains! The EPG is not being overwritten with anything new for day B
Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correct. The content of this forced EPG data must be identical to that of the forced EPG request issued before the epg.dat and epg cache deletion but then it didn’t result in the EPG data being corrected.
Doesn’t this suggest that not only is there a bug in the way the start time is (was) being calculated in that the a ghost time is generated 36 hours in the future but also a bug that prevents more up to date (and correct) source data from the broadcaster also updating a previously corrupted EPG cache?
As has been pointed out if branded sky and freesat boxes don’t have this problem, nor does fetching IEPG in a different way, then the broadcasters are not sending the equivalent of a startsecond value greater than the number of seconds in a day. Startseconds being such a large value is the bug and the fix should be to find out why rather than compensating for the problem at a later stage in processing. Bugs fixed in the wrong place have a nasty habit of returning with other symptoms.
Gordon, your right the code makes no logical sense.Hence the need for a comment associated with the fix.
And the raw data is not fine. It's being sent over the air in this "perverted" format by OpenTV.
These are not old programs that can fixed by deleting epg.dat and clearing cache as some suggest, these are current/future programs that appear wrongly offset by 2^17. I've seen up to 6 future bogus events for this program on this channel over last few days.
If it is a current or future program incorrectly assigned to a future timeslot offset by 36 hours do you also see a hole in the EPG where it should have occurred?
Because of the “bug” some information from day A gets inserted into day B or day C, 36 hours later – but I don’t notice this at the time. The bug would also created holes in the EPG for day A. The same source data cannot establish two different time slots 36 hours apart. I haven’t noticed holes in the EPG for the current day, especially as they would be very noticeable with so may ghost/double entries in the future which should have created the equivalent number of holes.However, let's assume that when the data is first broadcast by OpenTV it is correct.
It ends up in your EPG.
At some future point (while the programme is still running?) this field becomes corrupted.
Now you have a "new" bogus future event.
On day B I turn on my box and notice the ghost/duplicate entries in the EPG. I now decide to update the source data from IEPG/OpenTV.
I’m now fetching data for days B, C, D, E, F, G and H – day A is already history so buggy data from day A cannot create ghost/duplicate data in the EPG for day B. It can't newly create them. But the refetch will not delete items - it (might) change events with the same eventID, but the bogus programme has now finished, so you will not get any update for it. Hence it remains in place.
Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correctBecause now you have removed the bogus entry and there is now no data to recreate it.
BrokenUnusableAccount
14-03-21, 04:26
If it is a current or future program incorrectly assigned to a future timeslot offset by 36 hours do you also see a hole in the EPG where it should have occurred?
In all the cases I noticed it should have occurred in the past so no hole seen.
In all the cases I noticed it should have occurred in the past so no hole seen.
I haven't noticed any holes either and I mostly use the grid view of the EPG where holes would be very noticeable. At some time some correct information must have populated these potential holes.
I don’t fully understand how the EPG data is managed on the boxes but my understanding is that the over the air Freesat EPG comes directly from the broadcast stream when tuned to, say, BBC1 as does the Sky FTA EPG when tuned into the IEPG channel.
The over the air data received by an Enigma box over air must be correct or else the same problem would be seen on Freesat and Sky branded boxes. It must be some software running on the Enigma box that is wrongly interpreting the broadcast EPG data and creating values of startseconds that are not sensible.
In all the cases I noticed it should have occurred in the past so no hole seen.Gemporia Craft has holes.
It can't newly create them. But the refetch will not delete items - it (might) change events with the same eventID, but the bogus programme has now finished, so you will not get any update for it. Hence it remains in place.As explained by LraiZer modified entries can't be removed because eventData type is only 8 bit so impossible for it to hold the identifier of OpenTV (0x4000).
https://www.world-of-satellite.com/showthread.php?64109-Weird-EPG-errors-EPG-Refresh-Recent-versions-of-OpenViX-e-g-5-4-008&p=511282&viewfull=1#post511282
Without the patch being applied first, the fix for updating old events was just removing more correct events that were overlapped by the bogus events, resulting in the event update fix making the first issue 10 times worse.
128 is an old removed free source type so i guess you could temp just change OPENTV=16384 to that as a bypass hack until update patch is fully tested and working?
BrokenUnusableAccount
14-03-21, 11:07
Gordon, you're right the code makes no logical sense.
As I tried to explain in #103 (https://www.world-of-satellite.com/showthread.php?64109&p=511383&viewfull=1#post511383), #105 (https://www.world-of-satellite.com/showthread.php?64109&p=511385&viewfull=1#post511385) (note: in #105 (https://www.world-of-satellite.com/showthread.php?64109&p=511385&viewfull=1#post511385) I wrote 249 when I should have written 239)
They are kind of treating the field I've called startSecond as a signed time.
However it doesn't roll from meaning a +ve number to meaning a -ve number in the usual place (when the MSB flips to a 1).
Instead it seems it should be treated as a signed negative number if it's greater than or equal to 86400.
Then just as when you interpret a normal negative signed 17 bit number as unsigned the result is 0x20000 too high.
Just for the curious, here is a debug data dump log of the current bogus events being fixed.
Data cycle appears to currently be running at 139 seconds to populate the epg ;)
Without the patch being applied first, the fix for updating old events was just removing more correct events that were overlapped by the bogus events, resulting in the event update fix making the first issue 10 times worse.
128 is an old removed free source type so i guess you could temp just change OPENTV=16384 to that as a bypass hack until update patch is fully tested and working?
Could you make a patch for this please?
The over the air data received by an Enigma box over air must be correct or else the same problem would be seen on Freesat and Sky branded boxes. It must be some software running on the Enigma box that is wrongly interpreting the broadcast EPG data and creating values of startseconds that are not sensible.
Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correct. The content of this forced EPG data must be identical to that of the forced EPG request issued before the epg.dat and epg cache deletion but then it didn’t result in the EPG data being corrected.
This is what I was trying to get at earlier by saying that I don't notice these issues with Sky UK when using CrossEPG to grab EPG data! Have previously seen when I switched from CrossEPG to EPGRefresh/OpenTV.
It's also looks like CrossEPG does some work-around maybe with epg cache as it creates a new instance of EPG cache.
Doesn't this, at least for Sky UK, potentially mean that broadcast data is correct and yes there could be multiple issues and EPG cache being other issue?
BrokenUnusableAccount
14-03-21, 13:27
This is what I was trying to get at earlier by saying that I don't notice these issues with Sky UK when using CrossEPG to grab EPG data! Have previously seen when I switched from CrossEPG to EPGRefresh/OpenTV.
It's also looks like CrossEPG does some work-around maybe with epg cache as it creates a new instance of EPG cache.
Doesn't this, at least for Sky UK, potentially mean that broadcast data is correct and yes there could be multiple issues and EPG cache being other issue?
I did look quickly at the OpenTV part of CrossEPG and as far as I could see the relevant part of the code looked very like that in OpenViX's opentv.cpp.
No obvious sign of any means for CrossEPG to already be correct that I noticed.
BrokenUnusableAccount
14-03-21, 13:41
As I tried to explain in #103 (https://www.world-of-satellite.com/showthread.php?64109&p=511383&viewfull=1#post511383), #105 (https://www.world-of-satellite.com/showthread.php?64109&p=511385&viewfull=1#post511385) (note: in #105 (https://www.world-of-satellite.com/showthread.php?64109&p=511385&viewfull=1#post511385) I wrote 249 when I should have written 239)
They are kind of treating the field I've called startSecond as a signed time.
However it doesn't roll from meaning a +ve number to meaning a -ve number in the usual place (when the MSB flips to a 1).
Instead it seems it should be treated as a signed negative number if it's greater than or equal to 86400.
Then just as when you interpret a normal negative signed 17 bit number as unsigned the result is 0x20000 too high.
Let me put that another way just in case some people still can't see it.
I think that startSecond (including the 0 LSB that's been added) is the bottom 17 bits from a larger (say 32 bit) signed integer that is only allowed to take on even values between -44672 and 86398.
They are kind of treating the field I've called startSecond as a signed time.
Which seems and odd thing to do.
Would you ever consider that I'm writing this at -11:00 on Monday?
It sounds more like and artefact of the way they process service suppliers data rather than being intentional. It's noticeable that it only seems to affect programmes which wrap between days at midnight. That's midnight UK, so for many Western European countries these do not span two days - they are completely within the second day.
But since the OpenTV spec isn't published we can only ever speculate, and just have work with whatever is sent out.
BrokenUnusableAccount
14-03-21, 14:34
Which seems and odd thing to do.
Would you ever consider that I'm writing this at -11:00 on Monday?
Perhaps the day is the day the program finishes?
Perhaps the day part is not allowed to be before today?
It sounds more like an artefact of the way they process service suppliers data rather than being intentional. It's noticeable that it only seems to affect programmes which wrap between days at midnight. That's midnight UK, so for many Western European countries these do not span two days - they are completely within the second day.
But since the OpenTV spec isn't published we can only ever speculate, and just have work with whatever is sent out.
But presumably genuine Sky boxes know what to do with it.
BrokenUnusableAccount
14-03-21, 15:34
While we have developer types in here can anyone tell me what's going on in opentv.cpp a few lines underneath where I was working?
uint8_t *otvt = new uint8_t[titleLength];
memcpy(otvt, buffer + 9, titleLength);
crc32 = opentv_crc(otvt, titleLength);
delete [] otvt;
WT* is it necessary to allocate memory to make a copy before working out the crc and then release it afterwards?
If you look at the definition of opentv_crc further up it can't modify it's input data.
It doesn't make any sense to me.
WT* is it necessary to allocate memory to make a copy before working out the crc and then release it afterwards?
If you look at the definition of opentv_crc further up it can't modify it's input data.
It doesn't make any sense to me.
Wasn't this also used and deleted somewhere instead in the early code creation during debug testing and then that code was removed and just a quick temporary delete added back here and then all this was forgot about?
Looks like you've got something else you'll have to fix :thumbsup:
Could you make a patch for this please?
[PATCH] New ENIGMA_EPG_V8 file structure
I think this patch should open up all the cache management code that was previously being blocked.
Will needs a lot of monitoring and testing over a period of time to see what it actually does or breaks!
Just for the curious, here is a debug data dump log of the current bogus events being fixed.
Data cycle appears to currently be running at 139 seconds to populate the epg ;)
If you inspect the stream can you see these erroneous values?
[PATCH] New ENIGMA_EPG_V8 file structure
I think this patch should open up all the cache management code that was previously being blocked.
Will needs a lot of monitoring and testing over a period of time to see what it actually does or breaks!
Thanks. That is now up on our repo for testing. :thumbsup:
I'm ready to test when software is available:)
If you inspect the stream can you see these erroneous values?
These erroneous values? e3 e0
They are in the stream buffer data that i output in the log file.
e3 << 9 = 116224
e0 << 1 = 448
116224 + 448 = 116672
1615766400 + 116672 = 1615883072
1615883072 - 131072 = 1615752000
BOGUS: Sewing Street / Yarn Lane REPEATS
mjd=59288(bcd:1615766400) bcd=1615883072 bcdfix=1615752000 sec=116672(E3 E0) duration=43200
b5
22
e3
e0
54
60
6b
Is it possible to get CRID from this stream? Simon is planning some code for programming series link.
There does seem to be some end-user documentation for an OpenTV API here:
https://opentv.nagra.com/sites/default/files/api/medialive_os_api/5.2.4/CCOM/index.html
Not sure how complete this is, but I can't find any reference to CRID data in it.
There does seem to be some end-user documentation for an OpenTV API here:
https://opentv.nagra.com/sites/default/files/api/medialive_os_api/5.2.4/CCOM/index.html
Not sure how complete this is, but I can't find any reference to CRID data in it.Definitely has it though because Sky boxes use it. :)
Definitely has it though because Sky boxes use it. :)Good point...
So we'll need another hack to use it.
No spurious epg entries today on build 5.004.009.005 (as expected). Good work Brian on tracking down that gremlin.
No spurious epg entries today on build 5.004.009.005 (as expected). Good work Brian on tracking down that gremlin.
..... and @something_fishy who spotted the 36 hours 24 minutes link.
BrokenUnusableAccount
24-03-21, 15:47
If anyone is able to check this change to OpenTV EPG doesn't cause problems with other services (other than Sky UK) that use OpenTV EPG, that would be good.
I only have 28.2E to test on. so cant test on others, but here is some code ideas for your pondering ;)
Interpretation of the data sent as a negative number, not a positive:
// HACK: startSecond is detected as being from the previous
// mjd date when the h:m:s is sent as greater than 1 day.
// when this occurs, shifted two's complement has a negative
// sign bit of a signed integer, and is not a positive number.
int32_t startSecond = UINT16(&buffer[2]) << 1;
// if h:m:s is sent as greater than 1 day in seconds,
// first bit is a negative sign bit without padding.
if (startSecond >= 86400)
startSecond = 0xFFFE0000 | (startSecond & 0x1FFFF);
startTimeBcd = ((startMjd - 40587) * 86400) + startSecond;
I only have 28.2E to test on. so cant test on others, but here is some code ideas for your pondering ;)
Interpretation of the data sent as a negative number, not a positive:
// HACK: startSecond is detected as being from the previous
// mjd date when the h:m:s is sent as greater than 1 day.
// when this occurs, shifted two's complement has a negative
// sign bit of a signed integer, and is not a positive number.
int32_t startSecond = UINT16(&buffer[2]) << 1;
// if h:m:s is sent as greater than 1 day in seconds,
// first bit is a negative sign bit without padding.
if (startSecond >= 86400)
startSecond = 0xFFFE0000 | (startSecond & 0x1FFFF);
startTimeBcd = ((startMjd - 40587) * 86400) + startSecond;
Is this tested code on 28.2E?
On another note, do you know about CRID on this transport stream? It would be a nice addition now that Simon is adding code in eEPGCache to process it.
https://github.com/OpenViX/enigma2/commit/c89c1e1802e0d4b841978c2e554e5be4134c495d#diff-b15024bd2b4ea380025955f8d690ba180059716f8f64bd2366 de9dcd0d44b0c0
Is this tested code on 28.2E?
Only tested by me so far based on how i now interpret the data can be a positive or a negative signed integer.
It correctly adjusted about 10 yesterday inc. all the regular occurring ones on our test channel Gempora Craft.
On another note, do you know about CRID on this transport stream? It would be a nice addition now that Simon is adding code in eEPGCache to process it.
https://github.com/OpenViX/enigma2/commit/c89c1e1802e0d4b841978c2e554e5be4134c495d#diff-b15024bd2b4ea380025955f8d690ba180059716f8f64bd2366 de9dcd0d44b0c0
I dont see CRID, as far as i am aware it uses URI as defined:
DVB original_network_id = 1 (0x1)
DVB transport_stream_id = 10 (0xa)
DVB service_id = 100 (0x64)
DVB event_id = 1000 (0x3e8)
Unique eventId string = 0001000a006403e8
DVB original_network_id = 1 (0x1)
DVB transport_stream_id = 10 (0xa)
DVB service_id = 100 (0x64)
Unique serviceId string = "0001000a0064"
event.serviceId = “serviceId_XYZ”
event.uri = “tv://channel.serviceId_XYZ”
event.eventID = eventId_XYZ
event.seriesID = seriesLink (we can now proccess)
event.episodeID = ??? (cant see this in the data)
.sourceURL = event.uri
Looking at OpenTV.nagra.com website, it references "seriesId" and "episodeId" as @LraiZer mentions...
https://www.google.co.uk/search?q=site%3Ahttps%3A%2F%2Fopentv.nagra.com+ser iesid&ei=G1VgYJ3tKouU8gLegLiABg&oq=site%3Ahttps%3A%2F%2Fopentv.nagra.com+seriesid&gs_lcp=Cgdnd3Mtd2l6EANQ1NYBWPDrAWD_7AFoAXAAeACAAew CiAGQEJIBBzQuMy40LjGYAQCgAQGqAQdnd3Mtd2l6wAEB&sclient=gws-wiz&ved=0ahUKEwjd5pfV3tLvAhULilwKHV4ADmAQ4dUDCA0&uact=5
https://docs.nagra.com/docs/os/524/series-linking-tutorial
BrokenUnusableAccount
30-03-21, 01:44
I only have 28.2E to test on. so cant test on others, but here is some code ideas for your pondering ;)
Interpretation of the data sent as a negative number, not a positive:
// HACK: startSecond is detected as being from the previous
// mjd date when the h:m:s is sent as greater than 1 day.
// when this occurs, shifted two's complement has a negative
// sign bit of a signed integer, and is not a positive number.
int32_t startSecond = UINT16(&buffer[2]) << 1;
// if h:m:s is sent as greater than 1 day in seconds,
// first bit is a negative sign bit without padding.
if (startSecond >= 86400)
startSecond = 0xFFFE0000 | (startSecond & 0x1FFFF);
startTimeBcd = ((startMjd - 40587) * 86400) + startSecond;
I'm not entirely sure it's any clearer now that we have a comment by the original code.
Though it does kind of process things in a more logical order.
You could write 0xFFFE0000 as (-0x20000) though that shouldn't be necessary since int32_t should be guaranteed to be 32 bits.
BrokenUnusableAccount
30-03-21, 01:51
Looking at OpenTV.nagra.com website....
This must be documentation for some piece of software that takes in EPGs and things and produces the "secret" OpenTV data that Sky transmit.
I certainly can't find much information in there to help with interpreting the raw transmitted data (such as EPG).
Perhaps all the APIs and things described are how the TV companies transfer their EPG data, and so on, to Sky.
(but it does, as bbbuk pointed out, give you a good idea of what's in there somewhere)
BrokenUnusableAccount
07-04-21, 01:13
Gemporia Craft has holes.
For a while after using the fix I think I wasn't seeing any holes in Gemporia Craft.
But now I am seeing holes, and also weird program ending times of the form xx:x7 just before the holes.
All other channels seem fine though.
I think the most liekly explanation is that Gemporia Craft are doing something weird.
Just thinking aloud ;)
Gemporia Craft take the piss with 20 hour events = 72000 seconds duration. I guess you could start your debug by checking opentv code for uint16_t getDuration limits of 65535. Maybe there could also be an upper int18_t limit on a (startSecond+duration) that also triggers startSecond as a negative, even when startSecond is not greater than 1 day?
For a while after using the fix I think I wasn't seeing any holes in Gemporia Craft.
But now I am seeing holes, and also weird program ending times of the form xx:x7 just before the holes.
All other channels seem fine though.
I think the most liekly explanation is that Gemporia Craft are doing something weird.
I've fixed the issue on OpenPLi where xx.x7 was seen for 20 hours (72000 seconds) events.
This was just a wrongly defined upper uint16_t limitation which subtracting 65535 from 72000 giving a false duration of 1 hour 47 minutes.
The issue of the holes now again on Gemporia Craft is that most events they send are sent correctly as GMT +0, but the ones that appear to be causing the issue are the new 20 hour events that are sent with an 1 hour too late startTime. I can only assume this is related to BST +1 change last week that they have not done correctly and so may only be a temporary issue once they realize their mistake?
Powered by vBulletin® Version 4.2.5 Copyright © 2024 vBulletin Solutions Inc. All rights reserved.