View Full Version : epgcache overlap problems
I assume that with more than one tuner that the EPG will be populated automatically by the use of other tuner in the background (if not actively being used for a recording). Is this correct? How often does it automatically fetch the EPG, and when? On coming out of deep standby or some set time? For example, if away on holiday for two or three weeks and the box doesn’t wake up from deep standby for a recording will this feature automatically wake up the box to fetch the EPG or does the user have to set up some form of wake up timer so that this feature works reliably?
Currently I have a repeat timer that wakes up the box for a IEPG recording at a time that I don’t usually use the box. The location for the recording is the trash folder and the box remains on IEPG for at least 10 minutes. This seems to work well except very occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced) BUT only for those 10 minutes at the ends of the listing. I have Freesat EPG disabled but I do have EIT enabled because I also have terrestrial tuners and want over the air EPG for Freeview.
except very occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced) BUT only for those 10 minutes at the ends of the listing.
An example
The double entry on BBC ONE (SD and HD) for "Election" and "BBC News". Note: even though my bouquet is labelled fav_freesat that is just a name I haven't changed when I changed to Sky FTA in ABM and disabled Freesat EPG in the appropriate settings.
61320
BrokenUnusableAccount
25-01-21, 01:51
occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced) BUT only for those 10 minutes at the ends of the listing. I have Freesat EPG disabled but I do have EIT enabled because I also have terrestrial tuners and want over the air EPG for Freeview.
Me too.
Never seems to happen with Freeview, just Sky EPG.
Presumably it's the now/next disagreeing with IEPG 1 data from some time ago, but I don't really know.
... except very occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced).
An example
The double entry on BBC ONE (SD and HD) for "Election" and "BBC News". Note: even though my bouquet is labelled fav_freesat that is just a name I haven't changed when I changed to Sky FTA in ABM and disabled Freesat EPG in the appropriate settings.
61320
Hmm. since installing 5.4.005. disabling my daily timer for IEPG and using opentvzapper instead I’m now seeing a LOT more overwritten EPG entries. Just checking before posting this, and after my box has been on for 3 hours, I have 12 channels showing the problem (in a bouquet with 40 channels) whereas before I would only see the occasional one of two, and not every day.
As I've mentioned before I also have EIT enabled to obtain terrestrail EPG data as I have both types of tuners in my box.
Seeing as how this just tunes the IEPG the same as using epgrefresh or directly I can't see how this would produce a different result.
I agree it should not make any difference but I wasn't using epgrefresh.
I was using a repeat timer to switch to IEPG at around 6:30am each day. It was a record timer but with the file directed directly to the trash folder. The timer ended by putting the box into standby and 10 minutes later a power timer put the box back into deep standby. This sequence ensured that the EPG was saved to disk
Did you disable Freesat?
Yes
Try disabling now and next.
I'm will experiment with a few EIT settings over the next week.
Also I don't understand how you think this is connected to terrestrial.
As I understand it, the EIT gets you the 7 day EPG on Freeview AND for now/next on satellite. As there is only a single option to enable EIT in the configuration menu then I’m assuming that enabling EIT will give both terrestrial AND satellite functionality for a box with the two types of tuner. I already know that disabling EIT kills to ability to get an over the air EPG on terrestrial Freeview.
This problem also raises another question. If I had 12 overwriten EPG entries this morning is the original EPG obtained from IEPG a few hours ago or the overwritten information obtained from maybe now/next the correct information? It one case on the taking pictures channel the film in the title EPG had been changed and another film was broadcast. Now/Next gave the correct film title so even with an overwrite of the start/stop timing other things are not overwritten, such as the description.
Is the IEPG data from sky so unreliable that it requires 12 off corrections from now/next with the same hour or two?
Whatever process you used to zap the box to IEPG is not relevant. What happened afterwards may be relevant. I.e. does the save to epg.dat and then sleep do some sort filtering of the data?
Whatever process you used to zap the box to IEPG is not relevant. What happened afterwards may be relevant. I.e. does the save to epg.dat and then sleep do some sort filtering of the data?
I downloaded 5.4.005 and installed then deleted the repeat timer I had for IEPG and enabled opentvzapper. I then deleted the EPG cache before rebooting the box to see if opentvzapper was working. The Sky FTA EPG data was populated soon afterwards.
So, this mornings' observations are with only opentvzapper running for IEPG on 28.2E (and EIT enabled) so unlikely to be my previous setup.
My box was switched on from deep standby at approx 8:30 am and the overwrite of the EPG data for 12 channels noticed just before I posted at around 1pm. Even if the EPG data from yesterday was read from the hard disk on switch on opentvzapper would have kicked in shortly afterwards to update the (I)EPG data. I have two satellite tuners and no recordings were taking place at anytime during the morning.
I realise that opentvzapper is just a background channel changer (zapper) but something is causing the EPG data in RAM cache to be overwritten, and not very well. I realise also that I have seen this problem, but to a much lesser extent, before opentvzapper and even possibly before epgrefresh.
Although a different method of obtaining over the air EPG I don’t recall seeing the problem on the Freeview terrestrial EPG which also suggests the problem is not how my box saves EPG to disk on shut down nor how it reads EPG from disk after startup from deep standby, and possibly not related to how the EPG is updated if a single method is used (only EIT in the case of Freeview). My box is in deep standby every night.
As I said in a previous post, I will disable EIT or now/Nex EIT to see if the problem goes away. This may take a week or so to reach any conclusions.
Give me a current example.
Give me a current example.
I cannot do this at the current time. After changing some of the EIT configuration this afternoon I deleted the EPG cache and ran opentvzapper manually to get new EPG data. I haven't seen the problem since. However, in the past I may go a few days or more before seeing any overwriting of EPG data. This morning the problem was not just one or two channels at any one time as seen before but this time the EPG data for 12 channels had problems. Possibly just an unrelated coincidence after installing 5.4.005?
Could this issue be caused by corrupt EPG data in cache?
CrossEPG looks like it handles cache. Don't think EPGrefresh did (when I first looked at its code when it first came on the scene) and quick look at OpenTVZapper doesn't look like it handles cache really (but could be wrong as was a quick look).
If there is a potential for corrupt EPG cache then could it coded into OpenTVZapper that it deletes EPG (cache) before it downloads the EPG. Could it then save EPG data once its done? This could enabled/disabled by a menu option to handle cache. So if enabled it would delete EPG cache etc before download etc and if disabled (default as is now) it wouldn't do anything. User choice?
BrokenUnusableAccount
29-01-21, 00:27
I downloaded 5.4.005 and installed then deleted the repeat timer I had for IEPG and enabled opentvzapper. I then deleted the EPG cache before rebooting the box to see if opentvzapper was working. The Sky FTA EPG data was populated soon afterwards.
So, this mornings' observations are with only opentvzapper running for IEPG on 28.2E (and EIT enabled) so unlikely to be my previous setup.
My box was switched on from deep standby at approx 8:30 am and the overwrite of the EPG data for 12 channels noticed just before I posted at around 1pm. Even if the EPG data from yesterday was read from the hard disk on switch on opentvzapper would have kicked in shortly afterwards to update the (I)EPG data. I have two satellite tuners and no recordings were taking place at anytime during the morning.
I realise that opentvzapper is just a background channel changer (zapper) but something is causing the EPG data in RAM cache to be overwritten, and not very well. I realise also that I have seen this problem, but to a much lesser extent, before opentvzapper and even possibly before epgrefresh.
Although a different method of obtaining over the air EPG I don’t recall seeing the problem on the Freeview terrestrial EPG which also suggests the problem is not how my box saves EPG to disk on shut down nor how it reads EPG from disk after startup from deep standby, and possibly not related to how the EPG is updated if a single method is used (only EIT in the case of Freeview). My box is in deep standby every night.
As I said in a previous post, I will disable EIT or now/Nex EIT to see if the problem goes away. This may take a week or so to reach any conclusions.
Maybe with Freeview there is a simple means in the received data (perhaps the EIT number mentioned in another thread) for the EPG store to automatically delete old EPG data when it gets changed.
While with IEPG data 1 perhaps there isn't such a simple way to detect that a newly received piece of EPG data replaces an old piece so both get stored. Presumably exactly identical records aren't stored twice. If the only way is to look for something like overlapping start and end times it may not be implemented, or not implemented well enough.
@adm
The attachment in post #11 looks like the duplicate occurred at midnight (ie a day boundary), and the entry you highlighted was the first programme for "today" as it was when you took the snapshot.
Do the others you've seen fit the same scenario?
(Hope it makes sense. :confused: )
@adm
The attachment in post #11 looks like the duplicate occurred at midnight (ie a day boundary), and the entry you highlighted was the first programme for "today" as it was when you took the snapshot.
Do the others you've seen fit the same scenario?
(Hope it makes sense. :confused: )
Until now I haven't had a systematic approach to the problem - it's just a minor annoyance that I've noticed at various times.
Currently I'm running the box with certain EPG settings disabled and checking what is happening with the 28.2E EPG data. If the problem disappears I will re-instate the disabled configurations to see if the problems re-appear. As the problem was previously not seen every day, now a few days with a clear EPG doesn’t mean that the change in EPG setting has necessarily cured the problem. It may take some time to come to meaningful conclusions.
@adm
The attachment in post #11 looks like the duplicate occurred at midnight (ie a day boundary), and the entry you highlighted was the first programme for "today" as it was when you took the snapshot.
Do the others you've seen fit the same scenario?
(Hope it makes sense. :confused: )
Other instances do not have the same scenario. Where I had 12 occurances at the same time all the problems relaeted to programs starting late morning.
.... I think we need to see when it's happening, with enough notice to be able to check if (as in post #11) the time ever was adjusted by 5 minutes.
The first problem seen since i changed settings
Settings
61342
01:35 to 2:45
Problem seen
61343
61344
61345
The entry for "The Professionals" should be from 1:35 to 2:45 and not overwritten by Auto Mundial (which was the program before The Professionals AND on my other box that doesn't show the problem River Monsters is not present in the EPG.
Note: the box has been on continiously for approx 18 hours.
My other box is a Zgemma H9S with a single satelitte tuner, with opentvzapper disabled and with a repeating timer set to switch the box from deep standby to the IEPG channel each morning at 6:30am
the setting for ths box is
61346
And the EPG is shown correctly for ITV4 at the same time as the above
61347
I'm now going to delete the EPG cache and change a setting to see if the problem re-appers in the next day or so.
After a bit of investigation I do believe the problem to be both EIT now/next and IEPG populating the same EPG data base.
If I disable all modes for getting data from IEPG, then delete the EPG cache the following setting will populate (or not) the (grid) EPG with now/next data
61348
So, if both Enable EIT EPG and Enable OpenTV EPG are both configured to yes the (grid) EPG for Sky FTA will be populated with now/next data.
Now, if the IEPG channel is also selected then the (grid) EPG data will be updated from this second source, as well as now/next
You could argue that for SKY FTA the Enable EIT EPG option can be turned off. This would give IEPG as the single source of EPG data and/or any Internet download as the single source of data.
However, I want Enable EIT EPG to be on to get the over the air terrestrial EPG for Freeview!
People with only satellite tuners may never see the problem if EIT EPG is configured to be off BUT the question arises about having an up to date EPG for Sky FTA. If IEPG or a Internet download is only updated once a day or every 6 hours the problem I found the other day with 12 TV channels having incorrect or overwritten data may indicate that now/next is possibly more up to date with respect to program changes.
The show EIT now/next in info bar option only seems to make a difference to the information shown in the Info Bar. In some cases in the table above where now/next doesn’t populate the EPG cache the now/next information is correctly shown in the Info bar.
The column of the table that explain what each yes/no means is missing.
The column of the table that explain what each yes/no means is missing.
The first row in the table is the headings for each column, the next 8 rows show all the possible settings for the three configurable menu items.
The last column in the table is the result - the observation about now/next data appearing in the EPG cache.
It shows that in
menu -> setup -> EPG -> settings
that option " Show EIT now/next in infobar" is a don't care as it is not controlling what appers in the EPG cache, and seen in the grid EPG
Only the combination of the options "Enable EIT data" and "Enable OpenTV EPG" both being set to "Yes" result in now/next appering in the EPG cache.
"Enable EIT data" and "Enable OpenTV EPG" both being set to "Yes" is the setting I need for over the air EPG for both satellite (Sky FTA) and terrestrial (Freeview) population.
So it turns out that previously epgcache OpenTV reader was using the event start time as the event_id. LraiZer has provided a patch to cure this which needs testing. https://github.com/OpenViX/enigma2/commit/0731baaaa16d7985cb42b003d3ec50dd5631b871
If you want a binary to use for testing I can provide an ARM version.
I'm copying this here as it is relevant to this thread.
Cache has no namespace or orbital position, just "uniqueEPGKey" from sid, tsid, and onid. If it matches between any type of T/S/C, then it rely's on the cache source type. These are priority checks performed before overwriting any current cached data.
Cache also has a type to check against where the epg came from, this is eg. if(source > type):
https://github.com/OpenPLi/enigma2/blob/3664db5f91860c05a80962ec9127f885daecd515/lib/dvb/epgcache.cpp#L40
The only epg with a lower write priority than opentv at the moment is EPG_IMPORT, but you may notice epg from this EPG_IMPORT sets a do not update from eit flag on its uniqueEPGKey's.
https://github.com/OpenPLi/enigma2/blob/103d129496df5b236473dc134d2554ba45456f9c/lib/dvb/epgcache.h#L249
Based on same uniqueEPGKey from different epg sources:
EIT schedule would overwrite FREESAT_SCHEDULE
FREESAT_SCHEDULE_OTHER would overwrite VIRGIN_SCHEDULE
OPENTV would not overwrite any of the above..
From here: https://www.world-of-satellite.com/showthread.php?62706-EPG-Refresh-iEPG-and-VPS&p=508597&viewfull=1#post508597
So it turns out that previously epgcache OpenTV reader was using the event start time as the event_id. LraiZer has provided a patch to cure this which needs testing. https://github.com/OpenViX/enigma2/commit/0731baaaa16d7985cb42b003d3ec50dd5631b871
If you want a binary to use for testing I can provide an ARM version.
Just for my own clarification...
I've been following this thread for days and noticed that @gort was indicating that the event id number generated by epgrefresh/opentv reader was different than that generated by CrossEPG and that this was causing issues for the VPS plugin when it was checking now/next EIT data and couldn't seem to match that particular event id with the one stored in the timer entry. I had a look at the actual EIT data on various 28.2 transponders and found that the event id info for most channels was a simple incrementing numeric sequence like 182, 183,184 etc. The info is stored in the table as a hex value. When I checked a timer entry for the same programme, it seemed to be a different, apparently random value which didn't match. Lraizer's post indicated that the event id was being generated in the OpenTV reader though an AND function using the start time of the event, whereas CrossEPG was actually using the even id stored by the broadcaster in the table. Am I correct in this?
It would explain the effect where small changes in the start time resulted in "ghost entries" being presented in the EPG display as they had different event ids generated by the AND function.
I'm further assuming that your amendment to the EPGcache code should now result in consistent use of the broadcaster event id by CrossEPG and the Opentv reader (and probably the Freesat reader) so that the issue of ghost entries should be minimised or eliminated and that the VPS plugin should work also?
@fat-tony, I'm only concerned that our built-in reader works properly. I haven't explored what might be happening in Cross.
Tests using LraiZer's patch show corresponding event_id in timers created from both EIT and OpenTV data. All that is needed now is to see what happens when there is a change in the data.
Yes - thanks for the changes. I think what's now happening is that the event ID values are now the same in Cross and in the OpenTV reader. I would hazard a guess that slight schedule moves were causing the overlapping entries in the EPG display due to the event id values being generated "on the fly". It will be interesting to see what will happen on a schedule change now!
I do apologize, and I know this is a little off the topic but after installing OpenViX 5.4.005 on my GigaBlue UHD QUAD 4K I have noticed when the receiver is in standby and OpenTV download (28.2) has updated the EPG the bouquet always defaults to 'Last Scanned' when I first turn on receiver. Is their a way to stop this happening and set it to the last bouquet I was was using as it has always been before?
I do apologize, and I know this is a little off the topic but after installing OpenViX 5.4.005 on my GigaBlue UHD QUAD 4K I have noticed when the receiver is in standby and OpenTV download (28.2) has updated the EPG the bouquet always defaults to 'Last Scanned' when I first turn on receiver. Is their a way to stop this happening and set it to the last bouquet I was was using as it has always been before?I can't reproduce this... but...
If you watch what is happening via OpenWebInterface you can see that the tuner is not released when the download completes.
https://github.com/OpenViX/enigma2/commit/75e530fa791db602499c8ccd511e04b06f3fc4e1
The above commit fixes that. Give it a try and see if it cures your problem.
Thanks Huevos for your reply,
I think that could be the issue as the front LCD display shows an active tuner on when in standby and nothing is recording. I am a complete novice using lynx. I know I can edit py files using Easy Python Decompiler, but how do you convert back to a py file to copy the file back over to the receiver? Thanks again.
You don't. You just take the py file from github, send it to your receiver, restart the receiver. Now the pyo on the box will be the latest version and you can delete the py.
Thanks Huevos for your reply,
I think that could be the issue as the front LCD display shows an active tuner on when in standby and nothing is recording. I am a complete novice using lynx. I know I can edit py files using Easy Python Decompiler, but how do you convert back to a py file to copy the file back over to the receiver? Thanks again.
https://www.world-of-satellite.com/showthread.php?56971-After-upgrading-to-5-0-016-AutoTimer-fails/page2#17
https://www.world-of-satellite.com/showthread.php?56971-After-upgrading-to-5-0-016-AutoTimer-fails/page2#16
Powered by vBulletin® Version 4.2.5 Copyright © 2024 vBulletin Solutions Inc. All rights reserved.