Extended EPG Data

[AB2022]

Is there a source (free or paid) of UK FTA EPG data that extends beyond 7days? OpenTV and Rytec only seem to cover the next 7 days and a couple of sites on the web (digiguide and tvguide) cover 7-14 days but don't look to have a download option.

[JonMMM]

This might help it uses tvguide

https://youtu.be/0ia9-Be_Fv4

[birdman]

The BBC used to supply a download of 2-week data, about 8(?) years ago.
Then this was moved to the Atlas service.
Which requires you to create an account and, AFAIR, the licence means it couldn't be integrated into Vix.

I did write something to download their data (back then I had a Toppy) but in the end gave up working on it, partly because of the licence requirements, but mostly as I found 7 days EPG was sufficient.
In the end the biggest thing I missed was the longer description of films, but if I really want that then an IMDb lookup is possible.

[AB2022]

JonMMM Tx for the suggestion - I've managed get an xmltv file that looks to contain 14 days of EPG data however until I can load it into the EPG I can't be sure - thats the next challenge. Unless someone knows of an "off the shelf" solution to get it into the EPG, Google indicates that I need to create custom sources and channel xml files to enable EPG importer to load the file. I'll post back with any findings.

[AB2022]

I now have a 14D EPG. It took a while to find all the information as it was scattered across several threads/posts so the following is my attempt to bring it all together in case others want to try it.
Note though this is specific to my requirements (14D EPG for UK FTA via Astra 28E using Windows 10) I hope there's enough detail to enable changes to be made for different sources/channels etc.

The 1st step is to source the EPG data. Thanks to JonMMM's posting above I've used mc2xml. Use the link to download mc2xml and to read the usage instructions.
As the windows version of mc2xml.exe doesn't need installing I placed it in its own folder (D:\EPE) which acts as the working folder to hold the files it creates.
Now run mc2xml to set the inital configuration (this only needs to be done once).
The first screen displayed is to select the EPG source - I used tvguide.co.uk (the country code should default to GB). Make sure the following options are selected
-U UTF8
-u UTC
-F Chan name first

Click on OK then choose the channel lineup that best represents your provider/region, I used Sky HD London and after clicking on OK, mc2xml will run using its built in defaults to pull 7 days data into a file called xmltv.xml.
To get 14D EPG you need to pass a parameter when calling mc2xml so in the working folder I created a batch file (EPE_Grab.bat) containing the following

D:\EPG\mc2xml.exe -d 336

D:\EPG\ is my location for mc2xml.exe and -d 336 is the parameter to specify the number of hrs of data to grab (336 = 14 days). If you look at the mc2xml usage instructions you will see other parameters you can pass when running the prog. For example, I added to my .bat file -o D:\EPG\EPGDATA.xml to redirect the output from the default xmltv.xml file to EPGDATA.xml
Run the bat file to get the 14d epg data (if you see some error messages about missing data its likely that the number of hrs being grabbed is greater than the number of hours available in the EPG data source - it can be ignored) then copy the resulting file to a directory on the Sat box - for convenience I put it into /etc/epgimport which is a folder used by EPGimport.

The 2nd step is create a custom.sources.xml file which will tell EPGimport where to find the EPG data and the location of a custom.channels.xml file which holds the channel mapping (see step 3). This file needs to be in /etc/epgimport/ and contain the following:

<?xml version="1.0" encoding="UTF-8"?>
<sources>
<sourcecat sourcecatname="AAAA">
<source type="gen_xmltv" channels="custom.channels.xml">
<description>BBBB</description>
<url>CCCC</url>
</source>
</sourcecat>
</sources>

Where:
AAAA = top level name that will be displayed in the EPGimport sources
BBBB = is the name displayed next to source selection check box.
CCCC = is the location & name of EPG Data file to use ie /etc/epgimport/EPGDATA.xml

The last step is to create the custom.channels.xml in /etc/epgimport/ which contains the mapping between the channel identifier(s) used by the EPG provider in the EPGDATA data file and the corresponding service reference which the box uses to tune to the channel. The format of the file is:

<?xml version="1.0" encoding="latin-1"?>
<channels>
<channel id="XXXX">1:0:19:3ED:A:1:FFFF0000:0:0:0:</channel>
<channel id="YYYY">1:0:19:3XX:A:1:FFFF0000:0:0:0:</channel>
</channels>
where
XXXX & YYYY are the channel identifiers in the EPGDATA
>1:0:19.... etc are the service references the box uses for each channel.

Unfortunately this is the tedious bit as you need to match the Enigma service reference with each channel identifier in the EPG data file and it looks like Enigma2 does not hold the service ref and channel name together in one place. The service ref looks to be in the bouquets and uses a Hex string (ie 3ED above) to link to the channel name in held in Lamedb5. There are tools like DreamboxEdit and E-channelizer which can display the service ref for each channel but the free versions don't allow bulk copy/exporting of the refs to a file that you can use as starting point for the custom.channels.xml so you need to copy/paste. An alternative (particularly if you have a lot of channels to deal with) is to work with the Enigma files directly and try to "semi-automate" the process. Though the exact approach will depend on your skill set and tools available, the "process flow" would look like:

1. Open/view the EPG data file and note/get the channel identifier & channel name for each of the channels of interest in the EPG data (the "Channel" section is the first part of the EPG Data file)
eg - <channel id="I101.713.tvguide.co.uk">
<display-name>BBC One London</display-name>

2. Use DreamboxEdit or E-channelizer to display the service ref for that channel.

OR (if "automating" and you want to pull the data directly from the Enigma files)

a. Open/view \etc\enigma2\lamedb5, find the row matching the channel name and note/get the hex characters following the "s:" (you may have rows starting with "t:" which I beleive are for terrestrial channels)
ie for BBC One London
s:189d:011a0000:07fd:0002:1:0:0,"BBC One Lon",p:BSkyB

The hex characters to note would be 189d

b. Open/view the "main" bouquet file in \etc\enigma2\ (mine was called userbouquet.abm.sat_282_sky_uk.main.tv) and note/get the row containing the hex characters from step a.
ie for 189d above the corresponding row in my bouquet file was
#SERVICE 1:0:1:189d:7fd:2:11a0000:0:0:0:

The string of characters after the #SERVICE is the service reference for that channel.

3. Create, then populate the custom.channels.xml file using the channel id from step 1 and the corresponding service ref from step 2.
ie the above examples would give the line
<channel id="I101.713.tvguide.co.uk">1:0:1:189d:7fd:2:11a00 00:0:0:0:</channel>

As I'm familiar with Excel, I used Notepad to open each of the files, copy/pasted the contents into an Excel workbook then used Excel's Find/Replace/Lookup functions etc. to create a sheet containing the all the required rows in the right format to copy/paste into the custom.channels.xml.

When all the files are created and in place, open EPGimport. Use the blue button to view the sources and you should see an entry matching whatever you used in place of AAAA above, expand that entry and you should see your version of BBBB above with a checkbox next to it. Select it and as a test use the yellow button to manually import the highlighted entry. All being well you should see the data being loaded into the EPG. Open the EPG and page forward 8-9 days and you should see program data for the channels in the EPGDATA file.
Note that EPGimport loads data in the order that it appears in the sources. If you are using one or more Rytec files in addition to the 14D EPG data file and there are any Channel/program duplicates between the two data sets, the last loaded file takes precedence.

Once you are happy that everything works you can either stick with a manual process or look at automating the EPG refresh etc.

[JonMMM]

Unfortunately for it to be automated and included as a plugin at the very least mc2xml would need to be rewritten to run in linux rather than windows. it would be very nice however.

But for me to be honest I just let my box get the EPG OTA seems to work pretty well

[LraiZer]

You can generate an up to date set of xmltv source and channel files to start to work with using one of the plugins from the feeds. For an automation solution, try install the RadioTimesEmulator plugin and zap to "IEPG data 1", this will generate a latest set of xmltv files in /tmp on the box to work with. You should only need to map channel id used in mc2xml to the real channel_ID in the stream data that this plugin grabs and uses.

[LraiZer]

So maybe it could work for automation by running mc2xml with "I101.713.tvguide.co.uk:282_2_2002" in its mc2xml.ren file to make the skyuk_28.2.channels.xml generated by radiotimesemulator a vaild match with the xc2xml generated xmltv epg?

If you want to rename some channel numbers or names you can create a "mc2xml.ren" text file (in the same directory as mc2xml.dat) listing the colon separated from:to channel numbers or names on separate lines:

(the "from" can also be a channel id, to distinguish duplicate channel numbers or names)

mc2xml.ren
605:2
BBC1:BBC News
I20.515292.microsoft.com:55

[birdman]

Unfortunately this is the tedious bit as you need to match the Enigma service reference with each channel identifier in the EPG data file and it looks like Enigma2 does not hold the service ref and channel name together in one place.

It does for me on Freeview.

It's in the lamedb* files. (And lamedb5 is much easier to use).

s:10bf:eeee0000:104d:233a:1:0:0,"BBC TWO"
s:10c0:eeee0000:104d:233a:1:0:0,"BBC THREE"
s:11c0:eeee0000:104d:233a:1:0:0,"BBC FOUR"
s:20c0:eeee0000:2002:233a:1:0:0,"Channel 4"

You might need to re-order the service fields, but it is all there.

[AB2022]

JonMMM There is a Linux version available via the link you gave but I don't know if it would run under enigma2.

LraiZer Tx for the suggestions, if I get some time I will try it out. At the moment my "automation" is mapping the box as a network drive via Samba then using Windows task scheduler to run the batch file each day at a time my laptop is usually on. The batch file has been amemded to put the output file directly into \etc\epgimport\. I've also set up a power timer to make sure the box is not in deep sleep when the tasks runs. Its early days but so far seems to work.

birdman though I did see this in the Lamedb5 I didn't catch on until you pointed it out that its the same data in a different order. I suspect with the amount of "slicing an dicing" to get the data in the right order it might take just as long, but good to know

[Huevos]

What "slicing and dicing"? That data is pretty simple to extract.

[LraiZer]

Have just tried using a mc2xml.ren file, it does not work as first thought. It only swaps out the channel number part of the mc2xml channel id.

It would probably be easier to swap the channel id in a custom_channel.xml to mc2xml format anyway, instead of swapping the entire epg data to radiotimes format.

If you first create a /etc/epgimport/mc2xml_channels.sed file with a list of your required sed swap commands to convert radiotimes to mc2xml format.

/etc/epgimport/mc2xml_channels.sed

s/282_2_2102/I101.14.tvguide.co.uk/g
s/282_2_2075/I102.1782.tvguide.co.uk/g

Then simply zap to "IEPG data 1" and run both these commands in a script, and it should create a valid mc2xml compatible custom.channels.xml file using the radiotimes emulator plugin generated channels.xml file as a donkey.

/usr/radiotimes_emulator/radiotimes_emulator -d /tmp/mc2xml

sed -f /etc/epgimport/mc2xml_channels.sed /tmp/mc2xml/skyuk_28.2.channels.xml > /etc/epgimport/custom.channels.xml

[birdman]

birdman though I did see this in the Lamedb5 I didn't catch on until you pointed it out that its the same data in a different order. I suspect with the amount of "slicing an dicing" to get the data in the right order it might take just as long, but good to know

The sort of thing that Perl was designed to handle with ease.