CrossEPG development thread for 6.5 images

[SkyFan]

I did a clean install of a new Vix image then I did this:

Menu > Setup > EPG > CrossEPG > OpenTV Providers

Press OK on "Sky UK OpenTV (Astra2 on 28.2)" and it should now be ticked. Press Yellow to download. In previous images the download would finish and I'd be back in the same menu. But on the most recent Vix image it attempts to download but crashes before it can do so. Log attached: Enigma2_crash_2024-03-25_10-11-02 = OpenTV Download.log

CrossEPG plugin also gave some error messages when installing it but it still said it installed.

[LraiZer]

I would guess it was maybe broken since time_t and time functions changed from 32 bit to 64 bit in openembedded?

see: defaultsetup: Enable largefile and 64bit time_t support systemwide for 32 bit platforms

There are differencies in the time related declaration types and structs between crossepg and enigma2.
For example in some struct start_time variables are defined as uint32_t and some are defined as time_t, so probably no longer have a matching struct size.

I would guess the cursor position within file read/write no longer matches up and are no longer set to the correct position for what crossepg writes and what engima2 expects to read.

eg.

fread(&title, sizeof(epgdb_title_t), 1, headers);

[LraiZer]

Not tested or build with these changes, but an initial quick look at the code suggest that at least these two lines in the code that crossepg uses to write are maybe not in line with what enigma2 code expects to read in the eEPGCache::crossepgImportEPGv21 function.

diff --git a/src/common/epgdb/epgdb.c b/src/common/epgdb/epgdb.c

index 230caed..9962603 100644
--- a/src/common/epgdb/epgdb.c
+++ b/src/common/epgdb/epgdb.c
@@ -32,7 +32,7 @@ typedef struct epgdb_title_header_s
 {
 	uint16_t	event_id;
 	uint16_t	mjd;
-	uint32_t	start_time;
+	time_t		start_time;
 	uint16_t	length;
 	uint8_t		genre_id;
 	uint8_t		flags;
@@ -194,7 +194,7 @@ bool epgdb_save (void(*progress_callback)(int, int))
 		if (progress_callback != NULL)
 			progress_callback (progress_count, progress_max);
 	}
-	fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (uint32_t) * 2), SEEK_SET);
+	fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (time_t) * 2), SEEK_SET);
 	fwrite (&channels_count, sizeof (uint32_t), 1, fd_h);
 	fflush (fd_h);
 	fsync (fileno (fd_h));

[abu baniaz]

Hopefully done correctly. Bitbake file changed to point to point my repo.
https://github.com/AbuBaniaz/e2openp...50a7710fe52350

I got this crash

[LraiZer]

I'll do a debug test build some time this week if possible to see if i can see why it is crashing.

I would guess the crash occurs somewhere around..
gmtim(time_t) and toBCD(time->tm_hour) code in engima2.

[LraiZer]

Missed some obvious bits in last patch..
OK, so just tested this patch and crossepg opentv epg downloaded for 28.2e and loaded ok into GUI for me.

diff --git a/src/common/epgdb/epgdb.c b/src/common/epgdb/epgdb.c
index 230caed..ac24b6b 100644
--- a/src/common/epgdb/epgdb.c
+++ b/src/common/epgdb/epgdb.c
@@ -32,7 +32,7 @@ typedef struct epgdb_title_header_s
 {
 	uint16_t	event_id;
 	uint16_t	mjd;
-	uint32_t	start_time;
+	time_t		start_time;
 	uint16_t	length;
 	uint8_t		genre_id;
 	uint8_t		flags;
@@ -194,7 +194,7 @@ bool epgdb_save (void(*progress_callback)(int, int))
 		if (progress_callback != NULL)
 			progress_callback (progress_count, progress_max);
 	}
-	fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (uint32_t) * 2), SEEK_SET);
+	fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (time_t) * 2), SEEK_SET);
 	fwrite (&channels_count, sizeof (uint32_t), 1, fd_h);
 	fflush (fd_h);
 	fsync (fileno (fd_h));
@@ -285,7 +285,7 @@ bool epgdb_load ()
 	char tmp[256];
 	unsigned char revision;
 	uint32_t channels_count, i, j, aliases_groups_count, indexes_count;
-	uint32_t now = time (NULL);
+	time_t now = time (NULL);
 	
 	epgdb_index_init ();

diff --git a/src/common/epgdb/epgdb.h b/src/common/epgdb/epgdb.h
index 6f36ed3..1566586 100644
--- a/src/common/epgdb/epgdb.h
+++ b/src/common/epgdb/epgdb.h
@@ -16,7 +16,7 @@ typedef struct epgdb_title_s
 	/* same elements of epgdb_title_header_t */
 	uint16_t	event_id;
 	uint16_t	mjd;
-	uint32_t	start_time;
+	time_t		start_time;
 	uint16_t	length;
 	uint8_t		genre_id;
 	uint8_t		flags;

[Ev0]

Missed some obvious bits in last patch..
OK, so just tested this patch and crossepg opentv epg downloaded for 28.2e and loaded ok into GUI for me.

diff --git a/src/common/epgdb/epgdb.c b/src/common/epgdb/epgdb.c
index 230caed..ac24b6b 100644
--- a/src/common/epgdb/epgdb.c
+++ b/src/common/epgdb/epgdb.c
@@ -32,7 +32,7 @@ typedef struct epgdb_title_header_s
 {
     uint16_t    event_id;
     uint16_t    mjd;
-    uint32_t    start_time;
+    time_t        start_time;
     uint16_t    length;
     uint8_t        genre_id;
     uint8_t        flags;
@@ -194,7 +194,7 @@ bool epgdb_save (void(*progress_callback)(int, int))
         if (progress_callback != NULL)
             progress_callback (progress_count, progress_max);
     }
-    fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (uint32_t) * 2), SEEK_SET);
+    fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (time_t) * 2), SEEK_SET);
     fwrite (&channels_count, sizeof (uint32_t), 1, fd_h);
     fflush (fd_h);
     fsync (fileno (fd_h));
@@ -285,7 +285,7 @@ bool epgdb_load ()
     char tmp[256];
     unsigned char revision;
     uint32_t channels_count, i, j, aliases_groups_count, indexes_count;
-    uint32_t now = time (NULL);
+    time_t now = time (NULL);
     
     epgdb_index_init ();

diff --git a/src/common/epgdb/epgdb.h b/src/common/epgdb/epgdb.h
index 6f36ed3..1566586 100644
--- a/src/common/epgdb/epgdb.h
+++ b/src/common/epgdb/epgdb.h
@@ -16,7 +16,7 @@ typedef struct epgdb_title_s
     /* same elements of epgdb_title_header_t */
     uint16_t    event_id;
     uint16_t    mjd;
-    uint32_t    start_time;
+    time_t        start_time;
     uint16_t    length;
     uint8_t        genre_id;
     uint8_t        flags;

So with these changes, OpenTv now works for me (as in it downloads and saves the epg now).

However there is some random junk in the listings, so it seems more work is needed still.

Also Rytec does not seem to be working (I tried replacing all unit32_t entries with time_t in all files in crossepg) but that didn't seem to do anything.

[abu baniaz]

Quite a few commits here: Maybe some useful to us
https://github.com/OpenVisionE2/Cros...ommits/master/

[LraiZer]

dbmerge also still has hardcoded calculations, make it same as rest.

diff --git a/src/common/dbmerge/dbmerge.c b/src/common/dbmerge/dbmerge.c
index 197dbb5..34a97c1 100644
--- a/src/common/dbmerge/dbmerge.c
+++ b/src/common/dbmerge/dbmerge.c
@@ -158,7 +158,7 @@ bool dbmerge_merge (FILE *fd_h, FILE *fd_d, void(*progress_callback)(int, int))
 		return false;
 	}
 
-	fseek (fd_h, 22, SEEK_SET);
+	fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (time_t) * 2), SEEK_SET);
 
 	fread (&channels_count, sizeof (int), 1, fd_h);
 	for (i=0; i<channels_count; i++)

[Ev0]

dbmerge also still has hardcoded calculations, make it same as rest.

diff --git a/src/common/dbmerge/dbmerge.c b/src/common/dbmerge/dbmerge.c
index 197dbb5..34a97c1 100644
--- a/src/common/dbmerge/dbmerge.c
+++ b/src/common/dbmerge/dbmerge.c
@@ -158,7 +158,7 @@ bool dbmerge_merge (FILE *fd_h, FILE *fd_d, void(*progress_callback)(int, int))
         return false;
     }
 
-    fseek (fd_h, 22, SEEK_SET);
+    fseek (fd_h, strlen (MAGIC_HEADERS) + sizeof (unsigned char) + (sizeof (time_t) * 2), SEEK_SET);
 
     fread (&channels_count, sizeof (int), 1, fd_h);
     for (i=0; i<channels_count; i++)

Now it crashes for me:

<   204.041058> [CrossEPG_Loader] loading data with crossepg patch v21
<   204.041128> [epgcache][CrossEPG V21 patch] start crossepg import
<   204.041287> [epgcache][CrossEPG V21 patch] 0 aliases groups in crossepg db
<   204.041363> PC: 000c6580
<   204.041377> Fault Address: 00000008
<   204.041384> Error Code: 519
<   204.041516> Backtrace:
<   204.041689> /usr/bin/enigma2(_Z17handleFatalSignaliP9siginfo_tPv) [0x78920]
<   204.041770> /lib/libc.so.6(__default_rt_sa_restorer) [0xB5D48070]
<   204.041941> /usr/bin/enigma2(_ZN9eEPGCache20crossepgImportEPGv21ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE) [0xC6580]
<   204.042162> /usr/bin/enigma2(n/a) [0x1807F0]
<   204.042172> -------FATAL SIGNAL (11)

[Ev0]

Ok I have reverted that last commit, I then deleted the crossepg folder from /media/usb, then removed and reinstalled crossepg from the feeds.

It seems it works correctly now (no junk in the opentv listings) and also "some" rytec seems to work, but I don't think that is a plugin issue and more likely the servers / files not updated.

[LraiZer]

Why did you have to revert? The dbmerge patch above has nothing to do with your crash and is indeed 100% required!

dbmerge is only used in defragment and when the download protocol is set at xepg to download compressed crossepg database files instead of xmltv files.

It is ESSENTIAL that all crossepg database files be deleted from media when moving from oe-a 5.3 to oe-a 5.4 as they are not compatible due to the change from 32 to 64 bit time_t.

DO NOT PRESS defragment without the dbmerge patch or your crossepg folder will fill with new expediential database sizes in the background. Only a reboot would stop this process.

After the required dbmerge patch is added, would be a good idea to bump the version number so you know you must have version 0.9.1 to be compatible with oe-a 5.4 images.

diff --git a/VERSION b/VERSION
index 6b4eb6f..cbb513d 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-0.9.0
+0.9.1+gitr


diff --git a/src/enigma2/python/version.py b/src/enigma2/python/version.py
index 051849a..52117f1 100644
--- a/src/enigma2/python/version.py
+++ b/src/enigma2/python/version.py
@@ -1 +1 @@
-version = "0.9.0"
+version = "0.9.1+gitr"


diff --git a/src/version.h b/src/version.h
index 5e40ed5..9c09588 100644
--- a/src/version.h
+++ b/src/version.h
@@ -1 +1 @@
-#define RELEASE "0.9.0"
+#define RELEASE "0.9.1+gitr"

[Ev0]

Why did you have to revert? The dbmerge patch above has nothing to do with your crash and is indeed 100% required!

dbmerge is only used in defragment and when the download protocol is set at xepg to download compressed crossepg database files instead of xmltv files.

It is ESSENTIAL that all crossepg database files be deleted from media when moving from oe-a 5.3 to oe-a 5.4 as they are not compatible due to the change from 32 to 64 bit time_t.

DO NOT PRESS defragment without the dbmerge patch or your crossepg folder will fill with new expediential database sizes in the background. Only a reboot would stop this process.

After the required dbmerge patch is added, would be a good idea to bump the version number so you know you must have version 0.9.1 to be compatible with oe-a 5.4 images.

diff --git a/VERSION b/VERSION
index 6b4eb6f..cbb513d 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-0.9.0
+0.9.1+gitr


diff --git a/src/enigma2/python/version.py b/src/enigma2/python/version.py
index 051849a..52117f1 100644
--- a/src/enigma2/python/version.py
+++ b/src/enigma2/python/version.py
@@ -1 +1 @@
-version = "0.9.0"
+version = "0.9.1+gitr"


diff --git a/src/version.h b/src/version.h
index 5e40ed5..9c09588 100644
--- a/src/version.h
+++ b/src/version.h
@@ -1 +1 @@
-#define RELEASE "0.9.0"
+#define RELEASE "0.9.1+gitr"

Yes it's possible that I forgot to remove everything before I tested with the dbmerge changes, hence it crashed (I ran it 3 times and it crashed everytime), so I reverted and rebuilt and it's been fine since.

However @Twol has been testing with the dbmerge changes and says it is working fine for him, so it is quite likely I didn't remove everything first.

[twol]

@LraiZer - currently I cannot break it, but waiting until tomorrow when it has new Rytec data to load……
Providing it doesn,t crash, then I think your changes have fixed most issues, which will be great as I really like to have Crossepg available.

Thanks for all your efforts, fingers crossed for tomorrows scheduled update.

[LraiZer]

I think the only outstanding issue i can see, is when someone uploads xepg crossepg database files for download that have been created using images before oe-a 5.4 but you are now downloading them to dbmerge using the new oe-a 5.4 image. They will not be compatible. Even DB REVISION check increment from 7 to 8 would have no use if they had used 0.9.1 to create it.