Possibly malfunctioning h7s

[chrcoluk]

Some of you may remember before I reported I had to auto reboot the box every 3 days to keep it working.

Well I have been observing the past few months that on iptv I would get a picture freeze every 2-3 seconds, the obvious thought is a network problem right?

Well if I watch 'top' in the shell every 2-3 seconds the 'si' usage goes to 100%. The software interrupts counter.

I have just clean flashed the latest vix 5.4 image and its still happening.

Given how highly rated these units are I am starting to think mine is defective, but I am hoping someone may know of a workaround to the problem.

# top
top - 18:05:38 up 10 min,  1 user,  load average: 0.06, 0.04, 0.00
Tasks: 117 total,   1 running, 116 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
MiB Mem :   1004.0 total,    174.9 free,    729.9 used,     99.2 buff/cache
MiB Swap:    256.0 total,    256.0 free,      0.0 used.    241.3 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                         
 2051 root      20   0  145292  73808  15600 S   0.7   7.2   0:12.22 enigma2                                                         
 2193 root      20   0    6288   2276   1940 R   0.7   0.2   0:01.39 top                                                             
    7 root      20   0       0      0      0 S   0.3   0.0   0:00.07 rcu_sched                                                       
 1452 root      20   0       0      0      0 S   0.3   0.0   0:01.58 nxsched                                                         
    1 root      20   0    1604    328    284 S   0.0   0.0   0:01.39 init                                                            
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd                                                        
    4 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 kworker/0:0H                                                    
    6 root      20   0       0      0      0 S   0.0   0.0   0:00.08 ksoftirqd/0                                                     
    8 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_bh                                                          
    9 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/0                                                     
   10 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 lru-add-drain                                                   
   11 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 watchdog/0                                                      
   12 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                                                         
   13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1                                                         
   14 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 watchdog/1                                                      
   15 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/1                                                     
   16 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/1

[twol]

Which plugins have you installed???

[chrcoluk]

Can a plugin cause software interrupts?

List of plugins is.

crossepg
iptv bouquet maker
open webif

[chrcoluk]

On the new version of vix I have left the auto reboot disabled, and will see what happens on that particular issue.

The iptv is not confirmed working or not as I havent reinstalled the plugins for it.

I did find a thread elsewhere with people discussing hanging h7s boxes a few of them, and they all had faulty receiver's.

[chrcoluk]

A little update, I have identified over committing of memory, the unit even when firstly booted up has under 200 meg of free ram, the committed memory isnt on temporary things like cache is its fully committed to things that cannot be removed from memory if needed.

Due to the memory constraints I looked into ways to ease the problem. With a swap file installed and default vm.swappiness, it routinely will use the swap file setting the swapinness to as low as 5 is not utilising it, removing the swapfile causes the box to crash with OOM.

Part of the problem has also been identified the use of /var/volatile/tmp using a ram disk, its default sized to 500meg, this is where I made some progress, the disk is backed by swap, so wont error out when is not enough memory providing there is a swap device present, the default size of 500meg when the system only has circa 200 meg of memory to allocate is obviously a disaster waiting to happen, interestingly when I capped the ram disk to 100meg enigma crashed over night and went into a boot loop so I had proof enigma was doing something that required a lot of tmp storage (in ram), and it was epg updates.

I have since made a tmp dir on the hdd and bind mounted it over /var/volatile/tmp, existing /tmp is symlinked to /var/volatile/tmp, I thought about changing that symlink but I expect that was too risky with system updates undoing that change.

There has been some clear overconfidence in the capabilities of this unit in terms of ram, I feel selling a device with only 1 gig of ram in 2021 is a bit questionable, the manufacturers of this device probably should rethink that one. The amount of stories on the net of this unit been unstable is starting to make some sense, although at least some of those stories confirmed to be faulty receiver chips.

So will now see if this can stay stable overnight for a decent period of time, hopefully I wont report back for a week or two and only then to confirm no more problems. A usb stick is been ordered as using the hdd for tmp has a little risk if it fills up with recordings.

In addition cat /proc/interrupts shows a bunch of strangely named processes with varying length of 'b' in their name. Seems lots of oddly proprietary stuff running that is hard to debug including the earlier identified '999999999999999' process.

I am not sure if this is a clean install of vix as I did the upgrade via image manager and chose not to restore the backed up config, I may still do an install from the recovery image if instability prevails. Also htop oddly doesnt show the 100% usage that normal top shows under 'si' usage.

[ccs]

Where do you store epg.dat? If it's in flash, then move it to hdd or usb.

[ronand]

A little update, I have identified over committing of memory, the unit even when firstly booted up has under 200 meg of free ram, the committed memory isnt on temporary things like cache is its fully committed to things that cannot be removed from memory if needed.

Due to the memory constraints I looked into ways to ease the problem. With a swap file installed and default vm.swappiness, it routinely will use the swap file setting the swapinness to as low as 5 is not utilising it, removing the swapfile causes the box to crash with OOM.

Part of the problem has also been identified the use of /var/volatile/tmp using a ram disk, its default sized to 500meg, this is where I made some progress, the disk is backed by swap, so wont error out when is not enough memory providing there is a swap device present, the default size of 500meg when the system only has circa 200 meg of memory to allocate is obviously a disaster waiting to happen, interestingly when I capped the ram disk to 100meg enigma crashed over night and went into a boot loop so I had proof enigma was doing something that required a lot of tmp storage (in ram), and it was epg updates.

I have since made a tmp dir on the hdd and bind mounted it over /var/volatile/tmp, existing /tmp is symlinked to /var/volatile/tmp, I thought about changing that symlink but I expect that was too risky with system updates undoing that change.

There has been some clear overconfidence in the capabilities of this unit in terms of ram, I feel selling a device with only 1 gig of ram in 2021 is a bit questionable, the manufacturers of this device probably should rethink that one. The amount of stories on the net of this unit been unstable is starting to make some sense, although at least some of those stories confirmed to be faulty receiver chips.

So will now see if this can stay stable overnight for a decent period of time, hopefully I wont report back for a week or two and only then to confirm no more problems. A usb stick is been ordered as using the hdd for tmp has a little risk if it fills up with recordings.

In addition cat /proc/interrupts shows a bunch of strangely named processes with varying length of 'b' in their name. Seems lots of oddly proprietary stuff running that is hard to debug including the earlier identified '999999999999999' process.

I am not sure if this is a clean install of vix as I did the upgrade via image manager and chose not to restore the backed up config, I may still do an install from the recovery image if instability prevails. Also htop oddly doesnt show the 100% usage that normal top shows under 'si' usage.

I don't see a flood of stories on the net about the h7. Mine has never run out of memory with the default 256MB swap.

[chrcoluk]

Where do you store epg.dat? If it's in flash, then move it to hdd or usb.

on the hdd, that has no bearing on the memory consumption, during the process of creating the epg there is temporary files written to /tmp and /tmp is hosted on a ramdisk.

The vix crash logs were very useful in diagnosing this.

The initial crashes reported /tmp had ran out of space to store epg files (after I reduced it to 100meg thinking vix wouldnt need so much scratch space). The decision to reduce it was a bad one but did serve to help me debug what is going on, the ram disk only consumes ram for used space. It is backed by swap if there isnt enough ram.

Also thanks guys for confirming swap is on by default, as due to me been unsure if this is an actual clean vix install, I wasnt sure if I added the swap or it was default. It is 256meg, so I am assuming now its a default enabled swap.

After I did the bind mount moving tmp to my hdd, the crashes stopped, thus far since I made those changes the box has been fine and responsive, but I need more than 1-2 days to consider if its working ok, if it makes a week, then this is great progress as before I had to automate reboots every 2-3 days.

There is currently nearly 600 gig in my trash of recordings which will be deleted probably in next day or so and currently 40 gig of free space, so hdd wont be full for a while now. By then I will have the usb taking over /tmp duties.

[ccs]

on the hdd, that has no bearing on the memory consumption, during the process of creating the epg there is temporary files written to /tmp and /tmp is hosted on a ramdisk.

If it's on the hdd then there's no problem, but the default is to save an epg copy in a file called epg.dat in internal flash, which can cause problems.

[BrokenUnusableAccount]

I don't see a flood of stories on the net about the h7. Mine has never run out of memory with the default 256MB swap.

I'd never really investigated how much memory was free, or the swapfile.

It seems that by default it uses /dev/mmcblk0p7 as a 256MB swap partition.

Using emmc flash as swap can't really be a good idea, if you ever run something big that makes it really thrash it'll wear out the emmc.

However I can't work out how to permanantly stop it using /dev/mmcblk0p7 as swap.

swapoff /dev/mmcblk0p7

stops it right now, but next time it boots up there it is again 256MB swap using /dev/mmcblk0p7 !

I'd feel a lot happier using a swapfile on my hard drive.

[chrcoluk]

Hmm I honestly dont know if I tinkered with the swap in the past but my swap is on the hdd currently, I dont think its a good idea to use the flash as swap for the same reasons I was advised to make sure the epg is not on my flash.

root@zgemmah7:/tmp# cat /proc/swaps
Filename Type Size Used Priority
/media/hdd/swapfile file 262140 320 -1

in /etc/fstab (where it is set to apply at boot), there needs to be a swapfile created first.

/media/hdd/swapfile swap swap defaults 0 0

[BrokenUnusableAccount]

Hmm I honestly dont know if I tinkered with the swap in the past but my swap is on the hdd currently, I dont think its a good idea to use the flash as swap for the same reasons I was advised to make sure the epg is not on my flash.

root@zgemmah7:/tmp# cat /proc/swaps
Filename Type Size Used Priority
/media/hdd/swapfile file 262140 320 -1

in /etc/fstab (where it is set to apply at boot), there needs to be a swapfile created first.

/media/hdd/swapfile swap swap defaults 0 0

My H7 just completely refuses to do that.
It dosn't seem to care what's in /etc/fstab
I can't get anything except:

root@Zgemmah7:~# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/mmcblk0p7                          partition       262140  0       -1
root@Zgemmah7:~#

or no swap at all.


Oh well. At least mine seems to be almost never using the swap.

[chrcoluk]

Yeah at least you not using the swap, I got the swappiness set down to 5, and its still using 320k albeit a tiny amount. Currently only 62 meg of free ram with another 62 meg allocated to caching, its been creeping down so still curious if the box makes it a week.

Interestingly as well not all of the combined free+caching is available either. Just under 100M available. From around 200M on a fresh boot.

# free
total used free shared buff/cache available
Mem: 1028056 897756 64360 76 65940 99212
Swap: 262140 320 261820

[ronand]

On any linux box I've had (and I've had a few) the system seems to allocate most of the available RAM no matter how much there actually is. So I wouldn't read too much into that. If you are getting OOM then there must a plugin that you are using that it doing it - there is definitely enough resources for use as a satellite receiver.

[chrcoluk]

On any linux box I've had (and I've had a few) the system seems to allocate most of the available RAM no matter how much there actually is. So I wouldn't read too much into that. If you are getting OOM then there must a plugin that you are using that it doing it - there is definitely enough resources for use as a satellite receiver.

I agree though that as long as the box works, which is ultimately what matters then its ok, which is why I am seeing if it will work for the entire week.

How do I diagnose memory usage for plugins? right now in the UI there is only 3 basic plugins there. enigma itself is only using 110meg of ram. A lot of the memory usage seems tied up in proprietary kernel drivers.