Memory problem/leak?

[white_westie]

First the build info
Clean flash with 664, updated online to 665

Looking at dmesg output and seen that the enigma2 process had been killed by the oom routine twice in last few of days - every 2 days.
The output showed that it happened at 05:26 and that there was zero bytes free in the swapfile.
By default I always create a 32MB swapfile, just in case it is needed.

I use the Crossepg downloader to populate my epg, and have it scheduled to run at 05:23 daily.

I removed the schedule download from crossepg and started to run it manually.
Each time it completed, the swap space usage increased. On the 3rd iteration, I lost access to the box (had 4 putty sessions opened).
Box was still alive and the last free display that I could see did show that all swap space was used.
The spinner was running on the TV but in super slow motion, so gave it about 30mins, before I power cycled it get back control (and of course lost my messages file).

Deleted swapfile and recreated at 64MB, and ran the crossepg manually again.
Have run it about 6 times now and free shows the following output


root@et4x00:/var/volatile/log# free
total used free shared buffers
Mem: 236144 225808 10336 0 48
-/+ buffers: 225760 10384
Swap: 65532 54328 11204

Each iteration the swap space is increasing, but not freeing up. During the final epg reload step I noticed it has increased to nearly all used, but has not triggered the oom routine to kick in.

Any ideas?

I have been using crossepg from the start, with the same settings
France xepgdb repacked
Opentv 13
Opentv 28
Germany/Austria/Swiss xepgdb repacked

Also use the same setup on my ET9000 running bld 666, with no problems!

WW

[Rob van der Does]

Not sure what you mean by 'Each iteration the swap space is increasing, but not freeing up': 'freeing up' what exactly?

Also, as your experiments clearly show, Cross uses a lot of memory (especially during the parsing process). I'm not sure about the 4000: is it a low RAM box?

BTW: YYou do use a lot of EPG import. Cross is very good for Open-TV, but not for the rest. I would suggest to use the XMLTV-importer for the French, German and Austrian package (obviously set to an other time then Cross, preferably after 05.30 GMT).
And if the box still can't handle that, decrease the number of packages.

[white_westie]

Not sure what you mean by 'Each iteration the swap space is increasing, but not freeing up': 'freeing up' what exactly?
As i said I ran the download manually 6 times, and each time it ran the amount of swap being used increased - it never freed up.
So eventually if will use the full 64MB and I assume trigger the oom routine!

Memory in the ET4000 is 128/512MB - which is the same as my ET9000, but of course only has 1 core.

Also must add that my configuration was working fine versions with the 3.6 kernel and associated drivers....

[twol]

Not sure what you mean by 'Each iteration the swap space is increasing, but not freeing up': 'freeing up' what exactly?
As i said I ran the download manually 6 times, and each time it ran the amount of swap being used increased - it never freed up.
So eventually if will use the full 64MB and I assume trigger the oom routine!

Memory in the ET4000 is 128/512MB - which is the same as my ET9000, but of course only has 1 core.

Also must add that my configuration was working fine versions with the 3.6 kernel and associated drivers....

Why use flash when it is probably better to use a usb stick that handles any amount of epg and is pretty cheap to buy. Feel thta you are trying to fix a problem when there is an easier solution available, aprt from the intellectual challenge of your issue.

[Rob van der Does]

Not sure what you mean by 'Each iteration the swap space is increasing, but not freeing up': 'freeing up' what exactly?
As i said I ran the download manually 6 times, and each time it ran the amount of swap being used increased - it never freed up.
So eventually if will use the full 64MB and I assume trigger the oom routine!
Nope, that's not the way Linux handles memory. If you allow it to use SWAP it will do so, and there's no problem if it uses all of it.



Why use flash when it is probably better to use a usb stick that handles any amount of epg and is pretty cheap to buy. Feel thta you are trying to fix a problem when there is an easier solution available, aprt from the intellectual challenge of your issue.
Neither flash nor USB will be used.
When Enigma is alive, all EPG is in live-memory (RAM). USB or flash will only be used as a kind of hibernate file, to store the data when Enigma is asleep.

[twol]

Neither flash nor USB will be used.
When Enigma is alive, all EPG is in live-memory (RAM). USB or flash will only be used as a kind of hibernate file, to store the data when Enigma is asleep.
Yep, but when it creates/updates the EPG surely it saves the files on for instance the USB stick or why else do you get to say where it goes..... And stores a heap of files on my usb for sure,

[Rob van der Does]

Cross indeed creates a database (for reasons unknown); XMLTV importer doesn't.

[white_westie]

Just to confirm I am not using flash, but usb to store epg file.
The point about memory was to highlight that both my ET stbs have the same amount of physical memory.

Have been using Crossepg on different images for last 2 years on my et9000. Was actually downloading more epg packages than current 4 in use now - and never had any memory related problems.
Current problem I am seeing does not occur on et9000 stb and it is running the same image as on the et4000.
Besides the difference in cpu/chipset and hardware drivers, my understanding is that depending on what build options are used to configure/build the image, the 2 images should basically be the same and therefore act the same!

However from my experiences, ET4000 has a problem and ET9000 does not!

If the experts here think this might be driver related, then maybe they could highlight that issue to the appropriate forum.

Will post output from dmesg to see if it shows up anything unusual, when I get a chance to recreate it over the weekend.

Will also look into the suggested workaround to use XMLTV-importer instead, but a quick look at its functionality and it seems to be the same as crossepg!

WW

[Rob van der Does]

Will also look into the suggested workaround to use XMLTV-importer instead, but a quick look at its functionality and it seems to be the same as crossepg!
You can't have 'a quick look at it's functionality': you have to dig deep into the code to see how they work. And believe me: apart from using (more or less) the same sources there's nothing the same in the functionality. The XMLTV-importer has a much smarter, and less memory-hungry, way of parsing the imported data.

[white_westie]

Rob

Point taken re functionality/code.

Had some free time so was able to recreate triggering oom_killer routine - attached is output from messages file.

[white_westie]

Just to add to this issue I am seeing.

I ran a crossepg download a number of times on my ET9000, and could see the space used in the swap file increasing with each iteration.
The rate of increased usage was a lot slower than I saw when running the same type of test on the ET4000.
I think the max it reached was about 20MB. The main difference between the 2 boxes was that after a period of time, the used swap space on the ET9000 reduced back to where it started from (was about 282MB), whereas on the ET4000, this did not occur.

[Rob van der Does]

Did you create the swap-files via the GUI of the box in both cases? I ask so, as the GUI has been tweaked to use special parameters regarding the actual use of the swap-file (i.e. only if really needed, to prevent the box from slowing down because of the use of the swap). If I interprete your story correct, it looks like you created the swap on the 4000 via other means (manual).

[white_westie]

Always take the lazy option and use the gui.