OS Mini < 256Mb RAM available

[five384]

The OS Mini seems to be using only 256MB of it's 512MB RAM.

I understand about reduced net RAM due to reservations for framebuffers, device drivers, enigma usage etc., but that's not what I'm pointing out here. This seems to be a reduction in gross RAM due to highmem being disabled in the kernel, due to cache aliasing being set for the BMIPS4380 CPU core.

Best I can see, this was a change sometime around Linux 4.0, but I don't know if it's a permanent restriction or just a configuration error. This isn't unique to OpenViX, but presumably everything using the Edision kernel from GitHub (https://github.com/open-edision/meta-edision), including OpenATV.

I'm posting this here in case anyone reading knows more. It's probably a linux-mips or open-edision problem at its core?

Getting access to the full RAM would be great. Here are my two OS Minis with 10 days uptime:



root@sat1:~# free
total used free shared buff/cache available
Mem: 248772 192224 37324 816 19224 34244
Swap: 0 0 0

root@sat2:~# free
total used free shared buff/cache available
Mem: 248772 222664 12216 812 13892 6480
Swap: 0 0 0


I found old references to the BMIPS4380 core addressing working for >256MB in Linux 2.6 and 3.4. Around 2014/2015 there was a tidy-up of BMIPS with respect to HIGHMEM and cache aliasing. Since then, any satellite box dmesg log I can find on Linux 4.0+ seems hard-limited to 256MB gross on BMIPS4380.

Here's an OS Mini under OpenViX 5.0.014:


Linux version 4.11.0 (vix@ns3044876.ip-51-255-93.eu) (gcc version 6.3.0 (GCC) ) #1 SMP Tue May 16 11:44:48 BST 2017
CPU0 revision is: 0002a065 (Broadcom BMIPS4380)
FPU revision is: 00130001
MIPS: machine is Broadcom BCM97362SVMB
Determined physical RAM map:
memory: 10000000 @ 00000000 (usable)
Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
Primary data cache 64kB, 4-way, VIPT, cache aliases, linesize 64 bytes
This processor doesn't support highmem. -262144k highmem ignored
Zone ranges:
Normal [mem 0x0000000000000000-0x000000000fffffff]
HighMem empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000000fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
On node 0 totalpages: 65536
free_area_init_node: node 0, pgdat 80a5ec00, node_mem_map 8100b000
Normal zone: 512 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 65536 pages, LIFO batch:15
percpu: Embedded 16 pages/cpu @8120e000 s34192 r8192 d23152 u65536
pcpu-alloc: s34192 r8192 d23152 u65536 alloc=16*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 65024
Kernel command line: console=ttyS0,115200 ubi.mtd=rootfs root=ubi0:rootfs rootfstype=ubifs
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 248460K/262144K available (8530K kernel code, 330K rwdata, 1700K rodata, 312K init, 303K bss, 13684K reserved, 0K cma-reserved, 0K highmem)

Here you'll see the same on an OS Mini under OpenATV on Linux 4.8.0: https://www.opena.tv/edision-os-mini/29841-os-mini-booted-nicht-mehr-crashlog-gefunden.html



Linux version 4.8.0 (oe1@BUILD3) (gcc version 5.3.0 (GCC) ) #1 SMP Wed Oct 5 02:52:13 CEST 2016
CPU0 revision is: 0002a065 (Broadcom BMIPS4380)
FPU revision is: 00130001
MIPS: machine is Broadcom BCM97362SVMB
Determined physical RAM map:
memory: 10000000 @ 00000000 (usable)
Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
Primary data cache 64kB, 4-way, VIPT, cache aliases, linesize 64 bytes
This processor doesn't support highmem. -262144k highmem ignored
Zone ranges:
Normal [mem 0x0000000000000000-0x000000000fffffff]
HighMem empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000000fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
On node 0 totalpages: 65536
free_area_init_node: node 0, pgdat 809fc980, node_mem_map 8100b000
Normal zone: 512 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 65536 pages, LIFO batch:15
percpu: Embedded 14 pages/cpu @8120e000 s25104 r8192 d24048 u57344
pcpu-alloc: s25104 r8192 d24048 u57344 alloc=14*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 65024
Kernel command line: console=ttyS0,115200 ubi.mtd=rootfs root=ubi0:rootfs rootfstype=ubifs
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 248940K/262144K available (8211K kernel code, 317K rwdata, 1640K rodata, 256K init, 286K bss, 13204K reserved, 0K cma-reserved, 0K highmem)


Searching for "BMIPS4380 highmem" shows a 2015 commit thread for this CPU core https://www.linux-mips.org/archives/linux-mips/2015-05/msg00112.html
...and maybe a description (https://lwn.net/Articles/620810/) of the underlying issue:


Booting a HIGHMEM MIPS kernel on a system with cache aliases may fail if the system has >256MB of memory.

I found various old BMIPS4380-cored (not OS Mini) kernel bootlogs that weren't hard-restricted to 256Mb:

Linux 2.6, LG TV, 284MB net: http://openlgtv.org.ru/wiki/index.php/Broadcom_model_boot_log
Linux 2.6, DM520, 305MB net: https://dreambox.de/board/index.php?thread/21384-dm520-with-dvb-t-usb-rtl2832-issue/
Linux 3.4, DM520, 305MB net: https://dreambox.de/board/index.php?attachment/12907-dmesg-log/

[aido]

I've got a Technomate TM-Nano-SE M2 here which doesn't seem to have this problem albeit with a much different version of Linux (4.2.1)- here's the dmesg output from this box on 5.0.016:


Linux version 4.2.1 (vix@ns3044876.ip-51-255-93.eu) (gcc version 6.3.0 (GCC) ) #1 SMP Sun Apr 23 12:37:11 BST 2017
Fetching vars from bootloader... found 15 vars.
Options: moca=0 sata=1 pcie=0 usb=1
Using 512 MB + 0 MB RAM (from CFE)
bootconsole [early0] enabled
CPU0 revision is: 0002a065 (Broadcom BMIPS4380)
FPU revision is: 00130001
Determined physical RAM map:
memory: 10000000 @ 00000000 (usable)
memory: 10000000 @ 20000000 (usable)
bmem: adding 55 MB LINUX region at 8 MB (0x0378c000@0x00874000)
bmem: adding 192 MB RESERVED region at 64 MB (0x0c000000@0x04000000)
bmem: adding 256 MB LINUX region at 512 MB (0x10000000@0x20000000)
Zone ranges:
Normal [mem 0x0000000000000000-0x000000002fffffff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000000fffffff]
node 0: [mem 0x0000000020000000-0x000000002fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff]
On node 0 totalpages: 131072
Normal zone: 1024 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 131072 pages, LIFO batch:31
Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
Primary data cache 64kB, 4-way, VIPT, cache aliases, linesize 64 bytes
PERCPU: Embedded 11 pages/cpu @81407000 s12688 r8192 d24176 u45056
pcpu-alloc: s12688 r8192 d24176 u45056 alloc=11*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: rootfstype=ubifs root=ubi0:rootfs slab_max_order=0 rootflags=sync ubi.mtd=0 rw bmem=192M@64M hwtypenum=117

PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 314412K/524288K available (6490K kernel code, 353K rwdata, 1408K rodata, 244K init, 140K bss, 209876K reserved, 0K cma-reserved)

[five384]

Thanks aido. I wasn't able to find a 4.0+ log like that via google.

That kind of points to a configuration issue with the Edision kernel, rather than a BMIPS4380/Linux 4.0+ limitation. That's probably good news!

[aido]

It does seem specific to the Edision - I just jumped on the folks Miraclebox Micro and that has the full memory too and yet another kernel version (this is 5.0.014):


Linux version 4.0.1 (vix@ns202814.ovh.net) (gcc version 6.3.0 (GCC) ) #1 SMP Mon Feb 20 13:06:42 CET 2017
Fetching vars from bootloader... found 13 vars.
Options: moca=0 sata=1 pcie=0 usb=1
Using 512 MB + 0 MB RAM (from CFE)
CPU0 revision is: 0002a065 (Broadcom BMIPS4380)
FPU revision is: 00130001
Determined physical RAM map:
memory: 10000000 @ 00000000 (usable)
memory: 10000000 @ 20000000 (usable)
bmem: adding 54 MB LINUX region at 9 MB (0x036bd000@0x00943000)
bmem: adding 192 MB RESERVED region at 64 MB (0x0c000000@0x04000000)
bmem: adding 256 MB LINUX region at 512 MB (0x10000000@0x20000000)
Zone ranges:
Normal [mem 0x0000000000000000-0x000000002fffffff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000000fffffff]
node 0: [mem 0x0000000020000000-0x000000002fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff]
On node 0 totalpages: 131072
Normal zone: 1152 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 131072 pages, LIFO batch:31
Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
Primary data cache 64kB, 4-way, VIPT, cache aliases, linesize 64 bytes
PERCPU: Embedded 10 pages/cpu @81487000 s10384 r8192 d22384 u40960
pcpu-alloc: s10384 r8192 d22384 u40960 alloc=10*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129920
Kernel command line: ubi.mtd=rootfs rootfstype=ubifs root=ubi0:rootfs bmem=192M@64M console=ttyS0,115200n8
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 313080K/524288K available (7297K kernel code, 364K rwdata, 1400K rodata, 264K init, 136K bss, 211208K reserved, 0K cma-reserved)

[five384]

I've done some more digging around. It appears that Technomate and Miraclebox (aido's logs) use memory parameters passed from CFE/bootloader, whereas Edison uses devicetree bindings?

Combing through an open-edision kernel patch (https://github.com/open-edision/edision-kernel/releases/download/v4.9/edision-kernel-4.9.patch.xz) (linked), I see a single 256MB binding for bcm7362.dtsi (OS Mini), corresponding to the "10000000 @ 00000000" segment in the RAM map.

That might explain the missing RAM, except for the log line about the ignored 256MB due to highmem - where does it derive that from, if not CFE or the DTB?

...definitely out of my depth here.

It may only yield an extra 64MB of usable RAM for Linux - if the BCM97362SVMB always looses 192MB out of 512MB like aido's two logs - but that 64MB would make a useful difference for running the OS Mini without swap.

[five384]

One more thing: the baseline Broadcom 4.12 kernel DTS (https://github.com/Broadcom/stblinux/blob/master/arch/mips/boot/dts/brcm/bcm97362svmb.dts) (linked) still opens a potential for 1Gb mapped RAM on this SoC (with a 256MB hole at 0x1000000, like aido's logs), whereas the Edision overlay patches it back to 256MB max in line with that commit from 2015 that I linked in the first post.

There was a further comment on that thread about how the "Broadcom kernels enable the CPU's special "XKS01" feature to put 1GB of memory in ZONE_NORMAL": https://www.linux-mips.org/archives/linux-mips/2015-05/msg00103.html

I'm not in a position to actually try any of this out, but hopefully somebody reading this can check.