I've just run a build from scratch (kept sources) of 6.2.002.004.
On a system with 8 CPUs and 16GB of memory the maximum memory usage, according to sar, was 80% and there is no sign of paging.
I just let it run with the standard CPU count setting, as since both bitbake and several (many?) of the individual builds all try to use all of the CPUs they are always well overcommitted (my run-q size hovers around 40 and reached >70 at a maximum).
I believe you increase yours - which may be the cause of paging, as all it will do is leave more processes running but waiting (and hence consuming memory).