I would like to see stats beyond those provided by top. For example, memory bandwidth (broken into read and write bandwidth). IO bandwidth. CPU cache hit rates. Instructions per cycle. Energy consumption. AFAIK there is no really nice way to see all the true under the hood perf stats. Yes I know about perf, LIKWID, and other similar tools -- point being, I think there is an opportunity to put this kind of monitoring into a tool at the level of scout and truly have something informative.
I confirm that a high waiting time can be due to a damaged disk. I had a server that became slower over the time notwithstanding a constant number of requests. It took me a while to notice a 90% waiting time in top. A disk was dying, I got it replaced and the server got speedy again.
Where is time spent transitioning from userspace to kernel and back accounted for? And (perhaps relatedly), when does the accounting happen? After each timeslice?