This doesn't seem corroborated just from that chart. runtime.systemstack() is one of the calls that allows code to switch from the user stack to the system stack, so that non-preemptible (by the goroutine scheduler) code can run, but this doesn't only include garbage collection. It includes all system calls. The kubelet works by synchronizing node state with the definition retrieved from the apiserver across the network, so I would expect it spends a lot of time making system calls just to read from and write to whatever port and/or socket it uses to do that.
I’m still convinced GC is not a good idea as the single memory model for a language. It is possible to know about memory behavior at compile time and not pay at runtime for it in latency.
This is amusing next to "Go code that uses 20% of CPU time in GC is usually bad Go code," but how much does it matter? Kubelet runs on every machine in the cluster, but I'd expect few of each machine's cycles to be spent on it. I wouldn't be surprised if relatively little attention has been paid to optimizing its GC and/or if someone has deliberately tuned it to reduce RAM at the expense of GC CPU.<p>Put another way: this percentage uses the wrong denominator for ranking optimization targets. Don't rank by percentage of the binary's cycles but instead by percentage of overall cluster cycles.<p>This is particularly true for Kubernetes itself. It affects the efficiency of the cluster as a whole via its bin-packing decisions. Eg the whole cluster becomes more efficient when Kubernetes minimizes stranded capacity and also when it isolates workloads that heavily thrash the CPU cache to separate NUMA nodes or machines. Thus if I were to dive into Kubernetes optimization, I'd focus on bin-packing much more than its own GC cycles.