I was going to complain that a bug should be filed for this, but one of the blog comments says this issue could be fixed by [1].<p>[1]: <a href="https://github.com/golang/go/issues/54766" rel="nofollow">https://github.com/golang/go/issues/54766</a>
FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away.<p>I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn't what you want.<p>If a map has this behaviour, i would say that the most important thing is that it should be clearly documented (Java's isn't). The second most important thing is that there should be a way to get round it - either a .clearHarder() method which throws away the table, or a .compact() method which downsizes it while retaining the content.
Yes, Go maps never shrink. This is good for most use cases in practice.
Because in practice, map entry deletions happen seldom.
And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves.
For example, I only do map entry deletions in one of my projects,
In the project, I clear all entries of a map and re-use the map to avoid making new allocations. The current design satisfies my need well.<p>This is more an optimization than a memory leak.<p>To avoid the kind-of memory leak, just make a new map and discard the old one.
what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?
Huh, I didn't knew that... that might explain one slow "leak" I've been noticing in one of my apps.<p>Shame that quirk wasn't documented
This all looks like reasonable implementation behavior which will give optimal runtime performance in most "common" cases. If one really wants or needs a map that'll free memory as it shrinks (and sure, for some folks, that'd be super useful) one's always free to just implement your own.
> Also, in this case, the amount of required memory is less significant during peak times due to some optimizations to reduce the memory consumed.<p>Pretty sure it’s the same: because of collisions, maps always have a certain fraction of empty slots (much more so than vectors). I’ve not heard of Go maps being very high load factors, so they probably resize around 50% or so full. I didn’t check any of the numbers, I’ll assume 50% load factor, doubling on resize, and entries of {hash, key, value} with a word-sized (8 bytes) hash, but some and likely all of those are likely off.<p>Anyway with a load factor of 50% and doubling at any point you’d have 2n to 4n slots in the map total (depending whether you’re nearing resize or just went through one). And thus if you make entries smaller, you make all those overhead slots smaller as well.<p>hash + int + blob would be 8+8+128 144 bytes, a million of those is 144MB, x2 is 288 and x4 is 576, so the observed bounds are pretty close: IIRC Go maps use some form of separate chaining (though not “trivial” linked lists), I assume those do get reclaimed as entries are removed even if the map itself does not get shrunk, which would explain why memory consumption goes down as elements are removed from the initial map.<p>With a pointer each “entry” is 24 bytes instead, for a map overhead of 48~96MB (so clearly Go maps use either a higher load factor or a more efficient storage than the trivial scheme being assumed here, possibly both).<p>The actual data accounts for 128M plus some, there’s a 144M difference between the full and the emptied pointer-maps, which would account for some chain links being removed, and maybe some object headers / allocator overhead for the on-heap arrays.
< 293 MB <-- After we remove 1 million elements<p>Let's be kind and assume that prior to removal, the map has just trebled in size and the extra space wasn't used. Doesn't this imply that a map has an overhead of about <i>100 bytes</i> per key/value pair? How can this be so?
i don't see any issue here. same behavior can be achieved with slices:
foo := make([]int, 0, 1000)
for {
for k := range bar {
foo = append(foo, k)
}
foo = foo[:0]
}<p>the slice will grow as much as the largest dataset. map will be the same. you need to let go of it to be GCd and create a new one.