The Broken Promises of MRI/REE/YARV

I think this is a problem that exists across any VM that implements a GC, not just Ruby.<p>.NET CLR has the exact same problem (perhaps a harder one, since CLR has a moving GC), so anytime they touch GC references (pointers to objects that are collectible) it's always wrapped in an explicit GC stack frame (think GC struct that lives on the stack). Furthermore, all reads/writes are carefully done with macros (which of course expands to volatile + some other stuff) to make sure the compiler doesn't optimize it away.<p>On the one hand, this is nice because they don't need to scan the C-stack (it scans the VM stack and the fake GC frame stacks -- well it's one stack but you skip the native C frames), on the other hand this means that any time a GC object is used in C code (ok, actually it's C++) they have to be real careful to guard it.<p>Of course bugs crop up all the time where an object gets collected where it shouldn't have, it happens so often that there is a name for it -- "GC Hole".<p>Astute readers and users of p/invoke may remark that they don't have to set up any "GC frames" -- that is because this complicated scheme is not exposed outside of the CLR source. Regular users of .NET who want to marshal pointers between native/managed can simply request that a GC reference gets pinned, at which point I'm mostly sure it won't get collected until it's unpinned.<p>The bad news is I'm almost positive there is nothing you can do with just C here to make this problem go away. You'd want stuff to magically just happen under the hood, and C++ is the right way to go for that.<p>It's probably possible to create an RAII style C++ GC smart pointer that would be 99% foolproof at the expense of some performance. It gets a little bit trickier if we are doing a moving collector. I am thinking it could ref/unref at creation/destruction, and disallow any direct raw pointer usage not to shoot yourself in the foot.<p>Of course the people writing the GC still need to worry about this..

I do appreciate the technicality of the article, but I'm not sure to agree with the first point of conclusion: how does it makes MRI (and related) 'fatally flawed' though? (real question).<p>What makes it 1/ irreversible and 2/ bad for today's users?<p>EDIT: as well, I wouldn't stop using Ruby because of that; I would use JRuby or Rubinius or IronRuby (if I understand well, these ones are not affected?)

This post is a weird mix of careful technical analysis and douchey, Zed Shaw-style hysterical overstatement.<p>However, I would like to see Matz' response to the recommended steps for a fix at the end. Sounds like a reasonable goal to add for Ruby 2.0.<p>Note to self: Listening to Papoose while writing a technical blog post turns your otherwise important observations into a Chicken Littleish, end-of-the world rant.

I think this goes to a pretty simple point: anything you have to do by hand you will eventually get wrong. Thus, to a first approximation anything that can be automated, probably ought to. To show off this principle I'm going to show off some of the PyPy source code: <a href="https://bitbucket.org/pypy/pypy/src/default/pypy/module/select/interp_epoll.py" rel="nofollow">https://bitbucket.org/pypy/pypy/src/default/pypy/module/sele...</a><p>This is the implementation of `select.epoll`. Somethings you'll notice there's no GC details (allocations outside the GC of C level structs are handled nicely with a context manager), and we have a declarative (rather than imperative) mechanism for specifying argument parsing to Python level methods, this ensures consistency in readability as well error handling, etc.

Cute. The Boehm-Demers-Weiser collector has GC_reachable_here for this reason. Guile has scm_remember_upto_here since before it switched to libgc. I'm sure other systems have their things too.<p>That said, I like Handle, the RAII thing that V8 uses. It also allows for compacting collection. Too bad C doesn't do RAII.

Can someone dissect this a little more? My understanding is the pointer to str never gets written to the stack, and so str on the heap might get freed before zstream_append_input makes use of it. But how could the GC see this/what is the faulty assumption?

So, what this really seems to boil down to, is:<p>The Ruby C API is returning objects that are not correctly reference-counted for a short period of time and are incorrectly subject to GC.<p>This doesn't seem fatal to me, just not reasonably fixable from the GC side. It <i>might</i> be true, that a new API is needed to hold refs in the C side.

"Very few people out there know that the volatile type qualifier exists"? Only if there are "very few" kernel programmers, embedded programmers, and others who have used C for anything low-level and/or multi-threaded. Otherwise, no. Sorry, but knowing about it doesn't make you special.<p>"Volatile" is the wrong fix, by the way. That's just depending on yet another non-required behavior. There is in fact no further reference to "str" between the function call and the reassignment at the start of the next iteration, so there's nothing for "volatile" to chew on. This particular version of this particular compiler just happens to add an extra pair of stack operations in this case, but it's not truly required to. A real fix would not only mark the variable as volatile but also add a reference after the function call. The same "(void)str;" type of statement that's often used to suppress "unused argument/variable" warnings should count as a reference to force correct behavior here.

Well plus one for a blog post with a theme song, anyway.