On one hand, I understand commercial/IP interests may prevent full code release.<p>But this creates a strange situation where a paper presents detailed methodology and performance claims but code generating those results is withheld. The community can't verify or build upon the actual implementation.<p>The middle ground of sharing methodology without tools to reproduce seems to serve neither research nor commercial interests well.<p>If protecting IP is the priority, what's the incentive to publish at all?
This problem spans the sciences, not just ML. In the ML space I can see the IP issue people might claim given the commercial potential these days, but given that the same reproducibility issue arises in domains with no obvious IP protection motivation, that seems like a weak cause. In my experience the incentive just isn’t there to do the work to make stuff reproducible. Very few people get in much trouble if their work isn’t reproducible since most papers get very few reads and even fewer readers who would try to replicate it. I think publication venues (conferences, journals) should make artifacts a mandatory requirement for publishing.