Lots of great insight. Here’s one:<p>“Given the long timelines of a PhD program, the vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not.<p>Today, the vast majority of new ML researcher hires are freshly minted PhDs, who have only ever studied problems from the ML point of view. I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML.”
I work for Google Brain. I remember meeting Brian at a conference and I have nothing but good things to say about him. That said, I think Brian is underestimating the extent to which the Brain/DeepMind merger is happening because it's what researchers want. Many of us have a strong sense that the future of ML involves models built by large teams in industry environments. My impression is that the goal of the merger is to create a better, more coordinated environment for that kind of research.
>PyTorch/Nvidia GPUs easily overtaking TensorFlow/Google TPUs.<p>TF lost to PyTorch, and this is Google’s fault - TF APIs are both insane and badly documented.<p>But nothing comes close to performance of Google’s TPU exaflop mega-clusters. Nvidia is not even in the same ballpark.
because once Jeff Dean had solved Google's maslow problems (scaling web search, making ads profitable, developing high performance machine learning systems) he wanted to return to doing academic-style research, but with the benefit of Google's technical and monetary resources, and not part of X, which never produces anything of long-term value. I know for sure he wanted to make an impact in medical AI and felt that being part of a research org would make that easier/more possible than if he was on a product team.
Google has good engineers and a long history of high throughput computing. This, combined with a lack of understanding what ML research is like (versus deployment), led to the original TF1 API. Also, the fact that google has good engineers working in a big bureaucracy probably hid a lot of the design problems as well.<p>TF2 was a total failure, in that TF1 can do a few things really well when you get the hang of it, but TF2 was just a strictly inferior version of pytorch, further plagued by confusion due to TF1. In alternate history, if Google pivoted in to JAX much earlier and more aggressively, they could still be in the game. I speak as someone who has at some point knew all the intricacies and differences between TF1 and TF2.
> it is becoming increasingly apparent to Google that it does not know how to capture that value<p>To paraphrase, its the business model, stupid.<p>Inventing algorithms, building powerful tools and infrastructure etc is actually a tractable problem: you can throw money and brains at it (and the latter typically follows the former). While the richness of research fields is not predictable, you can bet that the general project of employing silicon to work with information will keep bearing fruits for a long time. So creating that value is not the problem.<p>The problem with capitalizing (literally) on that intellectual output is that it can only be done 1) within a given business model that can channel effectively it or 2) through the invention of totally new business models. 1) is a challenge: These billions of users on which AI goodies can surface are not customers, they are product. They don't pay for anything and they don't create any virtuous circle of requirements and solutions. Alas, option 2) inventing major new business models is highly non-trivial. The track record is poor: the only major alternative business model to adtech (cloud unit) was not invented there anyway and in any case selling sophisticated IT services whether to consumers or enterprise is a can of worms that others have much more experience in.<p>For a industrial research unit to thrive, its output must be congruent with what the organization is doing. Not necessarily in the detail, but definitely in the big picture.
> … the publication of key research like LSTMs in 2014 …<p>Minor nitpick, but LSTMs date to 1997 and were not invented by Google. [1]<p>[1] Hochreiter and Schmidhuber (1997). Long short-term memory. <a href="https://ieeexplore.ieee.org/abstract/document/6795963" rel="nofollow">https://ieeexplore.ieee.org/abstract/document/6795963</a>
My theory is that broadly, tech learned not to act like Microsoft in the 90s -- closed off, anti-competitive, unpopular -- but swung too far in the opposite direction.<p>Google has been basically giving away technology for free, which was easy because of all the easy money. It's good for reputation and attracting the best talent. That is, until a competitor starts to threaten to overtake you with the technology you gave them (ChatGPT based on LLM research, Edge based on Chromium, etc.).
"I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML. (This may be survivorship bias; the only chemists I encounter are those that have successfully learned ML, whereas I see ML researchers attempt and fail to learn chemistry all the time.)"<p>This is something that rings really true to me. I work in imaging and it's just very clear that there are groups of people in ML that don't want to learn how things actually work and just want to throw a model at it (this is a generalization obviously, but it's more often than not the case). It only gets you 80% there, which is fine usually, but not fine when the details are make or break for a company. Unfortunately that last 20% requires understanding of the domain and people just don't like digging into a topic to actually understanding things.
> Neither side “won” this merger. I think both Brain and DeepMind lose. I expect to see many project cancellations, project mergers, and reallocations of headcount over the next few months, as well as attrition.<p>This merger will be a big test for Sundar, who has openly admitted years ago to there being major trust issues [1]. Can Sundar maintain the perspective of being the alpha company while bleeding a ton of talent that doesn't actively contribute to tech dominance? Or will he piss off the wrong people internally? It's OK to have a Google Plus / Stadia failure if the team really wanted to do the project. If the team does _not_ want to work together though, and they fail, then Sundar's request that the orgs work together to save the company is going to get totally ignored in the finger-pointing.<p>[1] <a href="https://www.axios.com/2019/10/26/google-trust-employee-immigration" rel="nofollow">https://www.axios.com/2019/10/26/google-trust-employee-immig...</a> .
> I sat on it because I wasn’t sure of the optics of posting such an essay while employed by Google Brain. But then Google made my decision easier by laying me off in January. My severance check cleared...<p>I'm really baffled by how people think it's OK to write public accounts of their previous (<i>and sometime current!</i>) employers' inner workings. This guy got paid a shitload of money to do work <i>and to keep all internal details private, even after he leaves</i>. They could not be more clear about this when you join the company.<p>Why do people think it's OK to share like this? This isn't a whistleblowing situation -- he's just going for internet brownie points. It's just an attempt to squeeze a bit more personal benefit out of your (now-ended) employment.<p>Contractual/legal issues aside, I think this kind of post shows a lack of personal integrity (because he <i>did</i> sign a paper agreeing not to disclose info), and even a betrayal of former teammates who now have to deal with the fallout.
Organized, concise, and not wordy. Props to the writer, he shows a deep degree of written communication skills on a topic frequently cluttered with jargon.
> Today, thought leaders casually opine on how and where ML will be useful, and MBAs feel like this is an acceptable substitute for expert opinion.<p>Sounds like standard operating procedure.
This is great.<p>While Google is busy imploding the next generation of startups can flourish. I'm being hopeful that they decimate a lot of big tech and they don't just all get bought out.<p>Diversity might return to the Internet.<p>Wishful thinking, I know.
Hmm. Anything that slows Google down and maintains a diversity of leaders in the field is ok with me.<p>Imagine a host of "helpful" Google AI's, Facebook AI's, Amazon AI's, etc., that know their very existence depends them monetizing you more effectively than competitive AI's.<p>Of course, the first versions will be very helpful. But continuous efforts to remain "the most helpful" will cost a lot, and eventually need to pay for themselves.
> The next obvious reason for Google to invest in pure research is for the breakthrough discoveries it has yielded and can continue to yield. As a rudimentary brag sheet, Brain gave Google TensorFlow, TPUs, significantly improved Translate, JAX, and Transformers.<p>Except that these advances have made other companies an existential threat for Google. 2 years ago it was hard to imagine what could topple Google. Now a lot of people can see a clear path: large language models.<p>From a business perspective it's astounding what a massive failure Google Brain has been. Basically nothing has spun out of it to benefit Google. And yet at the same time, so much has leaked out, and so many people have left with that knowledge Google paid for, that Google might go the way of Yahoo in 10 years.<p>This is the simpler explanation of the Brain-Deep Mind merger: both Brain and Deep Mind have fundamentally failed as businesses.