Silicon Valley is turning into its own worst fear (2017)

If you’re not familiar with Ted Chiang, make sure you put Story of Your Life and Others on your reading list.

How unfortunate that this article aged so well.

Great article. Ted Chiang's "Understand" also happens to be my favorite short story of all time.

Problem is that for strawberries or diamonds or SV startup stuff there is a line where if you make something infinitely available it will also loose the value.<p>You cannot equate paper clip maximizer with SV companies. Because fitness functions are different.<p>Not saying that diamonds industry didn’t make world worse off but still somehow it didn’t take over the world to make people eat diamonds for breakfast. So the same there will be no entity that can make all people everywhere eat strawberries for breakfast all year long.<p>Scary part is finance industry that basically already is self conscious with all the rules baked in and no single person being able to grasp it.<p>Finance with AGI could already become paper clip optimizer - but it actually needs energy only. It doesn’t need humans anymore. So it would most likely fill in whole world with power plants and erase all other life just to have the electricity.

&gt;Then I realized that we are already surrounded by machines that demonstrate a complete lack of insight, we just call them corporations. Corporations don’t operate autonomously, of course, and the humans in charge of them are presumably capable of insight, but capitalism doesn’t reward them for using it. On the contrary, capitalism actively erodes this capacity in people by demanding that they replace their own judgment of what “good” means with “whatever the market decides.”<p>Excellent quote. We say we&#x27;re in a rational world where we make rational decisions in our societal game we&#x27;ve all been told we have to play - capitalism where money is the determinant of success vs failure for corporations, families, individuals.<p>But step back and when looking at the question of whether it&#x27;s rational to us as humans be playing this game and it is not rational at all. Why are we not deciding that food and places to live for everyone is the determinant of success of a country, society? Or happiness?<p>60 Minutes has a segment about Bhutan from a couple weeks ago [0] about this. They lived by something they named &quot;Gross National Happiness&quot;. Which feels weird to type but again stepping back, it&#x27;s because our whole lives we&#x27;re told that &quot;Gross Domestic Product&quot;, overall money, is the determinant of &quot;best&quot; and that&#x27;s so engrained for us.<p>On a different note, Ted Chaing&#x27;s short story books [1][2] are incredibly, incredibly good. I&#x27;m reading them again and read &quot;Story of Your Life&quot; earlier today. Being able to write fiction like that makes it much more trusting to listen to what someone has to say on other topics. And saying that seems like another topic - how we&#x27;re told to downplay fiction compared to non-fiction, when our brains evolved for stories. But that&#x27;s for another comment.<p>[0] - <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=7g_t1lzn-1A" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=7g_t1lzn-1A</a><p>[1] - <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Stories_of_Your_Life_and_Others" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Stories_of_Your_Life_and_Other...</a><p>[2] - <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Exhalation:_Stories" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Exhalation:_Stories</a>

Exceptionally good. Wow, going to be reading some of this fellow!

&gt; we are still a long way from a robot that can walk into your kitchen and cook you some scrambled eggs<p>I wonder if he&#x27;s seen the latest videos of staged demos where humanoid robots can fold clothes<p>edit: didn&#x27;t say 2017 when I commented.

The problem is not capitalism, it’s corporatism.<p>Or more precisely, it is that we allow a group of people, under the control of a single person or very small number of people, amass incredible power, and do so in an amoral framework in blind service to a goal of stock price growth.<p>The problem is a scale problem in my mind. If you limit the scale then all the other problems become manageable.<p>We wouldn’t need antitrust laws anymore if our tax laws made it unprofitable to own shares in a company with, for example, 50% of search market share or online retailing share.

The idea that AGI doom scenarios are really late-stage capitalism is interesting and strikes me as fundamentally correct. The difference between the AGI takeover and capitalism is just the choice of metric to optimize on.<p>But, I think, it&#x27;s the act of trying to optimize on a metric itself that is the source of the destruction. Unmeasurable human values can&#x27;t survive an optimization process focused on measurable ones.

&gt; This summer, Elon Musk spoke to the National Governors Association and told them that “AI is a fundamental risk to the existence of human civilization.”<p>And here I was going to suggest that billionaires, unbridled mega-corporations were the fundamental risk to the existence of human civilization.<p>&gt; Musk gave an example of an artificial intelligence that’s given the task of picking strawberries.<p>Also odd since it&#x27;s more likely that a corporation, in the name of maximizing profits, would make decisions that threaten humanity. We can start with Bhopal, India. If you find fault with that example I am sure there are plenty of others, some probably a good deal more subtle, that others can suggest.<p>Me, not worried at all about AI.

This article compares a strawberry picking machine killing all of humanity to increase strawberry fields with current corporations. This is stupid because corporations do have guardrails both internally and externally from society. All of the mega Silicon Valley corporations are not expanding by murdering people. Even in the ones that are expanding in ways that people question (social media), they are filled with humans who actually think they are doing the right thing.<p>Humans with morals are still very much in the decision chain and there is obviously a lot of debate about their morals, but them being there makes such a vast difference that the comparison to the strawberry AI is completely invalid. The strawberry AI isn’t even considering humans.<p>The article then builds on that false comparison for the rest of the article so there isn’t much to gain from the rest of it.<p>You can make the same lazy comparison to a completely socialist, centralized decision making by a government optimizing for a single metric (voter approvals, poverty levels, whatever). It has nothing to do with capitalism or the economic system.<p>TLDR; article says mega corps are the same as dangerous AI because they make optimizations in favor of profit that some people disagree with.

I love his science fiction but he was wrong on AI in 2017 and is even more wrong on today.