X Engineering Year Retrospective

I&#x27;m inclined to believe that they consolidated their code across a lot of their microservices and simplified their architecture since that was stressed a lot from the moment Musk acquired Twitter. But we also can&#x27;t really verify or disprove it.<p>I&#x27;m a bit confused about their so-called improvements to video recommendation quality and bot detection. I&#x27;ve seen a lot of sentiment from people that they see more bots, hate speech, and irrelevant content on their timelines. Maybe what I&#x27;m hearing is just anecdotal evidence or stories in a bubble?<p>The Sacramento data center migration to Portland is an entertaining story detailed here[1]. Here&#x27;s the Hacker News thread on it[2].<p>They have a GPU supercompute cluster?? It seems like they have the capability to do training and inference with state-of-the-art algorithms at massive scales then. Why have Twitter&#x27;s recommendations and ad revenue (even before the acquisition) been so poor then?<p>[1] <a href="https:&#x2F;&#x2F;www.cnbc.com&#x2F;2023&#x2F;09&#x2F;11&#x2F;elon-musk-moved-twitter-servers-himself-in-the-night-new-biography-details-his-maniacal-sense-of-urgency.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.cnbc.com&#x2F;2023&#x2F;09&#x2F;11&#x2F;elon-musk-moved-twitter-serv...</a><p>[2] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37470110">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37470110</a>

&quot;Shutdown the Sacramento data center and re-provisioned the 5,200 racks and 148,000 servers, which generated more than $100M in annual savings.&quot;<p>Yeah, about that.. <a href="https:&#x2F;&#x2F;www.cnbc.com&#x2F;2023&#x2F;09&#x2F;11&#x2F;elon-musk-moved-twitter-servers-himself-in-the-night-new-biography-details-his-maniacal-sense-of-urgency.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.cnbc.com&#x2F;2023&#x2F;09&#x2F;11&#x2F;elon-musk-moved-twitter-serv...</a><p>&quot;Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved.&quot;

&gt; Among the changes we made was a shift of all media&#x2F;blob artifacts out of the cloud, which reduced our overall cloud data storage size by 60%, and separately, we succeeded in reducing cloud data processing costs by 75%.<p>Huh. I feel like that&#x27;s the one place to <i>not</i> leave cloud. 90% of why I want to use AWS is S3.

I hope their Trust &amp; Security team has a roadmap that prioritizes topic and thread-jacking. The advent of GPT and their recent changes to creator compensation has created a massive drive for engagement by any means; the results are fairly predictable. Clicking on any trending topic usually surfaces completely unrelated videos with that topic&#x27;s hashtags (and all of the other trending topic hashtags) appended to the tweet.<p>Then you have the &quot;content creators&quot; who use GPT to summarize or add details to a post from a larger content aggregator in hopes of bandwagoning engagement. I see a lot of this type of behavior from popular History-focused accounts and the mega-accounts that post engagement bait content (think canned, &quot;desert island&quot;-type questions and polls). It&#x27;s less malicious but certainly reinforces cynicism towards the state of Twitter&#x2F;X and the broader social web.

I&#x27;ve seen this in other places I&#x27;ve worked. Too many people and too much money to spend can be a lot worse than too few and not enough.

&quot;Shrank site traffic by 80%, yielding cost savings and cratering the company as a whole&quot;<p>I hope everyone there is getting paid in advance for their work...

Changelog: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;cb_doge&#x2F;status&#x2F;1717668534078034024" rel="nofollow noreferrer">https:&#x2F;&#x2F;twitter.com&#x2F;cb_doge&#x2F;status&#x2F;1717668534078034024</a>

That project must be so much fun to work on. I&#x27;d pay just to shadow it.