TikTok reveals details of how its algorithm works

It&#x27;s a (seemingly) pretty simple matter of keeping track of how long someone looks at a video, and optimizing to show them videos they&#x27;ll spend a long time looking at. It&#x27;s pretty powerful too, I spend much more time looking at TikTok than any other digital content aggregator (to the extent I had to delete it from my phone).<p>They also do some things like sprinkle in random fresh videos, potentially unrelated to your interests, to your &quot;For You&quot; to get exposure to them and a base idea of how long people look at them, which is nice because it potentially boosts small creators to larger audiences.<p>Further, I&#x27;m pretty sure they see what creators are keeping the most people on their platform for the longest, and directly compensate them. This gives rise to a host of &quot;lifestyle accounts&quot; where folks can live doing the things that some chunk of the userbase wants to be doing, and they&#x27;ll get paid directly from TikTok, rather than needing to source a third party company to sponsor them.<p>All in all, I think it&#x27;s fantastically designed -- to the extent I&#x27;m not sure it should exist at all.

TikTok is like a highly compressed version of YouTube.<p>YouTube incentivizes creators to make artificially long videos, resulting in a huge amount of filler. So you get a lot of videos where you can skip the first 20%+ and not miss anything. The content is buried and spread out.<p>1 minute of content surrounded by 9 minutes of filler.<p>TikTok removed those 9 minutes, so it feels very refreshing in that respect. There&#x27;s no incentive to create filler - just the opposite.<p>Of course the downside is that there&#x27;s only so much you can fit into a 60 second package, so you&#x27;re not going to get a deep dive into anything. But for the kind of content that can be compressed like that, TikTok wins big time.

I don&#x27;t see any details about its algorithm, just that they use a recommendation engine? Things like this aren&#x27;t proprietary info, they&#x27;re just how recommendation engines work:<p>&quot;Once TikTok collects enough data about the user, the app is able to map a user&#x27;s preferences in relation to similar users and group them into &quot;clusters.&quot; Simultaneously, it also groups videos into &quot;clusters&quot; based on similar themes, like &quot;basketball&quot; or &quot;bunnies.&quot;&quot;<p>Although I do wonder, and maybe someone else with more experience could shed some light here, whether or not it is likely that TikTok has some fundamentally super advanced algorithm, or if they just do a better job of collecting data&#x2F;training &amp; evaluating their models?

Heh,<p>&quot;We&#x27;re a 2-year-old company operating with the expectations of a 10-year-old company,&quot; said Michael Beckerman, TikTok&#x27;s vice president in charge of U.S. public policy. &quot;We didn&#x27;t have the opportunity to grow up in the golden years of the internet, when tech companies could do no wrong. We grew up in the techlash age ,where there&#x27;s a lot of skepticism of platforms, how they moderate content and how their algorithms work.&quot;&quot;<p>Well, yes, Compete or die. It&#x27;s the same as a tobacco company start up.<p>Oh, wait, that&#x27;s Juul.<p>What happened to them?

Not much of a reveal.<p>What I find strange about TikTok are the waves of themes I get to see. Being a 50-something I&#x27;m probably not the target audience but at first all I saw was shuffle dance (enjoyed that) followed by Indian and Chinese manufacturing videos (loved those) and now it&#x27;s all cats (okay with that).<p>My daughters explained &quot;Elite TikTok&quot; and &quot;Beans TikTok&quot; so now I realize there&#x27;s a whole world built on top of gaming the algorithm in order to bolster your insider status.

How TikTok&#x27;s &quot;algorithm&quot; works is pretty obvious if you&#x27;ve spent any time on there.<p>It learns what you like by how much of a video you watch and how you interact with it, and it establishes some kind of weighted feature vector based on hashtags used in the description, words used in the comments, text drawn on the video, audio background, possibly some audio transcription of words said, if you commented on it, if you forwarded it to a friend, and so on. There may be some network based recommendations, based on who you follow but those seem to be weighted very weakly, and that makes sense if you&#x27;re trying to keep the platform from getting botted.<p>It also seems to do some non-dominant sorting to keep from only showing you things from the same type of video.<p>Facebook&#x27;s competing short video service is terrible in comparison. It only wants to show me Trevor Noah clips, and things tangentially related to things I may have &quot;Liked&quot; 12 years ago.

I think a huge reason, I didn’t see in the article, that TikTok is able to train their ML to be far more effective than any other video platform out there has to do with the size of those clips and how easy it is to flip through them.<p>If we are watching 30 second clips, flipping through them quickly, that’s 2 signals a minute and 20 signals every 10 minutes (the average size of a YouTube video) and that is assuming we watched 20 videos, we might have flipped through 60 in 10 minutes.<p>That’s 60 signals where YouTube would’ve gotten 1 signal. Whose ML will be better trained with your preferences? Obviously TikTok, I don’t think there is anything special about their algorithms it’s just a function of their product and the type of content.

I never really understood why TikTok is such a threat<p>Except for maybe collecting too much data, but that’s any app, and luckily Apple is taking their measurements for certain things

What&#x27;s been funny is how there&#x27;s been so many trends that relate to exploiting the engagement stats. For a long time, there were a lot of videos begging for likes or claiming that the heart would be purple on the particular video; this seemed to lead to a de-emphasis of likes. There&#x27;s a lot of videos&#x2F;sounds that involve a long build up leading to a short reveal, making sure the viewer finishes the video. One-frame image reveals encourage downloads and replays, and content hidden behind the interface can often lead to many downloads. Videos explaining how to repeatedly hit the share link button will sometimes have more shares than views. There&#x27;s endless numbers of alternative spellings of words like sex and porn to avoid the edgy content filters, but I suppose that&#x27;s a given. The hashtags are weird to me since #xyzcba seemed to actually have an effect for a while.<p>The things I didn&#x27;t know in this article were the stuff about device type (what is that used for?) and the initial 8 videos. Perhaps the next trend I&#x27;ll see is flexing on having an unlocked, AT&amp;T-branded Samsung G892A.

Can someone please indicate if there is truly some kind of AI magic going on here?<p>TT claims it&#x27;s a &#x27;key magical ingredient&#x27; - and apparently acquisition talks are held up ostensibly over &#x27;access to training data&#x27; etc..<p>But I find this maybe hard to believe.<p>TT learning is pretty good, but I suspect an algorithm could do it, or possibly some basic ML, not necessarily needed Deep Learning.<p>I suspect the &#x27;Deep Learning&#x27; they use may not even be very sophisticated, but provide cover to radically inflate their ostensible value to CEO acquirers who just love the concept of &#x27;real AI&#x27; often irrespective of it&#x27;s material value.<p>I&#x27;m just suspicious there isn&#x27;t really much there, and find it odd that an entity like Microsoft would be intimidated by trying to do that piece well.

Does anyone know of an open source recommendation engine? I would love to just explore how one works bet it seems like this solely in the realm of proprietary software at the moment.<p>I have a dream of naively making a open source recommendation engine using data from good reads and imdb.<p>There is a real utility here that I worry will be captured entirely by companies without an open source alternative.

Devil is in the details, a lot of which seem to have been lost in this explanation. This seems to be mostly a view into their value model (ie prioritize P(Click) * value + P(Heart) * value etc), but the hidden issue arise from the type of content that is likely generate clicks &#x2F; hearts &#x2F; long watch times.

To be fair, the algorithm they told is mostly what we would have perceived, knowing if that is Machine Learning driven.<p>Although, I am quite intrigued what they would be showcasing in their &quot;transparency centers&quot;. If they show what extent their data can be utilized. and not simply a simulation of the ML stuff.

The main reason for me why YouTube is out of choice is too many Ads interruptions and absence of Portrait mode video support. Even Telegram supports Vertical videos - you can zoom into any horizontal video, and I can move that window so that I can watch full screen on my phone.

I find that it takes device into account very interesting. What does me using an iPhone XS vs a Samsung Note vs a generic cheap Android phone say about me? I wonder if ends up being a proxy for income, age, or gender...

I didn&#x27;t see anything in there about how their algorithm prevents people from seeing videos involving poor people[1] or Hong Kong protests[2]. Maybe that&#x27;s part of the secret sauce.<p>[1] <a href="https:&#x2F;&#x2F;theintercept.com&#x2F;2020&#x2F;03&#x2F;16&#x2F;tiktok-app-moderators-users-discrimination&#x2F;" rel="nofollow">https:&#x2F;&#x2F;theintercept.com&#x2F;2020&#x2F;03&#x2F;16&#x2F;tiktok-app-moderators-us...</a><p>[2] <a href="https:&#x2F;&#x2F;www.washingtonpost.com&#x2F;technology&#x2F;2019&#x2F;09&#x2F;15&#x2F;tiktoks-beijing-roots-fuel-censorship-suspicion-it-builds-huge-us-audience&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.washingtonpost.com&#x2F;technology&#x2F;2019&#x2F;09&#x2F;15&#x2F;tiktoks...</a>

Well, this is not really &quot;revealing&quot; anything new on &quot;TikTok algorithms&quot;.<p>This is actually a very simple description of a very basic methodology. Not an algorithm.

One more tilt at the old windmill.<p>This is not an algorithm; this is a heuristic. An algorithm is (loosely) a method or procedure for achieving some specified end.<p>&quot;... avoid redundancies that could bore the user, like seeing multiple videos with the same music or from the same creator&quot; is the goal, and they have heuristics to try to work towards that, and algorithms and software that implement those heuristics.<p>I think the ship has sailed on this, but when you are in circles where both heuristics and algorithms are in play, this blurring of lines makes for very confusing conversations.

I mean it&#x27;s just Collaborative filtering no?

TLDR: We do some math <a href="https:&#x2F;&#x2F;xkcd.com&#x2F;1838&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;1838&#x2F;</a> On a more serious note I think there is a lot of sophistication and that is being left out from this very simplified explanation. To say something like &#x27;we show things from other clusters every so often&#x27; over looks so many questions about how far apart the clusters are, and how often these are shown. These values are foundational to the UX and understanding how to tune them should get a lot more focus than it does. In the future it would be good for the AI community and the associated HCI researchers in AIUX to focus on how these settings change the experience of a ML pipeline.

It’s rather comical, how for the past 20 years, Americans have been claiming themselves to be the pinnacle of creativity and innovation. And at the same time, denouncing the Chinese as a bunch of copycats-but-can-never-innovate. And they said the same thing about the Japanese a generation earlier.<p>Even Carly Fiorina, the witch CEO that burned down HP, said that the Chinese cannot innovate. [1]<p>And here you have a prime example of a Chinese company that found a good usage of the recommendation AI technique, and applied it in an innovative way, and created a whole new platform, that works better than YouTube or Twitter or Facebook.<p>And what happens? The white Americans steal it. Oh I’m sorry, their president declared it into law, like a dictator does, and force Tiktok to sell itself at firesale prices.<p>And what of all the supporters here on HN that have claimed that the Chinese can’t innovate. Well, their obvious answer is that Tiktok is obviously a spyware app for the Chinese government.<p>The American way to steal, is to force a firesale by edict, and get the patents for it. Then claim that it is their own.<p>The United States also did this to Toshiba and other Japanese semiconductor companies in the 1980s. Same dirty trick, different generation.<p>LOL. The hypocrisy of Americans.<p>But of course, you are free to prove me wrong.<p>Also, I expect to get flagged and downvoted for this comment.<p>[1] <a href="https:&#x2F;&#x2F;time.com&#x2F;3897081&#x2F;carly-fiorina-china-innovation&#x2F;" rel="nofollow">https:&#x2F;&#x2F;time.com&#x2F;3897081&#x2F;carly-fiorina-china-innovation&#x2F;</a>

so... wheres the code?

don&#x27;t see how tiktok is so game changing? How long can one look at random dance videos? What demographic would maintain sustained interest over years?<p>flavor of the month&#x2F;year?

Maybe I am just strange, but I find no appeal in TikTok, and would not like something that just shows me more of the same things. But I can see how it might appear to people who just want to be stimulated with stuff.

Does anyone actually trust any of this nonsense?<p>At this scale, TikTok, Google, YouTube, Twitter, Facebook.. it should legally be required to either:<p>- Open Source It<p>- Provide clear option to opt-out<p>- Provide the params to be 100% configurable<p>Wake up people!

Here&#x27;s the secret sauce: &quot;Using machine learning, the algorithm serves videos to users based on their proximity to other clusters of users and content that they like.&quot; It&#x27;s impressive how they arranged to be transparent for everything else, but kept this important part opaque.<p>Rumor is that the actual secret sauce here is human curation; people hand-select videos with high appeal and label them &quot;viral&quot;, &quot;popular&quot;, etc. in order to astroturf eyeballs and clicks. I suppose that admitting this directly would contrast sharply with the Chinese-harmonious-technocracy veneer that they work to project.