Warning: long somewhat related story that is basically humblebragging, but the summary is that bypassing Twitter ratelimits is not very hard.<p>I didn't feel like playing around with Twitter's annoying certificate pinning so I just uploaded the Twitter APK to Corellium, turned on what they call the "network monitor", opened the Twitter app since it lets you use Twitter without signing in. I clicked around, searched and viewed tweets. Then I looked at the requests in the log and saw it has a similar guest token process to the website but with a few differences. Anyways, if you recreate these requests, with one IP address you can generate a few OAuth tokens with no expiry per day. These tokens are for unauthenticated users so obviously they have no write privileges but that's not what was needed here. So if you have a proxy provider with a large pool of IPs where you can buy like 1GB of bandwidth you can use a very small percent of your bandwidth allowance and get thousands of tokens/secrets easily, all with their own separate rate limits. It doesn't even matter what IP you end up using the tokens on. Then I followed <a href="https://docs.google.com/document/d/1xVrPoNutyqTdQ04DXBEZW4ZW4A5RAQW2he7qIpTmG-M/edit" rel="nofollow noreferrer">https://docs.google.com/document/d/1xVrPoNutyqTdQ04DXBEZW4ZW...</a> and the fact that /statuses/lookup.json still allows you to return 100 (!) tweets at once to reconstruct something close to what the 50% Twitter firehose would look like. And Twitter doesn't even block datacenter IP addresses! Was going to display the data at <a href="https://firehose.lol" rel="nofollow noreferrer">https://firehose.lol</a> but the fact that it required a few hundred requests a second made me feel bad so I didn't end up running the program for more than a few minutes at a time and shut it down.<p>Looking at (a fraction of) the Firehose for a few minutes was interesting, originally I accidentally forgot to not display tweets labelled possibly_sensitive so I saw some pretty salacious material for a few seconds. Lots of Chinese gambling ads even though Twitter is blocked there, dubious investment promoters, accounts with usernames like FirstnameLastname3781264872 who would tweet three random words at each other every couple of seconds, and a handful of funny tweets.
This is the most impressive project in Nim I’ve seen yet. Rewriting any major front-end, complete with working authentication and handling idiosyncrasies of the private API is a herculean task. For context, Twitter would have a team of two dozen or more supporting what this does, effectively. Kudos to the author for accomplishing this feat!
Hey there, just wanted to thank you because you also fixed my Twitter Spaces downloader app[0]! After the API changes the default bearer token I was using (same as yours) stopped working, but after changing the same way you all's back to normal :D<p>0: <a href="https://github.com/Chiplis/moonbird">https://github.com/Chiplis/moonbird</a>
This is absolutely amazing, as you still can’t view profiles or replies on Twitter without logging in.<p>I’ve been a nitter user for 4 years now and will be as long as it works.
Is this really permanent? I'd love to know more about this bearer token.<p>Because in the other github issue thread it seemed like every time they found a way around Twitter's safeguards, it was shutdown.<p>It seems like they've literally hard coded a token into the source code. Meaning thousands of nitter-instances, thousands of users, around the world, will use the same token.<p>And potentially so will the AI companies.<p>So I just don't see how this can work.
It means anonymous access has been restored. Nitter does not use twitter API which is a goner anyway (but was restored for a little while).<p>Twitter should provide a noscript/basic (x)html interop www portal.
This is like the analog loophole. If data is transferred from Server to Client, it's always possible to scrape.<p>Though Apple is trying to make this harder: <a href="https://developer.apple.com/documentation/devicecheck/preparing_to_use_the_app_attest_service" rel="nofollow noreferrer">https://developer.apple.com/documentation/devicecheck/prepar...</a>
Awesome, now my light personal-use scraper works again. Didn't even take a restart!<p>Hope Twitter soon lets go of the temporary login restriction too. Given that this isn't completely blocked without a login, I'd expect that to be not far from now on. From what I've collected, I hope Twitter'd start selling dumps of their public data for a bit of a win/win with AI companies and Twitter itself.
This does not seem legal. Stealing an access token to bypass access controls is illegal and I suspect these people didn't get permission to just scrape anything they want.