Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

I run a MITM proxy for adblocking&#x2F;general filtering and within the past little while I&#x27;ve noticed CloudFlare and other &quot;bot protection&quot; tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that.<p>However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful with sharing stuff like this. Being forced to run JS is bad enough; profiling users based on other traits, and essentially determining if they are using &quot;approved&quot; software, is a dystopia we should fight strongly against. Stallman&#x27;s Right To Read comes to mind as a very relevant warning story.

&gt;impersonate Firefox 95<p>you should really be impersonating an ESR version (eg. 91). Versions from the release channel is updated every month or so, and everyone has autoupdate enabled. Therefore unless you keep it up to date, your fingerprint is going to stick out like a sore thumb in a few months. On the other hand, ESR sticks to one version and shouldn&#x27;t change significantly during its one year lifetime. It&#x27;s still going to stick out to some extent (most people don&#x27;t use ESR), but at least you have some enterprises who use ESR to blend into.

Cool, can&#x27;t wait for anti-bot protection to start rejecting me because I use firefox.

Given the relative market shares it might make more sense to impersonate Chrome.

&quot;Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block it.&quot;<p>As a user of non-browser clients (not curl though) I have not run into this in the wild.^1<p>Anyone have an example of a site that blocks non-browser clients based on TLS fingerprint.<p>1. As far as I know. The only site I know of today that is blocking non-browser clients appears to be www.startpage.com. Perhaps this is the heuristic they are using. More likely it is something simpler I have not figured out yet.

This might also be an interesting read for those curious about TLS fingerprinting: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29472624" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29472624</a>

Do you plan on getting this merged back into curl with an option to enable it? I can see that being useful for some people.

Currently, I cannot think about anything else other than &quot;noscript&#x2F;basic (x)html&quot; &#x2F;IRC to get us out of this, at least for sites where such protocols are &quot;good enough&quot; to provide their services to users over internet. But how? Enlighten the &quot;javascript web&quot; brain-washed devs to make them realize how toxic what they do is? regulations (at least for critical sites)? And how to deal with the other sites: those which devs are scammers and perfectly aware of how toxic they are and keep doing it.<p>In my own country, for critical sites, I will probably have go to court since &#x27;noscript&#x2F;basic (x)html&quot; interop was broken in the last few years.

Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare&#x27;s enterprise options was easily blocking all of the main http libraries due to the identifiable TLS handshake.

Very cool! Thanks for sharing - it’s always nice to learn about fingerprinting tricks and workarounds, from both a privacy and a “don’t unintentionally look like a bot” perspective.<p>What inspired the project?

This is cool, but is it really needed that often?<p>There are some industries (virtually all of Wall Street, for example, and certain parts of government) where the company needs to surveil 100% of what their employees do on the web from inside the office. These companies have been running MITM proxies for decades.<p>Wouldn&#x27;t any website that rejects a non-browsery TLS client be blocking out these people as well?

Very cool. I would have used Puppeteer&#x2F;Playwright in a similar scenario, but thanks for sharing the bot detection trick they employ.

Handy. Is the TCP handshake, or other details about socket behavior, ever get used for assessing the remote process, and in turn libraries written to mimic known patterns?

Great work mate, one of my team-mates showed me this library and we might use it in near future.