I've dealt with some spammers to various degrees. I think one of the most effective ways of dealing with spammers is to - "shadowban" them. Allow them to use your service, but don't indicate to them that you've identified them as malicious. For instance, when dealing with chat spammers - allow them to chat, but do not show their chats to other users. Another level would be to allow them to chat, but only show their chat to other shadowbanned users. For the author's use case, perhaps something like - if the ip address that created the link shortener accesses the link, they get the real redirect, and if a different ip address accesses it, they get the scam warning page. If the malicious actor doesn't know they've been marked as malicious, they do not know they need to change their behavior.<p>The second most effective thing is making the malicious actor use some sort of resource. Such as a payment (the author uses), or a time commitment (eg new accounts can only create 1 link a day), or some other source of friction. The idea is that for legitimate users the friction is acceptably low, but for consistent spammers the cost becomes too high.<p>The 3rd thing I've found effective is that lots of spam comes from robots - or perhaps robots farming tasks to humans. If you can determine how the traffic is coming in and then filter that traffic effectively without indicating failure, robots can happily spam away and you can happily filter away.
<p><pre><code> A big problem that came up at the domain level was what I'd call
a _trustworthy domain with untrustworthy subdomains_, specifically
where those subdomains represent user-generated content.
</code></pre>
The Public Suffix List (PSL) [1] to the rescue! It can help with this kind of disambiguation.<p>Paraphrasing, it's a list of domains where subdomains should be treated as separate sites (e.g. for cookie purposes). So `blogger.com` on the list means `*.blogger.com` are separate "sites".<p>[1] <a href="https://en.wikipedia.org/wiki/Public_Suffix_List" rel="nofollow">https://en.wikipedia.org/wiki/Public_Suffix_List</a>
What's the benefit of a link shortener, these days?<p>It made sense back before Twitter had one of their own. And I know that some people use it to get link analytics. I've also occasionally seen it used for printed materials, to get pretty URLs that are easy to hand-type.<p>People also use it for malicious purposes, such as hiding malware, or disguising referral links, or otherwise trying to obfuscate where a link is going. (Note: I'm not calling referral links malicious, I'm calling <i>disguised</i> referral links malicious.)<p>Other than printed materials (which need pretty URLs and thus often need a dedicated first-party URL shortener) and analytics, what are people using third-party URL shorteners for today?
No disrespect to the folks at y_gy who are clearly doing their best. But link shorteners, even when used by good faith actors, are problematic because they hide the destination of the link, and of course that's an invitation for bad faith actors to exploit, so the battle will be endless. Shorteners got popular on Twitter back in the days when all the characters in the URL counted against a very short limit. But there's less need to use them these days, and I am very reluctant to click on shortened links and don't think that this is unusual.
PSA: <a href="https://en.wikipedia.org/wiki/URL_shortening#Disadvantages" rel="nofollow">https://en.wikipedia.org/wiki/URL_shortening#Disadvantages</a><p>(Also note the difference between the length of the "Advantages" and "Disadvantages" sections)
Semi related. When I worked at Visa, I developed some ideas around making QR codes slightly more resilient to malicious hijacking when used in the context of a payments or commerce usecase. The idea was for the scanning app to look not just for a QR but also look for adjacent payment acceptance marks (e.g. branded Visa, MC, PayPal, or a merchant's brandmark etc.) and then dynamically only resolve URLs to registered domains associate with those marks. The idea was that QR codes not human readable, and URLs are a lot to ask the average person to reliable parse. So instead, have the scanner also see and understand the same contextual cues that the human can see and understand. And for the human, give them the confidence to scan QRs that will take them to a domain they would expect, and not to a Rick Astley video or worse.
I can really relate to this article! I created T.LY URL Shortener in 2018, and I've encountered all these issues and more! I found out the hard way when my hosting company shut down my servers for malicious content about a week into launching the site. Malicious actors will go to all sorts of lengths to achieve their goals.<p>Be careful relying on Stripe to prevent these users. Next they will start using stolen credit cards to create accounts then you will face disputes. If you get too many, Stripe will prevent you from processing payments.<p>About a year ago, I launched a service called Link Shield. It's an API that returns risk scores (0-100) on URLs. It uses AI and other services to score if a URL is malicious. Check it out and let me know if you would be interested in trying it linkshieldapi.com/
What worries me the most about things like these is that it makes it seem like it's impossible to make "free for all" products like these anymore if you're not an established player already. You will get blacklisted and you will receive emails from your host telling you to shut it down...<p>Established players like bitly and tinyurl didn't have all the resources to deal with the problem when they started out either, and they arguably still don't, yet they get favored by the antivirus vendors and "safe"search blacklists, since they're well-known services. It doesn't seem fair.<p>Is this really the way it should be? I wonder if they could've explained the situation to the antivirus vendors: The site itself doesn't host malware and doesn't allow the discovery of said malware through its service. It requires a user to receive an exact URL, just like they could've received any other link, and the blocklists should operate on what's hidden <i>behind</i> it instead of the redirect in front. Maybe y.gy could've been hooked into the safesearch API to automatically nuke any URLs blacklisted already by them, or another antivirus vendor.
While "honeypot" was mentioned in the title, there seems to be no useful outcome from caught bad actors, like reporting malicious websites, so browsers could block them.
Very cool read!<p>For the malicious links, did you have a chance to track whether the malware actors verify that their links do not work, e.g. by setting a cookie when they make a link and checking it later ?<p>I wonder if making these malicious links silently work <i>only</i> for the people that submitted them (and to say “no such link” for everyone else) ought to create a degree of confusion and slow them down to some extent at least…
Kudos for the great site and ethos. You still seem pretty buoyed by the experience. Ultimately, I found the article pretty depressing. Your initial free offering with a good UX and relatively little ongoing maintenance was destroyed by an army of criminals. It ended up wiping out weeks of your time developing increasingly complex cat and mouse techniques. Ultimately resulting in you abandoning most of the free plan altogether.<p>Kinda sad that this is what the online world has become.<p>And we just put up with it.<p>Imagine if walking down the road each day was like this – people lining up ready to swindle you or manhandle you in order to steal your things. There would be outrage. But online we have just sort of reached a weird state of acceptance I guess.
I generally always run any shortened link through a link checker before opening. So they are an inconvenience to me.<p>The time it took you to write all this evidences the problem with hosting the service publicly.<p>Yesterday I ran into problem with sharing a link to a simplex.chat group which was so long my website builder translated it incorrectly. I looked at link shorteners publicly available and now understand from your writeup why they are somewhat limited now. I found it easier to just spin up my own link shortener on my webserver using Shuri. It took less than a minute for me install. I won't publicize its availability now that I have read this.
My first thought after reading the title: I would never create a link shortener service, too complex and too much responsibility -- can it handle the traffic? what analytics can I provide? should it be a paid service (or rather, can it survive without being a paid service)? how to fight off scammers? what if some day the site goes down or permanently stops running, does that mean all those links are now useless?<p>My thoughts after reading the article: I was so right.
Getting a chargeback in Stripe is costly. As soon as a dispute is started there's a fixed $25 that won't be refunded even if you win the dispute.<p>So for a service at $4 a month which is likely to get a lot of fraudulent payments I wonder if it's really viable.<p>One thing he should do is immediately cancel accounts and refund subscriptions when there's an early fraud warning. They are usually accurate and help avoiding those fees.
Is there a name for this phenomenon? It's sort of like the dark forest, but not exactly. As soon as a free service becomes discovered, it is immediately swamped by scammers and spammers.<p>Many many years ago I ran a small forum for a small webcomic, and one day it was just full of low effort scams and spam. For an audience of, I dunno, a dozen people? I just shut the whole thing down because it wasn't worth our time to do anything about it.<p>We just can't have nice things, and if you run across something that is actually nice, make sure to thank whoever runs it for all their behind the scenes effort to deal with the scumbags that clog everything, and I mean everything, up with s(p|c)am.
A couple years ago a client asked me for their own URL shortener service. I found YOURLS (<a href="https://github.com/YOURLS/YOURLS">https://github.com/YOURLS/YOURLS</a>) and reluctantly installed it on a cheap, shared, hosting account.<p>Thankfully after a couple years, I convinced them (it took several tries) to use a 3rd party hosted provider.<p>Bullet dodged.
Amusingly, I thought this website was broken in a myriad of weird ways - I kept getting incomplete response errors and bad SSL errors.<p>As it turns out, my ISP was simply doing a rubbish job at blocking the site. After a few 10s of tries it eventually managed to redirect me to their warning page and prompted me to turn off settings in my account config. Thanks Virgin Media.
This is a great writeup. If you are just looking to deter scammers I bet $1 would have the same affect. I don't think scammers are worried about the price as much as having to give any amount of information to you. I could be wrong though as I am not a scammer!
I made a link shortener in 2010 and it was such a terrible experience. Constant notices from my hosting company about child porn links, repeated ominous emails from the FBI and their counterparts in other countries, having my server temporarily shut down repeatedly. I abandoned it after 6 months because the amount of time it took to continually adapt countermeasures to all the scummy abusers was too overwhelming. In so doing, I'm sure I contributed to all the link rot out there.
Tangentially, it's kinda funny how people really don't realize how much websites/companies/social system implement user-unfriendly behavior <i>because of scammers</i> or other bad actors. (Admittedly, it's something that I also did not understand when I was younger and more naive. Hell, I had to explain this to my 70y-old parent just a few weeks ago!)<p>The price of success is you then need to deal with moderation in some form. (and on that note: "it is easier to automate bad behavior than it is to police it")<p>Right now, "enshittification" is (rightly) on many people's minds, but before that the reason any company makes a process difficult is because some assholes ruined it for the rest of us.
this is the second time I'm seeing someone point out Replit being used for obvious phishing and I'm pretty sure I've even seen it myself before
See also: <a href="https://danluu.com/diseconomies-scale/" rel="nofollow">https://danluu.com/diseconomies-scale/</a>