There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook.<p>* Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be able to deal with that.<p>* Timeouts (slowloris): the webhook target could be sending back one byte at a time, with 1 second pauses inbetween. If you are using, say, the "requests" python library for making HTTP requests, the "timeout" parameter will not help here<p>* Private IPs and reserved IPs: you probably don't want users defining webhooks to <a href="http://127.0.0.1:<some-port>" rel="nofollow">http://127.0.0.1:<some-port></a> and probing your internal network. Remember about private IPv6 ranges too<p>* Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP. It is not enough to just validate webhook URLs when users set them up.<p>* HTTP redirects to private IPs. If your HTTP client library follows HTTP redirects, the attacker can set up a webhook endpoint that redirects to a private IP. Again, it is not enough to validate the user-supplied URL.<p>* Excessive HTTP redirects. The attacker can set up a redirect loop - make sure this does not circumvent your timeout setting.<p>My current solution for all of the above is to use libcurl via pycurl. I wrote a wrapper that mimics requests API: <a href="https://github.com/healthchecks/healthchecks/blob/master/hc/lib/curl.py" rel="nofollow">https://github.com/healthchecks/healthchecks/blob/master/hc/...</a> (may contain bugs, use at your own risk :-)
Hello all! I'm one of the creators of webhooks.fyi over at <a href="https://ngrok.com/" rel="nofollow">https://ngrok.com/</a><p>Happy to answer any questions!<p>First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole.<p>Next, yes it's ridiculously hard to build webhooks correctly. There are so many shortcuts that feel "okay" and you don't really think about until you realize that it didn't quite work as you had planned and now you have a gap. We're hoping the site can help people move more of that thinking earlier and make better decisions.<p>Finally, we're missing a bunch of webhooks! While we looked at 100+ in our own research, we only had time to add ~50 to this initial pass. If you'd like to add your favorites, pull requests welcome: <a href="https://github.com/ngrok/webhooks.fyi" rel="nofollow">https://github.com/ngrok/webhooks.fyi</a><p>And yes, seriously, you can use ngrok to verify webhooks pretty easily:<p><i>ngrok http 80 --verify-webhook=slack --verify-webhook-secret=[secret]</i>
As this page makes very clear, it's actually pretty hard to make a robust webhooks implementation!<p>What alternatives are there? I've looked at:
* Publishing AWS EventBridge events to other accounts.
* /events instead of webhooks <a href="https://blog.sequin.io/events-not-webhooks/" rel="nofollow">https://blog.sequin.io/events-not-webhooks/</a>
* ???
This is a fantastic resource! Thank you to the folks at ngrok for putting this together! As this site makes clear: webhooks are harder than they appear. Even just consuming webhooks it's easy to get bogged down dealing with issues around rate limits or recovering from bugs that cause missed events! Missed events being particularly painful with platforms that don't offer replay / retry.<p>Disclaimer: I work at <a href="https://hookdeck.com/" rel="nofollow">https://hookdeck.com/</a> & I shamelessly plug our tool for giving you an awesome developer experience working with webhooks and helping deal with some of the concerns brought up on webhooks.fyi.<p>And if you are interested in webhooks at large a couple more resources worth checking out is the awesome-webhooks[1] list and the r/webhooks[2] subreddit (I just got ownership of the sub and started dusting it off this week after being neglected for the past few years! Please, come join!)<p>[1] <a href="https://github.com/realadeel/awesome-webhooks" rel="nofollow">https://github.com/realadeel/awesome-webhooks</a><p>[2] <a href="https://www.reddit.com/r/webhooks/" rel="nofollow">https://www.reddit.com/r/webhooks/</a>
If only there was a single standard for webhook subscription verification. Cloud services invent their own authentication protocols. Webhook verification in Dropbox is different from that in Trello. The lack of a single standard makes it hard to design a universal incoming webhook service (we faced this problem).
Great resource! I’ve encountered most of these problems at some point.<p>It would be great to see advice about implementations. Things like when to process in a queue and when not to, tying idempotency tokens to database transactions, etc. These are subtle issues, but when done well can help make very robust systems.