If you are working at a < 30 people engineering team company what is the alerting stack that you guys are using ?<p>We tend to miss a lit of critical alerts that come to simply because the alerting is not set up properly.
I'm surprised how often I speak to technical teams that do not utilize PagerDuty (or an equivalent alternative). As PagerDuty integrates with nearly any external system, it separates the collection of telemetry from the incident response lifecycle, i.e. what is wrong? who should be or is looking into this? what did we learn from this? how often is this happening?<p>Personally, I find notifications in Slack to be an anti-pattern: a lot of teams expect someone to just "pick up" the incident based on their availability or expertise and _maybe_ the resolution is documented. Assigning direct responsibility by component and on-call schedule appending the RCA reduces the time-to-resolution and overall toil of the process.
Custom built monitoring on top of CloudWatch logs: we subscribe to the log groups and parse the logs.<p>Errors are reported in dedicated slack channels<p>The “MVP” was built in 1 week after we were faced with an outrageous bill from an observability vendor and decided to give a shot at implementing it ourselves.<p>In total I’d say that we invested 2 additional weeks of man-hour to get to where we are today.<p>It has worked extremely well for us and has needed little maintenance (granted we pay AWS to not have to do that maintenance)
StatusCake has a feature to call me. It's a horrible artificial voice "your website $name is down" but I'm fine with anybody shouting at me at 3am. The phone number is from the United States and I don't need to add it to my phone book because that's the only US phone calling me. (For people inside the US you might think it's another robocall)
i keep it simple with an uptime monitoring service that monitors all the elements of my stack and run tests every minute:<p>- regular http monitoring for websites<p>- run test queries on my sql & mongo databases<p>- check that rabbitmq queues are not overflowing<p>- check that docker container are up<p>If something goes wrong, email & telegram alerts.<p>fwiw i'm using <a href="https://uptimefunk.com" rel="nofollow noreferrer">https://uptimefunk.com</a>
Uptime monitoring + cron job monitoring via OnlineOrNot (dogfooding my own product), with alerts going to PagerDuty (set up to email -> SMS -> call me if I don't acknowledge), and a "public" alert in a Slack channel.