TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Show HN: I built an open-source tool to make on-call suck less

319 点作者 aray0710 个月前
Hey HN,<p>I am building an open source platform to make on-call better and less stressful for engineers. We are building a tool that can silence alerts and help with debugging and root cause analysis. We also want to automate tedious parts of being on-call (running runbooks manually, answering questions on Slack, dealing with Pagerduty). Here is a quick video of how it works: <a href="https:&#x2F;&#x2F;youtu.be&#x2F;m_K9Dq1kZDw" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;m_K9Dq1kZDw</a><p>I hated being on-call for a couple of reasons:<p>* Alert volume: The number of alerts kept increasing over time. It was hard to maintain existing alerts. This would lead to a lot of noisy and unactionable alerts. I have lost count of the number of times I got woken up by alert that auto-resolved 5 minutes later.<p>* Debugging: Debugging an alert or a customer support ticket would need me to gain context on a service that I might not have worked on before. These companies used many observability tools that would make debugging challenging. There are always a time pressure to resolve issues quickly.<p>There were some more tangential issues that used to take up a lot of on-call time<p>* Support: Answering questions from other teams. A lot of times these questions were repetitive and have been answered before.<p>* Dealing with PagerDuty: These tools are hard to use. e.g. It was hard to schedule an override in PD or do holiday schedules.<p>I am building an on-call tool that is Slack-native since that has become the de-facto tool for on-call engineers.<p>We heard from a lot of engineers that maintaining good alert hygiene is a challenge.<p>To start off, Opslane integrates with Datadog and can classify alerts as actionable or noisy.<p>We analyze your alert history across various signals:<p>1. Alert frequency<p>2. How quickly the alerts have resolved in the past<p>3. Alert priority<p>4. Alert response history<p>Our classification is conservative and it can be tuned as teams get more confidence in the predictions. We want to make sure that you aren&#x27;t accidentally missing a critical alert.<p>Additionally, we generate a weekly report based on all your alerts to give you a picture of your overall alert hygiene.<p>What’s next?<p>1. Building more integrations (Prometheus, Splunk, Sentry, PagerDuty) to continue making on-call quality of life better<p>2. Help make debugging and root cause analysis easier.<p>3. Runbook automation<p>We’re still pretty early in development and we want to make on-call quality of life better. Any feedback would be much appreciated!

36 条评论

dclowd990110 个月前
&gt; It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts.<p><i>grimace face</i><p>I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and we can certainly use more competition in this space, but this aspect of it is basically enabling bad cultural practices and I wouldn’t highlight it as a main selling point.
评论 #41098153 未加载
评论 #41098517 未加载
评论 #41097360 未加载
评论 #41101454 未加载
评论 #41097341 未加载
评论 #41098675 未加载
评论 #41105187 未加载
评论 #41097971 未加载
评论 #41101876 未加载
评论 #41097661 未加载
评论 #41095815 未加载
评论 #41097255 未加载
评论 #41098664 未加载
jedberg10 个月前
People do not understand the value of classifying alerts as useful <i>after the fact</i>.<p>At Netflix we built a feature into our alert systems that added a simple button at the top of every alert that said, &quot;Was this alert useful?&quot;. Then we would send the alert owners reports about what percent of people found their alert useful.<p>It really let us narrow in on which alerts were most useful so that others could subscribe the them, and which were noise, so they could be tuned or shut off.<p>That one button alone made a huge difference in people&#x27;s happiness with being on call.
评论 #41102119 未加载
评论 #41102954 未加载
评论 #41102851 未加载
aflag10 个月前
It feels to me that using LLM to classify alerts as noisy is just adding risk instead of fixing the root cause of the problem. If an alert is known to be noisy and have appeared on slack before (which is how the LLM would figure out it&#x27;s a noisy alert), then just remove the alert? Otherwise, how will the LLM know it&#x27;s noise? Either it will correctly annoy you or hallucinate a reason it figures that alert is just noise.
评论 #41100913 未加载
评论 #41101338 未加载
评论 #41099719 未加载
ravedave510 个月前
The goal for oncall should be to NEVER get called. If someone gets called when they are oncall their #1 task the next day is to make sure that call never happens again. That means either fixing a false alarm or tracking down the root cause of the call. Eventually you get to a state where being called is by far the exception instead of the norm.
评论 #41103032 未加载
评论 #41101476 未加载
评论 #41101481 未加载
Jolter10 个月前
Telecoms solved this problem fifteen years ago when they started automating Fault Management (google it).<p>Granted, neural networks were not generally applicable to this problem at the time, but this whole idea seems like the same problem being solved again.<p>Telecoms and IT used to supervise their networks using Alarms, in either a Network Management System (NMS) or something more ad-hoc like Nagios. There, you got structured alarms over a network, like SNMP traps, that got stored as records in a database. It’s fairly easy to program filters using simple counting or more complex heuristics against a database.<p>Now, for some reason, alerting has shifted to Slack. Naturally since the data is now unstructured text, the solution involves an LLM! You build complexity into the filtering solution because you have an alarm infrastructure that’s too simple.
评论 #41101474 未加载
评论 #41101894 未加载
mads_quist10 个月前
Founder of All Quiet here: <a href="https:&#x2F;&#x2F;allquiet.app" rel="nofollow">https:&#x2F;&#x2F;allquiet.app</a>.<p>We&#x27;re building a tool in the same space but opted out of using LLMs. We&#x27;ve received a lot of positive feedback from our users who explicitly didn&#x27;t want critical alerts to be dependent on a possibly opaque LLM. While I understand that some teams might choose to go this route, I agree with some commentators here that AI can help with symptoms but doesn&#x27;t address the root cause, which is often poor observability and processes.
RadiozRadioz10 个月前
&gt; Slack-native since that has become the de-facto tool for on-call engineers.<p>In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used.<p>Other comment threads are already discussing the broader issues with using IM for this job, so I won&#x27;t go into it here.<p>Regardless, well done for making something.
评论 #41097598 未加载
评论 #41095688 未加载
评论 #41095639 未加载
throw15675422810 个月前
I don&#x27;t want to be relying on another flaky LLM for anything mission critical like this.<p>Just fix the original problem, don&#x27;t layer an LLM into it.
评论 #41099739 未加载
评论 #41100099 未加载
Terretta10 个月前
Note that according to StackOverflows dev survey, more devs use Teams than Slack, over 50% were in Teams. (The stat was called popularity but really should have been prevalence, since a related stat showed devs hated Teams even more than they hated Slack.) Teams has APIs too, and with Microsoft Graph working you can do a lot more than just Teams for them.<p>More importantly, and not mentioned by StackOverflow, those devs are among the 85% of businesses using M365, meaning they have &quot;Sign in with Microsoft&quot; and are on teams that will pay. The rest have Google and&#x2F;or Github.<p>This means despite being a high value hacking target (accounts and passwords of people who operate infrastructure, like the person owned from Snowflake last quarter) you don&#x27;t have to store passwords therefore can&#x27;t end up on Have I Been Pwned.
voidUpdate10 个月前
Filtering whether a notification is important or not through an LLM, when getting it wrong could cause big issues, is mildly concerning to me...
nprateem10 个月前
Almost all alerting issues can be fixed by putting managers on call too (who then have to attend the fix too).<p>It suddenly becomes a much higher priority to get alerting in order.
asdf696910 个月前
I don’t really understand the use case. If there’s a way to programmatically tell that it’s a false alarm then there must also be a way to not create the alert in the first place<p>I’ve never seen an issue that’s conclusively a false alarm without investigating at all. Just delete the alarm? An LLM will never find something like another team is accidentally stress testing my service but it does happen<p>Another perfect example is when the queen died and it looked like an outage for UK users. Can your LLM read the news? ChatGPT doesn’t even know if she’s alive<p>I expect you will need AGI before large companies will trust your product.
makmanalp10 个月前
Underrated oncall problem that needs solving is scheduling IMHO:<p>- We have a weekday (2 shifts) &#x2F; weekend (1 slightly longer shift including friday morning to allow people to take long weekends) oncall rotation as well as a group-combined oncall schedule which gets finnicky.<p>- When people join or leave the rotation, making sure nothing shifts before a certain date or swapping one person with another without changing the rest and other things are a massive pain in the butt<p>- Combine this with a company holiday list - usually there&#x27;s different policies and expectations during those. - Allow custom shift change times for people in different timezones.<p>- We have &quot;oncall training&quot; &#x2F; shadowing for newbies, automate the process of substituting them in gradually, first with a shared daytime rotation and then on their own etc.<p>- Make oncall trades (if you can&#x27;t make your shift simpler)<p>Gripes with PD:<p>- Pagerduty keeps insisting I&#x27;m &quot;always on call&quot; because I&#x27;m on level N of a fallback pager chain which makes their &quot;when oncall next&quot; box useless - just let me pick.<p>- Similarly, pagerduty&#x27;s google calendar export will just jam in every service you&#x27;re remotely related to and won&#x27;t let you pick when exporting, even though it will in their UI. So I can&#x27;t just have my oncall schedule in google calendar without polluting it to all hell.
评论 #41102884 未加载
lmeyerov10 个月前
Big fan of this direction. The architecture resonates! The base lining is interesting, I&#x27;m curious how you think about that, esp for bootstrapping initially + ongoing.<p>We are working on a variant being used more by investigative teams than IT ops - so think IR, fraud, misinfo, etc - which has similarities but also domain differences. If of interest to someone with an operational infosec background (hunt, IR, secops) , and esp US-based, the Louie.AI team is hiring an SE + principal here.
CableNinja10 个月前
I get your sentiment, but theres another side of this coin that everyone is forgetting, hilariously.<p><i>You can tune your monitoring!</i><p>Noisy alert that tends to be a false positive but not always? Tune alert message to only send if the issue continues for more than a minute, or if the check fails 3 times in a row. Theres hundreds of ways to tweak a monitor to match your environment.<p>Best of all? It takes 30 seconds at most. Find the trigger, adjust slightly, and after maybe 1-2 tries, youll be getting 1 false positive sometimes, and actual alerts when they happen, compared to 99% false alerts, all the time.<p>Oh and did you know any monitoring solution worth its salt can execute things automatically on alerts, and then can alert you if that thing fails?<p>Also, Slack is not a defacto anything. Its a chat tool in a world of chat tools
jpb010410 个月前
I love this space; stability &amp; response! After my last full-time gig, I was also frustrated with the available tooling and ONLY wanted an on-call scheduling tool with simple calendar integration. So I built: <a href="https:&#x2F;&#x2F;majorpager.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;majorpager.com&#x2F;</a> Not OSS, but very simple and hopefully pretty straightforward to use. I&#x27;m certainly wide open to feedback.
solatic10 个月前
In my current workplace (BigCo), we know exactly what&#x27;s wrong with our alert system. We get alerts that we can&#x27;t shut off, because they (legitimately) represent customer downtime, and whose root cause we either can&#x27;t identify (lack of observability infrastructure) or can&#x27;t fix (the fix is non-trivial and management won&#x27;t prioritize).<p>Running on-call well is a culture problem. You need management to prioritize observability (you can&#x27;t fix what you can&#x27;t show as being broken), then you need management to build a no-broken-windows culture (feature development stops if anything is broken).<p>Technical tools cannot fix culture problems!<p>edit: management not talking to engineers, or being aware of problems and deciding not to prioritize fixing them, are both culture problems. The way you fix culture problems, as someone who is not in management, is to either turn your brain off and accept that life is imperfect (i.e. fix yourself instead of the root cause), or to find a different job (i.e. if the culture problem is so bad that it&#x27;s leading to burnout). In any event, cultural problems <i>cannot</i> be solved with technical tools.
评论 #41095770 未加载
评论 #41096421 未加载
评论 #41095686 未加载
评论 #41095835 未加载
评论 #41095669 未加载
评论 #41095724 未加载
maximinus_thrax10 个月前
Nice work, I always appreciate the contribution to the OSS ecosystem.<p>That said, I like that you&#x27;re &#x27;saying out loud&#x27; with this. Slack and other similar comm tooling has always been advertised as a productivity booster due to their &#x27;async&#x27; nature. Nobody actually believes this anymore and coupling it with the oncall notifications really closes the lid on that thing.
评论 #41095711 未加载
topaztee10 个月前
co-founder of merlinn here: <a href="https:&#x2F;&#x2F;merlinn.co" rel="nofollow">https:&#x2F;&#x2F;merlinn.co</a> | <a href="https:&#x2F;&#x2F;github.com&#x2F;merlinn-co&#x2F;merlinn">https:&#x2F;&#x2F;github.com&#x2F;merlinn-co&#x2F;merlinn</a> We&#x27;re also building a tool in the same space with the option of choosing your own model (private llms) + we&#x27;re open source with a multitude of integrations.<p>good to see more options in this space! especially OS. I think de-noising is a good feature given alert fatigue is one of the repeating complaints of on-callers.
deepfriedbits10 个月前
Nice job and congratulations on building this! It looks like your copy is missing a word in the first paragraph:<p>&gt; Opslane is a tool that helps (make) the on-call experience less stressful.
评论 #41096278 未加载
Arch-TK10 个月前
We could stop normalising &quot;on-call&quot; instead.
评论 #41100903 未加载
snihalani10 个月前
can you build a cheaper datadog instead?
评论 #41100653 未加载
评论 #41101809 未加载
tryauuum10 个月前
every time I see notifications in Slack &#x2F; Telegram it makes me depressed. Text messengers were not designed for this. If you get the &quot;something is wrong&quot; alert it becomes part of history, it won&#x27;t re-alert you if it&#x27;s still present. And if you have more than one type of alert it will be lost in history<p>I guess alerts to messengers are OK as long it&#x27;s only a couple manually created ones, and there should be a graphical dashboard to learn the rest of problems
评论 #41096750 未加载
评论 #41095584 未加载
评论 #41096717 未加载
评论 #41095641 未加载
评论 #41095291 未加载
T1tt10 个月前
is this only on the frontpage because this is an HN company?
c0mbonat0r10 个月前
if this is open-source project how are you planning to make this a sustainable business? also why the choice of apache 2.0
T1tt10 个月前
how can you prove it works and doesnt hallucinate? do you have any actual users that have installed it and found it useful?
lars_francke10 个月前
Shameless question tangential related to the topic.<p>We are based in Europe and have the problem that some of us sometimes just forget we&#x27;re on call or are afraid that we&#x27;ll miss OpsGenie notifications.<p>We&#x27;re desparately looking for a hardware solution. I&#x27;d like something similar to the pagers of the past but at least here in Germany they don&#x27;t really seem to exist anymore. Ideally I&#x27;d have a Bluetooth dongle that alerts me on configurable notifications on my phone. Carrying this dongle for the week would be a physical reminder I&#x27;m on call.<p>Does anyone know anything?
评论 #41095841 未加载
评论 #41096926 未加载
评论 #41096436 未加载
7bit10 个月前
&gt; * Alert volume: The number of alerts kept increasing over time. It was hard to maintain existing alerts. This would lead to a lot of noisy and unactionable alerts. I have lost count of the number of times I got woken up by alert that auto-resolved 5 minutes later.<p>I don&#x27;t understand this. Either the issue is important and requires immediate human action -- or the issue can potentially resolve itself and should only ever send an alert if it doesn&#x27;t after a set grace period.<p>The way you&#x27;re trying to resolve this (with increasing alert volumes) is the worst approach to both of the above, and improves nothing.
protocolture10 个月前
I feel like this would be a great tool for people who have had a much better experience of On Call than I have had.<p>I once worked for a string of businesses that would just send <i>everything</i> to on call unless engineers threatened to quit. Promised automated late night customer sign ups? Haven&#x27;t actually invested in the website so that it can do that? Just make the on call engineer do it. Too lazy to hire off shore L1 technical support? Just send residential internet support calls to the the On Call engineer! Sell a service that doesn&#x27;t work in the rain? Just send the on call guy to site every time it rains so he can reconfirm yes, the service sucks. Basic usability questions that could have been resolved during business hours? Does your contract say 24&#x2F;7 support? Damn, guess thats going to On Call.<p>Shit even in contracting gigs where I have agreed to be &quot;On Call&quot; for severity 1 emergencies, small business owners will send you things like service turn ups or slow speed issues.
评论 #41101379 未加载
EGreg10 个月前
One of the “no-bullshit” positions I have arrived at over the years is that “real-time is a gimmick”.<p>You don’t need that Times Square ad, only 8-10 people will look up. If you just want the footage of your conspicuous consumotion, you can easily photoshop it for decades already.<p>Similarly, chat causes anxiety and lack of productivity. Threaded forums like HN are better. Having a system to prevent problems and the rare emergency is better than having everyone glued to their phones 24&#x2F;7. And frankly, threads keep information better localized AND give people a chance to THINK about the response and iterate before posting in a hurry. When producers of content take their time, this creates efficiencies for EVERY INTERACTION WITH that content later, and effects downstream. (eg my caps lock gaffe above, I wont go back and fix it, will jjst keesp typing 111!1!!!)<p>Anyway people, so now we come to today’s culture. Growing up I had people call and wish happy birthday. Then they posted it on FB. Then FB automated the wishes so you just press a button. Then people automated the thanks by pressing likes. And you can probably make a bot to automate that. What once was a thoughtful gesture has become commoditized with bots talking to bots.<p>Similar things occurred with resumes and job applications etc.<p>So I say, you want to know my feedback? Add an AI agent that replies back with basic assurances and questions to whoever “summoned you”, have the AI fill out a form, and send you that. The equivalent of front-line call center workers asking “Have you tried turning it on and off again” and “I understand it doesn’t work, but how can we replicate it.”<p>That repetitive stuff should he done by AI and build up an FAQ Knowledge Base for bozos and then only bother you if it came across a novel problem it hasn’t solved yet, like an emergency because, say, there’s a windows BSOD spreading and systems don’t boot up. Make the AI do triage and tell the differencd.
LunarFrost8810 个月前
Really cool!
racka10 个月前
Really cool!<p>Anyone know of a similar alert UI for data&#x2F;business alarms (eg installs dropping WoW, crashes spiking DoD, etc)?<p>Something that feeds of Snowflake&#x2F;BigQuery, but with a similar nice UI so that you can quickly see false positives and silence them.<p>The tools I’ve used so far (mostly in-house built) have all ended in a spammy slack channel that no one ever checks anymore.
评论 #41101875 未加载
Flop733110 个月前
Is this for missile defense systems or something? What&#x27;s possibly so important that you need to be woken up for it?
评论 #41099234 未加载
评论 #41097617 未加载
theodpHN10 个月前
What you&#x27;ve come up with looks helpful (and may have other applications as someone else noted), but you know what also makes on-call suck less? Getting paid for it, in $ and&#x2F;or generous comp time. :-)<p><a href="https:&#x2F;&#x2F;betterstack.com&#x2F;community&#x2F;guides&#x2F;incident-management&#x2F;on-call-pay&#x2F;" rel="nofollow">https:&#x2F;&#x2F;betterstack.com&#x2F;community&#x2F;guides&#x2F;incident-management...</a><p>Also helpful is having management that is responsive to bad on-call situations and recognizes when capable, full-time around-the-clock staffing is really needed. It seems too few well-paid tech VPs understand what a 7-Eleven management trainee does, i.e., you shouldn&#x27;t rely on 1st shift workers to handle all the problems that pop up on 2nd and 3rd shift!
评论 #41096622 未加载
throwaway98439310 个月前
Don&#x27;t send an alert at all unless it is actionable. Yes, I get it, you want alerts for everything. Do you have a runbook that can explain to a complete novice what is going on and how to fix the problem? No? Then don&#x27;t alert on it.<p>The only way to make on-call less stressful is to do the boring work of preparing for incidents, and the boring work of cleaning up after incidents. No magic software will do it for you.
sanj00110 个月前
Using LLMs to classify noisy alerts is a really clever approach to tackling alert fatigue! Are you fine tuning your own model to differentiate between actionable and noisy alerts?<p>I&#x27;m also working on an open source incident management platform called Incidental (<a href="https:&#x2F;&#x2F;github.com&#x2F;incidentalhq&#x2F;incidental">https:&#x2F;&#x2F;github.com&#x2F;incidentalhq&#x2F;incidental</a>), slightly orthogonal to what you&#x27;re doing, and it&#x27;s great to see others addressing these on-call challenges.<p>Our tech stacks are quite similar too - I&#x27;m also using Python 3, FastAPI!
评论 #41096349 未加载
评论 #41098567 未加载
评论 #41100672 未加载
评论 #41096426 未加载