I want to learn about software monitoring and observability but not able to find many good resources to learn about it. I want to learn it's concepts, what to do, what not to do, how to setup from an existing software, etc.<p>Looking for some good recommendations.
I'd recommend reading content by copyconstruct on their medium blog and also content by Honeycomb and it's CEO/CTO. Lots of first principles content that you can learn and apply to any system.<p>Once you're familiar with the basics, you can pickup some docs/videos around OpenTelemetry to read about how it is done in the real world. Pretty much all obs systems today have otel support.
I found the first few chapters of the google SRE book a good primer, in particular Part 2[0]. The end of each chapter has cited sources which allowed me to dive deeper.<p>0-<a href="https://sre.google/sre-book/part-II-principles/" rel="nofollow">https://sre.google/sre-book/part-II-principles/</a>
Not exactly a getting started guide, but if you want to learn about Prometheus and PromQL, I've recently made a tool to spin up an instance and send fake metrics data to it: <a href="https://github.com/pb82/prometheus-toolbox">https://github.com/pb82/prometheus-toolbox</a><p>This sets you up to play around and learn PromQL.