I’ve been running a few distributed web services, but except for a few rudimentary nginx access logs, I have no idea how they’re functioning.<p>I have the following goals and questions regarding implementing a logging & monitoring system to get better insights of them:<p>- What are the best practices to instrument source code to collect general logs and exceptions?
- How to determine if the services and databases are performing efficiently? More specifically, what I can do to discover if they are doing unnecessary work or there are any hotspots?
- Are the servers being run on overloaded? If so, what are overloading them?
- How do I know if some one is trying to break into the servers?
- How can I be alerted whenever a bad thing previously mentioned happens?<p>And then there is the business logic side of things. like how many users are online, how many transactions are currently being processed, etc. I don’t suppose directly querying the production database is a good idea.<p>My own research online surfaced a great deals of tools like prometheus, ELK stack, fluentd, Nagios, bugsnag, New Relic, Datadog, etc, which overwhelmed me, and I reckon without a good understanding of logging and monitoring in general, I’m likely to pick the wrong tools.<p>This feels like a really big topic. Any books/resources that have a comprehensive introduction?