We switched a number of core processes to AWS Lambda and my team is now spending WAY more time debugging.<p>Our strategy has been to add more logging and metrics to better find root causes of issues - but that's getting unruly.<p>How are you debugging? What dev tools are you using understand what's happening within your distributed system? If you're logging...<p>- Where does your state go?<p>- Are you tracking startup latency?<p>- Are you tracking limits in memory, concurrency, execution duration?<p>Super interested in what others are doing for this, as well as other problems like testing, deployment, discovery.
A few pointers from our own experience:<p>- centralized logs (we tried an ELK stack and have moved to Datadog since they added logging). Using a correlation ID helps tracking the flow as it crosses Lambda / services boundaries<p>- using datadog also gives us metrics and dashboards for free, although there’s not always the data you’d want<p>- we started to experiment with X-Ray to track start-up time and how much time is spent where. I’d definitely advise you to try it if you’re tracking down performance issues. It’s a bit of a pain to get working though<p>- testing : as described in Yubl’s road to serverless (link in another comment), we have a switch to call the code locally or remotely through whichever service triggers the Lambda. This usually insures that the logic is sound before deploying and that remote bugs are mostly linked to integration or rights issues<p>- deployment : we rolled our own with Ansible and CloudFormation / SAM but if you fit in the Serverless use cases you should probably try that first<p>- discovery : we use SSM parameter store as a distributed key/value DB and a poor’s man discovery service: if we want to reach a given lambda or service we look up it’s name or arn in SSM PS.<p>I’m in the process of writing a post (or more likely a series) on our experience and will post to HN when ready<p>Edit: also, decoupling. If your Lambdas are calling each other directly, consider putting a queue or SNS topic in between. Makes it easier to test each unit independently, can manage timeout / retry issues on your behalf, and gives you a convenient observation point for inter-service traffic
I use SAM CLI so I can run my code locally in a realistic environment. So you can skip the build/zip/upload/deploy process.<p><a href="https://github.com/awslabs/aws-sam-cli" rel="nofollow">https://github.com/awslabs/aws-sam-cli</a>
Yubl's road to serverless has a section on testing:<p><a href="https://hackernoon.com/yubls-road-to-serverless-part-2-testing-and-ci-cd-72b2e583fe64" rel="nofollow">https://hackernoon.com/yubls-road-to-serverless-part-2-testi...</a>
Hi feel free to check this product (like you debug lambda in production easily and fast): <a href="https://techcrunch.com/2018/06/04/rookout-releases-serveless-debugging-tool-for-aws-lambda/" rel="nofollow">https://techcrunch.com/2018/06/04/rookout-releases-serveless...</a><p>full disclosure I work for Rookout - feel free to ask questions
We have a similar problem, and I'm really interested in the answer to this question... our solutions so far are Testing and monitoring (doesn't make it any easier)