TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Fast and flexible observability with canonical log lines

221 点作者 mglukhovsky将近 6 年前

12 条评论

ianstormtaylor将近 6 年前
Great article! I always love hearing Stripe talking about their internals.<p>I&#x27;ve been using this practice and I agree that it&#x27;s incredibly useful. I think because people tend to think in terms of &quot;logs&quot;, they end up overlooking the much more useful construct of &quot;canonical logs&quot;. Many fine-grained logs themselves are almost always less useful than the fewer fully-described canonical logs. Other observability tools often call these &quot;events&quot; instead of &quot;logs&quot; for that reason.<p>There&#x27;s a tool call Honeycomb [1] that gives you exactly what this article&#x27;s talking about in a really nicely designed package out of the box. And since it handles all of the ingestion and visualization, you don&#x27;t have to worry about setting up Kafka, or the performance of logplexes, or teaching everyone SQL, or how to get nice graphs. I was a little skeptical at first, but after using it for over a year now I&#x27;m completely converted.<p>If you record fully-described &quot;events&quot; for each request, and you use sub-spans for the smaller segments of requests, you also get a waterfall-style trace visualization. Which eliminates the last need for fine-grained logs completely.<p>If this article seems interesting to you, I&#x27;d highly, highly recommend Honeycomb. (Completely unaffiliated, I just think it&#x27;s a great product.)<p>[1]: <a href="https:&#x2F;&#x2F;www.honeycomb.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.honeycomb.io&#x2F;</a>
评论 #20569957 未加载
评论 #20572641 未加载
评论 #20570642 未加载
评论 #20571104 未加载
评论 #20570798 未加载
firethief将近 6 年前
It&#x27;s interesting that they&#x27;ve found denormalizing their log data so useful. I&#x27;m suprised to hear that that performs better for practical queries than a database with appropriate indexes, and that they&#x27;ve been able to build more ergonomic interfaces to query that than the standard relational approach a lot of people already have experience with. But I don&#x27;t know much about log management at scale, so I&#x27;m only mildly surprised.
评论 #20569306 未加载
评论 #20570728 未加载
评论 #20569324 未加载
manigandham将近 6 年前
Strange that they went with plain text when the industry is converging on (newline delimited) JSON logs for structured data. This also serves as the backbone of observability with metrics and tracing also being folded into and output as JSON.<p>Call them events and you can claim all the event-sourcing buzzwords too.
评论 #20570889 未加载
chrisweekly将近 6 年前
Related tangent: I can&#x27;t say enough good things about [lnav](<a href="https:&#x2F;&#x2F;lnav.org" rel="nofollow">https:&#x2F;&#x2F;lnav.org</a>). It&#x27;s like a mini-ETL powertool at your fingertips, w&#x2F; an embedded SQLite db and a terrific API. As of mid-2016 when I first used it, querying logs was extremely easy, and reasonably fast (w&#x2F; up to several million rows). Highest recommendation.<p>Disclaimer: I have no affiliation w&#x2F; the project or its maintainer -- but out of gratitude I mention it pretty much every time it&#x27;s appropriate.
osswid将近 6 年前
We&#x27;ve been using logging like this but with jsonl lines. Still easy to grep as straight text, but very handy to be able to parse with jq or other tools and be able to have rich values (or even substructures) as part of the log lines.
edsiper2将近 6 年前
Log structure is really important, from the examples provided I would suggest the same approach can be used using a full &#x27;logfmt&#x27; style, so timestamp and the event type can be set as keys, e.g:<p><pre><code> ts=&quot;2019-03-18 22:48:32.990&quot; event=&quot;Request started&quot; http_method=POST http_path=&#x2F;v1&#x2F;charges request_id=req_123 </code></pre> the main difference is that you make easier the parsing since many tools can parse lOgfmt without problems.<p>One interesting use-case here for &#x27;me&#x27; is the ability to perform queries in a schema-less fashion and I will do a quick speech on what we are working on Fluent Bit[0] (open source log project), pretty much the ability to query your data while is still in motion (stream processing on the edge[1]). Consider the following data samples in a log file:<p><pre><code> ts=&quot;2019-03-18 22:48:32.990&quot; event=&quot;Request started&quot; http_method=POST http_path=&#x2F;v1&#x2F;charges request_id=req_123 ts=&quot;2019-03-18 22:48:32.991&quot; event=&quot;User authenticated&quot; auth_type=api_key key_id=mk_123 user_id=usr_123 ts=&quot;2019-03-18 22:48:32.992&quot; event=&quot;Rate limiting ran&quot; rate_allowed=true rate_quota=100 rate_remaining=99 ts=&quot;2019-03-18 22:48:32.998&quot; event=&quot;Charge created&quot; charge_id=ch_123 permissions_used=account_write team=acquiring ts=&quot;2019-03-18 22:48:32.999&quot; event=&quot;Request finished&quot; alloc_count=9123 database_queries=34 duration=0.009 http_status=200 </code></pre> so if I wanted to retrieve all events associated for user 123 I would process the file as follows:<p><pre><code> $ fluent-bit -R conf&#x2F;parsers.conf \ -i tail -p alias=data -p path=canonical.log -p parser=logfmt \ -T &quot;SELECT * FROM STREAM:data WHERE user_id=&#x27;usr_123&#x27;;&quot; -o null -f 1 </code></pre> the output is:<p><pre><code> [1552949312.991000, {&quot;event&quot;=&gt;&quot;User authenticated&quot;, &quot;auth_type&quot;=&gt;&quot;api_key&quot;, &quot;key_id&quot;=&gt;&quot;mk_123&quot;, &quot;user_id&quot;=&gt;&quot;usr_123&quot;}] </code></pre> the results are in a raw mode but can be exported to stdout in json, to elasticsearch, kafka or any output destination supported.<p>One of the great things of the stream processor engine is that you can create new streams of data based on results, use windows of time (tumbling) for aggregation queries and such.<p>[0] <a href="https:&#x2F;&#x2F;fluentbit.io" rel="nofollow">https:&#x2F;&#x2F;fluentbit.io</a><p>[1] <a href="https:&#x2F;&#x2F;docs.fluentbit.io&#x2F;stream-processing" rel="nofollow">https:&#x2F;&#x2F;docs.fluentbit.io&#x2F;stream-processing</a>
sethammons将近 6 年前
This is not unlike what we&#x27;ve been doing for years. We generate billions of log lines like this daily as json and inspect them with splunk. By having consistent values across log lines, we can query and do neat things. &quot;What was our system timing in relation to users who have feature x?&quot; &quot;What correlations can we find between users whose requests took too long and were not throttled? -&gt; ah, 99% of those requests show $correlation_in_other_kv_pair!&quot;
评论 #20569595 未加载
thomas536将近 6 年前
What&#x27;s the primary use case for this? I almost always only look at logs to debug things; very rarely to perform some sort of event math&#x2F;analysis.
评论 #20571676 未加载
perq将近 6 年前
Very similar to what logsense.com does with logs except you don&#x27;t need to have canonical log line as patterns are found automatically.
msoad将近 6 年前
is there any legal restriction of how long you can keep internal systems logs? if it&#x27;s done right they don&#x27;t contain PIIs but they _can_ be used to track people if you have enough logs.
评论 #20569304 未加载
评论 #20569368 未加载
评论 #20571177 未加载
winrid将近 6 年前
This kind of debugging is one reason I loved Sumologic&#x27;s Join query feature.
chasers将近 6 年前
We do exactly this with Logflare and send stuff to BigQuery.