hey hn, supabase ceo here<p>this one is a long-time coming and it's a continuation of our acquisition of Logflare[0]. Since the acquisition we've be open-source-ing the server, which you can find here: <a href="https://github.com/Logflare/logflare">https://github.com/Logflare/logflare</a><p>Logflare handles about 1.5 billion log-events everyday on supabase. It's built with Elixir and has no problems with that workload.<p>This is really just the start of the Logflare updates. All logs are currently ingested into BigQuery, and we are adding support for Clickhouse and other backends (focusing primarily on open source backend). Over time Logflare will function very much like an open source Sentry alternative, where you can ingest data from various sources.<p>The team will be around if you have any questions about the technical implementation<p>[0] acquision: <a href="https://supabase.com/blog/supabase-acquires-logflare">https://supabase.com/blog/supabase-acquires-logflare</a>
> Logflare was available under a BSL license prior to joining Supabase. We’ve since changed the license to Apache 2.0, aligning it with our open source philosophy.<p>I wish more companies were like this!<p>(note, the vector.dev link is broken)
It would be awesome if this could use Quickwit as a backend which is a new promising alternative to Elasticsearch, I’ve been using it internally and it’s much more lightweight and easier to run.
<a href="https://github.com/Logflare/logflare/tree/staging">https://github.com/Logflare/logflare/tree/staging</a><p>> Simply provide your BigQuery credentials and we stream logs into your BigQuery table while automatically managing the schema<p>I didn't know BigQuery was capable of accepting streaming log data - in my mental model of the world it was the kind of database that you update using the occasional batch job, not from a streaming source of data.<p>Looks like that's the tabledata.insertAll method which has been around for quite a few years - though it's now called the "legacy streaming API" on <a href="https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery" rel="nofollow">https://cloud.google.com/bigquery/docs/streaming-data-into-b...</a> which suggests using the more recent Storage Write API instead: <a href="https://cloud.google.com/bigquery/docs/write-api" rel="nofollow">https://cloud.google.com/bigquery/docs/write-api</a>
I tried LogFlare (which is now Supabase Logs) in January, but it didn't work well for what I wanted.<p>Supabase Logs / Logflare seems primarily interested in creating graphs from logs rather than using logs for diagnostic purposes.<p>I've been looking for a log solution that's good for the use case of high retention but low volume.<p>I have a few small apps that generate a few MB of logs per month, so basically nothing. But I still want to have all my logs searchable in one place.<p>Most logging solutions set retention based on time rather than data size. So regardless of how much you're logging, they throw away your logs within somewhere between 7-30 days unless you're on an insane Enterprise plan.<p>I was excited about LogFlare because it supports unlimited retention, but I ran into too many issues and had to cancel my subscription:<p>* To search your logs, you need to write a SQL-like query in LogFlare's DSL. You can't just put in a route (e.g. /api/auth) like you can with other log analytics.<p>* Search only shows the matching lines. Usually, what I want to see is the log line in context. For example, if I search "catastrophic error" I want to see the log lines leading up to that, not just that specific line.<p>* Search is limited to a maximum of 100 results. If you want to see more results, you need to rewrite your query rather than just scroll up or hit a "load more" button.<p>* When you do adjust the query to a larger time window, the query will fail because it can't generate a graph unless you also adjust the group_by in your query to match the new time window's limits. This is an annoying obstacle if you don't care about graphing the results and are just trying to diagnose an issue in your logs.<p>I found support lacking as well. I emailed support to ask if I was misunderstanding how to use Logflare or if it was just designed for a different use case. I was on a paid plan, but I still had to wait 3 business days for a response. When the response came, they just said that it was designed for me but didn't address any of the issues I brought up.<p>I do like that Logflare/Supabase let you bring your own BigQuery. That's nice for customers like me who want low-volume, high retention. I hope they continue iterating because it has potential.<p>In the meantime, I've found LogTail to be a pretty good alternative, but they're limited to 30 days of retention even on the highest tier plan.
What are some of the largest production applications built using Supabase? I know it's popular for whipping something up for a hackathon, but how battle tested is it?<p>Also does anybody know what they're doing behind the scenes with the database? I know their storage uses s3, functions (I think) use Deno, this uses BigQuery. Is their db on RDS/Aurora? If so how do they claim max DB size of 1024 TB while Aurora is 128 TB?
If sentryio has a self hosted version and logflare has one too, why would I pick logflare? Are there any differences? I tried sentryio and it's really convenient, at least the hosted version.