Introducing S2

IANAL,but naming your product S2 and mentioning in the intro that AWS S3 is the tech you are enhancing is probably looking for a branding&#x2F;copyright claim from Amazon. Same vertical &amp; definitely will cause consumer confusion. I&#x27;m sure you&#x27;ve done the research about whether a trademark has been registered.<p><a href="https:&#x2F;&#x2F;tsdr.uspto.gov&#x2F;#caseNumber=98324800&amp;caseSearchType=US_APPLICATION&amp;caseType=DEFAULT&amp;searchType=statusSearch" rel="nofollow">https:&#x2F;&#x2F;tsdr.uspto.gov&#x2F;#caseNumber=98324800&amp;caseSearchType=U...</a>

This is a really good idea, beautiful API, and something that I would like to use for my projects. However I have zero confidence that this startup would last very long in its current form. If it&#x27;s successful, AWS will build a better and cheaper in-house version. It&#x27;s just as likely to fail to get traction.<p>If this had been released instead as a Papertrail-like end-user product with dashboards, etc. instead of a &quot;cloud primitive&quot; API so closely tied to AWS, it would make a lot more sense. Add the ability to bring my own S3-Compatible backend (such as Digital Ocean Spaces), and boom, you have a fantastic, durable, cloud-agnostic product.

Help me understand - you build on top of AWS, which charges $0.09&#x2F;GB for egress to the Internet, yet you&#x27;re charging $0.05&#x2F;GB for egress to the Internet? Sounds like you&#x27;re subsidizing egress from AWS? Or do you have access to non-public egress pricing?

So is this basically WarpStream except providing a lower-level API instead of jumping straight to Kafka compatibility?<p>An S3-level primitive API for streaming seems really valuable in the long-term if adopted

These folks knowingly chose to spend the rest of their careers explaining that they are not, in fact, S3.

It looks neat but, no Java SDK? Every company I&#x27;ve personally worked at is deeply reliant on Spring or the vanilla clients to produce&#x2F;consume to Kafka 90% of the time. This kind of precludes even a casual PoC.

I do like this. The next part I&#x27;d like someone to build on top of this is applying the stream &#x27;events&#x27; into a point-in-time queryable representation. Basically the other part to make it a Datatomic. Probably better if it&#x27;s a pattern or framework for making specific in-memory queryable data rather than a particular database. There&#x27;s lots of ways this could work, like applying to a local Sqlite, or basing on a MySQL binlog that can be applied to a local query instance and rewindable to specific points, or more application-specific apply&#x2F;undo events to a local state.

Roughly ten years ago, I started Gazette [0]. Gazette is in an architectural middle-ground between Kafka and WarpStream (and S2). It offers unbounded byte-oriented log streams which are backed by S3, but brokers use local scratch disks for initial replication &#x2F; durability guarantees and to lower latency for appends and reads (p99 &lt;5ms as opposed to &gt;500ms), while guaranteeing all files make it to S3 with niceties like configurable target sizes &#x2F; compression &#x2F; latency bounds. Clients doing historical reads pull content directly from S3, and then switch to live tailing of very recent appends.<p>Gazette started as an internal tool in my previous startup (AdTech related). When forming our current business, we very briefly considered offering it as a raw service [1] before moving on to a holistic data movement platform that uses Gazette as an internal detail [2].<p>My feedback is: the market positioning for a service like this is <i>extremely</i> narrow. You basically have to make it API compatible with a thing that your target customer is already using so that trying it is zero friction (WarpStream nailed this), or you have to move further up to the application stack and more-directly address the problems your target customers are trying to solve (as we have). Good luck!<p>[0]: <a href="https:&#x2F;&#x2F;gazette.readthedocs.io&#x2F;en&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;gazette.readthedocs.io&#x2F;en&#x2F;latest&#x2F;</a> [1]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=21464300">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=21464300</a> [2]: <a href="https:&#x2F;&#x2F;estuary.dev" rel="nofollow">https:&#x2F;&#x2F;estuary.dev</a>

This is a very useful service model, but I&#x27;m confused about the value proposition given how <i>every</i> write is persisted to S3 before being acknowledged.<p>I suppose the writers could batch a group of records before writing them out as a larger blob, with background processes performing compaction, but it&#x27;s still an object-backed streaming service, right?<p>AWS has shown their willingness to implement mostly-protocol compatible services (RDS -&gt; Aurora), and I could see them doing the same with a Kafka reimplementation.

Seems like really cool tech. Such a bummer that the it is not source available. I might be a minority in this opinion, but I would absolutely consider commercial services where the core tech is all released under something like a FSL with fully supported self-hosting. Otherwise, the lock-in vs something like kafka is hard to justify.

I look at the egress costs to internet and it doesn’t check out. It’s a premium product dependent on DX, marketed to funded startups.<p>But if I care about ingress and egress costs, which many stream heavy infrastructure providers do.. This doesn’t add up.<p>I wish them luck, but I feel they would have had a much better chance from the start by getting some funding and having a loss leader start, then organising and passing on wholesale rates from cloud providers once they’d reached critical mass.<p>Instead they’re going in at retail which is very spicy. I feel like someone will clone the tech and let you self host, before big players copy it natively.<p>It’s a commodity space and they’re starting with a moat of a very busy 2 weeks from some Staff engineers at AWS.

Just you wait, I am launching S1 next year!

Wow, imagine Debezium offering native compatibility with this, capturing the changes from a Postgres database, saving them as delta or iceberg in a pure serverless way!

I wish more dev-tools startups would focus on clearly explaining the business use cases, targeting a slightly broader audience beyond highly technical users. I visited several pages on the site before eventually giving up.<p>I can sort of grasp what the S2 team is aiming to achieve, but it feels like I’m forced to perform unnecessary mental gymnastics to connect their platform with the specific problems it can solve for a business or product team.<p>I consider myself fairly technical and familiar with many of the underlying concepts, but I still couldn’t work out the practical utility without significant effort.<p>It’s worth noting that much of technology adoption is driven by technical product managers and similar stakeholders. However, I feel this critical audience is often overlooked in the messaging and positioning of developer tools like this.

1. Do you support compression for data stored in segments?<p>2. Does the choice of storage class only affect chunks or also segments?<p>To me the best solution seem like combining storing writes on EBS (or even NVMe) initially to minimize the time until writes can be acknowledged, and creating a chunk on S3 standard every second or so. But I assume that would require significant engineering effort for applications that require data to be replicated to several AZs before acknowledging them. Though some applications might be willing to sacrifice 1s of writes on node failure, in exchange for cheap and fast writes.<p>3. You could be clearer about what &quot;latency&quot; means. I see at least three different latencies that could be important to different applications:<p>a) time until a write is durably stored and acknowledged<p>b) time until a tailing reader sees a write<p>c) time to first byte after a read request for old data<p>4. How do you handle streams which are rarely written to? Will newly appended records to those streams remain in chunks indefinitely? Or do you create tiny segments? Or replace and existing segment with the concatenated data?

This is a very interesting abstraction (and service). I can’t help but feature creep and ask for something like Athena, which runs PrestoDB (map reduce) over S3 files. It could be superior in theory because anyone using that pattern must shoehorn their data stream (almost everything is really a stream) into an S3 file system. Fragmentation and file packing become requirements that degrade transactional qualities.

My first thought: &quot;introducing? The S2 has been out for a while!&quot;<p><a href="https:&#x2F;&#x2F;www.sunlu.com&#x2F;products&#x2F;new-version-sunlu-filadryer-s2" rel="nofollow">https:&#x2F;&#x2F;www.sunlu.com&#x2F;products&#x2F;new-version-sunlu-filadryer-s...</a>

This is cool but I think it overlaps too much with something like Kinesis Data Streams from AWS which has been around for a long time. It’s good that AWS has some competition though

In the long-term, how different do you want to be from Apache Pulsar? At the moment, many differences are obvious, e.g., Pulsar offers transactions, queues and durable timers.

so the naming convention for 2024-25 products seems to be &lt;letter&gt;&lt;number&gt;.<p>o1, o3, s2, M4, r2, ...

In terms of a pitch, i&#x27;m not sure i understand how this differs from existing solutions. Is the core value proposition a simpler api?

Seems really good for IoT no? Been a while since I worked in that space, but having something like this would have been nice at the time.

I had an idea like this a few years ago. basicly emitting a stream interface to a cloud based fs to enable random access seeking on bystreams. I envisioned it to be useful for things like loading large files. would be amazing for enabling things like cloud gaming, images processing and CAD<p>kudos for sitting down and makin it happen!

Definitely a useful API but not super compelling until I could store the data in my own bucket

So is this a &quot;serverless&quot; named-pipe-as-a-service cloud offering? Or am I misreading?

How is this compare to <a href="https:&#x2F;&#x2F;github.com&#x2F;deuxfleurs-org&#x2F;garage">https:&#x2F;&#x2F;github.com&#x2F;deuxfleurs-org&#x2F;garage</a> ?<p>Seems like there are a lot of more lite weight self-hosted s3 around now days. Why even use S3?

I really liked the landing page and the service, but it took me a while to realize it wasn&#x27;t a AWS service with a snazzy landing page.

Apparently this is “S2, a new S3 competitor” not “S2, the spatial index system based on heirarchical qaudrilaterals”.

How does this compare to Kafka? Is the primary difference that this is a hosted solution?

Is it possible to bring my own cloud account to provide the underlying S3 storage?

Really interesting service and bookmarked.<p>I&#x27;d really love this extending more into the event sourcing space not just the log&#x2F;event streaming space.<p>Dealing with problems like replay and log compaction etc.<p>Plus things like dealing with old events. Under GDPR, removing personal information&#x2F;isolating it from the data&#x2F;events themselves in an event sourced system are a PITA.

Would this be like an alternative to Delta? Am I thinking about that right?

Pretty bad branding! It should have at least been S4!

S2 is, in my opinion, the sweet spot of PRS&#x27;s lineup.

This would sell much better is was S5 or S6 next level thing.<p>Wow man are you stil stuck on S3?

&quot;Making the world a better place through streamable, appendable object streams&quot;

Scribe aaS? ;)

Kafka as a service ?

Can someone tell me what does this do? And why its better.

Serverless pricing to me is exactly like the ETH gas pricing !