TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Show HN: AWS S3 Connector with DuckDB – Query AI/ML Batch Results Directly in S3

2 pointsby nagstler11 months ago
[ <a href="https:&#x2F;&#x2F;github.com&#x2F;Multiwoven&#x2F;multiwoven">https:&#x2F;&#x2F;github.com&#x2F;Multiwoven&#x2F;multiwoven</a> ]<p>Multiwoven, our Open Source alternative to Hightouch, Census and RudderStack, has always been about making data available where it&#x27;s needed.<p>We&#x27;ve added a new AWS S3 connector as a data source to Multiwoven, This data source connector has been a highly requested feature from the community.<p>We believe we&#x27;ve not only added AWS S3 as a data source, but also optimized the performance of querying data stored in S3 buckets.<p>We&#x27;ve integrated DuckDB, an in-memory analytical database, to provide fast and efficient SQL query execution on large datasets directly in S3.<p>-&gt; Features:<p>1. IAM and Role-based Access - Securely connect to AWS S3 buckets using IAM or role-based permissions.<p>2. File Format Support - Native support for CSV and Parquet file formats.<p>3. DuckDB Powered Performance - Utilizes hashtag#DuckDB, an in-memory analytical database, for fast and efficient SQL query execution on large datasets directly in S3.<p>4. Native SQL Interface - Execute SQL queries directly on data stored in S3 buckets, eliminating the need for intermediate scripting steps or data movement to a separate database.<p>-&gt; Use Cases:<p>* Query and Transform - Convert ML model batch results stored in S3 buckets into actionable insights.<p>* Sync Data - Sync log data or event streams from S3 to business applications like Salesforce, Google Sheets, or other destinations for real-time analytics.<p>[ <a href="https:&#x2F;&#x2F;github.com&#x2F;Multiwoven&#x2F;multiwoven">https:&#x2F;&#x2F;github.com&#x2F;Multiwoven&#x2F;multiwoven</a> ]<p>Refer to our GitHub repository for more information &amp; hit the star button to show your support :)

1 comment

8organicbits11 months ago
Isn&#x27;t DuckDB is an in-process database, not an in-memory database?
评论 #40909441 未加载