TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

S3 as a Git remote and LFS server

197 pointsby kbumsik7 months ago

16 comments

mdaniel7 months ago
All this mocking when moto exists is just :-( <a href="https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3&#x2F;blob&#x2F;v0.1.19&#x2F;test&#x2F;remote_test.py">https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3&#x2F;blob&#x2F;v0.1.19&#x2F;test&#x2F;r...</a><p>Actually, moto is just one bandaid for that problem - there are SO MANY s3 storage implementations, including the pre-license-switch Apache 2 version of minio (one need not use a bleeding edge for something as relatively stable as the S3 Api)
评论 #41889525 未加载
评论 #41889194 未加载
评论 #41896008 未加载
Scribbd7 months ago
This is something I was trying to implement myself. I am surprised it can be done with just an s3 bucket. I was messing with API Gateways, Lambda functions and DynamoDB tables to support the s3 bucket. It didn&#x27;t occur to me to implement it client side. I might have stuck a bit too much to the lfs test server implementation. <a href="https:&#x2F;&#x2F;github.com&#x2F;git-lfs&#x2F;lfs-test-server">https:&#x2F;&#x2F;github.com&#x2F;git-lfs&#x2F;lfs-test-server</a>
评论 #41891514 未加载
CGamesPlay7 months ago
If you are interested in using S3 as a git remote but are concerned with privacy, I built a tool a while ago to use S3 as an untrusted git remote using Restic. <a href="https:&#x2F;&#x2F;github.com&#x2F;CGamesPlay&#x2F;git-remote-restic">https:&#x2F;&#x2F;github.com&#x2F;CGamesPlay&#x2F;git-remote-restic</a>
zmmmmm7 months ago
Just remember, the mininum billing increment for file size is 128KB in real AWS S3. So your Git repo may be a lot more expensive than you would think if you have a giant source tree full of small files.
评论 #41896027 未加载
评论 #41894455 未加载
评论 #41896785 未加载
doctorpangloss7 months ago
<a href="https:&#x2F;&#x2F;alanedwardes.com&#x2F;blog&#x2F;posts&#x2F;serverless-git-lfs-for-game-dev&#x2F;" rel="nofollow">https:&#x2F;&#x2F;alanedwardes.com&#x2F;blog&#x2F;posts&#x2F;serverless-git-lfs-for-g...</a><p>I’ve used this guy’s CloudFormation template since forever for LFS on S3.<p>GitHub has to lower its egregious LFS pricing.
x3n0ph3n37 months ago
Wow, AWS <i>really</i> wants to get rid of CodeCommit.
Evidlo7 months ago
For the LFS part there is also dvc which works better than git-lfs and natively supports S3.
评论 #41896488 未加载
评论 #41898528 未加载
评论 #41892558 未加载
milkey_mouse7 months ago
You can also do this with Cloudflare Workers for fewer setup steps&#x2F;moving parts:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;milkey-mouse&#x2F;git-lfs-s3-proxy">https:&#x2F;&#x2F;github.com&#x2F;milkey-mouse&#x2F;git-lfs-s3-proxy</a>
philsnow7 months ago
I&#x27;m surprised they just punt on concurrent updates [0] instead of locking with something like dynamodb, like terraform does.<p>[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3?tab=readme-ov-file#concurrent-writes">https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3?tab=readme-ov-file#...</a>
评论 #41890916 未加载
评论 #41890892 未加载
评论 #41890047 未加载
kernelsanderz7 months ago
I’ve been using <a href="https:&#x2F;&#x2F;github.com&#x2F;jasonwhite&#x2F;rudolfs">https:&#x2F;&#x2F;github.com&#x2F;jasonwhite&#x2F;rudolfs</a> - which is written in rust. It’s high performance but doesn’t have all the features (auth) that you might need.
fortran777 months ago
Amazon has deprecated Amazon Code Commit, so this may be an interesting alternative.
评论 #41894558 未加载
tonymet7 months ago
how does it handle incremental changes? If it’s writing your entire repo on a loop, I could see why AWS would promote it.
评论 #41894524 未加载
WhyNotHugo7 months ago
git-annex also has native support for s3.
评论 #41896403 未加载
xena7 months ago
How do you install this? Homebrew broke global pip install. Is there a homebrew package or something?
评论 #41896032 未加载
mattxxx7 months ago
This seems wrong, since you can&#x27;t push transactionally + consistently in S3.<p>They address this directly in their section on concurrent writes: <a href="https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3?tab=readme-ov-file#how-s3-remote-work">https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3?tab=readme-ov-file#...</a><p>And in their design: <a href="https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3?tab=readme-ov-file#how-s3-remote-work">https:&#x2F;&#x2F;github.com&#x2F;awslabs&#x2F;git-remote-s3?tab=readme-ov-file#...</a><p>But it seems like this is just the wrong tool for the job (hosting git repos).
Havoc7 months ago
Does this work with other s3 implementations like minio?