TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

AWS S3 SDK breaks its compatible services

190 点作者 ulrischa3 个月前

29 条评论

js23 个月前
Okay, but I&#x27;d never expect an AWS SDK to remain backwards compatible with third-party services and would be leery about using an AWS SDK with anything but AWS. It&#x27;s on the third parties to keep their S3-compatible API, well, compatible.<p>On the client side, you just have to pin to an older version of the AWS SDK till whatever compatible service you&#x27;re using updates, right?<p>Also, this is the first I&#x27;ve heard of OpenDAL. Looks interesting:<p><a href="https:&#x2F;&#x2F;opendal.apache.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;opendal.apache.org&#x2F;</a><p>It&#x27;s had barely any discussion on HN:<p><a href="https:&#x2F;&#x2F;hn.algolia.com&#x2F;?q=opendal" rel="nofollow">https:&#x2F;&#x2F;hn.algolia.com&#x2F;?q=opendal</a>
评论 #43120746 未加载
评论 #43119260 未加载
评论 #43120425 未加载
评论 #43119311 未加载
评论 #43123190 未加载
评论 #43119728 未加载
profmonocle3 个月前
Treating a proprietary API as a standard is risky - this is a good example of why. From Amazon&#x27;s point of view there&#x27;s no reason to keep the S3 SDK backwards compatible with old versions of the S3 service, because they control the S3 service. Once this feature was rolled out in all regions, it was safe to update the SDK to expect it.<p>Amazon may not be actively hostile to using their SDK with third party services, but they never promised to support that use case.<p>(disclaimer: I work for AWS but not on the S3 team, I have no non-public knowledge of this and am speaking personally)
评论 #43119871 未加载
评论 #43119654 未加载
评论 #43122244 未加载
dougb3 个月前
I got bit by this a month ago. You can disable the new behavior by setting 2 environment variables<p><pre><code> export AWS_REQUEST_CHECKSUM_CALCULATION=when_required export AWS_RESPONSE_CHECKSUM_CALCULATION=when_required </code></pre> or adding the following 2 lines to a profile in ~&#x2F;.aws&#x2F;config<p><pre><code> request_checksum_calculation=when_required response_checksum_validation=when_required </code></pre> Or just pin your AWS SDK to version before the following.<p>&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-go-v2&#x2F;blob&#x2F;release-2025-01-15&#x2F;service&#x2F;s3&#x2F;CHANGELOG.md#v1730-2025-01-15">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-go-v2&#x2F;blob&#x2F;release-2025-01-15...</a>&gt;<p>&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;boto&#x2F;boto3&#x2F;issues&#x2F;4392">https:&#x2F;&#x2F;github.com&#x2F;boto&#x2F;boto3&#x2F;issues&#x2F;4392</a>&gt;<p>&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-cli&#x2F;blob&#x2F;1.37.0&#x2F;CHANGELOG.rst#L19">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-cli&#x2F;blob&#x2F;1.37.0&#x2F;CHANGELOG.rst#L19</a>&gt;<p>&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-cli&#x2F;blob&#x2F;2.23.0&#x2F;CHANGELOG.rst#2230">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-cli&#x2F;blob&#x2F;2.23.0&#x2F;CHANGELOG.rst#223...</a>&gt;<p>&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-java-v2&#x2F;releases&#x2F;tag&#x2F;2.30.0">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-java-v2&#x2F;releases&#x2F;tag&#x2F;2.30.0</a>&gt;<p>&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-net&#x2F;releases&#x2F;tag&#x2F;3.7.963.0">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-net&#x2F;releases&#x2F;tag&#x2F;3.7.963.0</a>&gt;<p>&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-php&#x2F;releases&#x2F;tag&#x2F;3.337.0">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-php&#x2F;releases&#x2F;tag&#x2F;3.337.0</a>&gt;<p>and wait for your S3 Compatible Object store to add a fix to support this.
评论 #43121841 未加载
aaronbrethorst3 个月前
<i>Many S3-compatible services are recommending that their users use the S3 SDK directly, and changing the default settings in this way can have a direct impact on their users.</i><p>This is wholly predictable; AWS isn&#x27;t in the business of letting other companies OpenSearch them.
评论 #43119668 未加载
评论 #43119165 未加载
riknos3143 个月前
As a user of S3, but not any service with an S3 compatible API, this execution of the change is perfect for me as I get the benefits with 0 effort - including needing to learn about the availability of the feature.<p>AWS is beholden first and foremost to their paying customers, and this is the best option for most S3 customers.
评论 #43119204 未加载
评论 #43121016 未加载
joshstrange3 个月前
I&#x27;m not sure what the other option is here? Keep old defaults and hope users update?<p>I wouldn&#x27;t be happy to find out they did it &#x2F;just&#x2F; to break third-party S3 providers but it seems like it&#x27;s an easy enough thing to turn it off right?<p>I&#x27;m just not sure how comfortable I am with the phrasing here (or maybe I&#x27;m reading too much into it).
评论 #43122261 未加载
benmanns3 个月前
Another case of Hyrum&#x27;s Law, where the entire functionality of the S3 SDK and any competing service provider borrowing from it becomes Amazon&#x27;s problem to fix at their own cost. Maybe it&#x27;s time for a non-Amazon but S3 API compatible library to emerge among the other cloud storage providers offering S3 compatible APIs. OpenDAL looks interesting. Also another reminder to run thorough integration tests before updating your dependencies.
评论 #43119127 未加载
xena3 个月前
This bit us pretty hard at Tigris, but we had a fix out pretty quickly. I set up some automation with some popular programming languages so that we can be aware of the next time something like this happens. It also bit me in my homelab until I patched Minio: <a href="https:&#x2F;&#x2F;xeiaso.net&#x2F;notes&#x2F;2025&#x2F;update-minio&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xeiaso.net&#x2F;notes&#x2F;2025&#x2F;update-minio&#x2F;</a>
femto1133 个月前
&gt; the AWS team has implemented it poorly by enforcing it<p>This is whiny and just wrong. Best behavior by default is always the right choice for an SDK. Libraries&#x2F;tools&#x2F;clients&#x2F;SDKs break backwards compatibility all the time. That&#x27;s exactly what semver version pinning is for, and that&#x27;s a fundamental feature of every dependency management system.<p>AWS handled this exactly right IMO. Change was introduced in Python SDK version 1.36.0 which clearly indicatesbreaking API changes, and their changelog also explicitly mentions this new default<p><pre><code> api-change:``s3``: [``botocore``] This change enhances integrity protections for new SDK requests to S3. S3 SDKs now support the CRC64NVME checksum algorithm, full object checksums for multipart S3 objects, and new default integrity protections for S3 requests. </code></pre> <a href="https:&#x2F;&#x2F;github.com&#x2F;boto&#x2F;boto3&#x2F;blob&#x2F;2e2eac05ba9c67f0ab285efe5050fe0d3eb03bd2&#x2F;CHANGELOG.rst#L252">https:&#x2F;&#x2F;github.com&#x2F;boto&#x2F;boto3&#x2F;blob&#x2F;2e2eac05ba9c67f0ab285efe5...</a>
评论 #43120399 未加载
评论 #43120265 未加载
julik3 个月前
Sometimes people forget that the S3 API is not an industry standard, but a proprietary inspectable interface the original author is at liberty to modify to their liking. And it does, indeed, have thorny edges like &quot;which headers are expected&quot;, &quot;which headers do you sign&quot;, what the semantics of the headers are and so forth.<p>It is also on the implementors of the &quot;compatible&quot; services to, for example, not require a header that can be assumed optional. If it is not &quot;basic HTTP&quot; (things like those checksums) - don&#x27;t crash the PUT if you don&#x27;t get the header unless you absolutely require that header. Postel&#x27;s law and all.<p>The mention in the Tigris article is strange: is boto now doing a PUT without request content-length? That&#x27;s not even valid HTTP 1.1
robocat3 个月前
Any third party is one update away from an external business shock should Amazon change their API.<p>Setting up a business so that all your customers fail at the same moment is a poor business practice: nobody can support all their customers breaking at once. I&#x27;m guessing competitors compete on price, not reliability.<p>Amazon has the incentive to break third parties, since their customers are likely to switch to Amazon. Why else use the Amazon code unless you&#x27;re ready to migrate or the service is low importance?
评论 #43120537 未加载
merb3 个月前
actually the new default is sane. its WAY WAY WAY WAY better than before. especially for multi uploads. it&#x27;s basically one of the features where gcloud had a insane edge. another thing was If-Match in cloud storage.
评论 #43120685 未加载
everfrustrated3 个月前
Time for the community to support a standards body effort to define S3 compatibility under its own brand and standard.<p>Having a way for vendors to publish some Level of compatibility would be a great help. Eg Tier 1 might mean basic upload&#x2F;download, Tier 2 might support storage classes and lifecycle policies etc. Right now is just a confusing mess of self-attestation.
评论 #43120424 未加载
jonasdoesthings3 个月前
There are great alternative libraries for AWS &amp; AWS-compatible services like S3<p>Been really happy with aws4fetch in TypeScript (for Cloudflare R2, generating presigned URLs &amp; sending mails via SES) after getting much frustration out of the official JS SDK.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;mhart&#x2F;aws4fetch">https:&#x2F;&#x2F;github.com&#x2F;mhart&#x2F;aws4fetch</a>
kagitac3 个月前
&gt;OpenDAL integrates all services by directly communicating with APIs instead of relying on SDKs, protecting users from potential disruptions like this one.<p>Implying that the SDKs don&#x27;t communicate directly with the APIs? This &quot;problem&quot; could have happened in OpenDAL just as it did in AWS SDKs.
donatj3 个月前
I&#x27;ve always been very uncomfortable with the S3 API becoming a defacto protocol, and this shows why.
earth2mars3 个月前
I see the same thing could happen with all the LLM providers depending on the OpenAI API .
dastbe3 个月前
with the open sourcing of smithy and the raw s3 specs, open source should be able to build clients in just about any language they care about while keeping relatively high fidelity with implementations s3 releases.
amazingamazing3 个月前
I wish more services would become de-facto standards and there were many implementations of the same API. I&#x27;d love more (cheaper) DynamoDB compatible APIs.
评论 #43120807 未加载
评论 #43120076 未加载
fulmicoton3 个月前
This bug hit us, and yes, I hadn&#x27;t thought of just switching to opendal. That&#x27;s indeed a great reminder.
imclaren3 个月前
I was bitten by this using aws’s golang sdk.<p>The nuance is that Amazon updated their sdk so that the default is that the sdk no longer works for third party services that do not use the new checksum. There is a new configuration option in the sdk that either: (1) requires the checksum (and breaks s3 usage for unsupported services); or (2) only uses the checksum if the s3 service supports it.<p>The sdk default is (1) when this issue could have been avoided if the sdk default was (2).<p>Agree with all the comments that Amazon has never said or even implied that updates to their sdk would work on other s3 compatible services. However, the reality is that s3 has become a defacto standard and (unless this is a deliberate Amazon strategy) it would be relatively easy for Amazon to set this default that allows for but does not require changes to s3 compatible services or, if possible, to loudly flag these types of changes well in advance so that they don’t surprise users.
_def3 个月前
I always thought the property &quot;S3 compatible&quot; was fishy - here we see why.
cyberax3 个月前
Stupid. A lot of people using the actual Amazon S3 for production, are also using S3 API with minio for local testing.<p>minio at least was updated to always emit the header, so it&#x27;s simply an upgrade away.
评论 #43119489 未加载
richwater3 个月前
The expectation that S3 should not improve functionality for paying customers because other people use the SDK for something completely unrelated is asinine.
deskr3 个月前
Does AWS S3 SDK have a list of &quot;compatible services&quot; ?
评论 #43119655 未加载
ydnaclementine3 个月前
I guess you could say the s3 compatible services are no longer s3 compatible. But don&#x27;t update your client and give them a sec to update
Vosporos3 个月前
Y&#x27;all put blind trust in a proprietary API SDK and all the eggs in the same basket? Just like that??
评论 #43119799 未加载
nisten3 个月前
just use bun s3client, it has native support now
przemub3 个月前
I&#x27;m confused. Why Cloudflare et al. won&#x27;t release a package in pypi named boto-r2 or something, pinned to the last compatible version with the added benefit of setting the endpoint by default? It seems asinine to rely on Amazon as not to break things from time to time.
评论 #43123050 未加载