TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

A journey of optimization of cloud-based geospatial data processing

28 pointsby BrainBuzzer5 months ago

4 comments

jandrewrogers5 months ago
As a meta-topic, the standard formats for geospatial are virtually all deeply suboptimal to the point of being arguably broken for modern large-scale geospatial analysis. What we have is a giant collection of narrow solutions to narrow problems, none of which efficiently interoperate with each other. It hasn’t modernized much at all, the tool chain is basically the same tech we had 25 years ago with a fresh coat of paint.<p>Fixing this is a difficult and extremely deep engineering and computer science problem, which is why there has been so little progress. Most improvements have been driven by users, but they aren’t able to make the investments to fix the problems underneath their problem, they have other work to do. Any fundamental improvements would materially break existing workflows, so adoption would be slow at best.<p>The entire spatial domain is stuck in a deep local minima.
评论 #42424551 未加载
scottcha5 months ago
I wonder how this compares to the use of zarr format which seems to have similar goals and design and is already well integrated in several libraries (particularly xarray)?
评论 #42424470 未加载
whinvik5 months ago
As someone who deals with geospatial processes like this daily, I have 2 notes. 1. STAC implementations are already complicated. Not everyone has good catalogues, or that work uniformly well for all types of queries. 2. Using STAC geoparquet on top and then another layer on top would mean we have to self host yet another catalog, essentially a new standard.<p>In short, even though I believe STAC adoption is what we should aim for, in reality we usually end up building workarounds.
评论 #42424224 未加载
评论 #42424504 未加载
lmc5 months ago
Interesting ideas. If you do some subsequent post on this, it would be great to see how the perf benefits scale with different sizes of AoIs.
评论 #42428456 未加载