TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Show HN: Serverless USGS historical topographic map tiles

134 点作者 kylebarron大约 4 年前

9 条评论

kylebarron大约 4 年前
Hi all, I was invited to repost this (originally posted last year) to be included in the second chance pool!<p>The USGS created topographic maps by hand from 1884 to 2006 (when they started making their topographic map series digitally). I stumbled upon the fact that all 183,000 of their hand-drawn maps were digitized and uploaded to a public S3 bucket [0] in Cloud-Optimized GeoTIFF format [1].<p>Because they&#x27;re in this format, it&#x27;s possible to quickly mosaic images _on demand_, in a process called &quot;dynamic tiling&quot; [2]. The dynamic nature of it gives you a lot more flexibility, because you aren&#x27;t creating static tiles once.<p>In contrast, the hardest part about this project is the time variation. The USGS created many different scales of maps at many different time periods, so figuring out how to combine them into a few mosaics is quite tricky. With more time invested in this, it would be ideal to have a search capability to show user-specified time horizons. (Edit: I talk about this process a bit more in the Github README [3]).<p>More recently I&#x27;ve been spending time doing similar dynamic tiling projects, but focusing on satellite imagery, to enable analytic processing of satellite imagery in the browser [4].<p>[0]: `aws s3 ls s3:&#x2F;&#x2F;prd-tnm&#x2F;StagedProducts&#x2F;Maps&#x2F;HistoricalTopo&#x2F;GeoTIFF&#x2F;`<p>[1]: <a href="https:&#x2F;&#x2F;www.cogeo.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.cogeo.org&#x2F;</a><p>[2]: <a href="https:&#x2F;&#x2F;kylebarron.dev&#x2F;blog&#x2F;cog-mosaic&#x2F;overview" rel="nofollow">https:&#x2F;&#x2F;kylebarron.dev&#x2F;blog&#x2F;cog-mosaic&#x2F;overview</a><p>[3]: <a href="https:&#x2F;&#x2F;github.com&#x2F;kylebarron&#x2F;usgs-topo-tiler" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kylebarron&#x2F;usgs-topo-tiler</a><p>[4]: <a href="https:&#x2F;&#x2F;www.unfolded.ai&#x2F;blog&#x2F;2021-04-28-raster-layer&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.unfolded.ai&#x2F;blog&#x2F;2021-04-28-raster-layer&#x2F;</a>
评论 #27070809 未加载
评论 #27070042 未加载
评论 #27071307 未加载
评论 #27071696 未加载
评论 #27071661 未加载
eharmon大约 4 年前
This is great! I&#x27;ve been fascinated with tiling historical maps for quite some time. In fact, I made a very similar (but far less advanced) site focusing primarily on historical USGS maps of the SF Bay Area [0].<p>I&#x27;m curious about this:<p>&gt; To get around this, I apply a &quot;hack&quot; to the MosaicJSON format. Instead of just encoding a URL string, I encode the url and the bounds of the map as a JSON string.<p>In my effort I struggled to automatically assemble bounds from tilted and angled scans. Is this source data set clean enough to make simpler assumptions, or did you also devise a way to automatically determine the map boundaries within the scan?<p>[0]: <a href="https:&#x2F;&#x2F;eharmon.net&#x2F;bayquads&#x2F;" rel="nofollow">https:&#x2F;&#x2F;eharmon.net&#x2F;bayquads&#x2F;</a>
simonw大约 4 年前
I&#x27;m assuming the magic is this bit?<p><a href="https:&#x2F;&#x2F;us-west-2-lambda.kylebarron.dev&#x2F;usgs-topo&#x2F;11&#x2F;335&#x2F;790@1x.jpg?url=dynamodb:&#x2F;&#x2F;us-west-2&#x2F;usgs-topo_medium_oldest.v2" rel="nofollow">https:&#x2F;&#x2F;us-west-2-lambda.kylebarron.dev&#x2F;usgs-topo&#x2F;11&#x2F;335&#x2F;790...</a><p>Which figures out how to return a JPG of that specific 11&#x2F;335&#x2F;790 tile.<p>What&#x27;s Dynamo DB being used for here?
评论 #27070135 未加载
dirtyid大约 4 年前
Somewhat related, there was a great video on USGS cartography department from the 60s from a deleted youtube archival channel that went through the steps of map making and administration. Ive been trying to refind for years, wonder if anyone has come across it or know where to look. It looked like an official gov video, so must be archived somewhere.
smashah大约 4 年前
Wow excited to use this!
nonameiguess大约 4 年前
This is great work, but it almost makes me sad that you have to do this. I spent the earliest parts of my software career implementing full spectrum geospatial intelligence product generation algorithms for US intelligence agencies and the military. Part of the code base was software that does exactly this, but not with the historical, hand-drawn maps, but with the most current maps generated from aerial and space-borne radar sensing, as well as non-USGS data of, say, the ocean floor generated by US Navy sonar surveying programs. We could generate a map of any arbitrary coordinate boundaries of any part of the earth in seconds, using data that was at most a few years old.<p>There is absolutely no reason any of this code needs to be classified, as the maps themselves are not, the GeoTIFF format and tiling procedures are completely open standards, and the NGA has even open sourced some libraries for doing this kind of thing, but so much of the code that does this really quickly and efficiently, directly on your own workstation using C++ native executables into formats that can be rendered by a standard image viewer, is tied up in proprietary and even classified code bases just because of the monorepo nature of early work and the fact that development was done entirely on the classified systems because of the difficulty in developing on unclassified systems and then transferring across the air gap.<p>This is slowly changing, and in fact what I am presently working on is automated transfer and assurance mechanisms so more development that doesn&#x27;t need to be classified can be done on open systems and work shared, but it&#x27;s unfortunate to see so many wheels being reinvented when there are already really, really good tools to do these kinds of things but they are kept hidden.<p>We also have up-to-date indexed catalogs of exactly which tiles exist in which scale ranges to solve the problem you noted of trying to query and getting no data for poorly mapped regions. We can automatically retrieve the best data available and interpolate to the desired scale range instead. And rather than only getting the GeoTIFF, you can also get the raw altitude data intended for machine consumption, which is the main geoint use case for doing projections from sensor ephemeris to ground point.<p>I can see you also forked proj4js. This is of course also the kind of thing we had heavily optimized code for doing, since you have to project from lat&#x2F;lon to ECF in order to project from ground to coordinates in orbit, which are ECF (we had a need to do this before the open source libproj for C existed).<p>In any case, it&#x27;s unfortunate that you seemingly need to work for free doing this to get any attention and I hope you get a job out of it eventually. Sad to say I don&#x27;t think any of the big aerospace and intelligence players would give you a second look because of a basic requirement that everyone has a degree in engineering, physics, applied math, or computer science, but hopefully a more innovative, disruptive, and agile company will give you a look. I&#x27;m not sure you&#x27;d even want the work, considering many people probably find working for spy agencies to be unethical, but I&#x27;d have hired you if it was up to me.
scardycat大约 4 年前
amazing
pickelchips大约 4 年前
wow
ytdytvhxgydvhh大约 4 年前
Channeling Mitch Hedberg: There used to be hills here. There still are, but there used to be too.