See also smart_open: <a href="https://github.com/RaRe-Technologies/smart_open" rel="nofollow">https://github.com/RaRe-Technologies/smart_open</a> which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: <a href="https://github.com/RaRe-Technologies/smart_open/issues/579" rel="nofollow">https://github.com/RaRe-Technologies/smart_open/issues/579</a><p>Personally I wasn't too impressed with fsspec, though I am using it. I had to wrap files with Python's io wrappers to get acceptable performance, and found that different fsspec implementations still have significant differences that you can't ignore. They don't seem interested in supporting or documenting use cases outside of Pandas and Dask.
fsspec is used in dask (<a href="https://github.com/dask/dask" rel="nofollow">https://github.com/dask/dask</a>) if you want to see it in action.<p>The author of fsspec also created fastparquet (<a href="https://github.com/dask/fastparquet" rel="nofollow">https://github.com/dask/fastparquet</a>), a native Python implementation of the Parquet file format.<p>I'm really appreciative for Martin's extensive contributions to the PyData ecosystem.
How does it compare to pyfilesystems [1]. In our startup using pyfilesystems as a generic interface for our webapp which makes it cloud agnostic and the app can use amazon, gcs, NFS, gzip, tar, azure or regular file system.<p>PyFilesystem is a Python module that provides a common interface to any filesystem. It supports many filesystems [2].<p>[1] <a href="https://www.pyfilesystem.org/" rel="nofollow">https://www.pyfilesystem.org/</a><p>[2] <a href="https://www.pyfilesystem.org/page/index-of-filesystems/" rel="nofollow">https://www.pyfilesystem.org/page/index-of-filesystems/</a>
Our team has been migrating DVC to fsspec. We've even started working on creating fsspec compatible wrappers for SSH, Alibaba cloud, etc.<p>There were challenges indeed, and some storages will require more work (e.g. things like GDrive) but I would say maintainers are responsive and helpful.
One feature of a filesystem library I would like is to convert a filename into one that only includes permitted characters for a given filesystem type (for example, ":" and "?" are allowed in xfs but not exfat). For instance, mystring = fs.util.convert_name(u"What time is it?.txt", fs.fstype.exfat, "_") where "_" is the character to be substituted.
Just a heads up - fsspec has an (optional) dependency on s3fs which has a requirement on aiobotocore, which in turn is currently locked to an ancient boto3 version. Versioning fsspec[s3fs] in projects with other boto3 dependencies is a nightmare.
it's super convenient, but watch out for bugs and performance issues. it's caching mechanism, how it uploads to S3, and how it seeks unseekable files conceal pretty big performance bottlenecks.
this is what the web's upcoming File System Access[1] really enables, but stealtily so. it's advertised as an implementation, as the capabilities to interact with the filesystem. but it's also an interface too. JS having a interface for filesystems is going to be extremely great.<p>[1] <a href="https://wicg.github.io/file-system-access/" rel="nofollow">https://wicg.github.io/file-system-access/</a>