Perkeep: personal storage system for life

<a href="https:&#x2F;&#x2F;perkeep.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;perkeep.org&#x2F;</a> is the new(ish) name for Camlistore, created by Brad Fitzpatrick and with a lot of active developers.<p>From the home page (rather than the linked overview):<p>&gt; Perkeep (née Camlistore) is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era. Data may be files or objects, tweets or 5TB videos, and you can access it via a phone, browser or FUSE filesystem.<p>Things Perkeep believes:<p>+ Your data is entirely under your control<p>+ Open Source<p>+ Paranoid about privacy, everything private by default<p>+ No SPOF: don&#x27;t rely on any single party (including yourself)<p>+ Your data should be alive in 80 years, especially if you are

I&#x27;ve recently started using Fossil[0] to archive all my personal data. It works rather brilliantly. Technically you can use any VCS but Fossil is unique in that the entire repo is a single SQLite db, so it&#x27;s very easy to backup and restore. Not to mention the web UI to have a quick glance before checking out any files. Even better I can sync flawlessly between multiple hard drives and computers. I&#x27;ve a few separate branches for Docs&#x2F;Photos etc. I checkout the related branch and just add more files whenever needed. After files are added to the repo, I just remove the working copy. There are some limitations though like files larger than 2GB aren&#x27;t supported.<p>[0] - <a href="https:&#x2F;&#x2F;www.fossil-scm.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.fossil-scm.org&#x2F;</a>

TDLR: This is a content addressed data store similar to IPFS (although this project is older). You can configure one of several backends such as local file storage, S3, SSH, etc. It includes an organization system based on tags, and other meta data. You can construct a fuse filesystem representation based on a query. A web UI exists allowing exploration of existing files, uploading, etc.

I&#x27;m looking for something like perkeep, but with the ability to add (scientific) metadata. Oftentimes when doing science for the university your research fund is attached with clauses that obligate you to store all data of your research for a timespan of 10-20 years and to do (who would have guessed) scientific research - which entails saving information with every data point: When was the data obtained, how was it obtained, who generated it, for which experiment, what&#x27;s the copyright on this, is it anonymized, pseudonymised, is it connected to any other research, what&#x27;s the doi&#x2F;arxiv&#x2F;ark-id connected to it,....<p>An archive where you drag and drop your files that can upload everything to a s3 storage (no not amazon s3) and tag metdata to it would be a dream. Right now there is no good solution for this and in the beginning I took a deep look at camlistore and hoped for a solution in it. (I looked at upspin, ipfs and other solutions as well). If someone as a solution for this or if perkeep could be expaned (or has the option somehow hidden somewhere) I would be very happy if somebody could point me in the right direction.

It seems weird that deletion is prohibited. As we grow as people, sometimes we no longer want to associate something with ourselves. A photo we don&#x27;t want to remember, for instance. This feels like an unnecessary restriction.

The bottom of page says last updated in 2013, but the name has been changed and the latest version does seem to be 0.10. This was previously called camlistore.<p>Is it still the case that you can&#x27;t delete anything? Although rarely needed, that seems like a showstopper these days. Irreversible actions are bad UI.

Is there a user guide anywhere?<p>I&#x27;m having trouble finding one. The &quot;Getting Started&quot; page just says &quot;run the daemon&quot; and not much more. There are pages on how to set the many configuration options.<p>What if I just want to use Perkeep, or find out what the experience of using it is like? Is there a friendly walkthrough or tutorial? Or an introduction to the concepts one needs to understand as a user, not as a developer?

Looks like a pretty interesting project, and it&#x27;s been consistently worked on for seven years, which is definitely something:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;perkeep&#x2F;perkeep&#x2F;graphs&#x2F;code-frequency" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;perkeep&#x2F;perkeep&#x2F;graphs&#x2F;code-frequency</a><p>Anyone have a testimonial from the perspective of a user or hacker on it?

It&#x27;s really worth thinking about the idea of not having filenames by default. They give a good example: if you take photos you don&#x27;t want to name them, instead you want automatically collected metadata (like creation time) and some UI for easily searching by that metadata.<p>So it&#x27;s basically a correct idea, but I want to know what is needed to make it work.<p>I remember the Palm Pilot tried to do this by pretending not to have files, and having &quot;databases&quot; instead. The result was that the palm-pilot database just became an obscure, inconvenient file format.<p>On the other hand, modern big giant internet storage service do a pretty good job of &quot;freeing&quot; you from filenames, letting you get photos, docs stuff.<p>On the other, other, hand, there might be something about the <i>personal</i> aspect of perkeep that makes it more like the palm-pilot.

There was some discussion earlier about the former Camlistore, and how it differs from the Upspin project in a couple threads here (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13700492" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13700492</a>) but maybe the authors can chime in here and restate what the different usecases would be between Upspin and Perkeep -- it seems like they are targeting the same audience: personal users wanting to back up data. The biggest point of emphasis is that these are <i>not</i> to be used for enterprises, and using them as such would be an anti-pattern, but curious as how the breakdown goes after that.

Where does it store the data?

The thing about files is that they are never going away and they are simple like a rock. If you want to avoid any type of lock-in ever, just store things in files.

Past discussion[0].<p>[0]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15928685" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15928685</a>

&quot;You are in control of your Perkeep server(s), whether you run your own copy or use a hosted version.&quot;<p>Can the perkeep server be an SSH&#x2F;SFTP login ? Or is there a server side component that would need to be running ?<p>I&#x27;ve thought in the past about the intersection between (camlistore) and rsync.net but it&#x27;s not obvious what that looks like ...

I&#x27;ve been looking for a system that lets me track replication of online&#x2F;offline data, as well as a search tool + format obsolescence report on files. I once started writing such a thing using Python + SQLite. It&#x27;s kind of trickier than it seems.

This is in the same spirit as some OSS work I did a few years back, to enable similar scenarios<p><a href="https:&#x2F;&#x2F;github.com&#x2F;briangu&#x2F;cloudcmd" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;briangu&#x2F;cloudcmd</a>

This looks like an article from 2013. &quot;Last updated 2013-06-12&quot; is in the footer

Hmmm how is this different from Box&#x2F;Dropbox etc?