"So the point of my keeping a notebook has never been, nor is it now, to have an accurate factual
record of what I have been doing or thinking. That would be a different impulse entirely, an instinct for
reality which I sometimes envy but do not possess. At no point have I ever been able successfully to
keep a diary; my approach to daily life ranges from the grossly negligent to the merely absent, and on
those few occasions when I have tried dutifully to record a day's events, boredom has so overcome me
that the results are mysterious at best. What is this business about "shopping, typing piece, dinner with
E, depressed"? Shopping for what? Typing what piece? Who is E? Was this "E" depressed, or was I
depressed? Who cares?"<p>Joan Didion, "On Keeping a Notebook"<p>Apologies for the long quote but this is pretty much how I feel about all attempts at "personal analytics" so far. As other posters have mentioned, analytics are great for businesses who have their FOMs and TPMs and KPPs and whatever to maximize already, but on a personal level the best use of diaries and journals is to force yourself to realize what your goals are, not to help achieve them.
I've built this and have used it for years. Initially I decided never to make a product out of it so that I could allow myself to experiment and really just solve my own problems.<p>Its exactly focused on insight and de-cluttering the mind and finding focus. Being happier and more creative.<p>It really works and lead me to significant periods of peace and contentment and working on meaningful things.<p>In the last year I did a lot of work to turn it into a proper business, but got caught up in another startup which took up all my time.<p>I am planning on making it into a product with great design.<p>That's really the primary work now: crafting a design that is communicates the data in a clear way.<p>Some messy bells and whistles thing with all the data and graphs and features.... nope. Not interested. I want clarity.
I'm building a mobile app, called DidSum, that focuses on your #2 "Avoid Data Entry, but Make it Easy" (and putting an API around the data). It's almost ready for broader use.<p>I use DidSum to track running, blogging, eating healthy, sex, coffee consumption, and more.<p>DidSum allows you to define the actions you want to improve at. You can track publicly with friends and family and encourage/compare/compete with each other. You can also track privately (some actions I only track with my wife). It has basic analytics, which will get more powerful with time.<p>We're in a closed beta right now, if you're interested in helping test I'd love anyone interested in this thread's feedback: <a href="http://didsum.com/auth/register.html" rel="nofollow">http://didsum.com/auth/register.html</a>
> You need as much data as possible.<p>You need to separate the signal from the noise, and ramping up the noise isn't the solution. Quality takes precedence over quantity. Benjamin Franklin identified thirteen 'virtues', and those signals seemed to work for him.<p>> I propose a simple data entry framework: click a button, and that event gets tracked.<p>The problem with data entry is that it requires the motivation and initiative of the user, which is precisely what the user lacks. That's the reason people get personal trainers and life coaches: you need someone else to keep you accountable and motivated.<p>> Machine Learning Matters<p>Here's where I think almost all of these personal-analytic type applications go wrong: it's trying to solve a very subjective human problem with a very objective machine program.<p>Here's an alternative startup idea: a web application for life coaches and their clients, focused on providing them the tools they need to communicate better with one another. I'm betting that life coaches use email, text messages, phone calls, a calendar, and maybe a spreadsheet. They probably could use some help integrating all those tools and all the data they are collecting.
> If you consider the number of people that want to lose weight, quit smoking, or get better at something, I’m probably wrong about the market size. It’s huge.<p>Everyone wants to improve something (or everything!) about themselves. Most people have a sense of what that is, and what they <i>should</i> be doing. Stop smoking, go to the gym, get more sleep, eat better, spend less money, make more money, etc. A personal analytics system may help someone <i>optimize</i> their life from seeing the hidden details, but I claim that the steps to self-improvement for most people are fairly obvious. The problem in self-improvement is not lack of data, it's how to execute on the obvious known fixes (how to stop smoking, etc).
I've published some code and articles about this concept. My idea is to have something like:<p>friends <- [user.alice, user.bob, user.carol] # array of friends
friends_movies <- apply(friends, movies)<p>When a friend updates its movies everything is recalculated like in a spreadsheet. The core idea behind this is to end with emergent behaviour from the small scripts of millions of people.<p>To simplify the programming language everything can be written as an s-expression. I've published the parser (using o-meta) with an article here: <a href="http://blog.databigbang.com/parsing-s-expressions-in-c-using-ometa/" rel="nofollow">http://blog.databigbang.com/parsing-s-expressions-in-c-using...</a> more food for thought in:<p>i) Egont, A Web Orchestration Language: <a href="http://blog.databigbang.com/ideas-egont-a-web-orchestration-language/" rel="nofollow">http://blog.databigbang.com/ideas-egont-a-web-orchestration-...</a><p>ii) Egont Part II: <a href="http://blog.databigbang.com/egont-part-ii/" rel="nofollow">http://blog.databigbang.com/egont-part-ii/</a>
This is exactly the problem we're trying to solve with <a href="http://exist.io" rel="nofollow">http://exist.io</a>.<p>For the moment we're starting small and just trying to pull in activity tracker data (Fitbit, Jawbone, Withings etc) and some secondary services that provide useful quantifiable data - things like Foursquare checkins, Gmail, task management tools, productivity trackers, etc. It's hard to interface with every service when you're writing custom API clients for each of them. And on the other hand, if you try to dictate a common standard without any clout to back it up, you just end up with <a href="http://xkcd.com/927/" rel="nofollow">http://xkcd.com/927/</a>. We're working on an API to export all your data too, so people can build visualisations and apps on the aggregates of their data.<p>Tracking things has to be as frictionless as possible, so I'm wary of manual tracking unless you're already quite committed, or it takes mere seconds - the interface in #2 looks like a good way to approach this. But we're avoiding this for now.<p>The other thing we strongly agree on is #4 - you need insights, not just data! Numbers alone are of limited value, and a lot of people already into Quantified Self end up with "number fatigue" where they're no longer motivated to put in the effort to track things, let alone work towards a goal, just so they can see the number go up. People need to qualify their data and make it part of a narrative about their life. They also need actionable feedback, which is where we're trying to focus on providing value - if you're trying to be more productive, on which days is that working, and what are the factors behind it? We'll try and tell you so you can focus on that. Same process for becoming more active or anything else.<p>It's a big problem to solve, but so many possibilities too.
I think the oversimplified nutrition analytics is actually killing one of the biggest potential sources of insight from personal analytics. The key then is having a large, accurate database of nutritional information so you can see which fundamental ingredients a users is constantly ingesting.<p>Other than that, I totally agree. Someone please make this app-- then let me do the machine learning for you. :)
What companies have been at least mildly successful in Personal Analytics? From where I am standing I can't think of any.<p>Yes Fitbit is a great pedometer that gets people to walk more but last time I checked they weren't recording that much data and linking to connected scales is useful but its barely scratching the surface.<p>One issue with Personal Analytics is we need more data and it needs to be EFFORTLESS to record. The ROI is only high enough when the effort is very low. This is a challenging problem that requires "true innovation". I don't think a tracking app on my mobile phone is going to be the answer.<p>Another issue is the ability to improve some facet of my life with this data(weight, sleep, etc). Even if I can correlate drinking coffee in the evening is causing me sleep problems unless I can change my habits its not useful information. We need to be able to change our habits and processes based on this data. There are scientific strategies to get people to change habits and simply telling them they need to walk more or eat less do not pass scientific muster. Trigger -> Habit -> Reward. If we need to change Habits we must replace them something. If I need to stop drinking coffee then I need to replace it with something.
4 months after our 2nd child was born my wife still wasn't getting enough sleep. Part of that was the child's sleeping pattern and part of it was her not being disciplined with sleep (not taking naps or staying up even later). Of course my telling her "you need more sleep" fell on deaf ears. So I recommended she get a fitbit since it claims to track one's sleep. It was a big reality check to see on her computer in a big red circle that she was only getting on average 4hrs of sleep each day. Just seeing it on the computer meant she could no longer ignore the truth. It's the same as a doctor's prognosis vs. a friends opinion. The former carries a lot more weight. She started getting more sleep the following week.<p>If someone can create a system that nearly automates the tracking of what you eat you'll make gazillions and help a lot of people along the way. Maybe a bluetooth food scale in conjunction with a tablet that can do "facial recognition" on a plate of food to estimate what I'm eating and the carb/protein/fat/calorie breakdown. Then based on my physical activity for the day and calorie consumption it could recommend I skip desert or have a carrot instead.
How would tracking all this be useful to you? Why would you pay for it? The reason why tracking is more common in business is that knowing the metrics gives you guidance on how to be profitable. Therefore you can optimize all your metrics to paint that picture. With personal analytics, there isn't one thing that delivers that kind of ROI, even if we're talking about the investment from inputing the data alone.
I feel that the most important part of this is having clear data ownership rules to assure users that they actually own their data. That is, they can take it in a machine-readable format whenever they wish, and with it remove the right of the company to continue holding the data. I am very much in favour of stepping up personal analytics, but there's no way I'm going to just hand over some of my most personal data and hope that the company doesn't use it as the ultimate lock-in tool.<p>If a web platform is developed to provide storage and an API, the only way I would trust it is if it were completely open source and users were encouraged to run their own nodes (as well as supporting easy data migration between nodes).
<a href="http://jcutrell.svbtle.com/a-simple-and-flexible-selfquantified-aggregation-idea" rel="nofollow">http://jcutrell.svbtle.com/a-simple-and-flexible-selfquantif...</a><p>I've written a response, but I suppose I am hellbanned for new posts. No clue why unfortunately.<p>The basic idea: prompting is of great importance in this field. You can't get GREAT data through passive measures, but you can get much more consistent data if you consistently prompt. (This is my personal method, at least.)<p>[edit] It's not so much of a response as it is related - a way to capture data about self. Not so much related to the startup viability discussed in OP's article.
"Every piece of content created: mobile camera roll, instagram, facebook, vine. Every piece of content consumed: amazon, itunes, netflix, spotify, rdio. Every place you go. Every dollar you spend. You won’t start with 100 services, but you’ll get there soon enough. Some people will be obsessed with adding more data"<p>And therein lies a potential headache. What if the API for those 100 services break? What if just 10 of them change their API every month? Are you going to update them? And oh btw, you're gonna keep doing this before you actually have paying customers.
I'm slowly creating something similar to learn rails.<p>I want to track how much spend, consume electricity or gas, what are my goals.<p>The problem I found that I forget to enter the data into the app so it is fairly sparse. I can only get meaningful data on a monthly base.<p>I imagined it would be capable to compare gas (heating) consumption with the weather on the same day but it is not really working (without some automatic data input, that I don't have).<p>Among others I though about adding mood tracking, attaching 1-3 picture/day so when you want to watch back on your vacation you are not overwhelmed with 5000 photos, tracking locations. I would like to think about it like a social network where I am alone or something like this. (Codename: Track My Sheet :-) )<p>Showing trends, adding watchers like "you now need 1 year 3 months on your Y Account to reach Z amount at your current rate +Q USD / day, but these kind of things are faster with an excel sheet. Here is a dashboard mockup: <a href="http://snag.gy/OozTi.jpg" rel="nofollow">http://snag.gy/OozTi.jpg</a><p>It is similar to the reportr.io app, actually I started mine on the same week before I first saw it on HN. :-)
I wrote a chrome extension to analyze the websites I visit on certain days and track how long I spend looking at each one and graphs it. It's called melytics and it is pretty buggy but it does the job.<p>Honestly I think this is too much data to make use of. I'm just one person. How much meaning can you get out of what I do? If it is a lot, do I really want that platform to exist on the web?
I worked on a concept for this for a while. I ran into the following contradiction I couldn't solve:<p>If the data entry is quick and easy, then I'm not gathering empirical usable data. If I try to gather good data the user will become annoyed.<p>I think data gathering apps work for a small scope when the user is engaged enough to provide the data. Fitness or body building for example. Or sports car races etc.<p>This was making me think that maybe a generic statistics app with user generated values/measures/timers might be interesting. But once again, non technical folks probably aren't interested in statics too much.<p>Maybe gamification and a great UI is the answer here or maybe one should start with a specific problem (workout, drinking, sex or whatever) and broaden the scope as he analyzes user interaction.<p>I would give such an app a shot, but I'm not sure if I wouldn't forget about it too quickly. Just as I forget about using Lift (the app) and other interesting stuff.
I'm the founder of <a href="https://zenobase.com/" rel="nofollow">https://zenobase.com/</a>, another startup in this space.<p>Completely agree that "graphs without the ability to manipulate data are worse than nothing"--it's hard to gain much insight without being able to drill down and filter the data. Simple questions such as "how strong is the correlation between room temperature and sleep" end up having to be refined e.g. with "ignore weekends" and "correct for different amounts of physical activity".<p>Being able to "get as much data as possible" sounds good, but garbage in, garbage out applies...<p>Finally, I don't think it's possible to create a single mobile app that suits everyone, which is why Zenobase has a flexible, generic data model and an API.
In case anyone hasn't already seen this amazing article by Stephen Wolfram about his own analytics...<p><a href="http://blog.stephenwolfram.com/2012/03/the-personal-analytics-of-my-life/" rel="nofollow">http://blog.stephenwolfram.com/2012/03/the-personal-analytic...</a>
Personal analytics is indeed a very interesting topic (cool article from stephen wolfram: <a href="http://blog.stephenwolfram.com/2012/03/the-personal-analytics-of-my-life/" rel="nofollow">http://blog.stephenwolfram.com/2012/03/the-personal-analytic...</a>).<p>The biggest challenge imo is privacy and security: How do you safely collect and process everything (from emails to texts)? While I could implement something for my personal use, would anyone give access to all their communications to a third-party service? I wouldn't.<p>Maybe there is some better way to collect this information (just metadata? do some analytics and store results instead of raw messages?).
My biggest issue with personal analytics is storage. I want to track everything, from browser history to keystrokes, but I don't want to share this information with a third party. I think solving the data problem is the first hurdle.
Quite an ambitious idea. I started small (by focusing on personal reading analytics): <a href="http://www.clarelegere.com/profiles/1" rel="nofollow">http://www.clarelegere.com/profiles/1</a>
There are already too many of these services that don't interoperate with each other, or sources of data in an open way.<p><a href="http://activitystrea.ms/" rel="nofollow">http://activitystrea.ms/</a> is a good first step in opening some of this stuff up for consumption.<p>The way it is now, each front-end service has to do a ton of work for each data source they want to integrate with. There should be services that translate data into a shared format. And then front-end services can focus on their idea instead of having to parse all the data.
I think social media platforms are slowly building towards generating massive personal data. I thought with the advent of Facebook timeline, user events (both macro and micro) would be well documented in an aggregate consumable form.<p>Where do you get your revenue for personal analytics like this? Therein comes super targeted ads with pinpoint accuracy, using predictive analytics at a personal level. If you buy a brand of milk only on Saturdays, rivals have a higher incentive to strut their wares on next Saturday.
Sorry for the following unsolicited ad. I work for a company in the Washington, DC area that is making almost this exact product. Reading this article was like hearing a stranger talk about what I do at my day job.<p>In addition to the Quantified Self crowd, we are also (for now, primarily) focusing on the educational space. We are currently hiring developers. Most of the code is Python, but we need front end help also. Contact me at personalanalyticsdc@drbacon.33mail.com if you are interested in learning more.
I've been thinking about this for a while. It seems almost impossibly easy and yet no one has really done it.<p>Maybe it's because the consumer startup crowd and the data science crowd don't overlap enough. If <a href="http://prediction.io" rel="nofollow">http://prediction.io</a> supported time series data you might see a few of these products pop up overnight.
This sucks.
We don't need some new "exciting" personal helper shit, what we need on hacker news is exciting object recognition algorithms, better nature language processing module, high accuracy text detection and recognition in natural scenes.
Why are there so many people building crappy stupid things that no one really need?
I remember starting a company several years ago focused on using advanced analytics to improve personal decision-making. This idea morphed over time to a specific use: budgeting. Visualization and making the process quick and super easy are of the utmost importance for "personal analytics".
I’ve enjoyed reading your post but I’d be more interested in seeing how you go about fleshing your ideas out. I personally keep a similar list but have trouble taking the ideas from rough works to completed business plans. I’d love to see what processes/frameworks other people use.
The quantified self movement should be far more extensive. I am linking to our blog again!<p>The Secret Digital Ocean: The Data We Aren't Addressing<p><a href="http://thoughtly.co/Blog/thesecretdigitalocean" rel="nofollow">http://thoughtly.co/Blog/thesecretdigitalocean</a>
Many efforts underway already. To add to the list
<a href="http://fluxtream.org" rel="nofollow">http://fluxtream.org</a>
<a href="http://lockerproject.org/" rel="nofollow">http://lockerproject.org/</a>
I built a website www.selfstats.com with this sort of thing in mind.<p>Ask yourself questions
Receive email everyday with the set of questions
Reply to email with answers<p>Login to your dashboard for an overhead view (charts, lists, etc).<p>An API and Mobile support are in the works.
Maybe the slow carb diet is a bad choice since it doesn't work... always results in bouncebacks (2 data points for you; 1 for me). Maybe only useful for a photo shoot or weeks leading up to a beach trip..
I'm basically working on that idea, too. It's built API first and the first iPhone App is almost done. <a href="http://goaly.co" rel="nofollow">http://goaly.co</a>
Lots of us working on this problem. We're focused on the insight and machine learning pieces.<p><a href="https://www.fitoop.com" rel="nofollow">https://www.fitoop.com</a><p>Send me a note for an invite.
My startup sympho.me is working on this challenge. Trying to help people get a holistic view of their lives though quantified self.
Please send feedback:D
Here's yet another (mine) attempt of addressing this problem.
<a href="https://pokelog.com/" rel="nofollow">https://pokelog.com/</a>