Ask HN: How do you deal with Rabbit Hole Syndrome?

If I had one piece of advice it would be: Descend the rabbit holes!<p>Rabbit Hole Syndrome is a symptom of having a curious, persistent, independent mind. If there were a drug that cured Rabbit Hole Syndrome, and everyone just worked on Shipping Their Products and Not Asking Questions, there'd be no one left to make the interesting discoveries that make the world better in the long run.<p>There are very famous quotes from Richard Hamming (see "You and Your Research") and Richard Feynman (see "Surely You're Joking!") about the importance of working on problems that seem trivial at first. Not only do they help you enjoy problem-solving for its own sake, but if you work on enough silly problems, then eventually the odds are good that you'll stumble upon something that other people will later think is really important. Incidentally if you read Thomas Kuhn you'll find that this tends to be how scientific revolutions happen: someone is bothered by some tiny little thing that doesn't quite make sense according to the prevailing theory, starts digging at the little cracks, and finally the whole system comes crashing down.<p>The major benefit of going down rabbit holes is that is opens yourself to serendipity: sometimes you'll turn out to be right about something for the wrong reason, or discover something that you later realize contains a solution to a seemingly unrelated problem. The more rabbit holes you've gone down, the more they start to connect.<p>In addition, frivolous research helps you develop a very "bottom-up" view of the world. If you know the details cold, then you are better able to see through high-level BS ("conventional wisdom") and evaluate things on their own terms. You'll find that regardless of what's optimal, most things are done a certain way only because they've always been done that way, and that regardless of what's true, most people believe things only because other people believe them.<p>Of course there are costs to all this -- in particular, "schleppers" will resent you for being irresponsible, and most people will think you're crazy if you ever explain what stupid little you've been working on lately. Which is why Rabbit Holers should try to should avoid actual responsibilities to the greatest possible extent and absolutely not care about what most people think about their work.<p>Anyway, to sum up, if you're lucky enough to be in a position to descend rabbit holes without impoverishing your family or bankrupting your company, I say go for it. It strengthens your most valuable asset (your mind), and who knows, maybe one day you'll discover something you can teach the rest of us.

You're driving along a road and you notice a pothole. You pull over to the shoulder, put your hazards on, open up the trunk, take out a reflective vest and tape measure, then you begin to analyze the pothole. You spend an hour analyzing the depth and formation of the pothole and determine that the cause is due mostly to poor mixing of asphalt. You call the town hall and learn that the road construction crew uses a Caterpillar BG500E wheeled asphalt paver. After some extensive research you determine that poor mixing occurs from the inferior design of the BG500E's auger and that upgrading to the BG600D with its improved auger would cause better asphalt mixing and produce paved road conditions less conducive to potholes.<p>By now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.<p>The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.<p>But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.<p>Stay on target. Make it to dinner.

To be honest I think that "it's not good enough until I can understand the class of problem and derive it for myself" kind of attitude is actually very positive. The problem with this attitude is if you do a "depth-first" kind of journey instead of refining the solution with successive passes.<p>So I suggest you to try to build the minimum viable working program ASAP. Along the way, write all the things you would like to improve in something like an Evernote note, just a few lines for every thing you want to address and make better.<p>Then if you have something working <i>ship it</i> ASAP, don't care about what other people will say about the sub-optimal parts of your work: many programmers trying to achieve perfection actually end with a mess of complexity that does not serve very well the purpose of the software, so there is little to be embarrassed for a programmer for shipping simple software.<p>In the second pass, refine every part with the same approach: find a solution that within the timeframe you have is better compared to the previous one, but will make you able to ship a new version.<p>Also when you face a problem, other than reading the existing literature, papers, and the proper way to do it, check if there is an intuitive solution that is comparable as a result (even if maybe not provable or not perfectly optimal) but much simpler to implement.<p>But IMHO the golden rule is: don't freaking care, ever, about what other people think about your work. Often perfectionism is just a form of insecurity.

Here are a couple of random suggestions:<p>1) Try the pomodoro technique, or some other form of time tracking. When you're about to take a side path which may or may not be a distraction, you can decide how long it'd be worth investing in it. Once the timer dings, you can stop and evaluate if it was worth it.<p>2) A few years ago I started repeating in my head the phrase "real artists ship". Embrace imperfect or partially finished solutions that are viable.<p>3) Keep a list of things that you'd like to investigate more. I've found that the act of writing down the idea lets me stop obsessing over it. Later when you revisit the list you'll be able to cross off the things that you thought were important, but turned out not to be. By delaying work on these items, you're able to better explore the most important parts of the problem you're solving, and so your future self will be in a better position to evaluate which areas need deep research.

I think you're not being ambitious enough in your side projects.<p>I'm like you. I write fairly vanilla production code, then I go home and work on alien technology. And I used to have the same problem as you: I'd get sidetracked and sidetracked, and somehow I went from writing a fart app to reading about type theory.<p>So I started skipping the fart app part, and started learning some more abstract theory. The nice thing abstract theory is that it's <i>abstract</i>---it's not incidentally connected to anything, so there are fewer places for you to get sidetracked.<p>So go sign up for a math course on Coursera, or learn the lambda calculus. They are so alien from your everyday programming experience that you won't have anything to connect them to---until you <i>do</i>. But then you'll be coming at the new topic in the direction of abstract to concrete ("a trivial application of x") rather than concrete to abstract ("there's a greater truth here and I MUST understand it!").

This is an experience problem. As you become more experienced, and see how you've wasted your time coming up with a "perfect" solution that ended up not mattering, then you'll be fine not having a perfect solution.<p>At some point you realize your best days are the days when you delete more lines of code than you write.<p>But that being said, chasing rabbits is what will eventually make you more skilled than your peers. Which is awesome, except now you'll have put yourself in a perpetual category of being paid less than your worth because you don't fit the same performance evaluation criteria as everyone else.<p>All anyone cares about is a) are you easy to work with, and do you b) get things done to contribute to profitability.<p>Continutally stressing yourself out by spending more time on problems than they deserve and eating away at your work/life balance does not contribute to a) or b).

&#62; I find myself striving for ideal and perfect solutions in parts of my work that might not matter much. Sometimes it's probably worth the time and detail, but admittedly, a lot of the time it isn't. It's just more fun and interesting to be "thorough."<p>This actually has a label: Maximizers vs Satisficers.<p><a href="http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More_Is_Less" rel="nofollow">http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More...</a><p>And a little something I wrote about it regarding programming languages:<p><a href="http://journal.dedasys.com/2006/02/18/maximizers-satisficers-and-programming-languages" rel="nofollow">http://journal.dedasys.com/2006/02/18/maximizers-satisficers...</a><p>I think the key is to "choose your battles", and be a maximizer where it really counts, and try and be more of a satisficer for the things that aren't so important.

As with others on this thread, I question the rightness of "curing" this tendency. But you can certainly improve the way you harness it.<p>Something that's helped me: make sure you have multiple rabbit holes available at any given time. i.e. several problems, any of which is interesting enough to tempt you. Then you can make reasoned decisions about <i>which</i> would be the most rewarding to work on, while still giving in to the temptation of rabbit holing.<p>Another benefit: maybe by solving one problem, you'll discover the other was totally irrelevant, or a special case of the other. (At least for me, "Turns out I didn't need this to be perfect" doesn't carry much weight, whereas "Turns out I didn't need this <i>at all</i>" is quite convincing.)

This happens to me often as well. Although, I am not certain I would describe it as something I need to fix. All my practical experience, going back 15+ years has formed this way.<p>As a recent example, I just spent a long time on a tiny plugin for CarrierWave (the Ruby/Rails file uploading gem). This took me through most of their source code (always good to read code, and I could probably PR on their project now), new Rails internals and techniques, and mocking and stubbing with RSpec (in the process, I turned up a bug with RSpec, which they promptly fixed; and I have a better understanding of mocking best practices and clean RSpec).<p>I do empathize with feeling I'm "not getting enough done/shipped" (if you feel that way?). To alleviate it, I try to cut corners and just get something out.<p>Time-boxing helps me do this -- "accomplish X in 1 hour". This doesn't happen too often, however. I know, like you, I prefer the learning itself; and, I view all my spent time as building experience and a critical mass to accomplish work faster in the future.<p>Also, I find pair programming helps immensely. I am sensitive to the other person's time, and thus naturally refrain from digression when pairing.<p>As for motivating myself to "schlep". I'm not sure what you mean -- menial tasks? I do those when I'm tired or having down time.<p>Finally, as in your case, I do this on personal projects and not on "work". That said, it can make it hard to get your startup and product launched - your own "work". Again, disciplined time-boxing helps. I have not mastered this, and find myself regularly looking for good time-boxing tools.

I write a ticket for someone else to take care of it. Even if someone else is only future me, it's enough for my mind to accept that it is taken care of and I can proceed with the really important stuff.<p>Most of these tickets get resolved as "doesn't matter / won't fix" two or three weeks later - but myself.

Catalog your rabbit holes if possible, and then review them before working on them.<p>What I mean is, if you see a problem to solve, and you are able to keep working on your current task and to solve the problem later, then do so... just note the problem. This will immediately make you much more focused.<p>Then at the end of the day, review your list of rabbit holes and try and determine which ones are necessary for the current project, which ones would be educational / you want to do, and which ones can be discarded.<p>Basically rabbit holes are a problem because they are long and narrow and do not offer an overview of the entire grounds, so before jumping down a rabbit hole force yourself to survey the big picture and to see if you can step over it instead.

You apparently enjoy the interesting stuff, and love the fact that there is actual practical benefit - though initially very limited - from doing something that's theoretically interesting. Letting that chance slip would feel like a waste.<p>It might help to limit going down the rabbit hole too much, by researching what could be improved without actually doing it right then. Save it in a list for later, do the schlep, then when things get bigger chances are you'll actually <i>need</i> to take on some of the challenges on that list. And the more useful they become, the more satisfying it'll be.

Kind of sounds like procrastination to me. You should have a clear goal, and work towards it, allowing no or few of these side-distractions to take you off the path. A real life deadline helps immensively as well.

What else do you enjoy? Do you derive satisfaction from solving problems for users? Maybe you're trying to perfect something that's less interesting to you than another problem on your plate.<p>If positive user feedback 'gets you off,' as it were, try releasing something that isn't as perfect as you'd like it to be and see if there's any measurable difference in sentiment. I bet there won't be. Do it a couple more times and hopefully it'll break the pattern.<p>For the other case, try time boxing yourself so you can get back to the more interesting challenge. Make that other thing 'perfect' if you won't.

Never happens to me. I'm always eager to finish, release and go on. Good enough for the users? Publish. Not so good? Publish. Some will bear it, the rest will swear it and I'll know what to fix.<p>I'd very much prefer to pick up an apple that isn't perfect than having no apple at all because of looking for the perfect one forever.

A couple possibilities. If you really enjoy doing these things for their own sake, you can treat your side projects as a hobby and stop beating yourself up about it. Spend all the time you want going down the rabbit hole, and enjoy that it provides a chance to think deeply, making something as close to perfection as possible, and be fully present in the experience of creating.<p>The other approach I'd suggest is at the complete opposite end of the spectrum. Start treating your side projects like a real business, and create some process that connects real incentives to them. There are harder and softer ways to approach this. You can quit your job and now you have the focus in your side projects that you need to ship to get paid. You could also keep your day job and set up some automatic funds transfers into a semi-liquid asset like a CD or 401k, such that you leave yourself enough to cover the basic expenses, but so that the marginal income from your side project becomes significant enough to be motivational. Revenue is a great feedback loop getting you to focus on what matters, and letting go of what really doesn't.<p>You could also bring in another person, perhaps as a business partner, or simply have one of your friends agree to call you up once a week and have you give them a report. The key here is to clarify and draw attention in your mind to the work that you're avoiding, and the outcome that you want. In this process, it's important to vividly color the positive outcome be emphasizing the real changes it will create in your life.<p>For example: "This week, instead of shipping my product which will enable me to quit working for other people, spend more time with my family, own the high-performance german cars I've always wanted, and visit 20+ countries a year, I instead spend 6.5 hours working on a sort algorithm, and 5.4 hours fiddling with the presentation styles on the country-list drop-down menu in my app."<p>I used all of the techniques above when I was starting my first company, and they made a big difference in helping me focus on what was important in getting to market, and not OCD-ing out on things that were, ultimately, minor details.

I think the key is to:<p>1. Set goals and fixate on them. Imagine what it will be like to have achieved the goal. Get excited about it and keep reminding yourself. When you're not making progress toward the goal, make yourself imagine the situation where you spend 6 months making unimportant things perfect and never achieve the goal. Imagine all the other goals you won't even be able to set because you're wasting your time.<p>2. Make other commitments. Make plans to meet a friend for drinks at 9pm. You only have 2 hours after work to get anything done – don't waste those 2 hours! If you can stay consistently busy, you'll notice quite quickly that not using your time effectively will lead you nowhere.

I've been very affected by this myself.<p>It happened the most while attempting to study research papers.<p>A lot of times I would be unable to finish reading what I had planned, because I had stumbled upon an interestingly-looking concept and then proceeded opening up another paper on that subject and so on.<p>And then, of course, having returned back to the original paper, I would have to re-start reading from beginning to freshen up my memory.<p>Terribly exhausting process, so I can perfectly understand your frustration.<p>The only solution I found was to un-clutter my computer 'work-space' as much as possible: close any non-essential apps, unplug internet cable and each time I have the urge to stop reading and go research a newly discovered subject, I remind myself that I am only allowed to do it after finishing what I'm currently reading.<p>Another helpful thing for me is to remind myself what I'm trying to accomplish with whatever I'm doing at that particular time and what my long-term goals are.<p>For instance, if I'm working on a project for a client, the goal is to get the work done as soon as possible and obviously get paid. I am not working on said project to primarily enrich my knowledge, but to make money.<p>I can use the time after the project is delivered to draw conclusions from the experience or do further research.<p>This may sound trivial, but it really does help to constantly remind yourself of what your goals are, it keeps you in check.<p>For extra effect, every time you have the urge to let your mind wander too much, try imagining the possible consequences of not completing your task (on time).<p>This can be particularly effective if you're doing client work. Imagine how embarrassing/unprofessional would have to explain to your client/employer that you won't be able to deliver on time because of something that you could've prevented.<p>Side-note: you have not provided enough info, but you may have obsessive-compulsive disorder, which is nothing to be ashamed of, but you can only 'solve' this with medication, so you would need to see a doctor.

Two ideas: (1) Find a partner for each side project, or at least 1 target user (real person you know) you commit to serving by a hard date. It's an uphill battle to maintain accountability without another human. (2) I draw a box on my whiteboard and write in it the essence of the project and only the 2-3 deliverables essential to achieving it. When I take on a new activity, I challenge myself as to whether it goes in the box or represents a branch from it. Forces me to visually acknowledge how I am spending my time (and thus define and stay to the essence of the project).

Discard the unnecessary. It isn't always easy to tell but there are often subtle signs around what is and isn't necessary. Unnecessary things usually feel sort of shadowy in your mind or require an additional effort that doesn't fit square into what you are doing. Think about what it's like to carry too many groceries into the house at once. Necessary things are solid and leap forward into your mind with evident emphasize. Be ruthless with your pruning of those unnecessary branches and your trip down the rabbit hole will be both smoother and more thrilling.

Great post. Thanks for taking the time to write this up.<p>I think you're probably not doing projects to help people, but instead projects that you think would be fun to do. If you start with a burning itch, or a person that needs help, you will probably find it much easier to stay on task. This is not to say that there's anything wrong with exploring interesting technologies, or thinking about "the right way" to solve certain problems you find interesting... It's just to say that when there's a burning need the interesting diversions tend to fall away.<p>Hope that helps :)

It's not necessarily that what you're doing is the interesting option; I'd say that it's the easier option instead.<p>Reading about interesting stuff is deeply satisfying, and so it's something you want to justify so you get to do more of it. Actually getting stuff done is hard.<p>It's a bit like why so many techies - myself included! - vastly prefer to code new app features rather than working on marketing or sales copy. One is has a definite end point, and we feel comfortable getting there.<p>The other one can seem like <i>work</i>!

If it's a side project, you have to ask yourself what kind of side project it is. If it's a business or something you want to turn into a business, just take any task that takes longer than an hour or two, put it down on a notecard or sticky or some digital equivalent thereof, and put them in a priority queue. If it's just for fun, don't worry about it--if you're not frittering away on things just to please yourself, why are you hacking anyway?

What happens when you implement a dreadful, obvious, hacky solution? Do you find yourself compelled to change it? Or do you not even get that far?<p>It sounds like at least part of your problem may be perfectionism. I did a bit of research on this a while ago, and it turns out there's a lot of literature on perfectionism and how to manage it. A quick look on Amazon under "Perfectionism" should bring up a few interesting books.

Just think pragmatically.<p>Whilst the desire to strive for perfection is in any craftsmen, ultimately what matters is getting the product to the customers and solving their needs.<p>Only after solving their needs should you start to indulge in the craft for what the customer cannot see, the internal quality.<p>I also sometimes describe this as "be lazy"... don't do anything that doesn't <i>need</i> to be done (for the benefit of the business and your customers).

I don't much have a solution to the rabbit hole syndrome, but at the very least I can point out that your rating system could be solved by a confidence interval made using a Bernoulli random variable, which is a useful and (fortunately for you) a relatively simple formula to derive and comprehend.

Actually, come to think of it, I don't know if my previous answer of "get more experience" solves this procrastination problem.<p>I think it's actually "have kids" that solves this procrastination problem. With having kids you get so many constraints on your time that procrastination is no longer an option.

I have the impression you are subject of a typical mild ADD syndrome. Explore that rabbit hole and you may learn alot about yourself and see what people do to handle this. An obvious solution is to just do what you like and you'll be much more efficient than the average person in that.

I think it's about finding a happy medium. You certainly don't want to cut this passion for learning &#38; consuming knowledge out completely but maybe you want go along the route of shipping your first solution and then improving incrementally based on further information.

Want to break the rabbit hole habit? Start your own business where you have to get things done. You'll either break the habit or go out of business.<p>BTW, going down the rabbit hole occasionally isn't a bad thing IMO. But if you go down every rabbit hole, it can slow you down.

Oh boy. In a sense I have the same problem. I've started so many projects but I never release them because I've always been afraid they're not perfect enough. It sucks I know. I guess that's why a lot of people propose the MVP, then continue from there.

as an engineer you have to realized there's not a best solution overall but a best solution for this particular problem. Each solution has pros and cons.<p>I think you spend time weighing each design because you're unclear on your final vision of your product. You haven't definitely answered what your values and needs are that you're trying to solve.<p>This may be a hard question to answer. This is why MVP are neat, you can get a product out to consumers quickly and use their response to develop values and a overall direction.<p>I find you if you think too far ahead, you don't leave any room for random disruptive opportunities that can occur in each step.<p>So know where you're going. Figure out and focus on the next step that gets you close to that.

At what time do you work on your side projects? I found out that if I wake up early and work on them before going to my day job I tend to be much more focused. That might be just because of the effort it requires to do so :)

Just remember that, most likely, not a single user will ever be execute this code.

This sounds like a good thing.<p>Over time, your rabbit holing will develop in a direction called a 'specialization.' Take note of which areas interest you and what areas have opportunity and rabbit hole that way.

Honestly? You don't care about your side projects. You care about the puzzles that make up your side projects. If you want this to stop, you need to find a side project you care about.

Short term hack: medication.<p>Long term hack: meditation.<p>How long with each, how much of each... that's going to take a lifetime to learn and figure out.

"It's not enough that we do our best; sometimes we have to do what is required." -- Winston Churchill

Possibly you are not that interested in the main topic? Possibly adult ADD?

I do it and don't try to stop, cause it is good to learn.

My mantra for this is "only solve problems".

Discipline.

I have been thinking about this a lot lately. It's been an immense problem for me and it's destroyed almost all my side projects.<p>The problem is that you're interested in the wrong aspects of your side project. You're interested in learning and solving the code problems that come with your project. You have to shift your perspective to the human side of your project. You have to learn to see the user's issues as the problem to be solved.<p>Let's use an "reviews" site as an example. Your users cant find reviews, that annoy them. What you're solving is not a programming problem; your problem is "make it easy for users to find reviews". That is your problem. Your problems is not what the best algorithm for ratings. None of the code stuff matters. These users do not care about code.<p>When you've shifted your perspective to the human side you'll start to see hacky solutions differently. A crappy algorithm for ratings isn't a poor solution, it's a perfect solution. You've solved the reviews problem. You have a perfect solution.<p>The problem is that most of aren't actually excited about the human side of our side projects. This is because we get excited about things that interest us, and what interests a programmer is usually programming. It's not surprising we choose projects with interesting code problems.<p>Start reflecting on the human/sociological side of things before you choose a project. You might be surprised at what you find. Projects that seemed interesting may suddenly become dull and the one's you thought were dull might suddenly become interesting. A human perspective doesn't exclude building stuff with a code focus, just make sure coders are your audience. You and your users just need to be excited about the same things.<p>There is this classic saying about "building something for yourself" resulting in the best products. As with most sayings, they left out some important information. They forgot to tell to you to make sure that what you're building for yourself is the same thing you're building for your users.<p>Make your goals align with your users. Otherwise, you'll find yourself trapped down the rabbit hole.<p>Some extra thoughts<p>Don't forget that all your tricks to solve programming problems work on human problems. For example, for a reviews site it's easy to phrase the problem as "make it easy for users to find the best reviews in the most efficient and enjoyable way possible while allowing them to simultaneously book and view and and and....". Break it down into it's simplest elements first, just like you would a programming problem. The simplest element of the problem is "make it easy for users to find reviews"; best reviews are a separate problem.<p>You're probably good at solving programming problems, breaking them down and being productive. You know all the 37signals posts, all the design patterns, and can quote re-work by heart. Use those principles you have learned and apply them to the human side. Everything your learned about productive programming applies to productive life.

I have like 20 ideas in queue I'd really want to prototype and they are of every kind... website, business, writing, drawing, mobile apps, artsy stuff...<p>Also sometimes feel I read way too much<p>One post that really catched my atention that's related was: <a href="http://www.ribbonfarm.com/2010/03/18/the-turpentine-effect/" rel="nofollow">http://www.ribbonfarm.com/2010/03/18/the-turpentine-effect/</a><p>I related to it in the way that I find it difficult to see myself as a super nerd technical robocoder, I need to create too, can't not think about creating and ~wholism~ in this because it's the reason I started to learn it in the first place, so I accepted that<p>So, well, I'll buy a board, outline stuff and start on them, maybe I'll do an Output Week in that I'll try cranking projects for a week and this is good because of learning and etc like other ppl said here, but learning how to focus is just as much important, so is actually getting-things-done... so it's like "there's no problem in running as long as you know where you're going"<p>I think the anxiety increases as you don't do stuff and the more you do the more you also learn what's worth doing, no? Before I tried setting out to do a complete project by myself I'd probably accept the invites of "Hey let's start this project, you code!" more, now that I know the work it takes I'm way more picky because I know it doesn't matter if I build a complete skeleton for this but then there's nobody up to even do a fucking design for it and I won't be willing too... so it rings "not worth it"<p>If it's something that's clearly not ever seeing sunlight, if it won't be at least instantly useful for me, if no one is buying/using it(in the future)... why do it? even a painting has to have a meaning and a purpose<p>Aren't the best actors the ones that are picky about work they'll take? Aren't the top industry guys who get teased by Facebook, Google and most promissing startups picky? I'm pretty sure they are... I'd guess it's because they know they are basically unable of half-assing stuff, so they need to choose well what they'll set out to do.<p>I'd say that's the positive version of people with this personality trait, the negative version of people with the same personality trait are the anxious, lost, starting-a-new-thing-everyday-and-never-finishing-anything guys... so IMO it's something for each person to work out