A lot of people are leaving comments critical of his goal of beating the market as playing a zero-sum game, just pushing wealth around, having no value for society, etc.<p>This strikes me as dubious and high-handed, to say the least.<p>First, as to the value of the endeavor. He is trying to, in his words, "come up with an algorithm that finds pockets of profitability in a cloud of probable randomness." If he could really do this, it would create capital flow, the ability to borrow, and so on, for people that could turn it into wealth that would not otherwise have access to that wealth. Financial markets serve a real function, and beating the efficient market hypothesis would improve their ability to fulfill that function.<p>Second of all, he's extremely interested in the problem. He's obsessed with modeling complex systems and sees this as a sort of holy grail. Surely the fact that he's interested in it and values it counts for a lot, especially if you grant the above point, that there is a value to what he's doing. He has his reasons for being interested in it, and it's a productive endeavor.<p>I think it's false that his lack of progress, or the reasons that he's frustrated at his lack of progress, have to do with these aspects of his goal. It's more that he's working on an extremely hard (arguably insoluble) problem where it's hard to see incremental gains. This is a problem that ambitious people in many different fields face.
<i>After eight years, I have nothing concrete to show for my efforts.</i><p>So sad, so true. This is what happens when you play a zero-sum game. Even if you win, someone has to lose.<p>Here's an idea. For the next 8 years, why don't you do something that helps others by making the pie bigger for everyone instead of just trying to game a bigger piece of a smaller pie for yourself.<p>When you do for others, you <i>always</i> have something concrete to show for your efforts. It may not be what you expected, but it will definitely be something worthwhile, I promise you.
Question: has <i>anyone</i> managed to do this? (eg. do hedge funds do this, or do they combine algorithms with their own ideas and quasi-insider knowledge, plus massive amounts of timely data - inaccessible to an individual).<p>My old supervisor (ACM fellow) comforted my failure to create AI by saying that lots of other clever people hadn't done it either. I have a feeling that you are attempting something <i>more difficult</i> that AI - ie, algorithmic inference <i>without the data that your competitors have</i>. Because of their better information, they will find a regularity (eg. 0.01% dip on Wednesdays) quicker than you - and trade it away.<p>It would be awesome if you could do it though - forget the philosopher's stone aspect of gold-for-nothing, this would be a very cool intellectual achievement.<p>Idea: focus on small markets/exchanges (unpopular metals, unpopular countries, even non-financial markets, like eBay) that don't attract the attention of big players. Easier to find arbitrage there; you can make some actual money, and <i>then</i> move onto a bigger market with your learnings, earnings, and confidence. Sort of a niche strategy.<p><i>Personal Note</i>: My AI research depressed me, and I (wisely) decided to shelve it. It's been incredibly more satisfying to work with others, to achieve doable goals, that benefit others. Fortunately, AI comes up in everything; and my current approach (of not doing AI directly) may well be the most effective way to approach the goal. It might be the same with you - both markets and inference are universal.<p><i>EDIT</i> So, Tantalus = low-hanging fruit + moving-goalposts <a href="http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus" rel="nofollow">http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus</a>
You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse.<p>I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints look really good. The problem isn't with the math, it's with an inappropriate application of the math based either on fundamental misunderstanding of extrapolation or on a fundamental failing of human psychology in not realizing that the model was built on the data, and not the other way around.
You've learned a lot, so you're obviously not a failure, but I think your post illustrates how draining a winner-take-all field like quantitative trading can be. You can never be sure if your wins or losses are real or just mirages caused by noise. Humans need the emotional encouragement that incremental success brings. Nothing get's me excited to work more than a good day of sales or a thoughtful comment from a customer. It takes a special type of personality to keep on persisting in the face of repeated failure. Certain start-ups also have winner-take-all properties. Anything that relies on network externalities or advertising revenue is more risky. The rewards can be big, but I prefer markets where the product's value isn't a function of the customer base and success is more incremental.
My mother was down on the floor of the NYSE the other month, and she got the opportunity to corner a trader who she noticed was relying heavily on algorithms.<p>She asked "What's the difference between a good trader and a good algorithm?" The trader said "nothing.If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful."<p>The issue here is the definition of "trader." For this guy, trader is a person who executes large trades for an investment manager. A trading decision is not the same as an investment decision, and in that vein many trader's motivations are simply to buy a desired quantity of a financial instrument as cheaply as possible.<p>If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful.
If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful.<p>Humans are probably better computers for figuring out which stocks to buy and sell to my mind, but there's certainly a place for algorithms if they are not too high-minded.
If you're doing what you love, you've already won.<p>I feel a similar way pretty often. I have a couple of visions of software I want to build that are far out of my technical reach. I've been prototyping them, on and off, for about eight years too, depending how you count.<p>They've all failed. I start with a flash of optimism, and code until I realize I was wrong. I then stop, and rethink things until I think of another guess.<p>This is exactly how great accomplishments happen. You may be working on an impossible problem, or you may be working on a 20 year problem. Or maybe you're lucky and it's a 10 year problem.<p>But even if your goal is impossible, it's very likely that if you work on it for 20 years, you <i>will</i> solve a 20 year problem. It might be slightly askew to what you thought you were solving, but it will happen. It's almost inevitable.<p>Regardless, I wholly support what you're doing. Keep going. And read widely... explore... make sure you're playing in the margins of your field, and exploring odd projects far outside the scope of your work. It's those places that you will make that accidental discovery... that in all these years of trying to "beat the market" you made a fundamental discovery in artificial intelligence, or network theory, or whatever.<p>And yeah, if you're doing what you love, you've already won.
Many smart people have tried to "beat the markets" and failed. From what I've seen of it, this sort of activity is at best a form of numerology. Also, in trying to move money around slightly more efficiently than the next market analyst you're not really generating wealth as such, and your ideas and activities will leave no lasting value for society.<p>You may not be a failure, since the skills acquired in analysing trends can be utilised in other areas. The process of having promising looking preliminary results, only to find that they are illusory is familiar to many scientists.<p>Ultimately it comes down to what you really want to do with your life. Is your life only about moving money, or something else?
Seems like it'd be a better bet to apply all your skills to creating value (wealth) instead of just trying to push it around a smidge more efficiently.<p>In other words: find a real problem and solve it, increasing global wealth in the process, and collect the economic reward.
I suspect you could get a pretty good job as programmer for some hedge fund. Not only would it leverage your command of this domain, you might learn a few things that will explain why it is difficult to impossible for you to win in a vacuum.<p>There are social engineering aspects to this game that you have not been exposed to yet. Also, you may just not be sophisticated enough, or you have pursued a dead end approach. This is a very rich field of study.<p>If it's your passion, I say stick with it, but expand your circle of knowledge and influence.<p>And screw all this 'zero sum' crap, capitalism is the least zero-sum game humanity has ever come up with. And speculation, market-making, betting on winners & losers is one of the most important driving forces of capitalism.
"He[Dan Zanger] holds the unofficial record in trading stocks by turning $11,000 into more than $18 million in 18 months in 1999-2000. He grew that to an incredible $42 million in less than two years and has the tax receipts to prove it."
<a href="http://www.investopedia.com/articles/trading/04/082504.asp" rel="nofollow">http://www.investopedia.com/articles/trading/04/082504.asp</a><p>"Williams won the 1987 World Cup Championship of Futures Trading from the Robbins Trading Company were he turned $10,000 to over $1,100,000 (11,376%) in a 12 month competition with real money. Ten Years later his daughter Michelle won the same contest"
<a href="http://en.wikipedia.org/wiki/Larry_R._Williams" rel="nofollow">http://en.wikipedia.org/wiki/Larry_R._Williams</a><p>Somebody has some sort of algorithm.
<i>I find myself excited over the preliminary results of my increasingly sophisticated simulations </i><p>This. The vast majority of the money being made in the market is being made by people with what you may consider a laughable lack of "sophistication".<p>Start simple. Complex modeling of stock market behavior only makes sense if you think the stock market has a complex underlying structure.
The term "complete failure" is more revealing about your draconian idea of success. You should take a look at your own words : "they have never found their Holy Grail" [therefore, they joined the ranks of Academics as a testimony of failure].<p>If you're running after the Holy Grail, you have no choice but to be a hero. What kind of pressure is that ? In fact, most of the answers are in front of you, in the way you express yourself.<p>For instance, the myths of the Holy Grail and Tantalus are ones of insoluble challenge and dilemma. Therefore, when you are wondering if the Holy Grail exist, where it is, and if you'll ever reach it... You are probably asking the wrong questions.<p>Instead, you should ask yourself : "Who am I ?". Do you want to be rich, or do you want to "beat the markets" ? That's 2 different things, one is a mere consequence of the other. Where is your priority ? As it's not clear from your writing, I suppose it's not clear in your mind either.
Look on the bright side. Many people dedicated their lives to the search for a process to turn lead to gold. This seemingly fruitless search lead to all kinds of neat discoveries. You've learned lots of valuable things that you wouldn't have learned otherwise as well. Keep searching if you must but start benefiting from that knowledge.<p>The real secret about turning lead to gold is that it <i>is</i> possible, but by the time you have mastered the skill to do so, you no longer need to or want to.
I hate to be the bearer of bad news but, based on the comments you've made in your post, you are most likely a smart fool (as you have so aptly put). In essence, you don't even know what you do not know.
I've worked at top-tier Wall St institutions and have had first hand experience working with several of the largest quant hedge funds out there.
Here's my advice: if you truly enjoy markets and are sure you want to take this path, you will be much better served by spending several years working at an established quant hedge fund. This will open your eyes to the theory, processes, data, and techniques required to earn alpha consistently and in a risk efficient manner. It's a long road and that road is full of extraordinarily hard working and intelligent people that one has to constantly compete against. If you can handle this though, you will learn a bunch and at some point in the future you can strike out on your own again. At that point you will at least have a chance of success.
Good luck!
I tried to beat the market using all kinds of AI stuff off and on for a number of years. Learned a lot about AI techniques, and became a better programmer, which helped in my day job, but never got anywhere. It's a fool's game.<p>Besides, most of these HFT systems hold positions for only a few minutes or less. There's no way someone without a huge account, to cover commissions from trading so much, and access to a lot of expensive real-time data streams could possibly make any headway at it.<p>Not to say there aren't people who do make money at trading, but they are swing traders and/or people who've figured something out about the market that the market doesn't know yet, and thus has not revealed at all in the price movements of the stock. Since price movements are the primary input into a lot of trading algorithms, most trading algorithms are blind to these developments.
Only very few people have commented that, you are only 25! (I guess given the nature of the HN community :)<p>I don't think anyone can consider themselves a failure at 25, having spent 8 years learning a lot of programming, algorithms, data mining / machine learning and complex systems modelling.<p>It would be easier to think that if you hated what you were doing. But it sure doesn't sound like you do. So you just need to decide what you really want to be doing.<p>If it's algo trading, I concur with another poster that says go work for a hedge fund (or bank, or prop trading shop). It is super competitive, but they have the technology infrastructure and most importantly the capital. Getting in is not easy, but simply show them all your work (it doesn't work anyway, but is indicative of your skills and way of thinking). You will learn a lot, you may hate the people and environment, or you may love it even more. And then yes after a few years of experience you will most certainly be in a better position to go off on your own again (or do something totally different, by then you will really know if you like it or not!). Many top hedge fund managers / traders only started their own thing at 30, 35, 40, even 50... I know a dentist who became a prop trader. Anything is possible.<p>A PhD would be a great option IF it's for the right reasons. But if you want to do a startup (sounds like you might quite like the idea and you posted here on HN, so...):
- you already live on ramen, so no lifestyle change there;
- bootstrapping something can't cost more than losing money with trading algorithms;
- you already have many of the requirements: coding / technical skills, low-cost living circumstance, love to solve tough problems and a huge amount of tenacity in the face of failure and overwhelming odds;
- bonus: your interest in social systems modelling etc ties in pretty nicely with what's big right now and in the near future.<p>So if that is what you really want to do, go for it either alone (or find a co-founder), or find a small/medium startup to work for. To make the transition a bit more natural perhaps focus on ones that are data-driven and have machine learning / modelling at the core of their business. Think recommendations, systems modelling (www.flightcaster.com) and weather (www.weatherbill.com). There are many many examples of YC and other startup companies of this nature (many focused on the social network space).<p>Good luck in whatever you do decide to do with the next 60+ years of your life. On your deathbed you can post about whether you think you are a complete failure or not.
John,<p>95% of people would agree with you. But they (and you) are wrong. What is your end game? Do you want to be worshiped on HN or do you want a six figure job some where?<p>If you want either of those, you could just as well write up all you know spit it out in blog posts and free eBooks. Someone is going to recognize your skills and take it from there. At this point the worst thing you can do is keep all your failures to your self, because then no one can benefit from them.
His approach to this - the goal of beating the market - has been totally wrong. It sounds as if he's been at this completely solo - he mentions in the comments that he worked alone as he didn't want to share his algorithms.<p>He would have benefitted enormously from working in an institution that specialises in this kind of work (hedge funds etc). Being largely self-taught and having worked alone since he was 17 pretty much guarantees he doesn't know anything institutional trading firms won't know or have studied - employing as they do advanced mathematicians, physicists etc to work at the cutting edge of algo research and trading.<p>Not only has he missed out academically, he's missed out operationally. "Playing-the-game" is as big a part - if not bigger part - of trading than pricing/valuation. With the resources of a large institution you can learn to play the market in a way you can't possibly by yourself. This is in fact largely how they make money - consistently, year on year.<p>Thirdly - even if he did discover a pricing inconsistency or whatever, it's doubtful he'd be able to leverage it in as profitable a way that he would be able to as a firm.
Well, he might just not be smart enough. Large financial institutions and many smaller hedge funds reap countless millions from the markets and much of this comes from proprietary trading (well, all of it in the case of hedge funds.) There's a massive brain-drain into financial institutions of many of our best and brightest, all in search of making a fortune by the time they're 30. So on the one hand you have top students from top institutions applying their considerable intellect and training into applying advanced mathematical methods against an extremely complex problem. On the other you have a guy who tinkers around himself hoping after hope that the amateur (for that is by definition what he is) attempts that he makes at beating the market are going to some day work out and make him his fortune.<p>Unfortunately the odds are stacked against you, and those are compounded by fact that you have little or no opportunity to employ any serious leverage.<p>"I would have probably, or at least possibly, been wealthy by other means by now". At _best_ "possibly", I think. At 25 that is an incredibly arrogant statement to make.
Look to none other than Long Term Capital Management[1] back in the 90s to see how dangerous a game it is to either algorithmically predict or automate a thing as emotionally wrought and, at times, institutionally manipulated as the market.<p>[1] <a href="http://en.wikipedia.org/wiki/Long-Term_Capital_Management" rel="nofollow">http://en.wikipedia.org/wiki/Long-Term_Capital_Management</a>
The most important thing in your life is you and the brain you are carrying around in your head. If your experiences have made you smarter, you DO have something concrete to show for the years you've put in.<p>Entire countries have had everything destroyed, but have come back to be very successful. And this happened because of what they carry around in their heads.
It sounds like a trolling headline - he himself acknowledges the value of what he has learnt.<p>Some suggested in his comments that he should join a firm so that knowledge and experience could be shared towards finding the algorithm[s]. His reply to Henrik, "I’ve been wary about taking jobs at firms like that because of IP concerns. I always believe I am on the cusp of something great, and wouldn’t want to share my algorithms", is a factor in his progress, or lack of.<p>By working alone, he's more likely to earn 100% of nothing, when he could collaborate and earn a small % of a very large number.<p>That said, if he finds the algorith alone, then big congratulations and respect are in order for sticking to his convictions.
I've met people who claimed to make money with their algorithms, which really surprised me. Your post gave me an idea, though: What if instead of running your algorithm on the whole set of stocks, you create several random subsets. One of them has a good chance of being successful, at which point you can claim that your algorithm works and sell it for $$$. (I am not entirely serious, of course - but maybe some of the successful algorithms work that way?).
Even people with very high level math knowledge fails to predict the market (remember the two noble prize scientists who failed?).Since you're out of school when you were 17, I doubt you'll able develop complex schemes (unless you are a very very rare genius).
"Every few months, I find myself excited over the preliminary results of my increasingly sophisticated simulations, only to be disappointed a few short weeks later to find that I was simply wrong."<p>Isn't this why algorithmic traders "refit" their algorithms on a daily basis?
To beat the market you HAVE to predict what trades others will make in the future. It is easy to use math to make it appear to be a numbers game but that is just abstraction. The reality is that people buy and sell stocks for 1000s of different reasons.
Any algorithmic method of beating the market can only be successful if it is a secret and if its trades are of insignificant volume. Otherwise, knowledge of this algorithm will be factored into the prices, defeating it.
An interesting post. I'd be interested hear what kind of background you've got.<p>I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship.<p>I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your background and what you know. If you think 'a random walk down Wall Street' is correct, it isn't. Have you seen the derivation of Black-Scholes? Do you know what most high frequency trading is based on?<p>I'd also be interested to see what type of stuff you like to read.
This type of thing has been done before:<p><a href="http://en.wikipedia.org/wiki/FatKat_(investment_software)" rel="nofollow">http://en.wikipedia.org/wiki/FatKat_(investment_software)</a><p><a href="http://www.idsia.ch/~juergen/finance.html" rel="nofollow">http://www.idsia.ch/~juergen/finance.html</a><p><a href="http://www.technologyreview.com/blog/guest/25308/" rel="nofollow">http://www.technologyreview.com/blog/guest/25308/</a><p>I'm sure there are other examples. I think part of the problem is this guy thinks he can get rich quickly with just the right set of programming instructions.