The problem with the paper is not overfit. They claim to have run their simulation with out of band ("live") data. The <i>actual</i> problem with the paper is that we have no idea if their simulator is any good, which means that their result (89% return in 50 days) could be totally bogus. In other words, we don't know if the actual bitcoin exchange would fill their orders at the same prices (if at all) as their simulator does. A decent simulator for a high-frequecy strategy like this is not trivial because you have to incorporate all the exchange behavior (documented or not) into your simulator, and then you have to validate your results by comparing your simulated results to the results of some actual trading. The fact that they spent none of the paper on the details of the simulator makes me extremely skeptical.
This is short-term trading. "Every two seconds they predicted the average price movement (on OKcoin) over the following 10 seconds. If the price movement was higher than a certain threshold, they bought a Bitcoin; if it was lower than the opposite threshold, they sold one; and if it was in-between, they did nothing." I don't see them allowing for commissions and fees.<p>OKcoin, at peak, had a trading volume so high that it's generally considered to be fake - the exchange operators manipulating the price. What this group at MIT may have done is reverse-engineered the fake trade generation algorithm.
This seems like massive historical overfit, which can lead to arbitrarily precise fit, but no predictive capability.<p>Any model, if given enough parameters, can be made to match historical data to an arbitrary degree.<p>I also run several Bitcoin bots. I can tell you that slippage is not insignificant. If you make transactions every ~10 seconds and incur 0.1% fees each time, this is an extremely significant effect in aggregate. Also bid-ask spreads, while usually small, often aren't in periods of high volume.
Sometime in 2013, before the bitcoin prices exploded, I downloaded some bitcoin historical price data and ran symbolic regression on it with Eureqa. It came up with a formula that fit the observed data fairly well, and wasn't very complicated.<p>But when I extrapolated it forward a few months, it predicted the price would explode to unreasonable levels. I was disappointed and threw it away, assuming that it must be wrong.
I'm late to comment, but something which I'd like to point out is that this is done by the same team behind the Twitter trending topic prediction technique from a few years back, as mentioned also in the article [1].<p>When their Twitter technique was released, I spend a few weeks reading through Nikolov's PhD thesis (the advisor gets most of the the fame in the press articles but Nikolov's thesis has all the details) and trying to implement it in R. My observations at the time: extremely simple algorithm which would be shot down by most peer reviewers for being not very novel (the affiliation helps a lot here). That said, I believe greatly in pragmatism, and the approach was actually working well. What I did find out however is that their was a great deal of data selection and pre-processing involved making the approach hard to implement in a real-life, real-time setup. I get similar feelings from this work.<p>[1]: <a href="http://newsoffice.mit.edu/2012/predicting-twitter-trending-topics-1101" rel="nofollow">http://newsoffice.mit.edu/2012/predicting-twitter-trending-t...</a>
The paper states that the strategy was simulated with live data and makes no mention of slippage. I've never traded bitcoin so I'm not sure how difficult it is to get fills, but that along with spreads are non-trivial components of real trading.
They didn't include any discussion of:<p>1) execution (are they expecting to buy on the bid and sell the offer?).<p>2) commissions. They only made 3,362 yuan on 2,872 trades. A yuan is about 12 cents, so they are making 15 cents USD per trade.<p>A .1% commission would cost them roughly 5 yuan per trade, but they are only making 1.17 yuan/trade.
Success in predicting markets is measured in profit.<p>This research team should start a company that offers a service that allows users to deposit bitcoins, which the company then invests according to their alleged predictions, and then pay interest on deposits, and keep a part of the profit for themselves.<p>Doubling the initial investment one time is one thing, but this hypothetical company being able to double its investment every 50 days for years is something else. I doubt they can. A doubling every 50 days is x160 every year.<p>I think claims of being able to predict market prices should be met with great skepticism. Especially prices of easily traded commodities, including bitcoins.<p>The only proper measure of an ability to predict market prices is profit, because profit also measures the extent of the predictions: how much can you move the market (by trading according to your predictions) until you can no longer predict what will happen? Obviously, there's a limit. No one can extract unlimited profit from any market. So there definitely <i>is</i> a limit to how much you can earn from your algorithm. If you can earn 10% p.a. on an investment of maximum $5000, your algorithm isn't really worth much. If you can earn 1000% p.a. on an investment of up to $100M, your algorithm is <i>great</i>. But without knowing these figures we really only have a claim, seemingly a claim of them being able to make <i>a lot</i> of money, but choosing not to do so.
I fixed the article's headline image: <a href="http://i.imgur.com/QVgcgNI.png" rel="nofollow">http://i.imgur.com/QVgcgNI.png</a><p>If everyone began using the paper's strategy, would the strategy still work?<p>Also, the strategy seems less effective than portrayed in the news article. If you look at the "results" section, it seems like the profit flatlined shortly after starting, then had success due to some major trading event, then eventually flatlined again: <a href="http://i.imgur.com/CBjEjgo.png" rel="nofollow">http://i.imgur.com/CBjEjgo.png</a><p>Wouldn't it be more accurate to say "this strategy is effective under some very specific circumstances"?<p>Also, does anyone know how equation 4 was derived? <a href="http://i.imgur.com/vkx8ZEC.png" rel="nofollow">http://i.imgur.com/vkx8ZEC.png</a><p>It seems like the key insight of the paper, but there's no mention of where it originated from. Is it a common equation in statistical modeling? I'd like to learn more about it. Does anyone have any suggested reading or coursework I should study?
How many of the people who are bashing the paper in this discussion are machine learning experts? Especially the overfitting crowd - looks like emotional attachment to one's favorite topics is not really impacted by said person's overall education and expertise.
... for the next 15 minutes.<p>When you predict the future of a market, you change the future of that market. People start investing on the basis of your predictions and whatever opportunity for profit you found is closed. This is why HFT people iterate constantly and also why they put their servers as physically close to the market as possible.
This kind of innovation is cool, but it's a zero sum game. The bitcoin markets are already driven by competing bots. Their profits will be reduced as other bots iterate on their algorithms.
They should have made more money rather than publishing more quickly. It used to be possible to do these sorts of things to the stock market but when these sorts of regularities are discovered the process of exploiting them also eliminates them once enough money is being made. Heck, a major trading firm got started by noticing that stocks went down on the weekend (and of course they don't any more).
Can a HFT-knowledgeable commenter chime in on the viability of the Sharpe ratio here?<p>From a physics perspective, it appears that the Sharpe ratio of 4.1 is roughly equivalent to a 4.1-sigma claim that their algorithm is better than random trading. I can't check easily, but I'd guess that the movement of Bitcoin prices isn't normally-distributed (looking at the paper's time series suggests that there's more low-frequency power there). If so, I'd guess that a more robust measure of the claim's significance would show it to be less significant.<p>Put differently, I'd guess more than 1 in 15,000 random sets of 2872 trades (their number of trades) would yield comparable profit. Furthermore, a simple buy-and-hold would've yielded a 20+% return over the same period.
The fact is anyone can *Predict the price of bitcoin, but will you actually put your own money to trade?<p>Like what we did here, <a href="http://financeai.com/forex/btc" rel="nofollow">http://financeai.com/forex/btc</a> it does show some degree of correlation between sentiment and the price.
Ever thought how our big data overlords (Google, Facebook, MS, Twitter, etc) just need to check for correlations between their users' data input and stock exchange movement?<p>They own the stock "matrix".
Those interested in automatic trading of bitcoin using algorithms should check out <a href="https://cryptotrader.org" rel="nofollow">https://cryptotrader.org</a>