I'm building an automated stock trading system for my personal investments. The plan is for the trades to be based on data ingested through past stock performance, current trend, fundamental analysis, performance compared to similar stocks, and other data sources added incrementally.<p>There is a learning curve in which I'm developing the initial algorithms that will help gain an advantage over simplistic human investing with the computer providing a more deterministic risk-reward management system that it will apply. Once this has been achieved, the goal would be to increase the ROI and maximize the final take-home profit (minimize tax where possible to increase effective ROI). I'm validating algorithms with a back testing approach, currently obtaining data from Quandl.<p>I'm planning to host this on AWS and use a combination of Python and R. I haven't decided on the datastore yet, but I'm open to suggestions.<p>Does HN have any suggestions or any tips from someone who has attempted to do something like this in the past, if so, please also briefly describe your past experience and current position with the project?<p>Thanks!
This is a bad idea.<p>The "simplistic human investing" you're talking about probably accounts for 0.01% of trades. The vast majority of trading these days is algorithmic, and the HFT guys will eat you for lunch.<p>What you won't see in back testing is how the market reacts to your strategy. The market isn't something that stays static: your activity changes things in subtle ways, and I guarantee there are people smarter than you who have designed algorithms to detect and exploit trading strategies such as yours.<p>Algo trading is very dangerous if you're not coming from an investment finance background. Just be very careful.
I once set down this path but stopped.<p>Your big problem is that the future market doesn't look like the past market. There is a statistical error inherent in excessively curve fitting past data. You could develop a system which says you'll return 25% per year and then lose 100% in he he first six months. So don't fall into that trap of excessively back testing.<p>What you should be spending time is your money allocation algorithms - the very same set of trades can return high amounts or wipe you out depending on what % of your trading equity you allocate. This is something you can test - by simulating actual trading returns - both upside and downside.<p>I'm not sure if you're looking for trading advice or development advice, but I woudo recommend these books while you are doing system development:<p>- <a href="http://www.amazon.com/Trade-Your-Way-Financial-Freedom/dp/0070647623/ref=asap_bc?ie=UTF8" rel="nofollow">http://www.amazon.com/Trade-Your-Way-Financial-Freedom/dp/00...</a> - trade your way to financial freedom by Van Tharp<p>- the entire 'market wizards' series (<a href="http://www.amazon.com/Market-Wizards-Jack-D-Schwager/dp/0887306101" rel="nofollow">http://www.amazon.com/Market-Wizards-Jack-D-Schwager/dp/0887...</a>) - there are three (or four?) books in the series, all are worth reading.<p>The final problem is designing a system which you truly understand, which you are sure gives you an edge. And which you are willing to stick with if there are periods of high drawdowns.<p>Also remember you are up against people with very deep pockets and huge computers and phd guys working on there same thing.<p>It's a great intellectual puzzle and a worthy challenge, but make sure you know what you are getting into.
If I could just give you one piece of advice it would be this: don't do this.<p>A bit longer: You're a pig headed for the slaughterhouse. Please do it 'for play money' only for the first 5 years or so, if your method makes money over that period then set yourself a goal and if you reach it <i>stop</i>.
"A mathematician plays the stockmarket" is an interesting read of a smart person who made a bunch of mistakes and lost money.<p><a href="http://www.amazon.com/gp/aw/d/0465054811/ref=mp_s_a_1_sc_1?qid=1424599137&sr=8-1-spell&pi=AC_SX110_SY165_QL70&keywords=mathematician+plays+stockmarket" rel="nofollow">http://www.amazon.com/gp/aw/d/0465054811/ref=mp_s_a_1_sc_1?q...</a>
Just a word of caution: the profitability of stock trading, regardless of the timeframe (high-frequency, daytrade, week long positions), heavily depends on access to information. So even if you build a system that mathematically is better than those built by a team of physicists at large financial institutions, it may easily under-perform the market due to its limited access to data.
Let's assume your strategies work, then you will earn by a magnitude more by doing this for a trading company than you will ever with your own funds.<p>You can go down that road, but this a whole full time career and it usually takes years to be good enough at it.<p>I could recommend this book (<a href="http://www.amazon.com/Building-Algorithmic-Trading-Systems-Website/dp/1118778987" rel="nofollow">http://www.amazon.com/Building-Algorithmic-Trading-Systems-W...</a>) to get an understanding for the craft.
I used to do it with bitcoin, but not to get rich, just for curiosity.<p>The most algorithms is just chart reading, playing around with periods, standard deviations etc. You can achieve something really quickly, but it only works if the market jumps up and down within a short time period and stays on the same level if watching a bigger time frame. What if the market drops by 90% suddenly because some bank decided to do something stupid?<p>The next time I won't work on HFT, but more of risk reduction and long-term investments:
- Markowitz diversification model
- Multi-Market trading, trade on market differences<p>Another interesting approach would be to exploit trading algorithms, since 99.9% of all trades are algo trades.<p>What you theoretically could do is use popular trading algorithms, develop a tool to monitor their decisions, adjust decisions weights and see how a simulated market performs on comparion to the real market. Now that you have something like a "slave" market and can go on to the next step and calculate scenarios e.g. 5min ahead and can 99.9% predict where the market will be in 5min.
I wrote a somewhat long-winded comment on this topic a while back, you may find it useful:<p><a href="https://news.ycombinator.com/item?id=6832589" rel="nofollow">https://news.ycombinator.com/item?id=6832589</a><p>The <i>tl;dr</i> version of it is:<p>Risk management, specifically position sizing, is commonly neglected and yet extremely important.<p>Psychology is everything, regardless of whatever your approach may be.
You have not given nearly enough information to give any reliable tips.<p>Start with a few questions. How often are you targeting trades? Do you anticipate trading thousands of times per day, a few times per day, a few times per week, a few times per month or a few times per year? Know that there are sophisticated algorithmic trading systems in all of those spaces.<p>Once you know how often you will be trading, you can make more appropriate decisions about execution platforms. What kind of data feeds do you want to use? How will you get the data into your systems, how will you get the outputs out of your system. How will you translate those outputs into orders? Who will take your orders? As an aside, most brokerages will want to know if a computer is making all the trades and will either outright ban fully automated trading or will want a kill switch (see the TSA for your broker of choice).<p>This leads us to the actual hardest part about automatic trading, operational risk. What are you going to do to prevent your system from making out trades? What is your shut down procedure? How is it triggered? How will you monitor the system? Know that managing automation risk is the last mile of human intervention in the system, and that lots of people have lost a lot of money by not intervening soon enough, or intervening too soon.<p>You also need to have a target benchmark to compare to. What are you going to benchmark against? How often are you going to evaluate against that benchmark? How long will you allow your system to underperform the benchmark? There are long game algorithmic trading systems that have built into their capital allocation models >10 year underperformance against benchmarks for instance.<p>Finally, ask yourself, what inefficiency are you taking advantage of, that will allow you to profit from trading? What is it that you bring to the table that isn't already there? Once you honestly assess that, ask why wouldn't that same efficiency be more valuable in some other field that is less crowded because automatic stock trading is a very low percentage game right now.
This old HN thread may be relevant: <a href="https://news.ycombinator.com/item?id=4748624" rel="nofollow">https://news.ycombinator.com/item?id=4748624</a>
I started doing this after reading one of the Motley Fools' investment books (Rule Breakers - Rule Makers) ... fortunately I never asked it to trade real money. A few years later and MF has completely disavowed the techniques in that book (and I would have lost whatever I put it).<p>Now I just throw darts ... unless you can get some insider knowledge like the big boys, you'll never be the early mover.
I want to clarify, the intention of this system is specifically to NOT be a HFT system because I'm aware of the large handicap associated with inferior data available at slower speeds. It's intended to primarily help with stock picks since it can look through more data than a human can. This would involve both fundamentals and technical analysis. The most frequent trades would be once a day and may end up being closer to once a month. It is intended to be a low-cost solution. I would like the trades to be done by the system but the approach will not require it.<p>Based on some of the comments it seems such a system will be the underdog and will be the prey for the large HFT shops. And that's one of the risks of such an experiment. Once I put in real money into this (not too much, but just enough to not be consumed by the fee) there will be a hard point after which I'm ready to throw in the towel. I think of this more like an investment management software that has more knowledge than myself (various trading techniques, tax laws, etc.).
Someone can please give me some trustworthy, scientific reading material to give people asking me to proof the point 'systematically beating index performance on the stock market, using only public available data, is not possible'? Intuitively i understand this is true, so no need for holywar. Same also applies to commodities trading.
My 2 cents - don't reinvent the wheel. Use quantopian.com - it has great community, where you can learn a lot. Their trading system called ZipLine[1] is open sourced, so I guess you can extend it to fit your specific needs.<p>[1] <a href="https://github.com/quantopian/zipline" rel="nofollow">https://github.com/quantopian/zipline</a>
Don't get discouraged by the negative comments. It's completely possible to make a profit (with low risk) without resorting to HFT.<p>I'd recommend you read Quantitative Trading by EP Chan for a great overview of the basic methods and some example implementations.<p>Disclaimer: I've done exactly this successfully for >2yrs now.
I've worked in the electronic trading aka hft market now almost 7 years. The big boys are optimizing their trades down to the nanoseconds, they are spending big money on bleeding edge network technologies, colocating their servers <i>inside</i> exchanges, etc.<p>If you expect to compete with them on AWS save yourself the loss and just stop right there. However, if your strategy is more or less buy and hold for weeks / months then this doesn't sound entirely unreasonable. From my experience, simply getting into this industry as a serious player is roughly a 10-12 million investment, and that is starting very very small in only a few exchanges.
Use paper trading for a while on a platform like Interactive Brokers before going live. You will find it's unlikely you discovered the holy grail but hey maybe you have. Just be wary of a common trap for both beginners (excusable) and longtime market pros managing billions (not so excusable) - lots of strategies look great while testing because they have a payoff matrix which yields +1$ 95% of the time and -$1000 5% of the time. Obviously this is a poor ex ante bet but you are unlikely to empirically observe/discover the asymmetry at first when it just seems like you are printing free money.
I'am working on this.<p>Only using python , ipython rethinkdb and postgresql.<p>We are working on a event driven fund to find pattern, but we still need human at the end (the program found pattern and we try to find why and if they are reproductible).<p>Good luck :)
Even though backtesting is only of limited use, you should try your algorithms on fresh (you haven't previously used them) time periods and see what types of returns you can get. When I did this with my algos I was playing with, even though it seemed positive, when I accounted for the cost of the trades and the cost of the data (near-real time market data is damn expensive), I wouldn't come out much ahead of a simple buy-hold index fund approach.<p>Invest the minimum amount you can in this project to learn that it isn't a great idea.
You should have a look at MetaTrader. It provide tools for back-testing and dynamic spread etc..<p>I have been building also my own platform (with nodejs as for high IO), but you easily loose focus when you have to build it ALL by you're self..<p>(opensource) platforms like MT4 / MT5 give you more time to spend on actual alghorithmes, then on data structures etc
Check out omnitrader.Com as a potential platform for building your algorithms. Whilst there also check their product myomnivest. If you want to go big-time as I have, check into their Nirvana club. Not cheap but their auto trading algorithms are extremely sophisticated.
One of my friend is working on a same project for a big company and some of my friends are interested in this.<p>I'am thinking does anyone will be interested by a specialised forum or a subreddit to share our experience and tips ?
I'am thinking btw to make a portal / forum / subreddit about this topic. Anyone interested ? What format will you prefer ? Subreddit / special forum ?
<i>gain an advantage over simplistic human investing</i><p>What makes you think there are any simplistic humans in the market for you to take advantage of?
The only thing you're going to automate is the emptying of your account.<p>> I'm planning to host this on AWS and use a combination of Python and R.<p>Yeah, great idea. NOT. Unless you're doing something like tens of trades per day, or less.
You may want to take a look at <a href="http://quantopian.com/" rel="nofollow">http://quantopian.com/</a>, who make an engine that does just that. A lot of brokers actually support this.<p>The reason appears to be that there are lots of stock trading packages that people want to use (and to be fair, most brokers' interfaces suck somewhat. Also one interface for 2-3 brokers sounds appealing to me).<p>Here is an article about one such trading API:<p><a href="http://www.quantstart.com/articles/Using-Python-IBPy-and-the-Interactive-Brokers-API-to-Automate-Trades" rel="nofollow">http://www.quantstart.com/articles/Using-Python-IBPy-and-the...</a><p>Most API's also let you play with the "fake money" accounts of the brokers. But keep in mind that they usually cheat on transaction costs in those modes.<p>You may want to read a lot of the algorithms on Quantopian. To be frank, the only one that really makes sense to me is the "auto-rebalancing" approach. What this does it it keeps your investment at what you set it.<p>Say you invest 50% in automotive stocks and 50% in nasdaq stocks. Nasdaq goes up 10% -> it sells 5% of the profit and invests it into automotrive stocks. This is one of the few algorithms that seems to :<p>1) beat the market<p>2) actually spreads risk around (or it seems to me it does, depends on your investment spread I guess)