The biggest current limitation with cloud providers when it comes to exchange tech is the lack of real multicast support. It is rare outside of exchanges, but extremely low latency L1 multicast market data has become the backbone of exchanges, both for fairness and for scalability.<p>Knowing you can saturate your entire network with 10G traffic and every participant will get the same market data packets at the same time[0], and there will be zero queuing or bottlenecks is very hard to do otherwise. There is a pretty good podcast episode about it out of Jane Street[1].<p>I know AWS have 'multicast support' but last time I tested it, it was clearly just uni-cast traffic with a software switch doing fan-out/copying, I assume using the same tech as their transit gateway, I think it was called hyperplane or something.<p>[0]: for some definition of the same time, at least low enough that you can't measure it without equidistant optical splitters or White Rabbit synced devices.<p>[1]: <a href="https://signalsandthreads.com/multicast-and-the-markets/" rel="nofollow">https://signalsandthreads.com/multicast-and-the-markets/</a>
One key consideration is “provable fairness”. It’s my understanding that exchanges use techniques like long, same length fiber optic cables to all racks within the exchange datacenter to convince customers that everyone is on a fair playing field.<p>This is a lot harder to do when a server is virtualized somewhere on some rack on EC2. Exactly as mentioned, people will try to optimize by spinning up/down instances as close to the exchange server as possible. Customers will be unhappy because they can’t prove that it’s fair, even if they have the closest server.<p>Overall great, thought provoking writing btw
Nice article. Wondering though why trading is not done in discrete batches, e.g. 5 second intervals? Trades in the same interval get filled equally or stochastically? Info about trades with that same 5 second batch delay? Is there some (theoretical) market efficiency thing at play? All this HFT feels wasteful and bad for 'regular' human investors.
Why would the exchanges want to move their colos into the cloud. They've spent the capex and they can charge lots of money to rent data center space, cross connects and other services to their customers. If they moved to the cloud, all that revenue would go from them to AWS/GCP/etc.<p>Doesn't seem like a profitable move for the exchanges. They make more $$$ with on prem setups.
Nothing to add but I found this to be a well written thought exercise on a space I realize I know very little technical detail of. Thanks for writing this!
If a big exchange goes to the cloud it won’t look like a regular company setting up an aws account and getting a bunch of ec2 instances in us-east-1. They would at least have dedicated racks.<p>I suspect the provider would end up with a plan where traders can get servers that all have the same network distance from the exchange’s nics (down to the same length of fiber).
I used to work in machine maintenance, so I always think about what will happen when the machines involved fail.<p>NYSE machines hosting the trading server fail: presumably they have hot backups they're ready to switch to but that takes time and will interrupt trading during the cut over. Not to mention that not all failures are hard failures, what if the NIC is downtrained to a lower speed, RAM is slower than it should be, or a single hard drive storing important data crashes? Lots of interesting failure modes. When the NYSE owns their own machines they can handle these cases directly. When they don't and Amazon is responsible for repairing these machines it might take a lot longer to get things fixed. I hope NYSE is thinking about hardware failures and building a system to check performance of their trading servers before letting them become the active host.<p>Thinking about failures on the side of the traders: basically if they get unlucky then there could be delays as Amazon rerprovisions them replacement servers in the case of failures. This likely impacts what trading strategies are viable, and could cause them to lose money if machines fail at unlucky times.
This is an interesting read but I think it leaves outside what kind of trading is the one that would benefit from ULL.<p>ULL and currently HFT seems to be very useful for market making (buying the ask and selling the bid and profiting from the bid-ask spread making parts of a cent per transaction, done a few million times a day), but there are other uses for HFT. One of them would be to execute very big orders over time to instead of drastically rising the price of the security they can get a better cost basis by performing a set of trades, letting the market absorb the impact and continuing with the order.<p>The thought of having the market in a cloud provider like AWS scares me! Although I’m sure that AWS might have pitched the idea already. If the markets could be controlled by a private company that could schedule “maintenance” at convenient times for them, that sounds like a recipe for market manipulation bay trillion dollar company. Sounds like something the SEC wouldn’t stand for.
About the market data & monitoring element, beyond the usual suspects Clickhouse and kdb+ - worth noting that Aquis mentioned in this article uses QuestDB ( <a href="https://questdb.io/case-study/aquis/">https://questdb.io/case-study/aquis/</a>)
I like the assumption that NYSE would just grab some EC2 instances and run an exchange on them, and that AMZN wouldn't bend several directions at once to deliver new products that just happen to exactly replicate the environment they're 'leaving'.
Orders should have some durability and it would probably change behaviors enough to make hft go away.<p>If you list a buy or sell order it just has to be in force for some period of time, say a minute or something.<p>HFT shops will say this would reduce liquidity, but it would only make clear what real liquidity was in the first place.
Interesting read, thanks! Some points:<p>> Hogging instances might pay, if this stops competitors getting good hosts. This would eat into PnL and is also wasteful on energy.<p>Aren’t reserved instances cheaper than spot?<p>> Bad players could do a ping test to many thousands of EC2 instances, find those which are also at very low latency to their good boxes (assuming these are competitors), and DDoS them during trading hours to hammer the hypervisor’s NIC. This would result in critical overhead occurring for competitors sending orders out.<p>Leaving out the logistics of how someone could do this (why are your instances reachable from the internet?), wouldn’t you have a good case with your exchange to get them kicked out?
It's my opinion that stock exchanges should batch trades every 30 seconds, or longer (depending on the market), so that millisecond arbitrage becomes impossible.<p>Front running the market in any manner should be illegal.
Fun thought exercise, thanks! My question is, what advantage would a large exchange find in moving to cloud? They’ve already got the personnel capable of managing their environment. They’re not a rapidly-growing startup in need of flexibility. They’re large enough to get at least decent deals purchasing gear. “The cloud” will naturally expect to make a profit on the deal, which likely eats up (and then some) any savings which might otherwise be delivered.<p>I “get” cloud in a lot of circumstances but it doesn’t seem to make much sense here.
The author of this article clearly knows a lot about the subject, but I think this would have been better titled "Low Latency Trading isn't Going To The Cloud and Here's Why". Or to put it a different way, infra peeps within exchanges have a very specialized skill set and priorities. General cloud infra peeps don't. No shade, but there's always going to be some business that doesn't make sense to switch to the standardized solution.
At first glance, it seems this would even the playing field. However, large players will allocate resources to spinning up instances and overloading machines. Similar to how we're seeing the DDOS shenanigans going on in crypto.<p>Net-net, it still benefits startup quant shops and sophisticated independents. Most retail isn't doing HFT or really any quant. But for people wanting to have their own shops, this is a better version than having to build hardware and colo.
I'm a total trading noob.
Can you explain why a low latency is worth so much? And how these traders "exploit" that advantage to make a profit?
Showing that I don't understand economics while also telling that I don't understand economics: It would probably do the world more good to tweak the structures making ULL trading profitable anyway, it's not like the trading in and of itself brings any value to the broader world, while consuming enormous amounts of resources that could have been spent on actually improving systems that create real value.
It's amazing to me that regulators have not required a minimum latency, or random latency dispersion in orders/trades to level the playing field.
> The NYSE runs out of a public data centre (called NY4) which is run by Equinix.<p>No. NY4 is in Secaucus. NYSE operates out of an ICE (NYSE parent co) owned facility in Mahwah about 25 miles north of there. They managed to pick out the one big US equities exchange operator _not_ running in an equinix facility.<p>Sorry but this whole post sounds like someone who is sort of HFT adjacent but doesn't really know what they are talking about. Sending orders at "09:29:59.9999971 at the hope your order arrives at 100ns past 9.30am." What?
IEX Exchange is building a cloud-first stock exchange that uses the concept of "slowed trading" to eliminate some of the worst practices of HFT. They even use a 38 mile loop of fiber to slow down connections that are "too close".<p><a href="https://en.m.wikipedia.org/wiki/IEX" rel="nofollow">https://en.m.wikipedia.org/wiki/IEX</a>
How would any cloud offering deal with something like, say, the full options data feed, which is close to 40 Gb/s of binary packed goodies? You need both a very fat pipe and ultra-low latency: does the cloud, any cloud, offer that?<p>Also: how often have you guys seen the stock market being down? What's the "x nines" availability of, say, the US stock market and US options feed?<p>Now: do we wanna talk about the various cloud outages that made the news? Sometimes lasting hours?<p>Also what I've seen with the cloud is websites are now displaying spinners everywhere, for the myriad of not-low-latency-at-all microservices often taking <i>seconds</i> to respond. And that'd be on an ultra low latency fiber to the home setup, with 2 Gb/s down (and the ISP really supporting that).<p>Why the heck do I have to wait <i>seconds</i> for oh-so-many things to display in my browser, on a last gen Ryzen ultra-speedy machine, with a super fat and low-latency Internet pipe? The worst offenders being all those banking websites showing a balance of 0 instead of "-" or "n/a" while fetching my info: nearly gives me heart attack every single time.<p>I take it it has to do with micro-services all contacting shitloads of other micro-services, all living in the not-low-latency cloud. The problem being compounded by an army, a generation, of programmers who have never learned anything about optimization or latency and who solve every problem they have with the only hammer they have: the cloud. All these programmers know are JSON (or, worse, XML)...<p>I mean: JSON vs 40 Gbit/s of interrupted bit-packed binary feeds? How could these two world ever reconcile?<p>Now I don't do HFT but I do trade options and I do it through a desktop app and that app also offers an API through which I can fetch prices, send orders, etc. It's a good old Java app. And it's more advanced than <i>any</i> website I've ever used.<p>Can we please not enshittify everything with countless micro-services and JSON files in the cloud?
There's no directly colocated hosting that I know of, but "direct connect" gets you close: <a href="https://www.equinix.co.uk/partners/aws" rel="nofollow">https://www.equinix.co.uk/partners/aws</a>
... is there some movement toward "upload the strategy and let the exchange run it"? which would provide a more level playing field, reduce energy and hardware costs, etc?
Low latency/rapid trading seems to have added no value to the stock market and eroded a large portion of the fair market evaluation of companies. Whether it's technically possible or not it's likely we'll need to artificially add delay into the system - ideally that can be done in a way that makes it fair to traders that have a naturally high latency connection to even the playing field.