Trading Program Ran Amok, With No ‘Off’ Switch

'Prediction is difficult, especially with regard to the future'.<p>It looks like they were using some new algorithm, which should have made them a lot of money, had the market gone up after their massive purchases. In that case, they would have pocketed fat bonuses and would not be on the news.<p>However, it has not happened, so the crying and the search for a scapegoat is on. It sounds like the case of the banking business as usual: 'heads I win, tails you lose'.<p>Ultimately, there is a really serious problem with the concept of limited personal liability for companies engaging in speculation. It is an assymetric arrangement, whereby the directors are entitled to the profits but are never personally responsible for the losses. With such rules of the game, it is advantageous to take crazy risks. Expect to see a lot more of this and many more taxpayer funded bailouts.

What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like those in, e.g., aviation.<p>On one hand, you'd think the QA in finance would be pretty solid, considering that the survival of the company could be at stake (witness Knight). On the other hand, I have a feeling that even there, people just don't take it that seriously.<p>Would love to hear from anyone with more experience writing software for these industries.

They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence.<p>At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.

So, some of the owners were looking for a way out, and magically this thing broke loose and started giving away (basically) free money to undisclosed receipients. In the meantime all the technicians were fast asleep and couldnt kick the machines down or something, while they were losing milions of dollars per minute. This article is a completely honest recap by completely honest people, about completely honest traders/bankers (bankers are not people).<p>Edit: on a COMPLETELY unrelated note, trading firms/banks are known to actively pursue the extraction of money from their clients with bogus trades/advice <a href="http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-goldman-sachs.html?pagewanted=all" rel="nofollow">http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-g...</a>

I would rarely suggest this, but if something is so incredibly broken that you're loosing money at a rate of 800 million dollars per hour, screw the customers.<p>Turn it off at any cost. If you are forthcoming and transparent, customers will understand.

There seems to be a lot of confusion around market making, brokering, execution algorithms and HFT in this thread.

I read the nanex article. Regardless whether it is true or not, the general trend is towards development of more sophisticated load testing programs.<p>The most benign ones were developed for use in IT systems. E.g. Apache bench. While these can cause disruptions if aimed at production services, this does not necessarily threaten the health of an entire enterprise.<p>However, the trend is that all software sectors are starting to adopt this particular technique of testing software with not sufficient regard to what happens if it is released into live systems.<p>For example, we have chaos monkey, from Netflix, which randomly shutdown services in a cloud based system.<p>What would happen if software which simulated meltdown at a nuclear facility was accidentally bundled into the build system by a tired operator? Or some one does the same with flight software?<p>The main software running trading platforms would presumably be supervised by another program to ensure that bad algorithms do not lose e company too much money. However there was no such tool for the component that generated the test data.<p>To me, it sounds like the supervision should be done at a higher level, e.g. A wrapper around existing APIs. All software running against live systems must call into the wrapper.<p>Secondly, test software should conduct some kind of verification. E.g. Check for evidence that it is testing against a Test system. This might be the presence of a nonexistent company, et c.<p>I am more than happy to compile any other ideas you may have so that the IT industry is able to build more fail safes into software.<p>We are starting to see some of these fail safes in practice. E.g. When you try to send out an email to everyone in the organization, email software may warn you if you are sure you want to do that. The problem is we haven't thought enough about these scenarios that we don't adequately address them.<p>Incidentally, over in Australia, the Commonwealth Bank suffered a major downtime when it's outsourcer HP accidentally pushed out system wide updates instead of doing this to select machines as originally intended.

Pure speculation. Maybe there was an off switch, which used to work, but not regularly tested, and silently broken? Wouldn't surprise me.

This was apparently an infrastructure problem of some sort: <a href="http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-business-1ZpjUmh0TlevIFSAZ2UwxA.html" rel="nofollow">http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-...</a><p>Infrastructure changes can be notoriously difficult to back out by simply using an "off" switch, particularly if this was some type of a firmware upgrade that impacted all of their production servers. Backing it out at a minimum would require some type of a reboot, which would cause problems with an active trades. It could very well be that they were running an Active-Active environment, they had to go Active-Passive, back out the changes from the passive environment, reboot, and surgically cut over to the passive environment. This could easily take 30 minutes.

Doesn't this mean that others made a killing, taking advantage of all the mispriced orders?

"Knight is also working with Goldman Sachs to help unwind the trades behind its extensive loss, according to people briefed on the matter.<p>"Goldman has agreed to buy, at a discount, the shares that the trading firm had accumulated. Such a move would help Knight by taking the portfolio off its hands and freeing up capital."<p>What does this mean? Why would GS do this? Why would Knight do this? Couldn't they just sell them on the open market at a better price instead?

Strange article. Lots of text but missing the main thing I was looking for. What kind "erroneus trades"? where did the money go? If you buy stock at the market you did not intend to buy, why not just sell them the next day?

Why isn't automated high-frequency trading banned already?<p>Does it not go directly against the spirit and purpose of having a stock market with proper investors?

Can someone smarter than me please explain why a system where trades got matched / executed at a granularity of once per second or once per several seconds wouldn't work? What would be the problem with exchanges accumulating and keeping secret buy and sell orders and executing them at a reasonable interval?

I wonder if the lack of a kill switch was linked to the power black outs in India - what if they set it going, got cut (power) and then only came back online 45 mins later? Could be that just one link - say a local exchange or power for an FTTP line failed.

I don't believe the problem was caused by "bug" - this reminds me of "rouge trader" stories.<p>It is kinda weird that all problems in Wall Street are caused by "bugs in software" or "rouge traders": while executives are never hold accountable.

This is an extreme example of what what can happen when what should be a software company thinks it is some other sort of company. I am sure they thought they were a trading company and software development was the necessary evil required to get things done.<p>Well 400 million odd dollars in the red later I doubt they still feel that.<p>I do feel sorry for them and they probably didn't deserve this huge loss. Hopefully valuable lessons can be learned.

Can anyone provide some context on this matter? What happened wednesday and where?