NYSE Provides Glimpse into Glitch Timeline

By Steven Norton and Kim S. Nash, The Wall Street Journal
9 July 2015
URL: http://blogs.wsj.com/cio/2015/07/09/nyse-provides-glimpse-into-glitch-timeline/

The New York Stock Exchange gave a glimpse into the pressure-cooker technology situation it faced Wednesday when a configuration issue led to a nearly four-hour trading outage.It began Tuesday evening, when the NYSE started to roll out a software release ahead of an industry test of a Securities Information Processor timestamp requirement, the exchange said in a statement. SIP is the system that consolidates quote and trade data for U.S. stocks. NYSE operates one for stocks on its exchange.The next morning around 7 a.m. ET, NYSE noted communication issues between customer gateways and the trading unit that had received the software release. The exchange then determined some trading gateways didn’t have the configuration compatible with the software release. Those gateways are basically “how the exchange interacts with the rest of the world,” said Larry Tabb, founder and CEO of TABB Group, told CIO Journal on Wednesday.The exchange then updated the gateways with a correct software version ahead of the market open at 9:30 a.m., it said. But those updates caused more communications issues that showed up mid-morning.A message alerting traders to the technical issue came at 11:09 a.m. When customers were still reporting odd system behavior shortly thereafter, the decision was made to suspend trading on NYSE and NYSE MKT.NYSE began to cancel open orders on those trading venues, restarted customer gateways and failed over to backup trading units in the Mahwah, New Jersey data center, the exchange said. Talking with regulators and industry leaders, NYSE decided to implement a complete restart and resume trading shortly after 3 p.m. Trading happened as scheduled, the exchange said, and closed normally at 4 p.m.A person familiar with the events told the Journal that the NYSE set up three “war rooms.” One of them was reserved for technologists trying to figure out what happened and how to restart markets. The other two were for senior managers and a group that dealt with customers. It took about an hour and a half to find the issue, a person familiar with events told the WSJ.Perfection in any system is impossible, said Steve Rubinow, a former CIO at NYSE Euronext. In the pressure-cooker of an outage, technology leaders must make decisions with incomplete information, he said. For example, sometimes systems self-correct and sometimes not. But a decision must be made how long to wait before intervening. Once the technology team identifies the problem, they can’t simply fix it and turn trading back on, he said. Checks for data integrity and peripheral system damage are necessary.“What they don’t want to do is bring it back and have to take it down again,” he said. “You lose a lot of confidence.”