Financial Firms Should Collaborate on Tech Testing Across Markets, Former NYSE CIO Says

By Kim S. Nash, The Wall Street Journal
15 July 2015
URL: http://blogs.wsj.com/cio/2015/07/15/financial-services-together-must-test-tech-for-interconnected-markets-former-nyse-cio/

The technology outage at the New York Stock Exchange that halted trading for four hours last Wednesday reflects an interconnected financial market where problems in one spot cause changes in another. Glitches are inevitable with the growing complexity of technology behind financial markets, but what matters more is what financial firms and exchanges do in the aftermath, said Steve Rubinow, a former CIO of NYSE and of FX Alliance.A world full of sensors that regulate networks and machinery across industries, not just in financial services, means technology problems are inevitable, said Mr. Rubinow, now CTO at Catalina, a data marketing firm.“The opportunity for failure is greater.”When he was CIO at FX Alliance, a foreign exchange ultimately acquired by Thomson Reuters, about $5 trillion worth of trading occurred per day. The exchange had no major outages in his time there, he said. But generally in financial services, every second of downtime produces uncertainty about whether transactions were executed properly.Before an outage, CIOs should have already worked with others in the C-suite, as well as internal auditors and legal staff, to create a contingency plan for how, and how quickly, to respond in an outage. That includes who to notify, for example, and which related, adjacent systems to monitor or perhaps shut down, Mr. Rubinow said. At NYSE last week, staff found the problem in about 90 minutes and spent much of the time after that deciding when and how to restart and test systems, according to a person familiar with the events.After an outage, the IT staff must reverse engineer the activity, and inactivity, to figure out what happened and to correct problems with trades, Mr. Rubinow said.IT leaders inside NYSE, Nasdaq and other markets might now study the ripple effects of last week’s outage to learn what traders did during the glitch and prepare for the next time. For example, volumes increased at Nasdaq as computerized trading systems shifted orders away from a downed NYSE.Stress testing could also be improved, he speculated. Financial services firms could use supercomputing power available through academic research institutes via the cloud to bombard internal systems with unpredictable traffic to identify weaknesses. “When you bump up the number of transactions per second to the hundreds of thousands, the systems may not respond the way you want them to,” he said.