Bombay Stock Exchange: world’s fastest stock exchange leverages real-time analytics to improve data governance and industry SLA’s

Challenge

BSE sought to establish itself as an agile, prompt and well-governed trading website. With data growing at a daily average of 400GB across the enterprise, BSE needed a robust and scalable ecosystem that could centralize structured and unstructured data from multiple sources and shorten the time taken for analysis and reporting. 

“Back in 2015, we were facing challenges with our existing warehouse solution. We used a proprietary technology platform and scaling those up with the existing solution involved heavy recurring capital expenditure with no guaranteed cost-benefit or returns on revenue for the foreseeable future,” said Kersi Tavadia, Chief Information Officer, Bombay Stock Exchange. “That’s when we started reassessing our options and looked at open-source platforms that would help us deliver real-time analysis by merging all incoming data on to a single data lake where it can be sorted and analyzed as per the user’s need.”

Solution

“We wanted to start small, but we definitely wanted to make an impact. Cloudera has been an early adopter in offering a strong community version of the platform, which is the main reason why we switched to it. The availability on all OS platforms is seamless, so its usage is very convenient and compatible. These two reasons made Cloudera the perfect enterprise data platform to begin with,” Tavadia added.

BSE leveraged the platform to set up their application, and in 2017 they successfully scaled up operations to create a secure trading enterprise data environment.

BSE uses Cloudera to respond to transactional queries at a warp speed of six microseconds from when orders hit the exchange gateway. At market open, the exchange gets an average of 100,000 orders per second, which is processed and transferred to Kudu for simultaneous analytics. Using a Kafka framework to ingest vast amounts of data into real-time surveillance systems helped them achieve this feat.

“As a stock exchange, we undertake the responsibility of managing extremely sensitive information. Real-time analytics and scalability are factors imperative to the sustainable growth of BSE. This will ensure that our critical systems are future proof so we can continue to enable the industry by building capital market flows,” said Tavadia. “Cloudera meets our custom requirement by providing us with industry-standard technology and infrastructural expertise that has helped us deploy the highest number of references with the lowest total cost of ownership among vendors.”

Results

BSE is better equipped for making decisions in real-time ever since the deployment of the enterprise data platform. The time required to bring together vital pieces of information that help teams make informed choices has drastically decreased; which in turn has helped BSE reduce operational costs in a significant manner, by up to INR 60 Million annually.

Cloudera’s easy-to-use, open-source architecture provided BSE with the optimal base to structure their enterprise-wide applications.

• Automatically scan print, online and social media sources using alert generation mechanism to notify BSE on any suspicious articles, allowing the stock exchange to quickly clarify this information with the relevant companies

• Reduce time spent on generating complex reports by 70 percent. (e.g. It takes 20 minutes to generate a sync table as opposed to six hours spent previously)

• Strengthen BSE’s security posture by implementing authentication, authorization, encryption and auditing solutions. The implementation resulted in improved data security, identity and access management, allowing for enhanced visibility on where and how data is used across the enterprise

• Reduce operational costs by INR 60 million annually (~$800K USD) by using the platform which eliminates the need for expensive storage facilities

“The speed, scale, and analytics capabilities of Cloudera’s open-source ecosystem has helped us in delivering remarkable real-time analytics. As a stock exchange, it’s imperative for us to highlight and intervene promptly in any wrongdoings that may occur, and this is where the speed of data being ingested into the system is incredibly important,” Tavadia added.