Analyzing credit card trends with Cloudera at JRI

The Japan Research Institute, Limited (JRI), part of the Sumitomo Mitsui Financial Group, is a comprehensive information service company with functions that include a think tank, consulting, and IT solutions. JRI provides innovative value to companies and society through diverse businesses, from addressing new issues to creating new markets and businesses.

JRI is responsible for developing systems for various group companies, including a bank (SMBC) and a credit card company (SMCC). Utilizing Cloudera’s platform, JRI analyzes credit card usage to extract various insights. This process enables them to manage and utilize their data effectively.

Securing highly sensitive financial services data with Cloudera

A well-known example of data analysis in the SMBC Group is “Custella”, a service provided by Sumitomo Mitsui Card Company, Limited (SMCC).  “Custella” anonymizes and statistically analyzes cashless payment data held by SMCC, aggregating and visualizing purchase performance data based on customer attributes, customer behavior, and other factors. "Custella" was developed in-house at SMCC in 2018 using a Cloudera base with contributions from JRI members. In 2020, the Cloudera infrastructure was transitioned to JRI. Kobayashi's team is mainly responsible for developing functions, known as enhancement development, and they are also responsible for building and operating a similar data analysis infrastructure within JRI to implement various services, utilizing Cloudera on a daily basis.

“Although the use of cloud computing is increasing in data analysis platforms, the safety of placing sensitive data in the cloud had not been fully assured as of 2018; therefore, we decided that Cloudera-based on-premises computing was the only way to manage data safely and securely, considering the future expansion of the scale. Even now, it is still difficult to determine whether it is safe to move sensitive information freely to the cloud, so I think we made the right choice, said Kobayashi.

Scaling Up: Handling the Surge in Data and Analytical Needs

The rapid growth of data, especially in credit card payment information, has accelerated significantly. Kobayashi highlights that what used to accumulate in five years now comes in a year. Scaling up hardware based on forecasts poses risks, while the distributed nature of the Cloudera platform offers flexibility by adding small servers incrementally. 

Kobayashi’s team addresses new analysis needs. For example, when a need arises for trade area analysis in combination with map information, it is necessary to increase the data brought in from upstream systems to enable new analysis. Current initiatives include replacing the existing data warehouse (DWH), offloading data from mainframes to reduce CPU costs, and democratizing data access while ensuring high security. Cloudera’s powerful capabilities surpassed expectations, proving efficient for high-speed data inquiries and offering a scalable, cost-effective solution compared to traditional appliance-type DWHs. Offloading tasks from mainframes to Cloudera ensures data security while reducing operational costs. 

Data democratization is another initiative made possible by Cloudera’s high level of security. With Cloudera, sensitive information such as card numbers can be masked based on user privileges, allowing different levels of data visibility for different users. Cloudera's integration with Active Directory helps to manage and authenticate users and resources within a network, ensuring secure data handling and streamlined administration. Kobayashi stated, “While there are other products that allow the same treatment at the application level, Cloudera is highly valued for its high level of security because it makes data invisible at the location where it resides." This approach has enabled a democratic work environment, allowing on-site workers to securely utilize the data in their own ways.

Keeping Pace with Technological Evolution

Cloudera's products are based on open-source software and are available as verified packages. Yoshiyuki Iio, General Manager of the Card Core Systems, Modernization Task Force and Data & Information Systems Division at JRI praised Cloudera saying , “With open source software from Apache Software Foundation or the Linux Foundation, it can be troublesome to get it to work because of small differences in versions or differences in the libraries it references, which can cause inconsistencies, whereas with Cloudera products, we can develop without stopping at trivial points because everything is guaranteed to work.“ At the same time, the ability to quickly adopt the results of rapidly evolving open-source software is an advantage in today's fast-changing environment, allowing for a system that always utilizes the latest technology. Cloudera also excels in support, offering training programs for technicians. He also appreciated Cloudera by saying, "There are few vendors that offer such ongoing training on open source, so it was very helpful.”

Regarding the latest technology, Cloudera also supports Apache Iceberg, which is currently attracting attention as an open table format. Iceberg allows for flexible adjustments by simply changing the metadata, unlike conventional databases, which require schema definitions and data re-registrations. JRI is considering introducing Iceberg and plans to conduct a verification test. "We expect that the deployment of Iceberg will enable us to take data management to the next level," said Kobayashi.

Cloudera also supports AI integration with various models on the platform side, addressing the rapid evolution to meet enterprise users’ data security needs. Both JRI and Cloudera share a deep interest in AI research and have been working on joint PoC projects.

Today, digital data is increasingly valuable as an essential element supporting people's daily lives and corporate activities. JRI, managing the information systems of one of Japan's leading megabank groups, creates new value by using Cloudera as its data infrastructure, linking various group companies through data.