TOKYO — Multiple backup mechanisms failed to activate during Mizuho Financial Group’s widespread system crash last week, Nikkei has found, with a reliance on numerous outside vendors complicating system management.
The crash was the group’s fifth in 2021. Japan’s Financial Services Agency is extending its on-site inspections as last week’s incident raised the possibility of fundamental problems with the banking system.
Mizuho’s accounts system was developed by Fujitsu, Hitachi, IBM Japan and NTT Data. But subsidiaries Mizuho Bank, Mizuho Trust & Banking and Mizuho Research and Technologies handle daily operations, with the last contracting the work to an outside partner, meaning no centralized framework exists for managing the system as a whole.
After detecting a disruption at 8:57 p.m. Thursday, Mizuho attempted to swap a damaged disk drive for a spare within its primary account system based in outer Tokyo. When that did not work, the bank tried to switch to the backup system at the same site, which also failed.
Mizuho then attempted to tap a different facility in neighboring Chiba Prefecture, positioned as a backup in case of a natural disaster. But the bank could not transfer the necessary data to its core system and only managed to activate the backup system there on Friday morning, after branches had opened for the day. Deposits and other transactions at teller windows were halted at roughly 450 Mizuho Bank locations.
Mizuho Bank announced a new incident Monday, saying up to 130 of its roughly 5,300 ATMs across Japan stopped functioning around noon. The issue is believed to have been caused by network problems, and was resolved within a few hours.
Companies often operate what is called a hot standby system — a backup that runs simultaneously with the primary system so it can take over at any time. In contrast, cold standby systems are switched on only after the primary system fails. Though Mizuho’s Chiba facility receives data updates from the primary system in Tokyo, an operator still needs to manually let it take over.
“The hardware in the hot standby system was supposed to take over” during last week’s crash, Satoshi Ishii, Mizuho Financial’s chief investment officer, told reporters on Friday. It remains unclear why this switch did not occur right away.
Japan’s other megabanks also operate hot backup systems in case of a failure. The Zengin Data Telecommunications System, which processes transactions between financial institutions nationwide, is powered by four simultaneously operating systems that eliminate the need for any separate backup.
“Mizuho says the issue was caused by an equipment failure, but the fact that its backups failed to function points to problems in its software as well,” said a source at a Japanese megabank. “I don’t know if the backup system was prepared properly.”
The FSA has ordered Mizuho to submit a report regarding last week’s crash, and the regulator will expand the ongoing inspections. Its past investigation focused on the bank’s protocols and response to the repeated system failures, but did not identify the fundamental causes behind Mizuho’s outages.
“The series of problems is a sign of structural issues in Mizuho Financial Group as a whole,” an FSA official said. “We will continue thoroughly investigating until we find the root cause.”
Disruptions at Mizuho, one of Japan’s three megabanks, hamper the country’s entire financial system. The FSA has ordered Mizuho to improve operations following each incident, though it remains to be seen whether the agency can identify what lies at the heart of Mizuho’s complex system failures.