Last week, RBS, Natwest and Ulster Banks in the UK suffered a catastrophic IT failure. This failure is still causing on-going problems for customers as well as those that don’t bank with these organizations. One bad software upgrade has potentially lost this organization, loyal customers, cost them financially, and damaged their reputation.
For those of you outside of the UK that have not heard about this failure, a software upgrade affected the scheduling of batch processing runs that deal with banking transactions, so millions of transactions were not processed. The banks guardian systems spotted a problem during the run on Tuesday and they backed out of this change quickly but since then the banks have had a backlog of payments to get through as they have to re-run batches of transactions in the right sequence and they only actually managed to perform a successful batch run three days later on Friday. One week later customers are still experiencing problems.
The impact of this failure has been huge. There are many stories emerging of families unable to withdraw money to pay for food and travelers stuck in far flung destinations unable to pay their hotel bills. In addition, the knock on effects are being felt by businesses that don’t themselves necessarily deal directly with these banks such as Estate Agents who have had house buying chains collapse due to inability for one person in the chain to verify deposits have been transferred.
The details about what occurred and the procedures performed to resolve the problem have yet to come from the banks themselves but there has been a lot of speculation about poor testing, inexperienced operators, layoffs removing staff with experience and offshoring operations. This coupled with other rumors about security hacks as the initial reason for this implosion have probably done little for this banking group’s reputation with more than 18 million customers. As their crisis policy kicked in, the banks took the unprecedented step of opening on the Sunday and longer hours during the week to help their customers. These costs plus promises of compensation to ensure customers are not out of pocket will undoubtedly hit their bottom line. Added to the banks woes has been the call for the Financial Services Authority to carry out an investigation as to why it’s taking so long to recover.
So a few thoughts on this crisis from an IT perspective:
Change and Release Management must live. While there have been a lot of recriminations about how long it’s taking to recover from this IT failure, imagine how much more trouble they would have been in without the ability to manage this change or roll the change back if needed. You may think that Change Management is not necessary in your IT support environment or its phase four on your Service Management plan but if you continue to implement changes and release software unmanaged you may not have a business left to support.
IT Business Alignment is dead. There still seems to be a lot of discussion about aligning IT to the business. I can’t think of a clearer example of how wrong this is. Without IT this banking group cannot run their financial business and if they didn’t realize just how important a role IT plays in their business, they do now. IT is an integral part of a business, not a bolt on that runs parallel and if seen as the latter when making business decisions then expect the consequences as above.
Customer Experience Management of IT. Check out my previous blog about managing the customer experience of IT. As this banking group now realizes the customer experience is not just about their frontline support staff operate, the UX of their online banking or for that matter the service provided to customers by branch banking assistants, it’s the whole end to end package.
And on that note, if you are a Natwest customer visiting a branch banking assistant, remember it’s not the tellers fault and they’re probably experiencing similar issues too. I would imagine many of them did not choose to work on Sunday – so be kind to them.