Last week was turbulent. A large mainframe customer faced a database crash, as a result of which operations were shut for more than 24 hours. Huge Business Losses, Operational inefficiencies, Blame games and heavy escalations were the order of the day. Two large tables of 300 Million records each went into an inebriated state as a result of which Online and Batch users were affected.
I was pulled in as a database expert to fix this. Our immediate priority was to make the business up and running and we did this.
However, though majority of the blame was on the legacy nature of the platform, my findings and recommendations for such an event not to occur, were completely contradictory. The platform shall never be blamed, rather the lack of a disciplined approach and technology practice, is what the root cause was. The same application running on a modern technology stack and server, lacking a disciplined approach and technology practice, would face a state of similar nature.
The customer had antiquated out of support software, modern features provided by the current version of the software were not used by the applications, Storage volume allocations were not disciplined in nature as a result of which new volumes were added frequently, without a clean-up being attempted, concept of partitioning or profiling of database records (what’s frequently accessed and what’s not) were not done as a result of which the database grew out of bounds, all indexspaces and tablespaces were overloaded into a single Bufferpool and last but not the least a thoroughly analyzed batch process with appropriate triggers were missing, as a result of which jobs which were not supposed to run in parallel, started running in parallel.
A major airline which suffered a similar crash some months back, disrupting its operations and leaving passengers stranded for almost two days, left the media reporting the legacy nature of the platform as the root cause.
Modernization is always an option for legacy customers, however a disciplined approach and technology practice, irrespective of the platform, is the key to the success of any business.
What are your thoughts?