Clive Bull 1am - 4am
IT outage fix deployed but ‘some time’ before all systems restored – CrowdStrike
23 July 2024, 01:08
The chief executive of the cybersecurity firm at the heart of the global IT outage has apologised for the incident.
The chief executive of CrowdStrike, the firm at the centre of the global IT outage, has said he is “deeply sorry” for the incident, but warned it would take “some time” for systems to be fully restored.
George Kurtz said a fix had been deployed for a bug in an update rolled out by the cybersecurity firm which affected Microsoft Windows PCs, knocking many offline around the world, causing flight and train cancellations and crippling some healthcare systems.
In an interview with NBC’s Today Show in the US, Mr Kurtz said the incident was not a cyber attack, but admitted that despite CrowdStrike identifying the bug which sparked the issue and rolling out a fix, it would still be “some time” before all systems returned to normal.
“We’re deeply sorry for the impact that we’ve caused to customers, to travellers, to anyone affected by this,” Mr Kurtz said.
“We’ve been on with our customers all night and working with them – many of our customers are rebooting the system and it’s coming up and operational because we fixed it on our end,” he said.
“Some of the systems that aren’t recovering, we’re working with them, so it could be some time for some systems that just automatically won’t recover, but it is our mission to make sure that every customer is fully recovered and we’re not going to relent until we get every customer back to where they were and we’ll continue to protect them and keep the bad guys out of their systems.”
Asked if he ever thought an outage of this scale was possible, the CrowdStrike founder added: “Software is a very complex world and there’s a lot of interactions, and always staying ahead of the adversary is a tall task.”
In a post to X, formerly Twitter, Mr Kurtz reiterated that the outage “was not a security or cyber incident”.
“Our customers remain fully protected,” he said.
Crowdstrike’s stock tumbled in value when the US markets opened on Friday as investors digested the impact of the outage. Shares slid by over 8% at the start of trading, knocking around 10 billion US dollars (£7.8 billion) off its market value. The largest financial markets in the US and UK fell during the trading session as other companies were also affected.
Industry expert Adam Leon Smith of BCS, the Chartered Institute for IT, warned that it could even take “weeks” for all computers and systems to be fully restored.
“The fix will have to be applied to many computers around the world. So if computers are getting blue screens and endless loops, it could be more difficult and take days and weeks,” he said.
“Microsoft Windows isn’t the main OS for mission-critical systems, that’s Linux – and so this could have been much worse.”
CrowdStrike had earlier confirmed that Linux and Apple Mac systems had not been impacted by the bug.
The flawed update caused major infrastructure to grind to a halt with computer systems knocked offline, and many devices were showing the so-called “blue screen of death” as they got stuck in an endless cycle of trying to reboot themselves, affecting key sectors across the country.
The outage is “causing disruption in the majority of GP practices”, NHS England said, and ambulance services also reported increases in calls from patients who are unable to contact other NHS providers because of the IT issues.
The health service said patients should attend appointments unless told otherwise and should only contact their GP in urgent cases.
Across England, GP surgeries reported being unable to book appointments or access patient records as their EMIS Web system went down.
The National Pharmacy Association said “services in community pharmacies, including the accessing of prescriptions from GPs and medicine deliveries, are disrupted today”.
In an update on its website, Microsoft suggested users of virtual machines – a PC where the computer is not in the same place as the screen – turn their devices on and off again up to 15 times to help reboot the device and fix the issue.
“We’ve received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage,” the tech giant said.
In the UK, Sky News briefly went off air on Friday morning and Britain’s biggest train company warned passengers to expect disruption because of “widespread IT issues”, as did many major airlines and airports.
Around the world, banks, supermarkets and other major institutions reported computer issues disrupting services, while many businesses were unable to take digital payments or access key databases.
At airports, flights were cancelled and staff forced to check in passengers manually on specific flights to help ease long queues.
And one impacted airline, Ryanair, urged passengers whose flights have been cancelled to leave the airport.
The airline said in a statement: “Unfortunately, we’ve been forced to cancel a small number of flights today due to this global third-party IT outage.
“Affected passengers have been notified and are advised to log into their myRyanair account once systems are back online to see their options.
“A full list of cancellations is available at ryanair.com . If your flight has been cancelled, we kindly request that you leave the airport as the IT outage means we cannot currently assist passengers at the airport.
“We sincerely apologise for any inconvenience caused by this global third-party IT outage, and we are working hard to minimise disruption and keep passengers informed.”
Earlier in the day, Govia Thameslink Railway – parent company of Southern, Thameslink, Gatwick Express and Great Northern – warned passengers to expect delays.
GP practices across England warned they could not access EMIS Web, the most widely used clinical system for primary care in the UK.
It enables GP practices to book appointments and examine records, and includes a clinical decision support tool as well as helping with admin.
Professor Ciaran Martin, the founding chief executive of the National Cyber Security Centre (NCSC) said the incident was an “incredibly powerful illustration of our global digital vulnerabilities and the fragility of core Internet infrastructure.”
Prof Martin, who now works at the University of Oxford, said it was hard to estimate how long it would take to recover from the outage.
“The underlying problem is fixed and the fixes are being implemented. Some industries can recovery quickly. But others like aviation will have long backlogs. That said, I’d be surprised if we were still facing serious problems this time next week.”
He added that the cyber industry also needed to “get better” at “finding and fixing these single points of failure across all core digital infrastructure” and “managing how we cope when IT services fall over”, saying the world faced “more of these types of events” if changes were not made.