AI Autopsy: The Long Road to Recovery for the British Library

Phil Muncaster uncovers a cautionary tale that will resonate with many organisations hit by ransomware

In October 2023, the British Library suffered a double extortion ransomware attack which knocked out much of its server estate and led to the theft of 600GB of internal data. As of March 2024, the government-sponsored public body had spent £1.6m recovering from the breach. The incident severely impacted the institution’s custodianship and research “purposes” – eroding value and reputation.

Yet out of crisis can come opportunity. The British Library, with a collection of over 25 million books, is the world’s most extensive library. It has decided to share a detailed cyber incident report in its aftermath. This 18-page document is a cautionary tale that offers CISOs plenty of food for thought.

What happened?

It’s still not 100% certain how the Rhysida ransomware affiliate gained initial access to the library’s IT network. The institution has evidence of unauthorised access on October 25 2023, and believes that the “most likely source” of the attack is the compromise of privileged account credentials. That means most likely either a phishing or brute-force attack. An extensive ecosystem of partners and suppliers makes this more likely.

“The server, although protected by firewalls and anti-virus software, crucially didn’t have multi-factor authentication (MFA) enabled”

“The library utilises numerous trusted partners for software development, IT maintenance, and other forms of consultancy, and whose staff have a variety of levels of access to our network or infrastructure dependent on their contract with us and the level of supervision or vetting that is undertaken,” the report reveals.

The library’s best guess is that this enabled access to an on-premises “terminal server” which, although protected by firewalls and anti-virus software, crucially didn’t have multi-factor authentication (MFA) enabled. The Rhysida threat actors stole around half a million digital documents from the library, including personally identifiable information (PII) on users and staff.

As the British Library laudably refused to pay a ransom, the PII was put up for sale and then dumped on the dark web. Around 60% of the data – belonging to finance, tech and people teams – was lifted wholesale from network drives. Most of the rest was obtained after the actors searched the library network via keywords such as “passport” or “confidential” – copying files from drives used by staff for personal purposes. The group also hijacked native utilities to create backup copies of 22 databases, which were subsequently exfiltrated.

However, the most damaging part of the raid, was the encryption of critical data and systems, and the destruction of some servers to disrupt system recovery and preserve the attackers’ anonymity, according to the report.

“While we have secure copies of all our digital collections, we have been hampered by the lack of viable infrastructure on which to restore it”

“While we have secure copies of all our digital collections – both born-digital and digitised content, and the metadata that describes it – we have been hampered by the lack of viable infrastructure on which to restore it,” it explains. “The re-build of our infrastructure, on equipment approved and purchased before the attack, has been under way since December 2023 and remains ongoing.”

Another major hurdle to recovery is that the library’s “major software systems” – including those which catalogue and ingest “non-print legal deposit” (NPLD) material, and inter-library loans – can’t simply be restored in their current form. That’s because they’re no longer supported or won’t work with this new infrastructure.

Time to rebuild

Following an initial crisis management phase, in which the library’s gold and silver crisis response teams leapt into action, leadership began a transition from response to recovery. Its ‘Rebuild and Renew’ programme features a six-month phase to “identify and implement interim solutions to restore services, internal processes and partnerships”. This is to be followed by an 18-month renew phase, designed to create new infrastructure through upgrades, adaptations and new technology.

The silver and gold crisis response teams briefly superseded normal management structures during the crisis phase – handling all technical responses, workarounds, expenditure, and internal/external comms. Importantly, they also recorded “practical operational-level lessons” about their disaster management efforts, which will help to improve future processes.

Takeaways for CISOs

The report’s main value for readers is in sharing the early lessons learned from the incident. They largely revolve around new technology, risk management processes, staffing and incident response.

1: Revamp technology

The report highlights that the library must:

• Improve network monitoring to ensure there are no coverage gaps
• Fully implement MFA on all internet-facing endpoints, regardless of technical difficulties
• Segment its network to limit the damage caused by a successful breach. Legacy network topology restricted this previously
• Eliminate legacy infrastructure and apps which are hard to maintain, secure and restore
• Prioritise remediating legacy tech issues at every level in the organisation

2: Revisit risk management

The library also acknowledges that it must enhance intrusion response processes, so that in-depth security reviews are commissioned after even the smallest sign of network intrusion. That’s because it’s easy for an attacker to hide in plain sight and achieve persistence after gaining initial access.

All cyber risk should also be flagged to senior management to ensure it is able to benefit from a holistic view, the library claims. Low-level risks might fly under the radar separately but could combine to indicate something more serious. Finally, business continuity plans should be run for individual systems/services and a mass outage of all systems.

Meeting minimum cybersecurity standards and regularly reviewing and auditing policies and processes is also essential for managing risk effectively. The library admits it fell out of compliance with Cyber Essentials Plus in 2022, due to legacy systems.

3: Get incident response right

The library warns that recovery must be prioritised alongside security. “Given that no security is perfect, the ability to quickly recover is essential when (not if) an attack is successful. Investment in security needs to be balanced against investment in back-up and recovery capabilities,” the report notes.

Having external security expertise on retainer is essential in this regard; accelerating incident response, boosting resilience and improving the depth of analysis in the early stages of an attack.

4: Don’t forget your people

Motivated security-aware employees are the final piece in the puzzle, according to the British Library report. That’s why the library recommends:

• Regular staff training and awareness communication, to cover evolving risks and tailored to role and level of expertise
• Proactive management of staff wellbeing as part of incident management, given that cyber attacks are “deeply upsetting” for staff whose data is stolen, and whose work is interrupted
• Reviews of acceptable IT use policies where, for example, employees use network storage for personal use
• Ensuring all senior executives and board members have “a clear and holistic understanding of cyber risk” so they can make the best investment decisions
• Current risks and mitigations should be frequently discussed and a board member or advisor with cyber expertise should be recruited
• Collaboration and information sharing about common threats and best practices with sector peers

No silver bullet

The British Library suffered one of the worst ransomware breaches in recent memory for a combination of reasons, and there’s no single silver bullet solution that would have kept it safe. From a high-level perspective, however, some common problems emerge.

The library’s on-premises infrastructure was far worse affected than its cloud systems, especially in the context of Rhysida’s destructive efforts. Its core, cloud-based email, finance, HR and payroll systems were undamaged, for example.

“The library admits it fell out of compliance with Cyber Essentials Plus in 2022”

The library’s technology department was also overstretched in the run-up to the incident, and there are concerns that skills shortages may create additional risk as the institutions looks to migrate to next-gen systems. Developing a security-by-design culture has so far proven elusive.

Ultimately, no organisation has unlimited resources, so understanding where to focus efforts first is key – which is where system visibility is important.

“Although the security measures we had in place on October 28 2023 were extensive and had been accredited and stress-tested, with the benefit of hindsight there is much we wish we had understood better or had prioritised differently,” the report concludes.

Hopefully, the insight it offers will help CISOs better develop their own risk management and incident response strategies, to avoid a similar fate.