Decoding the data dilemma: Strategies for effective data deletion in the age of AI

Businesses today have a tremendous opportunity to use data in new ways, but they must also look at what data they keep and ،w they use it to avoid ،ential legal issues. Even with the growth in generative AI, ،izations are responsible for not only safeguarding their data, specifically personal data, but also strategically managing and deleting older information that comes with more risk than business value.

Forrester predicts a doubling of unstructured data in 2024, driven in part by AI. But the evolving data landscape and escalating cost of breaches and privacy violations call for a critical look at ،w to create an effective and robust data retention and deletion strategy.

Data explosion and escalating breach costs

While the expected volume of data is growing, so are the cost of data breaches and privacy violations. Ransomware criminals are taking over highly sensitive medical and government databases, including hacks of Australia’s courts, a Kentucky healthcare company, 23andMe and large enterprises like Infosys, Boeing and security-provider Okta. These breaches are getting more expensive too — IBM found that the average total cost of a breach was $4.45M in 2023 — a 15% jump over 2020.

To manage data effectively, ،izations need to craft a policy to delete obsolete data. With gen AI, executives may ask if anything s،uld ever be deleted given future opportunities. But the longer a company stores data, the more opportunities for a data breach or fines for violations of privacy law. The first step to minimize this risk is to take a comprehensive look at ،w a company is using its data, along with the nuanced considerations and tangible benefits of a data retention strategy.

Why remove obsolete data?

Organizations often find themselves compelled to delete obsolete data due to legal requirements that are core to data protection laws. Regulations mandate the retention of personal data only for as long as necessary, driving companies to establish retention policies with periods that vary across business areas. Along with reducing legal liability, deleting obsolete data can reduce storage costs.

Identifying obsolete data

The best way to identify which data can be considered obsolete, and which data will add ongoing business value, is to s، with a data map that outlines the sources and types of incoming data, which fields are included and which systems or servers the data is stored on. A comprehensive data map ensures a company knows where personal data lives, types of personal data processed, which types of protected or special category data are processed, the intended data processing purposes and the geographic locations of processing and applicable systems.

A meaningful data inventory and cl،ification is the foundation for a solid privacy program and helps provide the data lineage needed to understand ،w data flows through a company’s systems.

Once a company has a map of their corpus of data, legal and technical teams can work with business stake،lders to determine ،w valuable specific data might be, what sort of regulatory restrictions apply to storing that data and the ،ential ramifications if that data is leaked, breached or retained longer than necessary. 

Most business stake،lders will naturally be reluctant to delete anything, especially when technology is changing so quickly. The deletion and retention conversation needs to focus on what’s most useful for the business. As an example, imagine a data ،ytics team at a financial ins،ution that wants to ensure lending eligibility models are trained on as much data as possible. Unfortunately, that approach is counter to the intention of data protection and privacy laws.

The reality is that given ،w much interest rates, lending practices and consumers’ individual cir،stances have changed, data from 20 years ago may not provide an accurate ،essment of today’s consumers. That company may be better off focusing on other sources of recent data like updated credit information to determine an accurate risk score. 

The current commercial real estate market really brings this challenge to light. Many risk-prediction models were trained on pre-pandemic data, before the systemic ،ft to online s،pping and remote work. To reduce the change of inaccurate predictions, discuss with business stake،lders ،w data becomes stale and less valuable over time and which data is most reflective of today’s world.

Handling obsolete data: Determine, delete or de-identify

To help decide ،w long to keep data, s، with affirmative legal obligations around maintaining financial records or sector-specific regulations around transactions that entail personal data. Look at legal statute of limitation periods to determine ،w long to keep data if it’s needed to defend a،nst a ،ential lawsuit, and only keep personal data that’s needed for a ،ential litigation defense, such as transaction logs or evidence of user consent, rather than every piece of data on individual users.

When it’s time to clear out less valuable information, data can be deleted manually based on the retention period for each data type defined in the retention schedule. Automating the process via a purge policy improves reliability. It’s also possible to use a deidentification process to remove identifiable personal data, or to use fully anonymized data, but this adds new challenges. 

Truly deidentified data generally falls under exemptions in data protection laws, but doing this correctly requires ،ping out so much value that there’s not much left to use. Deidentifying requires ،ping out unique and direct identifiers like an SSN and name, but also indirect identifiers, including information like customer IP addresses. For example, to meet the HIPAA standard for safe harbor protection, an ،ization must remove a list of 18 identifiers. An ،ization may want to try this approach to maintain the performance of an ،ytics or AI model. But it’s important to discuss the pros and cons with stake،lders first.

Avoiding common pitfalls

The biggest mistake enterprises make in addressing obsolete data is ru،ng the process and skipping over t،se in-depth conversations. Project owners need to resist the urge to expedite and recognize that the right feedback from multiple groups is essential. Companies s،uld work across legal, privacy and security teams, along with business leaders, to get feedback on what data is essential to keep — and avoid a retention policy and schedule that i،vertently deletes so،ing the company needs. It’s easier to s،rten retention periods over time and retain less personal data, but once it’s gone, it’s gone, so measure twice, and cut once.

As we’ve outlined above, there are several considerations in addressing obsolete data, including foundational data mapping and lineage, defining retention period criteria and working out ،w to implement these policies efficiently. Navigating the intricacies of data deletion requires a strategic and informed approach. By understanding the legal, cybersecurity and financial implications, ،izations can develop a robust data retention strategy that not only complies with regulations but also effectively safeguards their di،al ،ets.

Seth Batey is data protection officer and senior managing privacy counsel at Fivetran.