Spring Cleaning: Data Purging Best Practices (Part Two)

Spring Cleaning: Data Purging Best Practices (Part Two)

May 15, 2013

Authored by Alana Smith, TeamCain

If you missed Part One, you can read it here.

In the first part of this blog series, we looked at why you should purge and archive. Now that you’ve had time to think about your own situation and contemplate your system performance (is it up to snuff?), let’s look at WHEN is the best time to purge and archive.

The question “When to purge and archive and how far back?” is really a question for the business. From the IT side of the equation, you’re working with the data that business users have to operate with and so the timing is something that they need to decide upon. Fauchon Paris realised it was time for a purge when their database capacity was running out, the system was slowing down and the IT department was getting complaints from operational users. You’ll have to make sure to consider “going-forward” plans as well – in order to keep the business running efficiently, how often do you want to purge in the future (monthly, quarterly, annually)?

In terms of how much data to keep in your system, the most common situation is the current year plus 2 years in production. If you already have an archive, you can keep another 2 years in archived tables. We have seen some clients have up to 7 years in production but it really depends on what industry you’re operating in.

You’ll also need to consider legislative rules and regulations in order to determine when you purge and how accessible archived data needs to be. In the industry that you work in, are you required to keep “x” number of years of data? We’ve seen at some clients that want to keep their data as clean as possible for reverse legal purposes – in an industry that is litigious by nature, minimizing the amount of data in your system can help reduce the risk of exposure.

From the user’s perspective, there never will be a good time to purge and archive the data they need. It’s like when I was a kid. My parents told me to clean out my room, to sort my toys into keep, giveaway, and trash... “Trash?!” said I, in utmost horror. “But I NEED it!! I don’t care if Barbie is missing an arm and I re-coloured her hair with magic markers... she stays!” But why did I need to keep her? Clearly, she had had a good run and it was time to move on. So when approaching users with a purge and archive project, if you’re met with “No, we need it” be prepared to ask “Why?” Most of the time, there are alternative ways to handle what users deem as required data. They may think they need 22 years of AP data to catch the occasional duplicate voucher, but there are better ways to handle that. 

Make sure to explore the data retention policy that is set up in your organization. You have to factor in legislative, business, and legal requirements as well as consider a historical comparison (i.e. what information have you needed in the past that is important for the future?) The timing of a data purge and archive project depends on you, your situation, and how quick your data grows.

Stay tuned for Part Three of this Blog Series about what to purge and archive.

If you liked this blog post, you might be interested in:
Spring Cleaning: Data Purging Best Practices (Part One)
Are you about to embark on a data purge project? Here are 6 key things to factor in...
The Case for Data Purging in JD Edwards