Spring Cleaning: Data Purging Best Practices

Spring Cleaning: Data Purging Best Practices (Part One)

May 13, 2013

Authored by Alana Smith, TeamCain

It’s that time of year again... Spring Cleaning! It’s time to blow away the winter cobwebs and let the sun shine through. In honour of Spring Cleaning, we’re bringing you another 3 Part Blog Series (last time it was all about Business Intelligence) but this time we are focussing on data purging and archiving best practices. We’re going to look at the “Why”, “When” and “What” when it comes to data purging and archiving best practices.

We’ve found through our various surveys that both JD Edwards EnterpriseOne and World users haven’t been thinking about a data purge – 60% said they have never purged and archived. We also discovered that almost half of those users have over 10 years of data in their system. It’s always interesting to look at how companies approach a data purge project and via discussions we’ve had with customers, a purge and archive project usually falls to a B or C level project. There’s always ways to get around physically doing the purge (like increasing disk space) and so it gets put off for a couple years.

But by putting a purge project off, you’re running the risk of too much information which can hurt your system and business in the long run. So let’s explore WHY you should purge and archive...

JD Edwards is a batch system and transactional. Over time, performance can drop and degrade significantly... it’s subtle, but it does drop. By cleaning out dead data, you can increase performance. Fun fact: If you can trim just 10 minutes a day from your users’ time (likely waiting for batch jobs to finish, for reports to run, for online screens to switch etc.), you save the equivalent of a full time person!

Business Change
Mergers, acquisitions, closures, sold off lines of business... all of these add up to process changes that can impact your data. It impacts the data that you have and the data that you want to keep. If you don’t need it, there is no point in keeping it around unless you have to deal with strong legal or legislative reasons.

Disk & Backup
Disk is “cheap” but that doesn’t mean that it’s never-ending.  Eventually it runs out and you either have to get rid of items on that disk or buy more disks. With all that extra data, backups run the risk of interfering with other jobs and then everything takes longer and longer and longer and longer... you get it. We have had customers who were hitting the wall for back up due to the amount of data they had.

Risk Avoidance
We live in a litigious society and business world. If you have the data and you’re asked to share it for legal purposes, you need to turn it over. If you are in an industry that tends to be “exposed,” you can avoid potential legal risks with a data purge. We have had clients who are quite aggressive in clearing out old data for this very reason.

Spin Off Impact
If you offload or pull data from JD Edwards to update or populate information in other systems, like Business Intelligence or disaster recovery, then those systems will experience a hindrance in terms of performance and overall efficiency.

Through our experience, this is the most common reason why companies are thinking about a purge project, especially since many support dates are coming to an end. Purging before an upgrade can reduce risk and speed things up. Data file conversions are one of the longest parts of an upgrade but if a purge project has been completed, this is no longer an issue. You can always upgrade the archived files post upgrade to keep them in sync with your latest results. In a recent TeamCain webinar, Exide Technologies explained that the reason why they purged their JD Edwards World data was because they were migrating to EnterpriseOne, so they needed to remove as much data as possible to ease the data migration.

Human nature proves that if something is not done before, odds of doing it after drop. Thinking about your own situation, how many years of data do you have in your system? If you’re in the double digits, it is probably just time to get it done.

Stay tuned for Part Two of this blog series about when is the best time to purge and archive.

If you liked this blog post, you might be interested in:
What Are the Top Business Challenges You May Face in 2013?
Three Tips to Maximize your EnterpriseOne Upgrade Value
The Case for Data Purging in JD Edwards