The Case for Data Purging in JD Edwards

August 21, 2012

Authored by Michael Guerin, TeamCain

All too often we see and talk with customers who have not done a data purge in 7, 10, even 14 years (what’s your number?). I’d like to explore here the “whys” we hear the most, and what you should consider if they apply to you.

By the way, you are not alone – our surveys show that the main reasons why organizations don’t purge their JDE data are:

  • The project never gets approvedcompeting priorities, and a data purge typically ends up being a B or C item.   With ever expanding wish lists of projects, getting a purge and archive project past the gatekeepers can be difficult. 
  • JD Edwards is a big, interrelated system - you hold off because you don’t know what you will get (will the accountants or operations staff be outside your office with pitchforks because you ended up blowing away stuff they needed).
  • You don’t feel there is a needperformance is not suffering, the users are not griping. 
  • The users really, really need all of that datathey never know when they will need to grab a sales order from 2003 or do a year to year comparison between three and four years ago. 

What have your peers said?

Anecdotally, what we have seen from our conversations and surveys is: 

  • More than 80% of customers have more than 5 years of data in their files, and nearly half have 10 or more years (the largest we have seen is 23 years).
  • Half of the customers have been held back by concerns about whether their data would safely survive a purge (or would it look like a train wreck).  One CIO specifically stated that this concern was what kept him up at night … not fun. 
  • Half of the customers who have done purges used the standard JDE tools (like P00Purge); most did so for “spot” purges and were nervous about the implications (or did not realize them).  

Why should I purge anyway?

Lots of reasons!  Let’s look at them in terms of the reasons why purges don’t get done …

You don’t know what you will get (fear of the unknown)

Fearing the unknown is not a bad thing – that little voice at the back of your mind is normally what keeps you out of trouble (scientifically it is proven that this part of the brain is not fully developed in teenagers, which explains a lot doesn’t it!)  The fear is usually based on the fundamental intercorrectedness of JD Edwards.  What if I miss a table that needs to be purged?  What about the custom files we have?  How do I figure out which ones to purge? 

Unfortunately, this kind of fear will not go away, and in the meantime your data is growing.  What you need is a proven tool that removes the fear – based on how it is built, based on what other customers have been able to accomplish.  So… simply put… get over it!  There is really no validity to this fear these days.

Performance is not suffering and we have lots of disk space

This is an interesting one.  Performance is a relative thing.  Disk is cheap – right?  There is a fallacy in these types of statements though.  If your performance is not “suffering” … how do you define suffering?  It is hard to picture an F0911 file with 400 million records as one that will not be longer and harder to chew through.  The thing of it is this … JD Edwards is a transactional system, and batch process driven.  Most processes (reports, on line entries, inquiries) will be impacted by file size – it is really a given. 

Even if you “only” end up with a 5% improvement in productivity – how many other things could you do today that would give that?  In this day and age, everyone needs to do more with less ... what a perfect way to accomplish this.  Most often as well, the ones that really are impacted are the hard core users and the processes at critical times – like month end processes or nightly updates.

Don’t forget about backups either.  The more you have, the longer the backups take.  This is a perfect example of the benefits too by the way – if you have ten years of data, and can purge 3 and archive 5 – your data backup for your production files will go down by 70 – 80%.  Nice savings there …

And yes disk is cheap – but really … is throwing more disk the answer?  Add in the performance implications and it really does end up being a false security. 

The users really need to get to all of that data

Really?  Have you challenged this at all?  In some cases it might be true … they might need to get to data from years gone by. We see this more in job based customers, or ones with retention needs that are industry based (pharma, some municipal spring to mind).  Most often though, if you push through the initial “Yeah we need access to 12 years of data” comment, the real need comes through.  Here are some sample conversations in this area (*Note: these are not real, but based on working with customers… I learned a long time back in my 30 year consulting career that the “why” question, although annoying sometimes, does get to the real need if you ask it enough!)

Scenario 1
Me:  How many years of data do you want to keep in production?
User:  We need to keep 12 years.
Me: Why?
User:  Well, sometimes we need to go back and find records from that far back.
Me:  When did this last happen, and what was the business case?
User:  Last May the VP of Sales wanted a report of sales by customer category for the last 10 years.
Me:  What was he looking for with the information?
User:  He wanted to see how sales by category were going.
Me:  Do you have a business intelligence system in place?
User:  Umm, I think so...
Me: Do he could get it from that?
User:  Yes, I guess he could...

Scenario 2
Me:  How many years of data do you want to keep in production?
User:  We need to keep 12 years.
Me: Why?
User:  We need to check for duplicate invoices from suppliers – some of ours are terrible and submit duplicates from years ago.
Me:  How do you know if it is a duplicate?
User:  It is the same invoice number.
Me:  How far back could this go?  Do you have a policy in place for this?  How often does it happen?
User:  Well, it could go back forever!  We don’t have a policy regarding how far back … if the supplier sends an invoice from 3 years ago we check and see if it should be paid.  This happens several times a year.
Me:  Most places we work with have policies in place where if an invoice is submitted that is over 6 months old. Well, it just is not processed.  If you had a listing of invoices by supplier by year or for a group of years … or had a file you could look at that just had the key information … would that solve it?
User:  That would probably work.

What’s the main point here? Rarely have we seen situations where there is real, defined and justified need for years and years of data. 

The justification for purging

In the end, the real life benefits for purging your data include:

  • You are upgrading “to the 9s” In this case, it is likely that your upgrade will go a lot quicker and smoother with a purge before you upgrade.  An upgrade in most cases involves upgrading your data files too … and this can take a long, long time if you have a lot of data. (EnterpriseOne users might be interested in this TeamCain webinar, "Upgrading EnterpriseOne in 2013? Why YOU should purge & archive NOW!")
  • With the upgrade – what about the garbage data from years gone by?  It happens … the older the data, the less likely it is to be clean; it’s the nature of the beast.  If you leave the data there as part of the upgrade, someone is going to have to look into the “junk” that is there … do you have time for this?
  • Your system and processes will run better / faster / smoother.  This is simple reality … less data = faster jobs.  Can you ever have a level of performance that cannot be improved?  My guess is no.
  • Purging does not equal archiving.  You can have your cake and eat it too.  Dump the old data – how much of it do you really need? Archive newer data – staff can still get to it, but it won’t affect performance and backup times. 
  • Has the business changed? Yes!  We see this a lot … after 7, 10, 14 years … you have the equivalent of dusty old magazines in your files – the company that was sold off 8 years ago, the line of business that was shut down.  There is no reason to keep this data – so get rid of it (safely). 

Let me know what you think.  We’ve “been there, done that” and have the t-shirt!  Would love to use our experience to help out… We do have an elegant solution to your purge and archive problems – find out more here.