Data cleaning – should this be done before or after a system migration?

by | Jun 23, 2016 | Omnichannel

The decision to migrate a key system is not taken lightly.  ERP projects, for example, typically cause business disruption, loss of revenue, and reduced productivity – and that is when they go according to plan!  Examples such as Hershey Foods’ botched SAP deployment, which lost $100 million in Halloween sales and 8% of their share price in 1 day, and Nike’s $400 million worldwide i2 deployment that lost $100 million in sales, reflect that even well-managed companies are not immune to project management failures.  Key findings are that planning and system suitability are key to robust outcomes.

Traditional project management tenets require data to be cleansed prior to deployment of the new system, whereas current lean project management standards advocate an ongoing process to eliminate waste of resources.

Supporters of the traditional model believe that data cleaning should be conducted as a discrete project, generally asking employees to collaborate with suppliers over Excel lists of products, descriptions, and attributes, with the goal to cull and correct SKU lists.  Once this is completed, updated data is imported to the new ERP.  This method avoids import of dead SKUs and incorrect data, for a pristine commencement of the new ERP – at least in theory.  The broad criticism of this approach involves the amount of time the data cleanse process takes – which, in our experience, is 6+ months – and the practical reality that even after this huge investment of time and resources, quality problems persist.  In addition to data entry errors, which are virtually impossible to eliminate, a significant proportion of ‘corrected’ data will become invalid during the cleanse project cycle.  ‘Churn’ is impossible to eliminate.

Proponents of modern ‘lean’ project management, with which SPS Commerce agrees, find that any advantage of cleaning data prior to ERP migration is outweighed by the negatives.

Time: a discrete cleanse project delays the actual ERP migration by 6+ months
Productivity: team members in buyer, logistics & warehouse, and accounts teams see an overwhelming proportion of their work hours absorbed by the cleanse project
Inefficiency: every SKU must be checked individually for validity and compliance
Quality: team members focus on the cleanse task, to the neglect of core duties
Error: data entry errors occur as manual updates are transcribed to Excel spreadsheets, including data for new fields that don’t exist in the legacy ERP
Complacency: companies believe that the project will effectively eliminate database error, and plans for ongoing maintenance are neglected
Future needs: this method does not inherently lead to an automated system of attribute management

The alternative, correcting data after the system migration, leverages modern technologies and methods to avoid pitfalls associated with the traditional approach.  With SPS Commerce, for example, automated Fulfilment EDI solutions be used as trigger-points for data review, on an ad-hoc basis.

Time: the ERP migration is not delayed by data cleanse
Productivity: time allocated to data cleaning is spread across a longer period
Efficiency: team members review data which has been found problematic, rather than reviewing every SKU
Quality: data maintenance is considered an ongoing core responsibility and conducted in parallel with other duties
Error: error rates are minimised through direct in-system ERP entry
Procedural rigour: companies implement a continuous process of data review and management
Future proof: retailers and consumer product companies are readily able to connect assortment (attribute automation) to their information system architecture

Modern lean project management practices support a clean-after approach, for better resource management, a higher quality outcome, and future-proofing.  However, if you are considering an ERP deployment, please talk to our consultants or partners to recommend a solution that fits your precise needs.

SPS Commerce Blog Team