Cleaning up Dirty Data

As the saying goes, data is the new oil.

But when it is contaminated, it clogs up the delicate mechanism which is business.

What kinds of dirty data do we see in our business and how do we deal with it?

1. Insecure Data

Data security and privacy laws like POPIA and PAIA impose financial penalties on businesses that don’t follow these laws to the letter. With steep fines for non-compliance, insecure data is quickly becoming one of the most dangerous types of dirty data.

Digital consent, opt-ins, and privacy notifications are the new norm in the day-to-day business landscape. So non-compliance with privacy regulations will end up costing organisations more in the long run if ignored.

Non-compliance can also negatively impact on productivity, reputation and disrupt business operations.

Remain within data privacy regulations:

Disorderly databases are the most likely candidates to house insecure data. There are several data hygiene practices you can implement to combat insecure data.

• Delete outdated and unusable records;
• Merge duplicates to prevent fragmented profiles;
• Implement a Document Management System with a customised, well thought through, data architecture, records keeping and metadata to categorise sensitive data and enforces security; and
• Consolidate your databases as much as possible using the same data architecture referred to above.

With a clean, organised and updated DMS and single data lake, complying with data privacy regulations becomes far more straightforward.

2. Inconsistent Data

Inconsistent or non-standardised data looks different but represents the same thing. Just like duplicate records exist in various places within your database, multiple versions of the same data elements can exist across different records in your system, for example “street” could be written down or captured at “St”, “Str” or even “Ave”.

Standardising data:

First, create standard naming conventions and ensure your organisation follows them closely. Where possible, instead of free text, let any data capturer use a term store with standard formats in a dropdown. As for existing inconsistent records, tools can normalise records in batches for more unified field names and more accurate segmentation.

There are also external databases which can be overlaid to attempt to normalise data by proxy or GIS mechanisms. Let’s say, for example, the address is captured as 32 1^st Road, in Newlands Cape Town. By overlaying a GIS and having four data points viz number, street, suburb and city, the data cleaner can establish that while there might be a 32 1^st Road, in Rondebosch Cape Town, the 1^st street in Newlands Cape Town is an Avenue and therefore the address should read 32 1^st Ave, in Newlands Cape Town. The more comparable data points, the better the accuracy.

Incorporating a data management tool that can standardise data from multiple sources helps create a centralised approach to data management. This enables data to be processed, analysed, and leveraged across each department. Establishing a successful data-sharing strategy increases accessibility throughout your organisation.

3. Too Much Data

Many people hoard data. Maintaining a sleek (but not small) database is a big part of data hygiene. It drives alignment between departments and improves accessibility throughout the organisation.

How to reduce database size:

It might seem like “too much data” can never be a bad thing, often a good portion of the data simply isn’t usable. This means that you are spending excess time digging through the bad to the good.

Data hoarding and outdated data go hand in hand, so you’ll find these two types of dirty data can be solved at the same time: Deletion features allows users to delete thousands of records at once.

4. Duplicate Data

In your software, duplicates are the doubling of information — for example, a single employee showing up twice under different companies, or with different job titles. They can show up in your prospect lists, contact data, and sales accounts.

Duplicates happen when data is muddled with copies during data migrations and manual inputs.

Duplicates have no place in the system of any data-driven organisation. Ridding your database of duplicates should be a top priority in any data hygiene campaign.

How to clean and prevent duplicates:

Before the age of mass data accumulation, manpower alone was enough to merge duplicates and link leads to accounts. Nowadays, there are automated solutions for detecting and merging duplicates.

External solutions to de-duplicate data, allow users to match leads, contacts, and accounts based on customisable criteria. This way, it prevents duplicates at all points of entry into your database.

Keep data fresh and up to date:

Purging your database of records created before a certain date can help expedite the process of cleaning outdated records.

5. Incomplete Data

Any incomplete data will certainly poke holes in your data driven strategies and decision making.

Without attributes like industry type, job title, or last name, you risk excluding valuable sources in forecasting models.

Fixing incomplete data:

You could combat incomplete records is to manually conduct research to append the missing fields, but this strategy is neither realistic nor scalable.

Enriching your data with a service is the best way to automate the filling of empty fields and gain a more complete dataset.

6. Inaccurate Data

If your data is wrong, you run into all sorts of problems including inaccurate, propensity modelling reporting and decision-making:

It’s far cheaper to verify and cleanse data regularly than to do nothing at all.

How to clean incorrect and inaccurate data:

Keeping track of all data entry points and diagnosing the cause of inaccurate data is the first step. If the problem is caused by external data sources, such as web forms or connected systems, seeking an external solution is the best way to maintain accuracy.

Data enrichment software corrects mistakes and overrides dirty data with clean data sourced from the most reliable sources. By augmenting existing data with purchased third-party information, organisations can attain more accurate data that may not have been possible before.

What are the Consequences of Dirty Data?

1. Ineffective Strategies

Dirty data creates an inaccurate data modelling and poor decision making.

Inaccurate data skews your understanding of the market and your target audience, which has a domino effect as it negatively influences your strategies.

2. Poor Customer Experience

When bad data results in poor customer experience, you’ll lose out on valuable prospects and fail to retain current customers.

3. Damaged Brand Reputation

Dirty data can hurt your company’s reputation in more ways than simply encouraging negative customer feedback.

4. Misinformed Decision-Making

When bad data contaminates your metrics and reporting, it can hurt your business.

In the past, executives and key stakeholders relied on instinct and intuition to make important long-term business decisions. Now, clean data provides decision-makers with the tools they need for accurate and comprehensive reporting.

5. Decreased ROI from Technologies

Bad data prevents your technology stack from operating at its full potential. When you invest in IT, you do so to improve the effectiveness and efficiency of your initiatives.

 Prevent Dirty Data from Entering Your System

 Begin with regular health assessments. You can do this manually or partner with your data provider.

• Use a good mix of data sources — first-party and third-party.
• Cleanse your data regularly and fill in any gaps by enriching each field with the most reliable source possible.
• Practice ongoing data management.

The key is to identify the types of bad data in your data lake, clear them out, and replenish them with a stream of high-quality, actionable data.