Red Flags When Choosing a Data Partner

Have you encountered these red flags during a data evaluation?

In the past year, we've seen an increase in data suppliers:

• serving replay data from previous event dates
• licensing data they don't have the rights to
• manipulating data to make it seem more unique
• manipulating data to make it seem like there are more pings per device

Here are some warning signs to watch out for when you evaluate data for your organization.

Cherry-picking data samples 

Be wary of vendors who provide a hand-picked, overly positive representation of their data set for evaluation.

This is the easiest way to misrepresent overall data quality. 

What to do: Ask for a specific week of data or a live feed. Specific samples can provide a signal that the evaluation data set is a fair and accurate representation of the vendor's actual data offering.

If you can’t get a specific sample, then you should at least get a solid explanation as to why not.

Overemphasis on proprietary methodologies

Some vendors talk about proprietary methods and algorithms that allegedly enhance data quality and accuracy.

Usually, they consist of anomaly detection, deduplication of records, and a few other cleansing tactics. 

But these aren’t proprietary methods. They’re industry norms. 

What to do: Be wary of companies who try to sell you on a patented process. It's usually just cover for avoiding transparency around processes.

Vendors should be able to provide clear, straightforward explanations of their techniques and willing to answer your questions.

Going dark during anomalies or industry events

In data, anomalies happen. They’re a feature of the landscape and sometimes make up the very landscape themselves. Vendors may attempt to disregard inconsistencies or irregularities during the evaluation process.

Your data team will likely be on the lookout for patterns of missing or suspicious values. Can the vendor provide a reasonable explanation for these occurrences?

What to do: Ask your vendor about their track record of client communication during industry events. How do they handle supply changes? What teams and systems do they have in place when you need understanding around a policy change that affects data quantity or quality?

Again, it's not about downplaying the fact that anomalies arise or industry-changing events happen; it's about understanding how a partner responds during those uncertain times.

Exaggerated precision claims 

Some vendors might claim unrealistic levels of precision for their location data, even though true precision may vary depending on factors such as GPS signal quality or data source.

What to do: Ask vendors to explain how they measure precision and what variables can affect it. Request a precision metric sample for their data and, if possible, verify it against independent benchmarks.

It's also a good idea to inquire how they handle situations where precision may be lower, such as in areas with poor GPS signal quality or data sourced from less precise methods. 

The main takeaway: Ask hard questions.

Like any other relationship, a strong data partnership begins with transparency, honesty, and open communication.

A worthy partner will appreciate your diligence and provide clear, satisfactory answers to your questions, even if that sometimes means saying, “I don’t know.”

Keep these red flags in mind as you navigate your options, and never settle for less reliable data than your project deserves.