Analyzing structured data can aid in better diagnosis and patient care. However, analyzing unstructured data can fuel revolutionary medical breakthroughs and discoveries.
This is the gist of the topic we will be discussing today. It’s very interesting to observe that so many radical advancements in the space of healthcare technology have happened with just 10-20% of usable healthcare data.
Statistics reveal that over 90% of the data in this spectrum is unstructured, which translates to data that is less usable and more difficult to understand, interpret, and apply. From analog data such as a doctor’s prescription to digital data in the form of medical imaging and audiovisual data, unstructured data is of different types.
Such massive chunks of unstructured data are home to incredible insights that can fast-forward healthcare advancements by decades. Be it aiding drug discovery for critical life-consuming auto-immune diseases to data that can assist healthcare insurance companies in risk assessments, unstructured data can pave the way for unknown possibilities.
When such ambitions are in place, interpretability and interoperability of healthcare data become crucial. With stringent guidelines and enforcement of regulatory compliance such as GDPR and HIPAA in place, what becomes inevitable is healthcare data de-identification.
We have already covered an extensive article on demystifying structured healthcare data and unstructured healthcare data. There’s a dedicated (read extensive) article on healthcare data de-identification as well. We urge you to read them for holistic information as we will have this article for a special piece on unstructured data de-identification.
Challenges In De-identifying Unstructured Data
As the name suggests, unstructured data isn’t organized. It’s scattered in terms of formats, file types, sizes, context, and more. The mere fact that unstructured data exists in the forms of audio, text, medical imaging, analog entries, and more makes it all the more challenging to understand Personal Information Identifiers (PII), which is essential in unstructured data de-identification.
To give you a glimpse of the fundamental challenges, here’s a quick list:
- Contextual understanding – where it’s difficult for an AI stakeholder to understand the specific context behind a particular portion or aspect of unstructured data. For instance, understanding whether a name is a company name, the name of a person, or a product name can bring in a dilemma on whether it should be de-identified.
- Non-textual data – where identifying auditory or visual cues for names or PIIs can be a daunting task as a stakeholder may have to sit through hours and hours of footage or recording trying to de-identify critical aspects.
- Ambiguity – this is specifically true in the context of analog data such as a doctor’s prescription or a hospital entry in a register. From handwriting to limitations of expression in natural language, it could make data de-identification a complex task.
Unstructured Data De-identification Best Practices
The process of removing PIIs from unstructured data is quite different from structured data de-identification but not impossible. Through a systematic and contextual approach, the potential of unstructured data can be seamlessly tapped into. Let’s look at the different ways this can be achieved.
Image Redaction: This is with respect to medical imaging data and involves the removal of patient identifiers and blurring out anatomical references and portions from images. These are replaced by special characters to still retain the diagnostic functionality and utility of imaging data.
Pattern Matching: Some of the most common PIIs such as names, contact details, and addresses can be detected and removed using the wisdom of studying predefined patterns.
Differential Privacy Or Data Perturbation: This involves the inclusion of controlled noise to conceal data or attributes that can be traced back to an individual. This ideal method not only ensures data de-identification but the retaining of the dataset’s statistical properties for analyses as well.
Data De-identification: This is one of the most reliable and effective ways to remove PIIs from unstructured data. This can be implemented in one of two ways:
- Supervised learning – where a model is trained to classify text or data as PII or non-PII
- Unsupervised learning – where a model is trained to autonomously learn to detect patterns in identifying PIIs
This method ensures the safeguarding of patient privacy while still keeping human intervention for the most redundant aspects of the task. Stakeholders and healthcare data providers deploying ML techniques to de-identify unstructured data can simply have a human-enabled quality assurance process to ensure fairness, relevance, and accuracy of outcomes.
Data Masking: Data masking is the digital wordplay to de-identify healthcare data, where specific identifiers are made generic or vague through niche techniques such as:
- Tokenization – involving the replacement of PIIs with characters or tokens
- Generalization – by replacing specific PII values with generic/vague ones
- Shuffling – by jumbling PIIs to make them ambiguous
However, this method comes with a limitation that with sophisticated model or approach, data can be made re-identifiable
Outsourcing To Market Players
The only right approach to ensuring the process of unstructured data de-identification is airtight, foolproof and adherent to HIPAA guidelines is to outsource the tasks to a reliable service provider like Shaip. With cutting-edge models and rigid quality assurance protocols, we ensure human oversight in data privacy is mitigated at all times.
Having been a market-dominant enterprise for years, we understand the criticality of your projects. So, get in touch with us today to optimize your healthcare ambitions with healthcare data de-identified by Shaip.