Data Quality Framework

[kassyray]

Research data quality frameworks that are appropriate for checking data prior to ingestion into main pipeline.

Framework should at minimum include:

• Deduplication of client records (checking for uniqueness of records)
• Non-nullness
• Data type
• Accepted values
• Ensuring calculated ages in the immunization history are valid (not negative years).

This could be a good opportunity to use data quality tools outlined in the dev-guide, if relevant for task at hand.

[kassyray]

Resources:

Great Expectations in Kubeflow:

• https://medium.com/analytics-vidhya/data-quality-assurance-with-great-expectations-and-kubeflow-pipelines-d83449fbaa81
• Kubeflow pipelines: understanding greatexpectation component StatCan/aaw#1202

Monitoring Data Quality in Data Lake:

• https://towardsdatascience.com/monitoring-data-quality-in-a-data-lake-using-great-expectations-and-allure-built-serverless-47fa1791af6a/
• This is a good example of visual workflows for data quality and how data quality can fit in overall workflows.

Data Profiling:

• https://docs.profiling.ydata.ai/latest/