DQ Check for DataFrame - Complete Guide to Data Quality Validation

Introduction to DQ Check-

DQ Check (Data Quality Check) is the process of validating data to ensure it is accurate, complete, consistent, and reliable before analysis or machine learning tasks.

diagram of dq check for dataframe and data quality check with pandas .The type of dq validation in spark data quality framework , dataframe validation in python — type of data quality checks with example

In data engineering and data science projects, DataFrames (Pandas or Spark) are widely used. Performing DQ checks on DataFrames helps:

Detect missing or invalid values
Ensure correct data types
Identify duplicates
Improve ML model accuracy
Prevent pipeline failures

Why DQ Check is Important?

Poor data quality leads to:

Wrong business insights
Poor ML model performance
Data pipeline failures
Incorrect reporting

A proper DQ check ensures clean, trustworthy, and usable data for analytics and AI models.

Common Data Quality Checks for DataFrame

1.Null / Missing Value Check

Pandas Example

df.isnull().sum()

Spark Example

from pyspark.sql.functions import col

df.select([col(c).isNull().sum().alias(c) for c in df.columns]).show()

Purpose: Identify missing values that can affect model accuracy.

2. Duplicate Records Check

Pandas

df.duplicated().sum()

Spark

df.groupBy(df.columns).count().filter("count > 1").show()

Duplicate data leads to biased results and wrong aggregations.

3.Data Type Validation

Pandas
df.dtypes
Spark
df.printSchema()

Ensure correct types:

Age → Integer
Salary → Float
Date → Timestamp

4.Range Validation (Business Rules)

Example:

Age should be between 18–60
Salary must be greater than 0

df[(df['Age'] < 18) | (df['Age'] > 60)]

Helps detect invalid or corrupted values.

5. Unique Value Check

df['EmployeeID'].nunique()

Ensures primary keys are unique.

6.Outlier Detection

Q1 = df['Salary'].quantile(0.25)

Q3 = df['Salary'].quantile(0.75)

IQR = Q3 - Q1

df[(df['Salary'] < Q1–1.5 * IQR) | (df['Salary'] > Q3 + 1.5 * IQR)]

Detects abnormal salary or numeric values.

7.Consistency Check

df['Gender'].value_counts()

Valid: Male, Female
Invalid: M, male, FEMALE, null

Use mapping or standardization to fix inconsistencies.

8.Referential Integrity Check

Used when multiple tables exist.

df_orders[~df_orders['cust_id'].isin(df_customers['cust_id'])]

Ensures foreign key consistency.

DQ Check Using PySpark (Production Ready)

from pyspark.sql.functions import col

# Null check

df.select([col(c).isNull().sum().alias(c) for c in df.columns]).show()

# Duplicate check

df.groupBy(df.columns).count().filter("count > 1").show()

# Range validation

df.filter((col("Age") < 18) | (col("Age") > 60)).show()

Benefits of Data Quality Checks

Improves ML accuracy
Prevents ETL failures
Enhances business trust
Detects data drift
Ensures reliable reporting

Best Practices for DQ Checks

Run DQ checks before model training
Automate validations in ETL pipelines
Maintain DQ logs
Use tools like Great Expectations
Validate both source and target data

Conclusion

Data Quality Checks (DQ Checks) are a critical part of modern data engineering and analytics pipelines. Clean data ensures accurate predictions, better business decisions, and stable ML systems.

Bad Data = Bad Decisions
Clean Data = Reliable Insights

About the Author
I am a Data Science Engineer specializing in Machine Learning, Generative AI, 
Cloud Computing, Hadoop, Scala, Java, and Python With expertise in 
cutting-edge technologies, I share valuable insights, blogging tips, and
 tech tutorials on DeveloperIndian.com, helping developers and data enthusiasts 
stay ahead in the industry.

Read Similar article

Example on artificial intelligence

TheDailyVibe DeveloperIndian

Search This Blog