Data quality checks do not have to run only after all transformations are complete. In many applications, validating data at different stages makes problems easier to identify.
There are three useful approaches.
Pre-Validation
Pre-validation happens immediately after reading the source data.
For example, a pipeline can check whether required columns exist, whether the dataset contains records, and whether important identifiers are populated.
This helps identify upstream problems before the application spends time performing expensive transformations.
Inline Validation
Inline validation happens between transformation steps.
Consider a pipeline that joins customer records with order information. A validation check after the join can identify missing customer information or unexpected duplicate records.
This helps developers identify which transformation introduced a data quality problem.
Post-Validation
Post-validation happens after the final transformation and before writing data to the destination.
This is useful when the application must prevent invalid records from reaching reporting systems or downstream applications.
For production pipelines, combining these approaches provides better visibility into data quality throughout the processing workflow.
Handling Invalid Records Without Stopping the Entire Pipeline
Not every data quality problem needs to stop the complete Spark application.
For example, suppose an incoming file contains 100,000 records and 150 records have invalid order amounts.
Depending on business requirements, the application might continue processing the valid records while moving the rejected records into a separate location for investigation.
This is known as a quarantine approach.
A simple Spark implementation can separate records using application-level conditions:
from pyspark.sql.functions import col
valid_orders = orders_df.filter(
col("order_id").isNotNull() &
col("customer_id").isNotNull() &
(col("order_amount") >= 0)
)
invalid_orders = orders_df.filter(
col("order_id").isNull() |
col("customer_id").isNull() |
(col("order_amount") < 0)
)
valid_orders.write.mode("append").parquet(
"s3://my-data-bucket/orders/valid/"
)
invalid_orders.write.mode("append").parquet(
"s3://my-data-bucket/orders/quarantine/"
)
This example uses Spark DataFrame conditions to separate records. It is independent of the DQDL evaluation result.
The distinction is important: EvaluateDataQuality returns rule-level evaluation results. It should not be treated as an automatic mechanism for producing a complete rejected-record dataset.
If you need to identify individual invalid records, implement corresponding row-level validation conditions or use the supported row-level evaluation capabilities.
For production systems, include a reason for rejection, source filename, processing timestamp, and batch identifier in the quarantine output. This makes it easier to investigate and reprocess records later.
Managing Data Quality Rules in AWS Glue Data Catalog
Defining rules directly inside a Spark script is useful for smaller pipelines. However, larger data engineering teams may need to share validation rules across multiple applications.
For example, customer data might be processed by several Spark jobs. Each job should follow the same rules for customer identifiers, email addresses, and country codes.
Maintaining separate copies of these rules can lead to inconsistencies.
AWS Glue Data Quality supports managed rulesets that can be stored in the AWS Glue Data Catalog and reused by supported processing workloads.
This approach provides a centralized place to manage validation requirements.
A managed ruleset can be retrieved using the corresponding AWS Glue Data Quality APIs and then supplied to the Spark evaluation transform.
The execution role must have the required permissions to retrieve the ruleset.
For larger projects, keep business validation rules separate from application transformation logic. This makes rule changes easier to review and maintain without repeatedly modifying every Spark application.
Monitoring Data Quality Results
Publishing evaluation results to AWS Glue Data Quality provides a historical view of validation activity.
This is useful when a pipeline runs every day and developers need to understand whether data quality is improving or becoming worse.
For example, consider a daily sales processing job.
The team may want to monitor:
- Total records received.
- Missing customer identifiers.
- Duplicate transaction identifiers.
- Invalid transaction amounts.
- Changes in daily record counts.
The evaluation context helps group runs belonging to the same logical dataset.
Developers can review historical results through the AWS Glue console and supported Data Quality APIs.
Amazon EMR Data Quality also supports statistics and anomaly detection capabilities. These can help identify unusual changes in datasets without requiring every validation rule to use a fixed threshold.
For example, a sudden change in daily transaction volume may indicate an upstream ingestion problem even when all individual records satisfy the configured validation rules.
Best Practices for Running Data Quality Checks in Spark Jobs
Define Rules Based on Business Requirements
Do not add validation rules simply because a column exists.
Understand what makes the data valid for its intended purpose. For example, a negative transaction amount might be invalid in one system but acceptable for refunds in another.
Validation should reflect actual business rules.
Validate Before Writing Critical Data
Run important checks before publishing data to production tables or downstream systems.
This reduces the possibility of incorrect records reaching reporting and analytical workloads.
Separate Critical and Non-Critical Rules
Not every failed rule should have the same impact.
A missing primary identifier may require the job to stop, while a small percentage of missing optional fields may be acceptable.
Define which rules are mandatory and which allow a configurable failure threshold.
Use Consistent Evaluation Contexts
Use the same evaluation context for recurring runs of the same logical dataset.
This helps organize historical results and supports comparisons between executions.
Apply Appropriate IAM Permissions
Grant the EMR execution role only the permissions needed by the application.
Separate permissions for reading data, accessing managed rulesets, and publishing evaluation results where practical.
Avoid Repeated Full Dataset Validation
Validation checks can require Spark operations such as aggregations, distinct calculations, and scans across partitions.
Some rules may therefore add considerable processing work.
Choose rules carefully, avoid unnecessary repeated evaluations, and monitor the additional execution time.
Keep Validation Rules Maintainable
For smaller jobs, inline DQDL rules can be sufficient.
For larger environments, use centrally managed rulesets where appropriate. Keep rule names, descriptions, and evaluation contexts consistent across teams.
Common Mistakes When Implementing Data Quality Checks
| Mistake | Why it causes problems | Recommended approach |
|---|
| Using an unsupported EMR runtime | The evaluation transform may not be available | Verify the AWS Spark runtime version |
| Missing IAM permissions | Results may not be published or rulesets may not load | Review the execution role permissions |
| Writing data before validation | Invalid data can reach downstream systems | Validate before the final write |
| Treating rule errors as passed | Evaluation problems may go unnoticed | Handle Failed and Error outcomes |
| Using overly strict rules | Valid records may be rejected | Align rules with business requirements |
| Ignoring evaluation overhead | Additional Spark processing may increase job duration | Monitor execution time and resource usage |
| Assuming rule results contain rejected records | Rule-level results do not automatically provide a complete quarantine dataset | Implement separate row-level validation |
| Using different rules in different jobs | The same dataset may receive inconsistent validation | Maintain reusable rulesets |
Advantages and Disadvantages of Amazon EMR Data Quality
Advantages
- Data quality validation can run directly inside existing Spark applications.
- DQDL provides a consistent way to define validation rules.
- Developers can evaluate data before, during, or after transformations.
- Results can be published to AWS Glue for historical monitoring.
- Managed rulesets can be reused across supported workloads.
- Failed checks can be used to control downstream processing.
- The feature is available across EMR on EC2, EMR on EKS, and EMR Serverless when using a supported runtime.
Disadvantages
- It requires a compatible AWS runtime for Apache Spark.
- IAM permissions need to be configured correctly.
- Complex validation rules can add processing overhead.
- Teams still need to implement their own quarantine and recovery workflows.
- It is limited to Apache Spark applications on Amazon EMR.
- Incorrect validation thresholds can reject useful data or allow poor-quality records through.
Summary
Data quality should be part of the data processing workflow rather than something checked only after reports or downstream applications start showing incorrect results.
Amazon EMR Data Quality, powered by AWS Glue Data Quality, allows developers to evaluate Spark DataFrames using DQDL rules directly inside their EMR applications.
With the EvaluateDataQuality transform, teams can validate incoming records, check intermediate transformations, and prevent failed datasets from reaching downstream storage.
For smaller pipelines, inline rules provide a straightforward starting point. For larger environments, managed rulesets in the AWS Glue Data Catalog help maintain consistent validation requirements across applications.
The most important part is deciding how the application should respond when a rule fails. Some problems should stop the pipeline, while others can be handled through record filtering or quarantine.
When validation rules are designed around real business requirements and integrated at the right stages, data quality becomes a regular part of Spark processing rather than an additional step performed after problems appear.
Join the conversation! Your thoughts help the community grow.