Job Description

QA/Test EngineerExperience Required: 4–6 yearsPlease share skills matrix and make sure skill matrix matches the resumeETL testing, data quality testingWorkflow testing (Step Functions, Airflow)SQL (strong), Python, PySparkLogging and monitoring using CloudWatchCI/CD (Jenkins, GitHub Actions, CodePipeline)API testing and microservices validationSecurity testing (IAM, encryption, policies)Role Summary: The QA/Test Engineer is responsible for building and executing comprehensive test strategies for all ingestion and ETL pipelines. This role ensures that every pipeline meets the 98–99% data quality pass rate required by the SOW before production promotion, and that automated tests are embedded in the CI/CD process.Key Responsibilities:Design and implement a reusable test framework for unit, integration, and end-to-end testing of data ingestion and ETL pipelines.Write and maintain unit tests for individual transformation logic, connector behavior, and schema validation.Develop integration tests that validate end-to-end data flow from source ingestion through raw ? curated ? consumption layers.Build automated test suites that execute as part of the CI/CD pipeline, gating production deployments.Validate data quality check behavior: confirm that DQ rules (completeness, schema conformance, record counts, freshness) correctly trigger fail/alert actions.Perform staging environment validation using sample data provided by source owners before production promotion.Track and report test coverage, defect rates, and pipeline validation results for each milestone.Collaborate with Data/Cloud Engineers to define test cases, edge cases, and acceptance criteria for each data source.Support performance and load testing for streaming and high-volume batch sources where representative test data is available.Produce test artifacts, test plans, and validation reports as part of milestone documentation.Required Skills & Qualifications:4–6 years of experience in QA/test engineering with a focus on data pipelines, ETL processes, or data platforms.Strong proficiency in Python for test automation and data validation scripting.Experience with data testing frameworks (e.g., Great Expectations, dbt tests, pytest, or custom frameworks).Hands-on experience testing data pipelines on AWS (Glue, Spark, S3, Athena, Redshift).Understanding of data quality dimensions: completeness, accuracy, consistency, freshness, schema conformance.Experience integrating automated tests into CI/CD pipelines (GitHub Actions, CodePipeline, Jenkins).Ability to write SQL queries for data validation and reconciliation.Familiarity with test data management strategies for sensitive or regulated data.Strong documentation skills for test plans, test cases, and validation reports.Experience working in Agile/Scrum teams with 2-week sprint cycles.Preferred Skills:Experience with performance/load testing for data pipelines.Familiarity with contract testing for APIs and schema registries.Knowledge of data observability tools (Monte Carlo, Datafold, or similar).

Job Tags

Similar Jobs