Overview
We are searching for a skilled Data Engineer to join our team.
The ideal candidate will have hands-on experience in development and production implementation with Azure Databricks and Spark, proficiency in Python (PySpark) and SQL coding, and a strong understanding of data lake architecture, ETL processes, and data quality management.
Experience with Azure data analytics services and excellent technical writing skills are also required.
Responsibilities
- Develop and implement data processing pipelines using Databricks and Spark.
- Write efficient and optimized code in Python (PySpark) and SQL for TB-scale data processing.
- Design and maintain data lake architecture and ETL processes.
- Ensure data quality and integrity through robust data quality management practices.
- Utilize Azure data analytics services for data storage, processing, and analysis.
- Create comprehensive technical documentation, including requirements and testing documents.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Maintain version control using Git.
Requirements
- 5+ years of hands-on experience in development and production implementation with Azure Databricks and Spark.
- Proficiency in Python (PySpark) and SQL coding.
- Proven ability to optimize performance for TB-scale data processing.
- Strong understanding of data lake architecture, ETL processes, and data quality management.
- Experience with Azure data analytics services.
- Excellent technical writing skills for creating requirements and testing documents.
- Ability to communicate technical concepts clearly and concisely.
- Knowledge of Git.
Seniority level
Employment type
Job function
Industries
- IT Services and IT Consulting
#J-18808-Ljbffr