Direct message the job poster from Pyramid Consulting, Inc
**Job Description: Data Engineer**
We are searching for a skilled Data Engineer to join our team.
The ideal candidate will have hands-on experience in development and production implementation with Azure Databricks and Spark, proficiency in Python (PySpark) and SQL coding, and a strong understanding of data lake architecture, ETL processes, and data quality management.
Experience with Azure data analytics services and excellent technical writing skills are also required.
Responsibilities:
- Develop and implement data processing pipelines using Databricks and Spark.
- Write efficient and optimized code in Python (PySpark) and SQL for TB-scale data processing.
- Design and maintain data lake architecture and ETL processes.
- Ensure data quality and integrity through robust data quality management practices.
- Utilize Azure data analytics services for data storage, processing, and analysis.
- Create comprehensive technical documentation, including requirements and testing documents.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Maintain version control using Git.
Requirements:
- 5+ years of hands-on experience in development and production implementation with Azure Databricks and Spark.
- Proficiency in Python (PySpark) and SQL coding.
- Proven ability to optimize performance for TB-scale data processing.
- Strong understanding of data lake architecture, ETL processes, and data quality management.
- Experience with Azure data analytics services.
- Excellent technical writing skills for creating requirements and testing documents.
- Ability to communicate technical concepts clearly and concisely.
- Knowledge of Git.
Colombia $12,000.00-$24,000.00 4 weeks ago