Job Profile: Sr Data Engineer
Job Type: Long-time Contract Job Opportunity
Location: 100% Remote in Colombia
Job Description:
We are searching for a skilled Data Engineer to join our team.
The ideal candidate will have hands-on experience in development and production implementation with Azure Databricks and Spark, proficiency in Python (PySpark) and SQL coding, and a strong understanding of data lake architecture, ETL processes, and data quality management.
Experience with Azure data analytics services and excellent technical writing skills are also required.
**Responsibilities:**
Develop and implement data processing pipelines using Databricks and Spark.
Write efficient and optimized code in Python (PySpark) and SQL for TB-scale data processing.
Design and maintain data lake architecture and ETL processes.
Ensure data quality and integrity through robust data quality management practices.
Utilize Azure data analytics services for data storage, processing, and analysis.
Create comprehensive technical documentation, including requirements and testing documents.
Collaborate with cross-functional teams to understand data requirements and deliver solutions.
Maintain version control using Git.
**Requirements:**
5+ years of hands-on experience in development and production implementation with Azure Databricks and Spark.
Proficiency in Python (PySpark) and SQL coding.
Proven ability to optimize performance for TB-scale data processing.
Strong understanding of data lake architecture, ETL processes, and data quality management.
Experience with Azure data analytics services.
Excellent technical writing skills for creating requirements and testing documents.
Ability to communicate technical concepts clearly and concisely.
Knowledge of Git.