The Senior Data Engineer will play a crucial role in designing, developing, and maintaining scalable data solutions and infrastructure for our clients. This position requires a deep understanding of data engineering principles, technologies, and tools, as well as strong problem-solving skills. The Senior Data Engineer will collaborate closely with cross-functional teams to ensure data pipelines are efficient, reliable, and aligned with business objectives within our clients.
TECHNICAL KNOWLEDGE
Strong knowledge and extensive experience in:
- Proficient in big data technologies such as Apache Hadoop, Spark, and Kafka.
- Strong programming skills, particularly Python, Java, and SQL.
- Solid experience in developing and optimizing data pipelines, architecture, and data sets.
- Solid understanding of data warehousing concepts, ETL tools, and data modelling.
- Good experience with cloud-based data solutions like AWS (EC2, S3, EMR, Redshift, Glue), Google Cloud (BigQuery, Dataflow, Pub/Sub).
- Data pipeline orchestration and workflow management tools: Airflow, Luigi.
- Familiarity with Containerization and orchestration platforms: Docker, Kubernete
- Works independently on complex data engineering projects.
- Develop and maintain optimal data pipelines architecture.
- Collaborate with data scientists and Business Analysts to meet functional data requirements.
- Assist the Lead Data Engineer in planning and implementing data engineering projects and be involved in project management and strategic decision-making.
- Optimize and tune data pipelines for performance, scalability, and cost-effectiveness.
- Ensure data quality and implement data validation and cleansing processes.
- Monitor and troubleshoot data pipelines to identify and resolve issues promptly.
- Evaluate and recommend appropriate data storage and processing technologies.
- Develop and maintain documentation for data engineering processes, data flows, and system architectures.
- Stay updated with emerging technologies, industry trends, and best practices in data engineering and analytics.
- Provide technical guidance and mentorship to junior data engineers.
- 5+ years of experience in data engineering, with a focus on building large-scale data processing systems.
- Bachelors or Masters degree in Computer Science, Information Systems, or a related field.
- Shows promising leadership abilities.
- Excellent analytical and problem-solving skills
- Ability to work in a fast-paced, dynamic, and collaborative environment.
- Comfortable with ambiguity and able to adapt to change.
- Ability to work independently and as part of a team.
- Passion for learning new technologies and staying up to date with the latest trends in Data Engineering.
- Excellent written and verbal communication and collaboration skills
- Strong business acumen.