Data Engineer (Databricks Specialist)

We are seeking a skilled Senior Data Engineer with expertise in Databricks and a strong background in the financial services industry. This role will take ownership of data engineering within the IT department, specifically focusing on managing Databricks clusters and designing efficient data processing pipelines. As a Senior Data Engineer, you will play a crucial role in ensuring the reliability, scalability, and performance of our data infrastructure.

DUTIES & RESPONSIBILITIES:

• Proven experience as a Databricks Engineer, with a deep understanding of Apache Spark and the Databricks platform.
• Extensive knowledge of data processing concepts, ETL pipelines, and data warehousing principles.
• Hands-on experience in managing Databricks clusters, including provisioning, configuration, and optimization.
• Proficiency in programming languages commonly used with Databricks, such as Python or Scala.
• Familiarity with cloud platforms like AWS, Azure, or GCP, and a track record of deploying and managing Databricks in a cloud environment.
• Strong understanding of distributed computing concepts and parallel data processing.
• Solid knowledge of SQL and experience working with relational databases.
• Strong problem-solving skills, with the ability to troubleshoot and resolve issues related to data processing pipelines and cluster management.
• Excellent communication and collaboration skills, enabling effective teamwork in a cross-functional environment.
• Ability to thrive in a fast-paced, dynamic work environment and handle multiple priorities simultaneously.

Requirements

• Bachelors degree in Computer Science, Engineering, or a related field. A relevant combination of education and experience will also be considered.
• Proven experience as a Databricks Engineer, with a deep understanding of Apache Spark and the Databricks platform.
• Extensive knowledge of data processing concepts, ETL pipelines, and data warehousing principles.
• Hands-on experience in managing Databricks clusters, including provisioning, configuration, and optimization.
• Proficiency in programming languages commonly used with Databricks, such as Python or Scala.
• Familiarity with cloud platforms like AWS, Azure, or GCP, and a track record of deploying and managing Databricks in a cloud environment.
• Strong understanding of distributed computing concepts and parallel data processing.
• Solid knowledge of SQL and experience working with relational databases.
• Strong problem-solving skills, with the ability to troubleshoot and resolve issues related to data processing pipelines and cluster management.
• Excellent communication and collaboration skills, enabling effective teamwork in a cross-functional environment.
• Ability to thrive in a fast-paced, dynamic work environment and handle multiple priorities simultaneously.
• At least 5+ years of experience designing, developing, deploying, and supporting data pipelines.
• Minimum of 3 years of hands-on experience designing, developing, deploying, and supporting data pipelines in Databricks.
• Databricks Certified Developer or Databricks Certified Administrator certification (preferred).
• Experience with machine learning frameworks and data science workflows on Databricks (preferred).
• Knowledge of big data technologies such as Hadoop, Hive, or Kafka (preferred).

Next Steps:

Click apply and your application will be reviewed by our talent acquisition team who will contact you directly

Your Application:

If you have not had any response in two weeks, please consider the vacancy application unsuccessful. Your profile will be kept on our database for any other suitable roles.