Senior Data Engineer

Requirements 5 years of experience in Data Engineer or relevant experience Proven experience in designing and implementing data pipelines Experience with end-to-end testing of analytics pipelines Expertise in managing and optimizing Elasticsearch clusters, including performance tuning and scalability Strong proficiency with data extraction, transformation, and loading (ETL) processes Familiarity with data modeling and schema design for efficient data storage and retrieval Good programming and scripting skills using languages like Python, Scala, or Java Knowledge of DevOps and automation practices related to data engineering Experience with data engineering in an agile / scrum environment Familiarity with ksqlDB / Kafka or other stream processing frameworks Familiarity of data lakes and the querying thereof Experience with integrating machine learning models into data pipelines Tech Stack Kafka / ksqlDB Python Redis Elasticsearch, cluster management and optimisation AWS S3 PostgreSQL AWS Responsibilities Designing, implementing, and maintaining robust data pipelines, ensuring the efficient and reliable flow of data across our systems Developing and maintaining Elasticsearch clusters, fine-tuning them for high performance and scalability Collaborating with cross-functional teams to extract, transform, and load (ETL) data into Elasticsearch for advanced analytics and search capabilities Troubleshooting data pipeline and Elasticsearch issues, ensuring the integrity and availability of data for analytics and reporting Participating in the design and development of data models and schemas to support business requirements Continuously monitoring and optimising data pipeline and Elastic performance to meet growing data demands Collaborating with data scientists and analysts to enable efficient data access and query performance Contributing to the evaluation and implementation of new technologies and tools that enhance data engineering capabilities

Location: Cape Town City Centre, ZA

Posted Date: 10/4/2024