Job Description
We are seeking a talented and motivated Software Engineer with expertise in data engineering and a strong background in Python, data processing technologies, and modern data architecture concepts. The ideal candidate will have a solid understanding of various data tools and technologies such as Tableau, Kafka, Terraform, Flink, Airbyte, Spark, pySpark, Scala, SQL, Airflow, dbt, Trino, Cassandra, GraphQL, Eventing, Data Lakehouse, and Data Mesh architectures. The candidate should have a proven track record of delivering high-quality solutions and possess a passion for staying updated with the latest advancements in the field.
Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL processes using technologies such as Kafka, Flink, Spark, pySpark, and Airflow.
- Collaborate with cross-functional teams to understand data requirements, design data models, and ensure data accuracy and consistency.
- Build and optimize data integrations and data transformations to support analytics, reporting, and business intelligence needs.
- Implement and maintain data architecture patterns following best practices, including Data Lakehouse and Data Mesh concepts.
- Work on event-driven architectures and implement event-driven data processing using Kafka and related technologies.
- Develop and maintain data storage solutions utilizing technologies like Cassandra, GraphQL, and other relevant data storage technologies.
- Collaborate with data analysts and data scientists to provide them with accessible and well-organized data for analysis and modeling.
- Monitor and troubleshoot data pipelines, ensuring data quality, reliability, and performance.
- Stay informed about industry trends, emerging technologies, and best practices to continuously enhance the data engineering processes.
Qualifications:
- Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
- 3 to 8 years of professional experience in software development and data engineering roles.
- Proficiency in Python and Scala programming languages.
- Strong experience with data processing frameworks such as Kafka, Flink, Spark, and pySpark.
- Solid understanding of SQL and database technologies, including Cassandra and Trino.
- Hands-on experience with ETL tools like Airflow and dbt.
- Familiarity with modern data architecture concepts including Data Lakehouse and Data Mesh.
- Previous exposure to event-driven architectures and event streaming using Kafka.
- Experience in working with Tableau, Mode, and other data visualization tools.
- Knowledge of infrastructure-as-code principles using Terraform is a plus.
- Strong problem-solving skills and the ability to work in a fast-paced, collaborative environment.
- Excellent communication skills to work effectively with both technical and non-technical team members.
- A proactive attitude towards learning and adapting to new technologies and challenges.