Responsibilities:
• Design, develop, and maintain high-performance back-end systems, data pipelines, ETL workflows and streaming applications using open-source and proprietary event streaming and distributed processing frameworks
• Implement data storage solutions, both relational (PostgreSQL, MySQL) and NoSQL databases, to handle diverse data types
• Develop and optimize data storage and retrieval mechanisms, ensuring data integrity and security
• Real-time and batch data search and analysis, optimizing search performance
• Integrate AI and ML models into backend systems to provide data-driven insights
• Develop and manage managed services while utilizing managed services from AWS, Azure, GCP
• Collaborate with cross-functional teams to gather and understand data requirements
• Optimize data processing and streaming performance for scalability and efficiency
• Implement data quality checks and monitoring solutions to ensure data accuracy and reliability
• Troubleshoot and resolve issues related to data ingestion and processing
• Stay up-to-date with the latest Big Data technologies and trends
Requirements:
• Bachelor's or Master's degree in CS / CE
• Prior experience with Apache frameworks including Kafka, Flink, Spark, Superset, etc.
• Strong programming skills in Python, C++, Java and Scala, as well as experience with functional programming concepts
• Familiarity with open-source, high-performance databases for business intelligence (BI) applications
• Experience with data modeling, ETL processes, and data warehousing
• Familiarity with AI/ML with the ability to integrate machine learning models into data pipelines
• Prior experience with backend development and building scalable APIs
• Familiarity with MLOps / DevOPs
• Experience with containerization and orchestration technologies (e.g., Docker, Kubernetes) is a plus
• Experience with cloud platforms (e.g., AWS, Azure, GCP) is a plus
• Solid problem-solving and analytical skills
• Excellent communication and teamwork abilities
• Solid understanding of version control systems and collaborative software development