Position: Software Engineer - Data & AWS
We are seeking a Software Engineer to build and optimize data platforms leveraging AWS and key technologies like CDAP, Snowflake, and Databricks. You will design and implement robust and scalable data pipelines, ETL, and analytics systems in the cloud.
Responsibilities:
- Develop and enhance data pipelines, ETL processes using CDAP on AWS infrastructure.
- Build data integration flows to migrate large datasets into Snowflake data warehouse.
- Implement AWS infrastructure-as-code solutions for deployment automation.
- Instrument data pipelines and leverage monitoring for performance tuning and reliability.
- Work with data scientists to optimize data workflows and models on Databricks.
- Follow security best practices for access control, encryption, and auditing across data platforms.
- Participate in architecture reviews and technology selections.
- Continuously monitor and improve data platforms for scalability and costs.
Requirements:
- 5+ years of experience in backend development or data engineering.
- Hands-on experience with AWS services like S3, EC2, and EMR.
- Proficiency in SQL and experience with CDAP, Spark, and Kafka.
- Experience building scalable ETL processes and workflows.
- Strong programming ability with Python, Java, and unit testing.
- Infrastructure-as-code expertise with CI/CD pipelines.
- Ability to communicate complex topics clearly.
Nice to Have:
- Experience with Snowflake, Databricks, and GCP or Azure.
- Knowledge of streaming data architectures.
- Data security and compliance implementation.
- Machine Learning Operations (MLOps) experience.