استخدام Big Data Engineer
شرح موقعیت شغلی
Responsibilities
- Build reliable data pipeline to apply data science team alogirthms on input data using Apache Spark and Airflow
- Maintain and improve big data infrastructure according to requirements
- Design and implement state-of-the-art solutions to improve services quality
- Work closely with data science and DevOps teams
- Design and implement end to end solution
Qualifications
- 3+ years of industry experience in computer engineering/science
- Excellent experience with Python programming language
- Good experience with distributed processing using Hadoop and Apache Spark
- Good experience with ETL tools including Airflow or Nifi
- Familiar with stream prcessing engines including Apache Spark, Flink, Kafka
- Familiar with NoSQL technologies such as Cassandra, MongoDB, Redis
- Build reliable data pipeline to apply data science team alogirthms on input data using Apache Spark and Airflow
- Maintain and improve big data infrastructure according to requirements
- Design and implement state-of-the-art solutions to improve services quality
- Work closely with data science and DevOps teams
- Design and implement end to end solution
Qualifications
- 3+ years of industry experience in computer engineering/science
- Excellent experience with Python programming language
- Good experience with distributed processing using Hadoop and Apache Spark
- Good experience with ETL tools including Airflow or Nifi
- Familiar with stream prcessing engines including Apache Spark, Flink, Kafka
- Familiar with NoSQL technologies such as Cassandra, MongoDB, Redis
مهارتهای مورد نیاز
- Big Data
- Python
- ETL
- Redis
- kafka
حداقل سابقه کار
- سه تا شش سال
جنسیت
- مهم نیست
وضعیت نظام وظیفه
- مهم نیست