استخدام Data Engineer

Job Description:
Analysis, design, development and maintenance of data pipelines.
Identify and design manual and repetitive processes and automate them by tools such as Airflow and Spark.
Design the infrastructure required for optimal ETL from various sources of log files, SQL and NoSQL databases.
Microservice and event-sourcing based design for data flow processing.
Collaborate with various teams such as product, design and data analysis to provide the necessary infrastructure.
Implement all stacks with Docker-based tools.
Experience working with stream processing tools such as KSQLDB, KSQL and Kafka Stream.

Required abilities:
Having in-depth knowledge of SQL and Python, Scala or Java programming languages.
Familiarity with Apache bigdata tools.
Experience working with tools like Hadoop, Spark and Kafka
Experience working with SQL and NoSQL databases (PostgreSQL, MongoDB, etc.).
Experience working with Kafka connector tools for SQL and NoSQL databases.
Familiarity with automation tools such as Apache Airflow and Apache NiFi.
Experience working with stream processing tools such as Kafka Streams. 
Having Teamwork skills and familiarity with Agile methodology.
Ability to design and develop ML infrastructure and services.

Advantage skills:
Experience working with Kubernetes and Spark stacks together.
Experience working with the Apache Druid / Hadoop stack in operating environment with large amount of data.
Familiarity with ETL tools such as Apache Drill and Presto.