استخدام مهندس داده Data Engineer

پایشگران هوشمندساز کسب و کار | Business Intelligence Excavation
تهران، تهران

Responsibilities:

Work with structured, semi-structure and unstructured data
Monitor and troubleshoot performance issues on the enterprise data pipelines and the data lake.
Partner with various teams to define and execute data acquisition, transformation, processing and make data actionable for operational and analytics initiatives.
Assess the effectiveness and accuracy of new data sources and data gathering techniques.

Required Qualifications / Experience:

Intermediate in System Version Controls(Git, ...)
Experience with big data tools and technologies (Spark, Hadop,HDFS, kafka, ...)
Experience with and managing data across relational SQL and NoSQL databases like MySQL, Postgres, Cassandra, Redis, and Elasticsearch.
Experience working in a Linux environment.
Experience developing data ingest workflows with stream-processing systems: Spark-Streaming, Kafka Streams
Experience with ETL development and performance tuning of ETL jobs.
Experience developing flexible ontologies to fit data from multiple sources and implementing the ontology in the form of database mappings / schemas.
Experience with manipulating and transforming data
Architect and implement best practices of big data tools such as Spark, Airflow, Kafka, Presto and Timeseries Databases
Proficient in Spark application development.

Preferred Qualifications / Experience:

Experience with Microservices architecture components, including Docker and Kubernetes.
Experience developing microservices to fit data cleansing, transformation and enrichment needs.
Experience developing flexible data ingest and enrichment pipelines, to easily accommodate new and existing data sources.
Experience with continuous integration and deployment (CI/CD) pipelines and their enabling tools
Knowledge about security and best practices.

تمام وقت

۱۳۹۹/۱۰/۰۲ (منقضی‌شده)

مشاغل مشابه