استخدام Data Engineer
شرح موقعیت شغلی
Job Description:
As the data industry matures, more priorities are being placed on Data Pipelines and Infrastructure. So, design and building production data pipelines as well as automating data engineering (DE) workflows are the main tasks we are looking for assistance with. In a DE role, you’re responsible for extracting, transforming, and loading data from a wide variety of data sources using SQL and big data technologies
As the data industry matures, more priorities are being placed on Data Pipelines and Infrastructure. So, design and building production data pipelines as well as automating data engineering (DE) workflows are the main tasks we are looking for assistance with. In a DE role, you’re responsible for extracting, transforming, and loading data from a wide variety of data sources using SQL and big data technologies
Moreover, as a data engineer, you work closely with data scientists to make high-quality data available to them. Therefore, you are responsible for building and maintaining pipelines that feed data to the data scientists
Responsibilities:
- Understand several programming languages used in data science, specially Python
- Experience and knowledge on Google Cloud Platform for data management: Composer, PubSub, Google Storage, Dataflow, Big Query.
- Experience and knowledge on Apache Airflow and develop and deploy multi-container apps with docker compose
- Know the ins-and-outs of SQL and NoSQL database systems, specially Postgresql and Elasticsearch
- Be comfortable using ETL (extract, transform, load) systems
- Ability to work with stream-processing systems
- Knowledge of Data modeling and data warehousing
- Knowledge of software engineering best practices
- Graph databases including Neo4j is a plus
- Familiar with Blockchain core and/or cryptocurrency market is a plus
- Last but not least: Able to work in teams and in a fast-paced agile development environment
Requirements:
- Bachelor’s degree in computer science, engineering, mathematics, or a related technical discipline, or equivalent experience
- More than two years of industry experience in software development, data engineering, business intelligence, data science, or related field with a track record of manipulating, processing, and extracting value from large datasets
- More than a year of experience with traditional RDBMS technologies, SQL and SQL tuning
- More than a year of experience in data modeling, ETL development, and data warehousing
- Experience in one or more scripting or programming language, specially Bash and Python
مهارتهای مورد نیاز
- SQL
- ETL
- Python
- RDBMS
حداقل سابقه کار
- سه تا شش سال
جنسیت
- مهم نیست
وضعیت نظام وظیفه
- مهم نیست