استخدام Big Data Engineer

شرح موقعیت شغلی

• Design and Implement Big Data Architectures: Develop robust, scalable, and high-performance big data architectures to meet the evolving needs of our data processing and analytics requirements.
• Data Ingestion and Integration: Design and implement efficient data ingestion processes, ensuring the smooth integration of diverse datasets from various sources into our big data ecosystem.
• Data Processing and Transformation: Implement data processing and transformation pipelines using technologies such as Apache Spark, Apache Flink, or similar frameworks to extract valuable insights from large datasets.
• Optimization and Performance Tuning: Identify and implement optimization strategies to enhance the performance and efficiency of big data processing workflows, ensuring timely data delivery for analytics and reporting.
• Data Security and Compliance: Implement and maintain data security measures, ensuring the confidentiality and integrity of sensitive information. Adhere to relevant compliance standards and regulations.
• Collaboration with Cross-Functional Teams: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective data solutions that meet business objectives.
• Monitoring and Maintenance: Develop and implement robust monitoring solutions to proactively identify and address issues within the big data infrastructure. Perform routine maintenance tasks to ensure system reliability.
• Documentation: Create and maintain comprehensive documentation for all aspects of big data engineering processes, ensuring knowledge transfer and continuity.

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• Proven experience as a Big Data Engineer or similar role.
• Proficiency in Python for developing robust, scalable, and high-performance big data solutions.
• Hands-on experience with big data technologies such as Apache Spark, Apache Kafka, Apache Iceberg, and related ecosystems using Python.
• In-depth knowledge of data streaming and event-driven architecture using Kafka for real-time data processing.
• Experience with version control systems, particularly GitLab, for managing and tracking changes in the Python codebase and configurations.
• Proficiency in working with distributed query engines such as Trino (formerly PrestoSQL) for fast and interactive querying of data.
• Knowledge of containerization and orchestration tools, including Kubernetes, for deploying and managing scalable and resilient big data applications using Python.
• Expertise in working with object storage systems like MinIO for efficient and scalable storage of large datasets using Python.
• Strong SQL skills and expertise in working with relational and NoSQL databases using Python.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

• Competitive Salary
• Training & Development
• Private Health Insurance
• Attractive benefit Package ( Family, Health, Entertainment)
• breakfast and lunch
• Hybrid workplace
• flexible working hours

مهارت‌های مورد نیاز

• Big Data
• Python
• spark

حداقل سابقه کار

• سه تا شش سال

جنسیت

• مهم نیست

وضعیت نظام وظیفه

• معافیت تحصیلی معافیت دائم پایان خدمت

مشاهده آگهی‌های استخدام مشابه