As a Data Engineer on Bitpin's AI & Data team, you will help architect and operate the data platform behind our cryptocurrency exchange. You will own the data's journey from source to value building pipelines from our operational PostgreSQL databases into our ClickHouse analytical warehouse, serving governed data marts to BI consumers, and managing the data ecosystem for our Qdrant-powered AI applications.
Key Responsibilities:
ETL/ELT Pipeline Development: Design, build, and maintain robust, scalable data pipelines using Apache Airflow and PySpark to transform data from PostgreSQL sources into our ClickHouse analytical warehouse.
Real-Time & Streaming Ingestion: Build and operate streaming ingestion paths that move data from operational systems into the warehouse with low latency, ensuring correct ordering, deduplication, and delivery semantics.
ClickHouse Modeling & Optimization: Design performant ClickHouse data models and write highly optimized SQL for large-scale analytical workloads.
Performance Tuning: Continuously profile and optimize query performance, storage layout, and resource utilization across the analytical warehouse as data volumes grow.
Data Quality & Reliability: Implement validation, reconciliation, and completeness checks across pipelines to guarantee that downstream consumers can trust every number — no silent data loss, no unnoticed drift.
Monitoring & Observability: Instrument pipelines and data services with metrics, alerting, and freshness monitoring so that failures and anomalies are detected before stakeholders notice them.
Application-Level Service Administration: Take full ownership of the configuration and administration of our data services. This includes creating Kafka topics and managing ACLs, designing database schemas, managing users and permissions, and setting up S3-compatible object storage buckets.
Workflow Orchestration: Develop, schedule, and monitor complex data workflows (DAGs) in Apache Airflow, ensuring they are efficient, reliable, and well-documented.
Serving Analytics & Governance: Build and maintain governed data marts consumed by our BI layer (Metabase) and data catalog (OpenMetadata), including supporting access-control enforcement jobs.
AI Data Provisioning: Develop processes to prepare, embed, and load data into our Qdrant vector database, often orchestrating these jobs within Airflow.
Collaboration & Ownership: Work closely with software developers, analysts, and AI specialists to define their data needs and build the solutions to meet them.
What We're Looking For:
Proven experience as a Data Engineer.
Hands-on experience with ClickHouse and its feature set.
Experience with CDC pipelines (Debezium, Kafka Connect) and streaming data processing paradigms.
Expert-level proficiency in Python for data processing, with experience in PySpark for transformation workloads.
Deep knowledge of SQL and experience with relational databases (PostgreSQL is a must).
Hands-on experience developing, scheduling, and monitoring complex data pipelines using Apache Airflow.
Experience with ETL/ELT architecture, data modeling, and data warehousing concepts.
Familiarity with containerizing applications using Docker.
Bonus Points:
Familiarity with BI tools (Metabase) and data catalogs (OpenMetadata).
Experience with vector databases (Qdrant, Weaviate, etc.).
بیتپین از شرکت های مطرح در حوزه فناوری اطلاعات و ارتباطات و مدیریت مالی و سرمایه گذاری و پلتفرم حرفهای مبادلات رمزارزهاست. این پلتفرم یک اکسچنج و بازار آنلاین با ابزار حرفهای است که امکان مبادله ارزهای برتر و محبوب بر بستر فناوری بلاکچین را فراهم کرده ..
تیممون توی بیتپین از خبرگان در حوزه فینتک تشکیل شده و جذب افراد تلاشگر و متخصص سرلوحهی راهبردهای توسعه منابع انسانی ماست و این دلیلیه که شما می تونید به عنوان یه بیت پینی به جمع ما اضافه شین، یعنی اونجایی که دقیقا باید باشید. سعی ما در خلق محیطی سرشار از پویایی و زمینهساز رشد برای سرمایه انسانی مون با همراهی افراد توانمند و باهوش معنا و مفهوم می گیره و منتج به خروجی عالی می شه به خاطر اینه که ما دنبال آدمهای باهوش و خوش فکر و خلاق می گردیدم.
حضور در محیطی صمیمی ، اولین مزیت پیوستن به تیم ماست. کار در کنار افراد باانگیزه، متخصص و مسلط به حوزۀ کاری خودشون، بستر مناسبی برای رشد و بالندگی شغلییه که ما این امکان رو در کنار هم داریم و بهش می بالیم . برای پیوستن به تیممون همین الان اقدام کنید.