Data Engineer

Taraki is hiring for one of its clients.Location: RemoteExperience Level: 5 to 8 years in Data Engineering, Data Pipelines, and Cloud-based Data PlatformsDepartment: Data & AI EngineeringCompensation: PKR 600,000 to 850,000 (based on experience)Role Summary:The Data Engineer will design and build large-scale, high-performance data pipelines to support segmentation, pricing simulation, and offer decisioning. They will ensure efficient data ingestion from telco systems (CDRs, usage, recharge, offer purchase), transformation, and integration with ML models and orchestration modules.Key Responsibilities:Design and develop scalable ETL / ELT data pipelines to process 50M+ customer records daily.Ingest data from OCS, CRM, DWH, and Adobe RT-CDP or other customer data platforms.Build and maintain Customer Profile Store and Feature Store for real-time and batch processing.Implement data validation, quality, and lineage frameworks.Optimize query performance and cost efficiency for batch and streaming workloads.Collaborate with Data Scientists to prepare model training datasets and deploy inference pipelines.Integrate outputs with Decision Engine and Real-Time Offer Orchestration Module.Automate pipelines using CI/CD and maintain environment configurations across Dev, UAT, and Prod.Required SkillsStrong in SQL, PySpark, and DataFrame APIs for data transformation. Expertise in Data Modeling (customer-level, event-level, offer-level). Understanding of data partitioning, schema evolution, and performance tuning. Experience in stream processing (Kafka, Spark Streaming, Kinesis). Knowledge of data quality frameworks (e.g., Great Expectations, Deequ). Familiarity with ETL orchestration tools (Airflow, dbt, or Dagster). Ability to work with cloud-native data platforms and object storage.Tools & TechnologiesData Platform: Databricks, AWS Glue, Azure Data Factory, Snowflake, BigQueryStreaming: Kafka, Kinesis, Spark StreamingStorage: S3, Delta Lake, Parquet, HiveWorkflow Orchestration: Airflow, dbt, Dagster, PrefectScripting: Python, SQL, PySparkDevOps: Git, Jenkins, TerraformMonitoring & Validation: Great Expectations, Deequ, DataDogPreferred (Nice-to-Have)Experience with telecom datasets (Recharge, Usage, Balance, Offer Subscription). Knowledge of DecisionRules.io, n8n, or KNIME for orchestration workflows. Familiarity with Adobe AEP data schemas (XDM) or Pricefx integration. Exposure to real-time microservices (REST/GraphQL APIs) for data access.