Data Engineer

Position: Data Engineer - Agentic AI Data Pipelines & Automation

Location: Hyderabad (On-site)

Experience: 2-4 years

Must Have Skills: Data Engineering, ETL/ELT Pipelines, Python, SQL, Data Modeling, Data Warehousing, Process Automation, Workflow Orchestration, API Integration, Real-Time and Batch Processing, Azure Data Services, Data Quality and Governance

About the Role

We are looking for a capable Data Engineer to build and manage the data foundation required for agentic AI, intelligent automation, and software-led process transformation. The role will focus on turning fragmented, manual, and poor-quality data sources into reliable pipelines and reusable data assets that can support AI agents, workflow automation, analytics, and operational decision-making. The candidate will work closely with backend and AI teams to ensure that business processes have access to trusted, timely, and well-structured data in an Azure-first environment.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines that ingest, transform, validate, and publish data required for analytics, AI, automation, and operational workflows.
• Develop data pipelines for structured, semi-structured, and messy source systems, including ERP, CRM, support platforms, spreadsheets, flat files, APIs, logs, and internal repositories.
• Create reliable data foundations for agentic AI use cases by preparing clean, context-rich, governed, and accessible datasets for retrieval, workflow execution, and decision support.
• Build batch and near-real-time data processing solutions to support business-critical applications, operational dashboards, automation engines, and AI-driven process flows.
• Design and optimize data models, data marts, data lakes, and warehouse structures for performance, usability, and business traceability.
• Implement data quality checks, reconciliation logic, schema validation, monitoring, lineage awareness, and exception handling across pipelines.
• Integrate multiple enterprise systems and third-party sources so that fragmented business data can be consolidated into usable, trustworthy outputs.
• Work closely with AI engineers and backend developers to ensure downstream systems receive consistent, timely, and production-ready data.
• Support reporting, root cause analysis, and process improvement by exposing meaningful datasets and metadata to business and technology stakeholders.
• Troubleshoot pipeline issues, optimize performance, reduce failure points, and continuously improve data engineering standards and practices.

What You Should Bring

• Bachelor’s degree in Computer Science, Engineering, Information Systems, Data Engineering, or a related field.
• 2-4 years of experience in data engineering, pipeline development, and data platform implementation.
• Strong proficiency in Python and SQL, including data transformation, query optimization, and handling of large datasets.
• Hands-on experience designing ETL/ELT workflows and using orchestration tools such as Airflow, Azure Data Factory, or similar platforms.
• Experience with Azure data services such as Azure Data Factory, Azure Data Lake, Azure Synapse, Microsoft Fabric, Event Hub, or related tools.
• Strong understanding of data modeling, data warehousing concepts, partitioning, schema design, and pipeline performance optimization.
• Ability to work with low-quality, inconsistent, and fragmented enterprise data and convert it into structured and reliable information assets.
• Experience with API-based ingestion, file-based ingestion, incremental loads, data reconciliation, and production support for data pipelines.
• Good understanding of data governance, data quality, security, access management, and operational monitoring in enterprise environments.
• Strong analytical mindset, attention to detail, and ability to collaborate across business, AI, and engineering teams.

Preferred Skills

• Experience with streaming or event-based data processing using Kafka, Spark Streaming, or Azure Event Hub.
• Exposure to AI/ML data pipelines, RAG preparation flows, vectorization support processes, or data layers for agentic AI systems.
• Understanding of master data, reference data, metadata management, and enterprise integration patterns.
• Experience with BI and analytics tools such as Power BI, Tableau, or semantic models for business consumption.
• Familiarity with DevOps, CI/CD, testing practices, and version-controlled pipeline deployment.
• Exposure to financial, billing, customer operations, or service workflow data domains will be an advantage.