Avinash
Anad
Senior Data & Platform Engineer · Azure · AKS · Airflow · Databricks · MLOps · Agentic AI Architecture
Senior Data & Platform Engineer · Azure · AKS · Airflow · Databricks · MLOps · Agentic AI Architecture
Senior Data & Platform Engineer with 13+ years of experience designing, building, and operating cloud-native data platforms at scale, now extending that experience into agentic AI platform architecture.
My work focuses on Azure-based data ecosystems — Airflow orchestration on AKS, Databricks, and metadata-driven pipelines supporting multi-country, enterprise workloads. I care deeply about reliability, cost efficiency, and operational simplicity in production systems.
Over the years I've helped teams move from tightly coupled ETL systems to modular, scalable platforms with strong automation, observability, and governance — working closely with data scientists, product teams, and business stakeholders to deliver dependable analytics and MLOps pipelines.
My work aligns best with organisations building long-lived platforms that require careful design, collaboration, and steady engineering discipline at scale.
Recently, I have been extending that platform experience into agentic AI architecture: LLM internals, MCP-based tool invocation, structured planning, RAG and memory systems, multi-agent DAGs, browser automation, and desktop agents. The focus is practical platform design rather than isolated model experiments.
Certifications
Cloud & Infrastructure
Data & ML Platforms
Languages & Frameworks
Agentic AI & Automation
Applied platform engineering experience to modern agent architecture across transformer foundations, tokenization, tool invocation, structured planning, cognitive pipelines, memory systems, RAG, multi-agent coordination, browser automation, and desktop automation. Completed the first 10 implementation modules toward a production-ready agentic platform spanning MCP, A2A, A2UI / AG-UI, routing, observability, safety, and evaluation.
Designed and operated Azure-native MLOps pipelines using Apache Airflow on AKS. Built config-driven multi-country orchestration with KubernetesPodOperator-based execution, automated failure notifications, drift detection (PSI/CSI), and secure secret management via Azure Key Vault.
Deployed a Digital Twin of the banking system in Palantir Foundry, using Ontology and Workflows to assess ESG-based creditworthiness. Built PySpark pipelines for ETL, automated data governance and unit testing, and integrated Foundry with Azure Databricks UC. Team size: 3.
Implemented metadata-driven ETL pipelines using Azure Data Factory and Databricks for transformation. Integrated Azure DevOps for version control and CI/CD, implemented RBAC for data security, and adopted a Unified Communication strategy — delivering a robust, scalable, and secure data integration solution.
Senior governance and administration engineer for Spark analytics on AWS. Managed IAM, KMS, S3, and Lambda services; enforced IaC with Terraform across a team of 9.
Built custom Python modules for Spark — including an AES-256 CBC encryption wrapper and a Databricks-to-Postgres sync module (Databricks2pg). Led SQL to Databricks migration: analysed existing systems, identified migration challenges, and optimised data processing pipelines on the cloud platform.
Designed SQL data pipelines implemented as stored procedures, cursors, and triggers across structural databases and linked servers. Built an SQL trigger-based audit system and scheduled data integrity jobs to detect dirty data (cross-treatment and mis-randomisation) in clinical trial data. Team size: 5.