Sagar Soni Senior Data Engineer

5+ years building and modernizing enterprise data pipelines. Now at Protective Life, on a new Databricks lakehouse with dbt, Dagster and data contracts.

About

I'm a Senior Data Engineer working in life insurance. At Protective Life I lead technical spikes and stories on a new Databricks lakehouse (Lakeflow Connect, data contracts, dbt, Dagster) while supporting the production SQL Server, SSIS and Control-M pipelines.

I pick up new technology quickly. Within a few months of joining the Databricks platform teams I had shipped merged dbt models and led a Dagster integration spike.

I'm known for root-cause investigations across source systems, and I work directly with data governance, data architecture and business owners.

I'm also completing an Executive MBA.

Work experience

Senior Data Engineer

Protective Life (AL), Sep 2024 to Present

  • Lead root-cause investigations across 8+ source systems feeding underwriting, actuarial and regulatory reporting. Two examples: a formatting bug affecting 267,911 reinsurance records, and a hardcoded rule that sent every New York policy to full underwriting.
  • Converted full-reload pipelines to watermark-based incremental merges on two production feeds, catching a bug that would have reset the watermark on empty runs.
  • Removed about 40% duplicate records from a partner-facing report.
  • Asked by the Director of Data Governance to help draft a partner data contract.
  • Present findings at sprint reviews to product owners and source-system owners.
  • Write runbooks and setup guides used by engineers joining the Databricks platform.

The four projects below are from this role.

BI Engineer

PRO EMS (MA), Sep 2023 to Aug 2024

  • Set up the organization's BI and cloud data stack on Microsoft Fabric and Azure Data Factory.
  • Built Power BI (DAX) dashboards for executive decision-making.
  • Analyzed EMS operational data in R using statistical techniques, contributing to a 10% reduction in emergency response times through KPI tracking and trend analysis.
  • Improved scheduling efficiency by 15% through demand analysis and resource allocation recommendations.

Data Engineer

WebCraft IT (India), Jun 2019 to Jul 2021

  • Built internal data collection tools in Python (text mining, web crawling) with MySQL and MongoDB, supporting anomaly detection that reduced data management errors by 10%.
  • Created stored procedures, views and triggers.
  • Built Power BI and Tableau dashboards and reports for management.

Data engineering projects

All four are from my work at Protective Life.

How the pieces fit together

A simplified view of the platform work described below.

Source systems8+ feeding reporting Lakeflow Connect75 tables, CDC DatabricksDelta tables dbt modelsraw, prep, intermediate RUN AS DAGSTER ASSETS DATA CONTRACTS ยท 60 AUTOMATED CI CHECKS

Databricks Lakeflow landing

I joined the team that delivered the first source system on the enterprise Databricks platform, landing 75 tables through Lakeflow Connect on a fixed deadline.

  • 75 tables landed through Lakeflow Connect
  • 5.7M rows
  • Co-built the Databricks Asset Bundle for the 75-table Lakeflow Connect CDC pipeline and deployed it to the test workspace.
  • Documented deployment findings for the production cutover.

Lakeflow pipelines as Dagster assets

I led a spike, guiding three engineers and working with the platform team, that proved Lakeflow pipelines can run as Dagster assets using two pipeline tags and no custom code.

  • 3 engineers guided
  • 2 pipeline tags, no custom code
  • Tested queueing, cancellation and failure behavior.
  • Demoed it live at sprint review.
  • The team then planned the pipeline's production rollout.

Qlik Replicate replacement

I did the majority of the migration work replacing Qlik Replicate with in-house SSIS and Control-M pipelines.

  • ~$700K a year in license fees removed by the program

Data contracts, dbt and data quality

I author data contracts and dbt models for the raw-to-prep and intermediate layers, and review teammates' pull requests as a required approver.

  • 60 automated CI checks passing
  • 41 tables verified against SQL Server
  • Generate code from the contracts with contract tooling and AI coding agents, and verify it with scripted parity checks.
  • Merge work through pull-request review and CI with 60 automated checks passing.
  • Caught a silent-failure bug in a CI script during review.
  • Built an AI agent skill, now used in those reviews, that checks evidence freshness, approval timing and risky contract type changes.
  • Ran data quality and governance proofs of concept with DQX and Unity Catalog Data Classification on real source data: found placeholder dates that not-null checks miss, verified row counts and schemas on 41 tables against SQL Server, and flagged a PII false positive before auto-tagging.

Skills

Data Platforms

  • Databricks(Lakeflow Connect, Delta Lake, Unity Catalog, Asset Bundles, DLT, Workflows)
  • Azure Data Lake
  • Microsoft Fabric
  • SQL Server

Transformation and Orchestration

  • dbt
  • Dagster
  • SSIS
  • Azure Data Factory
  • Control-M
  • ETL/ELT
  • change data capture(CDC)
  • incremental loads

Data Quality and Governance

  • data contracts(datacontract-cli)
  • DQX
  • Unity Catalog Data Classification
  • PII detection
  • data profiling
  • validation and reconciliation
  • data lineage
  • MDM

Languages

  • SQL(T-SQL, Spark SQL)
  • Python(PySpark)
  • R
  • YAML

DevOps

  • Git
  • Azure DevOps
  • CI/CD pipelines
  • pull-request review
  • Flyway
  • sqlfluff
  • AI-assisted development(Claude Code, GitHub Copilot)

BI and Databases

  • Power BI(DAX)
  • Tableau
  • SSRS
  • Qlik
  • MySQL
  • PostgreSQL
  • MongoDB

Methods

  • Medallion architecture
  • dimensional modeling
  • SCD Type 2
  • Agile/Scrum(backlog refinement, sprint planning, story-point estimation)
  • technical leadership
  • code review
  • stakeholder communication
  • knowledge transfer

Education

Executive MBA (in progress)

New England College

Master of Science in Business Intelligence and Analytics

Stevens Institute of Technology, Hoboken, New Jersey, 2021 to 2023

Bachelor in Information Technology

Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore, India, 2016 to 2020

Writing

November 2026

Running Lakeflow pipelines as Dagster assets: what the spike showed

A short note on the spike I led, what we tested, and what the team did next.

Resume

My full resume is a three-page PDF.

Email for the PDF

Contact

Email is the quickest way to reach me.

sagarsoni112698@gmail.com

Or send a message

Ask about SagarQuick answers from this page. Not an AI.