Writing ยท November 2026

Running Lakeflow pipelines as Dagster assets: what the spike showed

Sagar Soni, Senior Data Engineer

At Protective Life I led a spike to test whether Lakeflow pipelines can run as Dagster assets. I guided three engineers and worked with the platform team.

What we found

Lakeflow pipelines can run as Dagster assets using two pipeline tags and no custom code.

What we tested

  • Queueing
  • Cancellation
  • Failure behavior

I demoed the result live at sprint review, and the team then planned the pipeline's production rollout.

Back to the projects