Share via

Expectations

  • Article

This page contains Python reference documentation for DLT expectations.

Expectation decorators declare data quality constraints on materialized views, streaming tables, or temporary views created in DLT.

The dlt module includes six decorators to control expectations behavior. The following table describes the dimensions on which these permutations differ:

Behavior Options
Action on violation
  • Include the row in the target dataset. The count of valid and invalid records is logged alongside other dataset metrics.
  • Drop the row before writing to the target dataset. The count of dropped records is logged alongside other dataset metrics.
  • Immediately stop the update. This expectation causes a failure of a single flow and does not cause other flows in your pipeline to fail.
Number of expectations A single expectation or multiple expectations.

You can add multiple expectation decorators to your datasets, providing flexibility in strictness for your data quality constraints.

When you use expect_all decorators, each expectation has its own description and reports granular metrics.

Syntax

Expectation decorators come after a @dlt.table() or @dlt.view() decorator and before a dataset definition function, as in the following example:

import dlt

@dlt.table()
@dlt.expect(description, constraint)
@dlt.expect_or_drop(description, constraint)
@dlt.expect_or_fail(description, constraint)
@dlt.expect_all({description: constraint, ...})
@dlt.expect_all_or_drop({description: constraint, ...})
@dlt.expect_all_or_fail({description: constraint, ...})
def <function-name>():
    return (<query>)

Parameters

Parameter Type Description
description str Required. A description that identifies the constraint. Constraint descriptions must be unique for each dataset.
constraint str Required. The constraint clause is a SQL conditional statement that must evaluate to true or false for each record. The constraint contains the actual logic for what is being validated. When a record fails this condition, the expectation is triggered.

The expect_all decorators require descriptions and constraints to be passed as a dict of key-value pairs.