In a perfect world we will never experience any failures. 1
Everything will work as expected and you will be the happiest engineer at work. 2
Sorry, but that's not the case at all. 3
As an engineer, we know how important it is to deal with failures 4
as they may happen at anytime breaking down you entire data pipeline. 5
Since Airflow is the orchestrator of your data pipelines, it is critical to be warned when a task didn’t 6
succeed or if one of your DAGs is getting slower to finish than us 7
and so on. 8
The good thing is, Airflow brings us many ways in order to monitor our DAGs and make sure that everything 9
works as it should. 10
There are two levels of failure detection, 11
one for DAGs and one for tasks. 12
Let’s begin by DAG failure detections. When you instantiate a DAG, you can specify all of these parameters 13
in order to check the execution of your DAGRuns. 14
The most basic parameter is dagrun_timeout which specifies how long a DagRun should 15
be up before timing 16
out, so that new DagRuns can be created. 17
Notice that this timeout is only effective for scheduled DagRuns, so it won’t work if you manually trigger 18
your DAG, 19
and only once the number of active DagRuns equals to the max_active_runs parameter. 20
So, what is max_active_runs? 21
Well, as we will see later in the course about distributing Airflow, max_active_runs is a configuration 22
property allowing you to fix the maximum number of active DAG runs per DAG. 23
For example, let’s say this property is set to 16, which is the case by default, and you trigger a DAG 24
from the past. Airflow will run up to 16 DagRuns in parallel for this DAG in order to achieve the backfilling 25
process faster. 26
That being said, it’s really up to you to choose the right value for the dagrun_timeout according to the 27
time you expect your DAG will take. After having triggered your DAG multiple times, 28
you should be able to have the average time for getting it finished and set that value for the parameter. 29
The next argument we have is sla_miss_callback. 30
I'm not going to explain this notion now as 31
there is a dedicated video about using SLA in your DAGs later in the course, 32
but keep in mind that sla_miss_callback allows to call a function when reporting 33
SLA timeouts. 34
Then, we have on_failure_callback as well as on_success_callback. 35
As you can guess by their names, the foremost is used to call a function when the DagRun of this DAG 36
fails whereas the second is used to call a function when the DagRun succeeds. 37
Both functions take a context dictionary as parameter where you can find information about the DagRun 38
as well at the task instances. 39
All right, 40
we have seen the ways of being alerted at the DAG level, 41
let’s see what we can do at the task level. 42
First, we have all the arguments related to the email alerting. 43
Indeed, if you set an email and set the parameters email_on_failure as well as email_on_retry to true, 44
we will get an email as soon as a task is retried or failed. As we did earlier for the forex data pipeline, 45
if you want Airflow to send emails on retries and failures, you have to configure an SMTP server 46
in the arflow.cfg configuration file. 47
Notice that you can set more than one email where to send the alerts. Next, 48
we have all the parameters related to the retry of a task. In the same order as shown here, 49
retries indicates the number of retries that should be performed before marking the task as failed, 50
retry_delay specifies the delay with a timedelta object between retries, retry_exponential_back 51
off when set to true, allows progressive 52
longer waits between retries. And finally, max_retry_delay defines the maximum delay interval between 53
retries using a timedelta object. 54
You can see this parameter as a timeout for the retries. 55
If for any reason, a task takes more time than max_retry_delay to be triggered again, the retry is 56
skipped. 57
Then, we have execution_timeout corresponding to the maximum allowed execution time of a given task 58
instance, expressed with a timedelta object. 59
If a task takes more than execution_timeout to finish, it is marked as failed. 60
The last parameters are the callbacks. Like for the DAG, we can specify a function to call when a task 61
is either failed, succeed or retried. 62
So that's it, 63
it was a pretty short video, but the interesting part is coming as we gonna put into practice some 64
of these parameters in the next video. 65
Let's take a quick break and see you in a minute.