Hello everyone!
Welcome to the backend master class!
Continue with the background worker topic,
Today, I’ll show you why it is important
to add a bit of delay to your async task.
If you still remember, in lecture 56 of the course,
we’ve implemented this CreateUser transaction
with a callback function: AfterCreate.
This function will be called after a new user is created,
to send an async task for email verification to Redis.
And here, you can see that we have a 10-second delay,
This means that the task will only be
picked up by the worker 10 seconds after it is created.
Now, to understand why this delay is important,
let’s see what will happen if I comment out this line.
Then, let’s open the terminal and do some tests.
First, let’s make sure both Postgres and Redis servers are up and running.
OK, they are!
So we can start our web server.
Next, I’m gonna open Postman,
And send a request to create a new user: alice10
Alright, the request is successful.
Now, let’s look at the web server’s logs.
It received a request, and enqueued a verify email task
And the task is processed immediately.
There was no problem at all.
So does that mean we don’t need to delay the task?
Well, not really!
It is OK this time, but there might be issues another time.
Let’s take a look at the implementation of the CreateUser transaction
Here, the AfterCreate function is called at the end of the transaction,
But since it’s written inside the execTx function,
This function is actually run before the transaction is committed.
As you can see here,
The function is called,
and if it returns an error, the transaction will be rolled back
Otherwise, it will be committed to the database.
So what if there’s high traffic on the db
that makes this commit statement take longer time to complete?
For example, we can add a time.Sleep 2 seconds here to simulate that scenario.
Let’s see what will happen in this case.
I’m gonna restart the web server.
And go back to postman to send a create-new-user request: alice 11.
It will take about 2 seconds to complete because of the sleep statement.
But now, if we look at the logs,
We will see that the task has been enqueued,
And it has already been processed,
But failed because the user couldn’t be found in the database.
Do you know why?
Well, that’s because it took the transaction 2 seconds to be committed,
But the send email task was enqueued without delays,
So it was processed by the worker immediately.
And because of this,
when the worker tried to look for the user in the database,
It could not find the record, since the transaction was not committed yet.
Also, in the code, if the user couldn’t be found,
we return the asynq.SkipRetry error,
so the task won’t be retried in this case.
And it’s a big issue because user will never receive the verification email.
So now you know the importance of this 10 seconds delay.
It will make room for the db transaction to be fully committed
before the async task is picked up by the worker.
Alright, let’s restart the server and give it a try!
I’m gonna change the username and email to alice12
And resend the create user request.
It will take 2 seconds to finish, just like before.
But this time, in the server log, we don’t see any errors
That’s because the task has been delayed for 10 seconds.
So it’s only been enqueued for now.
And will only be processed by the worker after 10 seconds.
At that time, the create user transaction has already been committed,
So the task will be successfully processed, without any issues.
And that’s how a bit of delay can help us easily solve this problem.
But make sure you set the right amount of delay for your task,
Because for example, here if we just delay the task for 1 second,
Which is lower than the 2 second that the transaction takes to commit,
Then the problem will still occur.
Normally, I would say 10 seconds should be quite enough
for the transaction to commit,
But we never know for sure,
In some rare cases, anything could happen.
So, one way to deal with that, is to always allow the task to retry.
We can simply comment out this skip retry statement.
So even if the user is not found,
the task will be retried again at another time,
And hopefully at that time, it will be successful,
when the transaction has been committed.
Of course, if the transaction fail to commit,
then the user will never be created in the DB,
So the task will always fail.
But it’s not a big deal, because it will run out of retries eventually,
Since we have set the maximum number of retries to 10 times in this case.
Alright, let’s restart the server and do 1 more test.
In postman, I’m gonna change the username and email to alice13,
And resend the request!
The request is successful after 2 seconds.
But this time, in the log,
the task has been processed 1 time, and it failed.
That’s because we set the delay to just 1 second.
But note that, we have removed the SkipRetry statement,
So this task will be retry again later.
OK, as you can see,
After a while, the task is retried,
And it is successfully processed.
Exactly as we wanted.
Before we finish, I’m gonna revert this delay time to 10 seconds,
And remove this time.Sleep statement in the execTx function.
I’ll keep this SkipRetry here, but commented out, as a reference.
And that wraps up this lecture
about the importance of delay when processing async tasks.
I hope it was interesting and useful for you.
Thanks a lot for watching,
Happy learning, and see you in the next lecture!