Hello everyone! Welcome back to the backend master class!
In the last lecture, we�ve learned how to use docker network
to connect 2 stand-alone docker containers by names.
Today I�m gonna show you how to use docker compose
to automaticall setup all services in the same docker network
and launch them all at once with a single command.
OK let�s start!
First, I�m gonna create a new file:
docker-compose.yaml at the root of our project
There are a lot of things you can config with a docker compose file
You can read all about them in the documentation
by going to docs.docker.com
open Reference
Compose file reference
version 3
On the right-hand side menu,
you can see a list of different syntaxes.
In this video, I will show you some of the most important one that we usually use.
Here in the center of the page, we can see an example compose file
Basically you define the docker compose version,
Then a list of services that you want to launch together.
In this example, there�s a redis service,
A postgres DB service,
And some other web services for voting and computing the result.
Now I�m just gonna copy the first 2 lines of this example,
And paste them to our docker-compose.yaml file.
Note that a yaml file is very sensitive to indentation.
At the moment, the file is using 4 spaces indentation,
But I just want to use 2 spaces instead.
So I�m gonna click on this button,
Select indent using spaces,
And choose 2 as the tab size.
OK, now if we press enter,
we can see that it is nicely indented with 2 spaces.
Next we have to declare a list of services we want to launch.
The first service should be postgres db.
For this service, we�re gonna use a prebuilt docker image
Hence, the image keyword,
followed by the name and tag of the image,
which is postgres:12-alpine in this case.
Now we will use the environment keyword
to specify some environment variables
for the username, password and db name
just like what we did in the github CI workflow.
First let�s copy the POSTGRES USER variable.
The syntax is a bit different,
we have to use the equal operator to assign the value.
Similarly, I gonna copy the POSTGRES PASSWORD variable
And finally the POSTGRES DB variable.
Alright, next step, we will declare the api service,
which will serve all API requests to our simple bank.
For this service, we should build its image from the golang source,
So, under the build keyword,
We specify the context to build the image:
This dot means the current root folder.
Then we use the dockerfile keyword to tell docker compose
where to find the docker file to build the image.
In this case, it�s just the Dockerfile at the root of the project.
Next, we should publish the port 8080 to the host machine
So that we can call the simple bank API from outside of the container.
One of the most important thing we must do is
to tell the api service how to connect to the postgres service.
In order to do that, we will set 1 environment variable: DB_SOURCE
As we�ve seen in the previous lecture,
Setting this environment variable will override the value we declare in the app.env file
And since all services in this docker-compose file will run on the same network,
They can communicating with each other via name.
Therefore, here, in this URL,
instead of localhost, we will use the name of the postgres service.
And that�s basically it!
The docker-compose file is completed.
Let�s try to run it!
If you have the latest docker CLI on your machine,
All you have to do is to run docker compose up
Then docker compose will automatically search
for the docker-compose.yaml file in the current folder
and run it for you.
As you can see here,
before running the service,
It has to build the docker image for our simple bank API service first.
Then after the image is successfully built,
docker compose will start both the postgres and the api service at once.
In the logs, we can see whether it comes from which service
by looking at the prefix of the line.
It is either postgres_1 or api_1 in this case.
Now if in another terminal tab, we run docker images
We can see a new simplebank_api image.
Basically, the image name is prefixed with simplebank,
which is the name of the folder containing our docker file.
And its suffix is the name of the service itself, which is api.
Now let�s run docker ps to see all running services.
There are 2 services:
Simple bank postgres 1
And simple bank api 1
Both of them have the simple bank prefix followed by the name of the service.
Now if we go back to the docker compose tab,
And scroll up to the top,
We can see clearly here that
a new network simplebank_default is created
before the 2 service containers are created.
Now if we try to inspect the simplebank_default network,
We can see that the 2 service containers are actually running on this same network.
That�s why they can discover each other by names.
Alright, now it�s time to send some real API requests
to see if the service is working well or not.
As the new database is completely empty,
I�m just gonna send the create user API.
Oops, We�ve got 500 internal server error,
And the reason is: relation users does not exist.
Do you know why?
Well, that�s because we haven�t run db migration
to create the db schema yet.
To fix this, we have to update our docker image
to run db migration before starting the API server.
It would be done in a similar fashion
as what we�ve done in the github CI workflow:
We will have to download golang migrate binary to the docker image
and use it to run the migrations.
So I�m gonna copy these 2 instructions
And paste them to the Docker file,
in the builder stage.
First we have to run this curl command to download and extract the migrate binary
Then I�m gonna move this second command to the run stage,
Where we will copy from builder the downloaded migrate binary to the final image.
We have to change the path of the original migrate file to /app
Because that�s the working directory in the builder stage
when we download and extract the file.
Then I�m gonna put this file in the same WORKDIR folder
in the final run stage image,
which is also /app.
Next, we also have to copy all migration SQL files
from the db/migration folder to the image.
So COPY db/migration
I�m gonna put it in the migration folder under the current working directory.
Now, one thing we must do is, to install curl in the builder stage image
Because by default, the base alpine image doesn�t have curl preinstalled.
To do that, we just have to add a RUN apk add curl instruction here.
Finally we have to change the way we start the app
So that it can run the db migration before running the main binary
I�m gonna create a new file: start.sh at the top of our project
Then let�s change mod this file to make it executable.
This file will be run by /bin/sh
because we�re using alpine image,
so the bash shell is not available.
We use set -e instruction to make sure that
the script will exit immediately
if a command returns a non-zero status.
First step, we will run db migration.
So we call the /app/migrate binary
Pass in the path to the folder containing all migration SQL files,
which is /app/migration.
Then the database URL,
which we will take from the DB_SOURCE environment variable.
So here, I just use $DB_SOURCE to get its value.
Let�s also use -verbose option to print out all details when the migration is run.
Finally the up argument is used to run migrate up all migrations.
After running migrate up, we will start the app.
All we have to do in this step is to call exec $@
It basically means: takes all parameters passed to the script and run it.
In this case, we expect it to be /app/main as specified in this CMD instruction.
To make it work, we will use the ENTRYPOINT instruction
And specify the /app/start.sh file as the main entry point of the docker image.
Keep in mind that when the CMD instruction is used together with ENTRYPOINT,
It will act as just the additional parameters
that will be passed into the entry point script.
So basically, it will be similar to running �/app/starts.sh�
with �/app/main� as the second argument.
But by separating the command from the entrypoint,
we have more flexibility to replace it
with other command at run time whenever we want.
You can read more about this in the docker documentation page,
In the CMD instruction section.
OK, now the docker file is updated,
Let�s try to run docker compose again.
But first, we need to run docker compose down
To remove all existing containers and networks.
And we should also remove the simplebank_api image
Because we want to rebuild the image with the new db migration scripts.
OK, everything is cleaned up.
Let�s run docker compose up again!
It will take a while to rebuild the image.
Oops, we�ve got an error:
Start.sh: no such file or directory.
Oh, that�s because I forgot to copy the start.sh file into the docker image
So let�s do that now!
Here in the run stage,
we just need to add 1 more instruction:
COPY start.sh .
And that�s it!
We should be good now.
Let�s run docker compose down to remove all containers
Run docker rmi to remove the old simplebank_api image
And finally run docker compose up to rebuild and relaunch everything!
This time, looks like the db migration script was run,
However it was still not successful.
The service exited with code 1.
And in the log, we can see the error: connection refused.
The app cannot connect to the database to run db migration.
That�s because it takes some time for the postgres server to be ready,
But the app tried to run db migration immediately when it starts.
At that time the postgres server was not ready to accept connection yet.
So to fix this, we have to tell the app to wait for postgres to be ready
Before trying to run the db migration script and start the API server.
Now although in the docker-compose.yaml file,
we can use the depends_on instructions to tell docker compose that
the api service depends on the postgres service.
This only makes sure that the postgres will be started before the api service.
It doesn�t ensure that the postgres is in ready state before starting the api service.
You can read more about this behavior in the docker compose documentation page.
Just search for depends_on in this page.
Then you will see here, they listed several things to be aware of
when using the depends_on instruction.
If we want to wait for a service to be ready,
We should follow this link to know how to control the start up order.
There are several tools that has been written to solve this problem,
But the one we should use in our use case is the sh-compatible wait-for script,
because we�re using an alpine-based image.
OK, so here is its github page.
Wait-for is designed to synchronize services like docker containers.
And its usage is pretty simple.
We just need to run wait-for, and pass in the host:port URL we want to wait
Here�s an example,
It�s basically waiting for the web page eficode.com port 80 to be ready
before executing the echo statement.
So what I�m gonna do is
Open its latest release version,
And click on this link to download the script.
The file will be downloaded to my Downloads folder.
So I�m gonna open the terminal
And copy that file to the root of our simple bank project.
I will rename it to wait-for.sh to make it clearer about the file type.
Note that this file is not executable yet,
So we should run change mod +x to make it executable.
Alright, now back to our Dockerfile.
I�m gonna and 1 more instruction
To copy the wait-for.sh file into the final docker image.
Then, in the docker-compose file,
We should override the default entry point and command
So that it will wait for the postgres service to be ready
before trying to start the api service.
Let�s add entrypoint here
And its content should be: /app/wait-for.sh
Then the host and port to wait for should be postgres:5432
Followed by 2 dashes,
Then finally the start.sh script to run after the postgres server is ready.
Now you should know that when the entry point is overridden like this,
It will also clears out any default command on the image,
meaning that the CMD instruction we wrote in the Dockerfile will be ignored.
So in this case, we have to explicitly specify the command we want to run
in the docker-compose.yaml file as well.
We use the command keyword for that purpose.
And the command we want to run is simply: /app/main
And that will be it!
I think now we have everything we need for the app to launch successfully.
Let�s try it one more time!
First let�s run docker compose down to remove all existing containers.
Run docker rmi to remove the simplebank_api image
And finally run docker compose up to rebuild and relaunch the services.
This time, you can see that the services are started in order.
First the postgres service will run,
And only when it is ready to listen for connections,
then the api service will start running the db migration.
Therefore, the migrations were run successfully,
First the init_schema migration,
Then the add_user migration.
Finally the API server was started successfully after the db migrations.
We can see the GIN logs saying that
the server is listening and seving HTTP requests on port 8080.
So let�s open postman to try it!
I�m gonna send this same create user request as before.
But this time, the request is successful.
A new user is created!
Awesome!
And with this I�m gonna conclude today�s lecture about docker compose.
Now you know you to use this powerful tool
to launch multiple services at once,
with specific startup orders.
Thanks a lot for watching,
Happy learning, and see you in the next lecture!