gracefully stop php laravel sqs worker in Docker on ECS Fargate

Using AWS SQS to process asynchronous messages is a great way to handle scheduled jobs, and work that doesn’t need to happen in real-time inside your user driven application. Containerizing a PHP Laravel app and using an orchestration service like ECS Fargate allows you to easily run thousands of job queue workers in an infinite and embarrassingly parallel fashion.

php artisan queue:work sqs

If your work queue is inconsistent in depth and rate (i.e. “bursty”) you’ll find you need to scale-out and scale-in containers based on how much work is available. Starting containers is no problem; just swipe your credit card and ECS delivers. The problem comes when needing to scale-in and stop containers after they are no longer needed, because ECS stops the php workers mid-job.

During autoscaling actions, when the ECS agent stops tasks it sends the equivalent of a docker stop to each container in the task. Underneath the covers it is sending the Unix process signal SIGTERM to the process inside the running container (PID 1). After the SIGTERM is sent, the ECS Agent waits 30 seconds for the process to exit, and if the process is still running after 30 seconds, the ECS Agent gives up and sends a SIGKILL. Sending SIGTERM (or SIGKILL) to the php process running the worker makes it immediately exit. This is expected but problematic because whatever the worker was working on is halted in the middle of what it was doing.

One solution to this problem is to wrap the php worker command inside of a bash script and use traps to catch the SIGTERM and give the worker some time to stop processing SQS messages and exit gracefully. A trap catches the signal sent to it, but it does not interrupt what the process is currently doing. The trap waits until the current process is finished, then it executes. Simply running the php worker with a trap is not enough, because the queue worker does not exit in between jobs, and php artisan queue:work sqs is a long running process. Because of this we use an infinite loop (while true; do; done;) and the –once flag to “single-run” php workers over and over. This means that for every SQS message (or empty receive) a new one-off php process is run. Doing it this way means that the trap can execute (and exit the script) in between jobs when the current job finishes processing. Something like this:

  echo "received SIGTERM, exiting..."
  exit 0

trap exit_trap SIGTERM

while true
  php artisan queue:work sqs --once

caveat emptor

  • Running php workers with –once means the entire framework has to bootstrap for every message, which may add some extra processing time. But honestly, if your framework takes a long time to load you have bigger problems.
  • Running a new php process for every message can solve the “leaky” nature of php or laravel or problematic code and long running php worker processes slowly consuming more and more memory.
  • Workers must go from listening, to working, to finished with current job within 30 seconds (or less depending on the default receive message wait time). If the worker takes longer than 30 seconds it will receive a SIGKILL mid-job and die.
  • Using the EC2 Launch Type instead of Fargate will allow you to tweak the docker stop grace period. This value is not configurable with the Fargate Launch Type and you are stuck with 30 seconds.
  • Bash is used here, and PID 1 becomes a bash script instead of a php command.
  • It’s even more important for jobs to be idempotent, and have the ability to be re-run trivially at any time.

list contents of all docker volumes

To list the contents of a docker named volume, run a temporary container and mount the volume into the container, then do a directory listing. Loop over all the volumes to see what each one holds.

~$ for i in `docker volume ls -q`; do echo "volume: ${i}"; docker run --rm -it -v ${i}:/vol alpine:latest ls /vol; echo; done;
volume: 140f898b1c69b85585942aa7f25cf03eba6ac66125d4a122e2fe99455c4a1a3f

volume: 1fa7b49173076a3a1fdb07ea7ce65d7187ff80e8b1a56e2fa667ebbbc0543f3a

volume: 5564a11a1945567ffcc231145c01c806afe13a02b3e0a548f1504a1cd36c9374

volume: 6d0b313416430d2abc0c872b98fd4180bbda4d14560c0a5d98f534f33b792164
app                 ib_buffer_pool      private_key.pem
auto.cnf            ib_logfile0         public_key.pem
ca-key.pem          ib_logfile1         server-cert.pem
ca.pem              ibdata1             server-key.pem
client-cert.pem     mysql               sys
client-key.pem      performance_schema

volume: 8b08da4a38ca8f5924b90220db8c84384d90fe331a953a5aaa1a1944d826fc68

volume: 976239a3528b8b0b074b6b7438552e1d22c4f069cf20582d2250fcc1c068dc4f

volume: aac262a93155286f4c551271d8d2a70f81ed1ca4cd56925e94006248e458895e

volume: b385ebee063b72350fbc1158788cbb43d7da9b37ec95196b74caa6b22b1c115b

volume: c73f2001f829e5574bd4246b2ab7a261a3f4d9a7ef89997765d7bf43883e5c24

volume: ce73f9f85c475b1fd9cf4fede20fd04250ee7702e83db67c29c7118055275c28

volume: foovolume1

volume: efc8a8855ac2c13d83c23573aebfd53e15072ec68d23e2793262f662ea0ae308

volume: foovolume2
auto.cnf                     ibdata1
ca-key.pem                   mysql
ca.pem                       performance_schema
client-cert.pem              private_key.pem
client-key.pem               public_key.pem
ib_buffer_pool               server-key.pem
ib_logfile0                  sys

volume: foovolume3

volume: phpsockettest

volume: foovolume4
auto.cnf            ib_logfile0         public_key.pem
ca-key.pem          ib_logfile1         server-cert.pem
ca.pem              ibdata1             server-key.pem
client-cert.pem     mysql               sys
client-key.pem      performance_schema  
ib_buffer_pool      private_key.pem

asdf   asdf1  asdf2

This installation has some test files, backing files from a few different mysql databases, a unix socket, redis files, and empty volumes.

make Makefile target for help or usage options

Using make and Makefiles with a docker based application development strategy are a great way to track shortcuts and allow team members to easily run common docker or application tasks without having to remember the syntax specifics. Without a “default” target make will attempt to run the first target (the default goal). This may be desirable in some cases, but I find it useful to have make just print out a usage, and require the operator to specify the exact target they need.

DE=docker-compose exec app

.PHONY: help
  @sh -c "echo ; echo 'usage: make <target> ' ; cat Makefile | grep ^[a-z] | sed -e 's/^/            /' -e 's/://' -e 's/help/help (this message)/'; echo"

  $(DC) up -d

  $(DC) stop

  $(DC) rm -v

  $(DC) ps

  $(DC) logs

  $(DE) sh -c "vendor/bin/phpunit"

Now without any arguments make outputs a nice little usage message:

$ make 

usage: make <target> 
            help (this message) 

This assumes a bunch of things like you must be calling make from the correct directory, but is a good working proof of concept.

Docker Compose static IP address in docker-compose.yml

Usually, when launching Docker containers we don’t really know or care what IP address a specific container will be given. If proper service discovery and registration is configured, we just launch containers as needed and they make it into the application ecosystem seamlessly. Recently, I was working on a very edge-case multi-container application where every container needed to know (or be able to predict) every other containers’ IP address at run time. This was not a cascaded need where successor containers learn predecessors’ IP addresses, but more like a full mesh.

In Docker Engine 1.10 the docker run command received a new flag namely the --ip flag. This allows you to define a static IP address for a container at run time. Unfortunately, Docker Compose (1.6.2) did not support this option. I guess we can think of Engine as being upstream of Compose, so some new Engine features take a while to make it into Compose. Luckily, this has already made it into mainline dev for Compose and is earmarked for release with the 1.7.0 milestone (which should coincide with Engine 1.11). Find the commit we care about here.

get the dev build for Compose 1.7.0:

# cd /usr/local/bin
# wget -q
# chmod 755 docker-compose-Linux-x86_64
# mv docker-compose-Linux-x86_64 docker-compose$(./docker-compose-Linux-x86_64 --version | awk '{print "-"$3$5}' | sed -e 's/,/_/')
# mv docker-compose docker-compose$(./docker-compose --version | awk '{print "-"$3$5}' | sed -e 's/,/_/')
# ln -s docker-compose-1.7.0dev_85e2fb6 docker-compose
# ls
lrwxrwxrwx 1 root root      31 Mar 30 08:38 docker-compose -> docker-compose-1.7.0dev_85e2fb6
-rwxr-xr-x 1 root root 7929597 Mar 24 08:01 docker-compose-1.6.2_4d72027
-rwxr-xr-x 1 root root 7938771 Mar 29 09:14 docker-compose-1.7.0dev_85e2fb6

In this case I decided to keep the 1.6.2 docker-compose binary along with the 1.7.0 docker-compose binary, then create a symlink to the one I wanted to use as the active docker-compose

Here’s a sample of how you might define a static IP address in docker-compose.yml that would work using docker-compose 1.7.0

version: "2"
    driver: bridge
      - subnet:

docker get list of tags in repository

The native docker command has an excellent way to search the docker hub repository for an image. Just use docker search <search string> to look in their registry.

# docker search debian
NAME                          DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
ubuntu                        Ubuntu is a Debian-based Linux operating s...   2338      [OK]       
debian                        Debian is a Linux distribution that's comp...   763       [OK]       
google/debian                                                                 47                   [OK]
neurodebian                   NeuroDebian provides neuroscience research...   12        [OK]       
jesselang/debian-vagrant      Stock Debian Images made Vagrant-friendly ...   4                    [OK]
eboraas/debian                Debian base images, for all currently-avai...   3                    [OK]
armbuild/debian               ARMHF port of debian                            3                    [OK]
mschuerig/debian-subsonic     Subsonic 5.1 on Debian/wheezy.                  3                    [OK]
fike/debian-postgresql        PostgreSQL 9.4 until 9.0 version running D...   2                    [OK]
maxexcloo/debian              Docker base image built on Debian with Sup...   1                    [OK]
kalabox/debian                                                                1                    [OK]
takeshi81/debian-wheezy-php   Debian wheezy based PHP repo.                   1                    [OK]
webhippie/debian              Docker images for debian                        1                    [OK]
eeacms/debian                 Docker image for Debian to be used with EE...   1                    [OK]
reinblau/debian               Debian with usefully default packages for ...   1                    [OK]
mariorez/debian               Debian Containers for PHP Projects              0                    [OK]
opennsm/debian                Lightly modified Debian images for OpenNSM      0                    [OK]
konstruktoid/debian           Debian base image                               0                    [OK]
visono/debian                 Docker base image of debian 7 with tools i...   0                    [OK]
nimmis/debian                 This is different version of Debian with a...   0                    [OK]
pl31/debian                   Basic debian image                              0                    [OK]
idcu/debian                   mini debian os                                  0                    [OK]
sassmann/debian-chromium      Chromium browser based on debian                0                    [OK]
sassmann/debian-firefox       Firefox browser based on debian                 0                    [OK]
cloudrunnerio/debian                                                          0                    [OK]

We can see the official debian repository right at the top. Unfortunately there’s no way to see what tags and images are available for us to pull down and deploy. However, there is a way to query the registry for all the tags in a repository, returned in JSON format. You can use a higher level programming language to get the list and parse the JSON for you. Or you can just use a simple one-liner:

# wget -q -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n'  | awk -F: '{print $3}'

Wrap that in a little bash script and you have an easy way to list the tags of a repository. Since a tag is just a pointer to a image commit multiple tags can point to the same image. Get fancy:

# wget -q -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | sed -e 's/^,//' | sort -t: -k2 | awk -F[:,] 'BEGIN {i="image";j="tags"}{if(i!=$2){print i" : "j; i=$2;j=$4}else{j=$4" | "j} }END{print i" : "j}'
image : tags
06af7ad6 : 7.5
19de96c1 : wheezy | 7.9 | 7
1aa59f81 : experimental
20096d5a : rc-buggy
315baabd : stable
37cbf6c3 : testing
47921512 : 7.7
4a5e6db8 : 8.1
4fbc238a : oldstable-backports
52cb7765 : wheezy-backports
84bd6e50 : unstable
88dc7f13 : jessie-backports
8c00acfb : latest | jessie | 8.2 | 8
91238ddc : stretch
b2477d24 : stable-backports
b5fe16f2 : 7.3
bbe78c1a : 7.8
bd4b66c4 : oldstable
c952ddeb : squeeze | 6.0.10 | 6.0 | 6
d56191e1 : 6.0.8
df2a0347 : 8.0
e565fbbc : 7.4
e7d52d7d : sid
feb75584 : 7.6
fee2ea4e : 6.0.9