Monitoring Docker Containers

Overview

Instructions for how to set up monitoring and alerting for a Docker host and containers. You can enable mist.io monitoring on the Docker host and get diagrams for system metrics, plus add custom rules to get alerted when a condition is met. By default you will reveive an email alert if the Docker host goes offline, or if network issues are experienced. As soon as mist.io servers don't receive data for 1 minute, an email alert is sent. If you want to monitor container's uptime you can achieve that by adding a simple collectd script and set custom rules.

Step 1: Enable Mist.io monitoring

Add the Docker host as a Docker Cloud and enable monitoring. Run the monitoring command and wait 1-2 minutes until the monitoring graphs are created and appear on the Machine page.

Step 2: Add a simple collectd script to monitor container uptime

You will add a custom metric for the Docker containers monitoring. Create file /opt/mistio-collectd/docker.sh and insert as content:

#!/bin/sh
list_of_containers="db-dev web-staging"
HOSTNAME="${COLLECTD_HOSTNAME:-localhost}"
INTERVAL="${COLLECTD_INTERVAL:-60}"

containers=`sudo docker ps -f status=running --format "{{.Names}}"`
for container in $list_of_containers
do
  if echo $containers |grep -q $container
    then  echo "PUTVAL \"$HOSTNAME/container-$container/gauge-status\" interval=60 N:1"
  else echo "PUTVAL \"$HOSTNAME/container-$container/gauge-status\" interval=60 N:0"
  fi
done

Make sure you put the container names on list_of_containers. In this example, you will monitor db-dev and web-staging containers.

Now edit file /opt/mistio-collectd/collectd.custom.conf and append to the end of the file:

LoadPlugin exec
<Plugin exec>
Exec "ubuntu" "/opt/mistio-collectd/docker.sh"
</Plugin>

You cannot put root here since collectd doesn't allow it, instead, you have to use a sudo user that can sudo without password (ubuntu in our case).

Make sure /etc/sudoers contains line

ubuntu ALL=(ALL) NOPASSWD:ALL

Where you replace ubuntu with the user you have. If you don't want to allow passwordless sudo access for this user, make sure this user can run the docker command with passwordless sudo. This need the full command path. Example

root@ip-172-31-21-199:~# which docker
/usr/bin/docker

So in our case the entry will be

mistio ALL=(ALL) NOPASSWD:/usr/bin/docker

Now make the script executable

root@user:~# chmod +x /opt/mistio-collectd/docker.sh

and restart collectd

root@user:~# /opt/mistio-collectd/collectd.sh restart

Step 3: View the container uptime diagram

Now visit the Machine page for the Docker host. You will see two new diagrams, one for each of the containers.

If the container is running, then values will be 1. If the container becomes unavailable, then values becomes 0 .

Step 4: Add rules to get alerted when containers are unresponsive

Now add a rule to get alerted in case the container becomes unresponsive. Click ADD RULE, select the container (name will be container NAME gauge statues where NAME is the name of our container) and set the condition as < 1

As soon as a container becomes unavailable an email alert is sent.

Step 5: Add more containers

If you want to monitor uptime for new containers, just edit file /opt/mistio-collectd/collectd.sh and add the container name on variable list_of_containers of the script (no need to restart collectd).