Listening to Traffic in a Docker Stack (Docker Swarm)
Sometimes logs just aren’t enough information to troubleshoot your software stack. Being able to see the network traffic as it flows through your app is crucial, especially in the case of a web application.
Docker Swarm can make things more difficult because the containers are running on a separate overlay network. And because your Docker containers are meant to only contain dependencies needed for that service, they are unlikely to have any network-related tools on them.
That’s where docker run comes in. You can tell docker run that you want to launch a container (with the networking tool(s) you want) and attach that container to the network of your choice. In our case, we would want to attach such a container to the same network as the container we’re troubleshooting.
Network Troubleshooting Images
There are plenty of images out there for the exact purpose of supplementing missing network utilities on a running container. A really great one is the nicolaka/netshoot image which contains a ton of network troubleshooting tools such as tcpdump and curl. I’ll let you dig into that image if you want but as a web application developer, I mainly care about the text contents of an HTTP request/response. I could use tcpdump for this, but for command line viewing, its output isn’t the easiest to digest because it is difficult to get an ASCII output that doesn’t contain a bunch of garbage from the TCP headers.
tcpflow to the rescue. This handy utility serves a very similar purpose as tcpdump but is specifically designed to display the data in a way that is convenient for debugging. Unfortunately, as of this writing, the netshoot container doesn’t contain tcpflow.
You can find images on your own which wrap tcpflow but the one I will use is the appropriate/tcpflow image.
Displaying the HTTP Traffic
First, we have to get the container ID to pass to docker run. That is, the container whose network we want to attach to:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1549756680c0 wordpress:latest "docker-entrypoint.s…" 13 hours ago Up 13 hours 80/tcp site_wordpress.1.tgsxe9wjyj468osi1s1vyrh50Next, pull the appropriate/tcpflow image:
$ docker pull appropriate/tcpflow
Using default tag: latest
latest: Pulling from appropriate/tcpflow
ff3a5c916c92: Pull complete
4f09f203e665: Pull complete
98a1dc21e80f: Pull complete
Digest: sha256:26ba6a023a31581d8a623d438c1f02b7664af81209f7743cbb11baca483c66eb
Status: Downloaded newer image for appropriate/tcpflow:latest
docker.io/appropriate/tcpflow:latestThe basic syntax to run a container using the network of another container is:
docker run -it --net container:<container name/id> <image name> <command>Where <container name/id> is the container whose network we want to attach to and <image name> is the image we want to use to spin up a new container. Finally, <command> is used to specify what to run on that newly created container. Putting it all together with the container we got from docker ps above, we can run tcpflow -c (-c just tells tcpflow to print everything to stdout) like this:
docker run -it --net container:site_wordpress.1.tgsxe9wjyj468osi1s1vyrh50 appropriate/tcpflow tcpflow -cAnd in practice, while making a simple API request to the running WordPress container:
$ docker run -it --net container:site_wordpress.1.tgsxe9wjyj468osi1s1vyrh50 appropriate/tcpflow tcpflow -c
tcpflow: listening on eth0
010.000.000.002.55600-010.000.000.008.00088: GET /wp-json/wp/v2/users/1 HTTP/1.1
Host: localhost:88
Connection: keep-alive
sec-ch-ua: "Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"
Cache-Control: no-cache
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36
sec-ch-ua-platform: "macOS"
Accept: */*
Sec-Fetch-Site: none
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ;)
010.000.000.008.00088-010.000.000.002.55600: HTTP/1.1 200 OK
Date: Sat, 25 Sep 2021 01:56:08 GMT
Server: Apache/2.4.48 (Debian)
X-Powered-By: PHP/7.4.23
X-Robots-Tag: noindex
Link: <http://localhost:88/wp-json/>; rel="https://api.w.org/"
X-Content-Type-Options: nosniff
Access-Control-Expose-Headers: X-WP-Total, X-WP-TotalPages, Link
Access-Control-Allow-Headers: Authorization, X-WP-Nonce, Content-Disposition, Content-MD5, Content-Type
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Allow: GET
Vary: Origin
Content-Length: 618
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/json; charset=UTF-8
{"id":1,"name":"spiker830@gmail.com","url":"http:\/\/localhost:88","description":"","link":"http:\/\/localhost:88\/blog\/author\/spiker830gmail-com\/","slug":"spiker830gmail-com","avatar_urls":{"24":"http:\/\/1.gravatar.com\/avatar\/75e600535103ebcdf59da6c943c6073d?s=24&d=mm&r=g","48":"http:\/\/1.gravatar.com\/avatar\/75e600535103ebcdf59da6c943c6073d?s=48&d=mm&r=g","96":"http:\/\/1.gravatar.com\/avatar\/75e600535103ebcdf59da6c943c6073d?s=96&d=mm&r=g"},"meta":[],"_links":{"self":[{"href":"http:\/\/localhost:88\/wp-json\/wp\/v2\/users\/1"}],"collection":[{"href":"http:\/\/localhost:88\/wp-json\/wp\/v2\/users"}]}}As you can see, only the data relevant to the HTTP communication is printed. If you’ve ever tried to use tcpdump for this purpose, you’ll know how wonderful this is. We can quickly see the HTTP GET line, all the headers, the full response JSON, etc.
One final note about tcpflow. Check the documentation, but what’s nice is the command line options are pretty intuitive if you’ve ever used tcpdump. For example, if you want to listen to any interface (tcpflow -i any) or a specific one (`tcpflow -i lo`). Filters are also supported if you need more control over the specific hosts/ports you want to capture data between.
