How to Communicate Between Separate Docker Stacks (Swarm Mode)

Docker whales (a Swarm) supporting a stack of docker containers

This article explains the basics of getting two Docker Stacks to communicate with one another. But first, I want to talk quickly about why you might want to do this.

Problem

For a recent project, I had the requirement to create a dummy authentication server which would later be replaced by a black box solution provided by the customer. At first, that sounded like a no-brainer, but the customer required that the service communicating with this authentication server and that dummy server itself both had to exist as a service inside of a Docker Stack. Exposing the dummy authentication server on the host’s network wasn’t an option either.

This isn’t a super common need because typically, when someone wants to separate out stacks in this way, you would also want that system to be exposed via a network outside of Docker as well, making the solution as simple as pointing your service to a predetermined URL.

Solution

The solution here is to use an overlay network. Now by default, when you spin up a Docker Stack, you get a stack-specific overlay network out of the box so you’re services can communicate with each other using their service name. What we want to do here though is use a user-defined overlay network that we can attach our two stacks to. If you’re using Docker Swarm Mode, you’re probably already using docker compose files to define these stacks so this is really easy to do. Basically, we just have to:

Create the overlay network
Add this network to your compose file’s networks
Add the network to your service definitions

1) Create the Overlay Network to Attach To

$ docker network create -driver overlay --attachable auth-network

I named the network auth-network to match the example use case above.

2) Add Overlay Network to Compose Files

This should be done in both compose files where the stacks need to communicate with each other:

networks:
  public:
    external:
      name: auth-network

3) Add Overlay Network to Service Definitions

In the compose file for the main stack, tell the relevant service that it should be attached to our user-defined overlay network:

services:
  backend:
    # Other service definition configuration...
    networks:
      - auth-network

Do the same for the service in the other compose file. So for the example use case above:

services:
  auth-server-backend:
    # Other service definition configuration...
    networks:
      - auth-network

With this configuration, when you spin up these two stacks, the “backend” service will be able to access the “auth-server-backend” at that hostname. All while running in two separate stacks!