Integration testing and debugging many microservices can be painful. Often, I need to debug a service on a staging environment. This article shows how we use Docker for Mac with Weave (an overlay network) to connect our local machine to our remote staging environments.

In my workflow, I usually create a WIP git commit, push it to staging, and try to debug with the Ubuntu server’s limited tools. I could set up a bunch of SSH tunnels to connect to all the remote services, but our stack changes too frequently and finding IP addresses for each service is a pain. My dev machine is a Mac, so most of the tools we use locally don’t work on Linux.

I wanted something better that would speed up my dev and debugging flow. The first thing I tested with the Docker for Mac beta was Weave integration. Weave creates an overlay network to connect containers across multiple hosts together, which is very useful when distributing containers across a Swarm cluster.

image

Requirements:

  1. Remote docker-engine. I used the latest engine (v1.10.3).

  2. Weave binary on local and remote machine:

    sudo curl -L git.io/weave -o /usr/local/bin/weave

    sudo chmod a+x /usr/local/bin/weave

  3. Docker for Mac installed locally (similar steps can be followed if connecting from a Linux machine).

Note: You don’t need to have Weave running beforehand; we can dynamically attach the Weave network to any existing container.

Step 1: Set up Weave network routers and proxy

The first step is to set up our Weave overlay network so that we can connect to it from our local machine.

Remote machine:

  1. Launch Weave router: [1]

    weave launch-router --password <password>

    This will pull the latest Weave images and start the Weave router container. The password field is optional but recommended to prevent others from accessing your network while you are debugging.

  2. Launch Weave proxy:

    weave launch-proxy

    This proxy will automatically attach your IP address. More info here.

Local machine:

  1. Test network connectivity of Weave ports:

    nc -z <remote_host_ip_address> 6783

    Ensure your firewalls and security groups allow Weave’s control port (TCP 6783) and data ports (UDP 6783/6784) on the remote machine to be accessible from your local machine. If you need to create an intermediate link, refer to footnote #2.

  2. Launch Weave router on your local machine:

    weave launch-router --password <password> <remote_host_ip_address>

    If you set a password on the remote machine, ensure the same password is set here. remote_host_ip_address should be the IP address of the remote machine you set up above.

  3. Verify Weave connection:

    weave status connections

  4. Launch Weave proxy on the remote machine:

    If we were connecting from a Linux machine, we could use weave launch-proxy. However, we need to change this for Docker on Mac. The workaround is to tell Weave to listen on an HTTP interface:

    WEAVEPROXY_DOCKER_ARGS="-p 9999:9999" weave launch-proxy -H tcp://0.0.0.0:9999

    WEAVEPROXY_DOCKER_ARGS tells Weave to publish to port 9999 so the host has access to it, and the -H tcp://0.0.0.0:9999 tells Weave to listen on port 9999. If port 9999 is used on your machine, you can change it to any port, but ensure you replace 9999 with your port throughout the following steps.

Step 2: Launch containers into Weave network

Now that we have the Weave routers and proxy set up, we need some containers to talk to. In this example I will use a MongoDB container.

Remote machine:

  1. Configure Docker client to go through the Weave proxy.

    eval $(weave env)

    That command will set the DOCKER_HOST environment variable to point to the Weave proxy. Note: You’ll have use the same terminal session, or that variable will get unset.

  2. Run MongoDB container

    docker run -d --name=remote-mongo mongo

    The --name value is important here. It will be the hostname used to connect to this container. Notice that I did not expose any ports. Exposing ports to the host is not needed since all traffic to this container will be going through Weave. If you want to connect to an existing container, refer to footnote #3.

Local machine:

  1. Configure our Docker client to go through the Weave proxy.

    We also need to setup the Mac Docker client to point to our proxy. This is a bit tricky since we did a custom proxy configuration.

    1. Obtain docker.local IP address:

      ping -c1 docker.local

    2. Set DOCKER_HOST environment variable:

      export DOCKER_HOST=<docker.local_ip_address>:9999

      docker.local_ip_address should be the IP address returned from the ping command. Same note from above — you have to say in the same terminal session or that variable will get unset.

  2. Create a port forward container

    docker run -e REMOTE_HOST=remote-mongo -e REMOTE_PORT=27017 -p 27017:80 -d djfaze/port-forward

    In order to gain access to the remote machine, we need to set up a simple port forwarding container on our local machine. I used this image, which requires the following options to be passed to the docker run command:

    1. -e REMOTE_HOST=<remote_container_name>

      where remote_container_name is set to the hostname or Weave IP you want to forward to. Since we started the remote container with --name=remote-mongo, Weave DNS will resolve that name to the Weave IP address.

    2. -e REMOTE_PORT=<remote_container_port>

      where remote_container_port is set to the port you want to connect to.

    3. -p <host_port>:80

      where host_port is set to the port on your local machine you want to map to the remote container.

    The above example maps the local machine’s port 27017 to the remote container’s MongoDB port 27017.

Step 3: Connect and Play

After everything is set up, docker.local:27017 will point to the remote MongoDB container. I can now run my sample app server locally with all my local tools and simply configure my application to docker.local:27017. Integrating this with Runnable allows me to develop on my local machine with my entire isolated production stack without affecting anyone else, speeding up my feature development and bug bashing.

Using this technique, you can easily connect your local development machine to a staging environment and debug services just like you’d debug code locally.

1: Weave’s default subnet is 10.32.0.0/12. If your subnet overlaps, you will run into some issues. Add --ipalloc-default-subnet <subnet> and --ipalloc-range <subnet> to the Weave launch command where subnet is a CIDR for a non-overlapping network. ↩

2: If your application is on a private network, you can create an intermediate link with a server on a public network that can talk to your private network. On the intermittent server, run weave launch --password=<your password> <ip_addr_of_private_host> and on your local, run weave launch --password=<your password> <ip_addr_of_intermediate_server>. ↩

3: If you already have a container running and would like to attach to it, use weave attach <container_ID>. Note that DNS will not work. You will have to use the IP address returned from the attach command to connect. When using the port forwarder, use -e REMOTE_HOST=<weave_ip> where weave ip is the address returned from the weave_attach command. ↩