Scenario / Questions
I have a server with two network interfaces that is running Docker. Docker, like some virtualization tools, creates a Linux bridge interface called
docker0. This interface is configured by default with an IP of
172.17.42.1 and all Docker containers communicate with this interface as their gateway and are assigned IP addresses in the same
/16 range. As I understand it, all network traffic to/from containers goes through a NAT, so outbound it appears to come from
172.17.42.1, and inbound it gets sent to
My Setup looks like so:
+------------+ / | | | +-------------+ Gateway 1 +------- | | 10.1.1.1 | / +------+-------+ +------------+ | | eth0 | / | 10.1.1.2 | | | | | | DOCKER HOST | | | | | Internet | docker0 | | | (bridge) | | | 172.17.42.1 | | | | | | eth1 | | | 192.168.1.2 | \ +------+-------+ +------------+ | | | | \ +-------------+ Gateway 2 +------- | 192.168.1.1| | +------------+
I want to route all traffic from/to any Docker containers out of the second
192.168.1.2 interface to a default gateway of
192.168.1.1, while having all traffic from/to the host machine go out the
10.1.1.2 interface to a default gateway of
10.1.1.1. I’ve tried a variety of things so far to no avail but the one thing that I think is the closest to correct is to use iproute2 like so:
# Create a new routing table just for docker echo "1 docker" >> /etc/iproute2/rt_tables # Add a rule stating any traffic from the docker0 bridge interface should use # the newly added docker routing table ip rule add from 172.17.42.1 table docker # Add a route to the newly added docker routing table that dictates all traffic # go out the 192.168.1.2 interface on eth1 ip route add default via 192.168.1.2 dev eth1 table docker # Flush the route cache ip route flush cache # Restart the Docker daemon so it uses the correct network settings # Note, I do this as I found Docker containers often won't be able # to connect out if any changes to the network are made while it's # running /etc/init.d/docker restart
When I bring up a container I cannot ping out from it at all after doing this. I’m uncertain if bridge interfaces are handled the same way physical interfaces are for this sort of routing, and just want a sanity check as well as any tips on how I might accomplish this seemingly simple task.
Find below all possible solutions or suggestions for the above questions..
You might have to look more on the iptables setup also. Docker masquerades all traffic originating from the container subnet, say 172.17.0.0/16, to 0.0.0.0. If you run
iptables -L -n -t nat, you can see the POSTROUTING chain under nat table which does this –
Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
Now, you can remove this rule and replace it with a rule which masquerades all traffic originating from the containers’ subnet to your second interface’s IP – 192.168.1.2, as that is what you desire. The delete rule will be, assuming that it is the first rule under POSTROUTING chain –
iptables -t nat -D POSTROUTING 1
Then you add this custom rule –
iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -j SNAT --to-source 192.168.1.2
A friend and I ran into this exact problem where we wanted to have docker support multiple network interfaces servicing requests. We were specifically working with the AWS EC2 service where we were also attaching/configuring/bringing up the additional interfaces. In this project, there is more than what you need so I will try to only include what you need here.
First, what we did was create a separate route table for
ip route add default via 192.168.1.2 dev eth1 table 1001
Next we configured the mangle table to set some connection marks coming in from
iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-xmark 0x1001/0xffffffff iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
Finally we add this rule for all
fwmarks to use the new table we created.
ip rule add from all fwmark 0x1001 lookup 1001
iptables command will restore the connection mark and then allow the routing rule to use the correct routing table.
iptables -w -t mangle -A PREROUTING -i docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
I believe this is all that is needed from our more complex example where (like I said) our project was attaching/configuring/bringing up the
eth1 interface at boot time.
Now this example would not stop connections from
eth0 from servicing requests through to
docker0 but I believe you could add a routing rule to prevent that.
The masquerade isn’t from 172.17.42.1 but rather
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
Which means this rule won’t work right.
ip rule add from 172.17.42.1 table docker
ip rule add from 172.17.0.0/16 table docker
Disclaimer: This has been sourced from a third party syndicated feed through internet. We are not responsibility or liability for its dependability, trustworthiness, reliability and data of the text. We reserves the sole right to alter, delete or remove (without notice) the content in its absolute discretion for any reason whatsoever.