当VM设置完floatingip后，VM还是不能连接外网，排查原因，发现是quantum中设置的问题：
quantum中设置外网为192.168.19.129/25，不设网关，allocation_pools为{“start”: “192.168.19.130”, “end”: “192.168.19.254”}。
root@controller:/usr/src/nova# ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.19.129 0.0.0.0 UG 0 0 0 qg-29c30020-2e
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-cd728374-d8
10.0.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-f915c799-96
192.168.19.128 0.0.0.0 255.255.255.128 U 0 0 0 qg-29c30020-2e
路由器的网卡却是：
root@controller:/usr/src/nova# ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1390 (1.3 KB) TX bytes:1390 (1.3 KB)
qg-29c30020-2e Link encap:Ethernet HWaddr fa:16:3e:10:18:21
inet addr:192.168.19.130 Bcast:192.168.19.255 Mask:255.255.255.128
inet6 addr: fe80::f816:3eff:fe10:1821/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:87 errors:0 dropped:0 overruns:0 frame:0
TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:12593 (12.5 KB) TX bytes:9608 (9.6 KB)
qr-cd728374-d8 Link encap:Ethernet HWaddr fa:16:3e:d7:5a:2f
inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fed7:5a2f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:64 errors:0 dropped:0 overruns:0 frame:0
TX packets:89 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9710 (9.7 KB) TX bytes:10627 (10.6 KB)
qr-f915c799-96 Link encap:Ethernet HWaddr fa:16:3e:96:89:3a
inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe96:893a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:594 (594.0 B)
这两个值是不同的，本应从192.168.19.130路由的数据包均发往192.168.19.129，导致VM无法出去。其实后者是quantum中与外网连接的port中的fixed_ips值：
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| 10d13e25-cc01-4edc-aba4-5e2b3a6dff80 | | fa:16:3e:e6:9e:30 | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.4"} |
| 29c30020-2e91-4ffa-91e3-a8acef553641 | | fa:16:3e:10:18:21 | {"subnet_id": "3f53264f-683b-45a8-a7ab-289afd2288b5", "ip_address": "192.168.19.130"} |
| 7e659611-43b3-4f52-b392-28ddd5051bca | | fa:16:3e:9e:84:c8 | {"subnet_id": "3f53264f-683b-45a8-a7ab-289afd2288b5", "ip_address": "192.168.19.131"} |
| 7f000789-2e36-4aef-8d08-acb700ddde9f | | fa:16:3e:07:92:81 | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.2"} |
| 91da98b9-e9df-4a2c-b97d-02299d33fe89 | | fa:16:3e:f7:42:d9 | {"subnet_id": "3f53264f-683b-45a8-a7ab-289afd2288b5", "ip_address": "192.168.19.132"} |
| a132b58c-238a-4b9f-92ce-c47521cda668 | | fa:16:3e:31:81:8e | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.3"} |
| b1a9afa6-6850-4044-a2b6-cca6c12fc6fa | | fa:16:3e:89:2e:fb | {"subnet_id": "0636c5f2-70ab-4fb9-a7d5-986c92eaf1aa", "ip_address": "10.0.1.2"} |
| b629349e-ad6e-427a-8aae-291f55ef4b32 | | fa:16:3e:31:a2:cf | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.5"} |
| cd728374-d89e-4f64-b437-b3e1580b49e9 | | fa:16:3e:d7:5a:2f | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.1"} |
| f915c799-96aa-40bf-a3aa-06d43bc1c284 | | fa:16:3e:96:89:3a | {"subnet_id": "0636c5f2-70ab-4fb9-a7d5-986c92eaf1aa", "ip_address": "10.0.1.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
如果设置该网络的网关为130，提示失败：
# quantum subnet-update userA-public --gateway_ip 192.168.19.130 Gateway ip 192.168.19.130 conflicts with allocation pool 192.168.19.130-192.168.19.254
在quantum代码中体现是：
agent/l3_agent.py
ex_gw_ip = ex_gw_port['fixed_ips'][0]['ip_address']
if not ip_lib.device_exists(interface_name,
root_helper=self.root_helper,
namespace=ri.ns_name()):
......
gw_ip = ex_gw_port['subnet']['gateway_ip']
if ex_gw_port['subnet']['gateway_ip']:
cmd = ['route', 'add', 'default', 'gw', gw_ip]
不知道为什么这里有两个：ex_gw_ip和gw_ip，不一致导致这个问题。
workaround很简单：
# ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 route del default gw 192.168.19.129 # ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 route add default gw 192.168.19.130
——————————我是分割线——————————————–
上面的workaround很是麻烦，每次重启l3agent都需要添加，我今天看了一下这个问题，其实还是因为我们对neutron网络不太理解造成的。我前面的方法是先将数据包扔到qg-f103a9f2-d6接口，然后在命名空间外的路由表中进行路由决策：
# ip netns exec qrouter-09be29ea-25f6-4a53-b3ab-8d0e13dc7198 route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.19.130 0.0.0.0 UG 0 0 0 qg-f103a9f2-d6 100.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-d8fcb028-ea 192.168.19.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-f103a9f2-d6 200.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-b422c431-d8 # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.19.254 0.0.0.0 UG 100 0 0 br-ex 20.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 30.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 192.168.19.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ex
数据包的流向是qr-d8fcb028-ea->(namespace routing)->qg-f103a9f2-d6->(routing)->br-ex->eth0->router
其实public-net本身就是一个外网，所以应该跟物理机的网络一致，也就是192.168.19.0/24，网关是物理网关192.168.19.254。这样，每次l3agent都会在命名空间中新建默认路由：
# ip netns exec qrouter-09be29ea-25f6-4a53-b3ab-8d0e13dc7198 route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.19.254 0.0.0.0 UG 0 0 0 qg-f103a9f2-d6 100.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-d8fcb028-ea 192.168.19.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-f103a9f2-d6 200.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-b422c431-d8
这样数据包到了这个命名空间后，直接经过路由决策从qg-f103a9f2-d6经过br-ex到eth0出去了。虽然数据包流向与前面的一样，但是从命名空间到物理网关还是在一个网络中流动。