VMwarePlayer+Ubuntu/KVM/OVSでOpenFlowテスト環境をつくる(6)
はじめに
第5回 に書いた OVS ブリッジ間接続について、
というコメントをいただいたので、veth 仮想リンクでつないでいた物を、OVS patch port で置き換えてどうなる物かやってみました。
OVS Patch Port?
ovs-vswitchd.conf.db(5) を見てみます。
Interface TABLE
An interface within a Port.
Summary:
(略)
Patch Options:
options : peer optional string
(略)
Details:
(略)
Patch Options:
Only patch interfaces support these options.
options : peer: optional string
The name of the Interface for the other side of the patch. The
named Interface's own peer option must specify this Interface's
name. That is, the two patch interfaces must have reversed name
and peer values.
(略)
System-Specific Details:
type: string
The interface type, one of:
system An ordinary network device, e.g. eth0 on Linux. Some-
times referred to as ``external interfaces'' since they
are generally connected to hardware external to that on
which the Open vSwitch is running. The empty string is a
synonym for system.
internal
A simulated network device that sends and receives traf-
fic. An internal interface whose name is the same as its
bridge's name is called the ``local interface.'' It does
not make sense to bond an internal interface, so the
terms ``port'' and ``interface'' are often used impre-
cisely for internal interfaces.
tap A TUN/TAP device managed by Open vSwitch.
(略)
patch A pair of virtual devices that act as a patch cable.
(略)…と思ったらわかりやすい解説があった
事前確認
veth はいったん全部消します。Patch Port の名前はこんな感じにしてみます。
patch0-1 patch1-0
+--------+ +--------+
| ovsbr0 +--------------+ ovsbr1 |
+---+----+ +----+---+
| patch0-2 | patch1-2
| +--------+ |
+-------+ ovsbr2 +-------+
+--------+
patch2-0 patch2-1veth 全部消した状態でチェック(VMは起動してある)。
stereocat@oftest03:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:0c:29:ea:f4:c0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:0c:29:ea:f4:ca brd ff:ff:ff:ff:ff:ff
4: ovsbr2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether 2e:32:b6:35:09:45 brd ff:ff:ff:ff:ff:ff
5: ovsbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether 00:0c:29:ea:f4:ca brd ff:ff:ff:ff:ff:ff
6: ovsbr1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether 4a:e1:d9:33:0c:46 brd ff:ff:ff:ff:ff:ff
7: ovsbr0p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 4a:99:17:4e:8b:27 brd ff:ff:ff:ff:ff:ff
9: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether fe:54:00:00:00:01 brd ff:ff:ff:ff:ff:ff
10: tap4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether fe:54:00:00:00:04 brd ff:ff:ff:ff:ff:ff
11: tap6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether fe:54:00:00:00:06 brd ff:ff:ff:ff:ff:ff
stereocat@oftest03:~$
stereocat@oftest03:~$ sudo ovs-vsctl show
e7f05e4d-46fb-4182-a6ee-d48ec9c232c5
Bridge "ovsbr0"
Controller "tcp:192.168.144.119"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "eth1"
Interface "eth1"
Port "tap1"
Interface "tap1"
Bridge "ovsbr1"
Controller "tcp:192.168.144.119"
Port "ovsbr1"
Interface "ovsbr1"
type: internal
Port "tap4"
Interface "tap4"
Bridge "ovsbr2"
Controller "tcp:192.168.144.119"
Port "ovsbr2"
Interface "ovsbr2"
type: internal
Port "tap6"
Interface "tap6"
ovs_version: "1.4.3"
stereocat@oftest03:~$
OVS Patch Port 設定
Patch Portの追加(各ポートで、対向側のポートを peer で指定する必要がある)。
sudo ovs-vsctl add-port ovsbr0 patch0-1 -- set interface patch0-1 type=patch options:peer=patch1-0 sudo ovs-vsctl add-port ovsbr1 patch1-0 -- set interface patch1-0 type=patch options:peer=patch0-1 sudo ovs-vsctl add-port ovsbr0 patch0-2 -- set interface patch0-2 type=patch options:peer=patch2-0 sudo ovs-vsctl add-port ovsbr2 patch2-0 -- set interface patch2-0 type=patch options:peer=patch0-2 sudo ovs-vsctl add-port ovsbr1 patch1-2 -- set interface patch1-2 type=patch options:peer=patch2-1 sudo ovs-vsctl add-port ovsbr2 patch2-1 -- set interface patch2-1 type=patch options:peer=patch1-2
再度確認。OVS内部の話なので、OS上には特に変化なし。
stereocat@oftest03:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:0c:29:ea:f4:c0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:0c:29:ea:f4:ca brd ff:ff:ff:ff:ff:ff
4: ovsbr2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether 2e:32:b6:35:09:45 brd ff:ff:ff:ff:ff:ff
5: ovsbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether 00:0c:29:ea:f4:ca brd ff:ff:ff:ff:ff:ff
6: ovsbr1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether 4a:e1:d9:33:0c:46 brd ff:ff:ff:ff:ff:ff
7: ovsbr0p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 4a:99:17:4e:8b:27 brd ff:ff:ff:ff:ff:ff
9: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether fe:54:00:00:00:01 brd ff:ff:ff:ff:ff:ff
10: tap4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether fe:54:00:00:00:04 brd ff:ff:ff:ff:ff:ff
11: tap6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether fe:54:00:00:00:06 brd ff:ff:ff:ff:ff:ff
stereocat@oftest03:~$OVS コンフィグはこうなる。
stereocat@oftest03:~$ sudo ovs-vsctl show
e7f05e4d-46fb-4182-a6ee-d48ec9c232c5
Bridge "ovsbr0"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "patch0-1"
Interface "patch0-1"
type: patch
options: {peer="patch1-0"}
Port "patch0-2"
Interface "patch0-2"
type: patch
options: {peer="patch2-0"}
Port del-controller
Interface del-controller
Port "eth1"
Interface "eth1"
Port "tap1"
Interface "tap1"
Bridge "ovsbr1"
Port "ovsbr1"
Interface "ovsbr1"
type: internal
Port "tap4"
Interface "tap4"
Port "patch1-0"
Interface "patch1-0"
type: patch
options: {peer="patch0-1"}
Port "patch1-2"
Interface "patch1-2"
type: patch
options: {peer="patch2-1"}
Bridge "ovsbr2"
Port "patch2-0"
Interface "patch2-0"
type: patch
options: {peer="patch0-2"}
Port "ovsbr2"
Interface "ovsbr2"
type: internal
Port "tap6"
Interface "tap6"
Port "patch2-1"
Interface "patch2-1"
type: patch
options: {peer="patch1-2"}
ovs_version: "1.4.3"
stereocat@oftest03:~$さてこれで動くか…と思いきや、動かない。topology-controller 動かしてみるが何も出てこない。いったん、各ブリッジからコントローラの設定を消して、"actions=NORMAL" で動かすようにしてみて、VM 間の通信が通るかどうかを確認してみるが通らない。
/var/log/openvswitch/ovs-vswitchd.log
Mar 04 22:59:11|00256|dpif|WARN|system@ovsbr1: failed to add patch1-2 as port: Address family not supported by protocol Mar 04 22:59:11|00257|bridge|WARN|patch1-2 port has no interfaces, dropping Mar 04 22:59:11|00258|bridge|INFO|destroyed port patch1-2 on bridge ovsbr1
DBを見てみる。
stereocat@oftest03:~$ sudo ovs-vsctl list Interface
(略)
_uuid : 5465cc19-768a-422a-b538-f8f54d45d6f8
admin_state : []
cfm_fault : []
cfm_mpid : []
cfm_remote_mpids : []
duplex : []
external_ids : {}
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : []
link_speed : []
link_state : []
mac : []
mtu : []
name : "patch1-0"
ofport : -1
options : {peer="patch0-1"}
other_config : {}
statistics : {}
status : {}
type : patch
(略)admin_state が "up" になってないし、ofportが -1 になっている? もう一度 ovs-vswitchd.conf.db(5) をチェック…すると、"OVS が ofport を -1 にセットしている場合、このインタフェースは追加されない" とある。うーむ。
ofport: optional integer
OpenFlow port number for this interface. Unlike most columns,
this column's value should be set only by Open vSwitch itself.
Other clients should set this column to an empty set (the
default) when creating an Interface.
Open vSwitch populates this column when the port number becomes
known. If the interface is successfully added, ofport will be
set to a number between 1 and 65535 (generally either in the
range 1 to 65279, inclusive, or 65534, the port number for the
OpenFlow ``local port''). If the interface cannot be added then
Open vSwitch sets this column to -1.
Patch Port の ofport=-1 になる問題の調査
OVS が patch port をサポートしてない?
OpenStack Quantum で Patch Port 使っているというので (OVS Quantum Plugin Documentation | Open vSwitch) ソースを見てみる。
def setup_tunnel_br(self, tun_br): '''Setup the tunnel bridge. Creates tunnel bridge, and links it to the integration bridge using a patch port. :param tun_br: the name of the tunnel bridge.''' self.tun_br = ovs_lib.OVSBridge(tun_br, self.root_helper) self.tun_br.reset_bridge() self.patch_tun_ofport = self.int_br.add_patch_port( cfg.CONF.OVS.int_peer_patch_port, cfg.CONF.OVS.tun_peer_patch_port) self.patch_int_ofport = self.tun_br.add_patch_port( cfg.CONF.OVS.tun_peer_patch_port, cfg.CONF.OVS.int_peer_patch_port) if int(self.patch_tun_ofport) < 0 or int(self.patch_int_ofport) < 0: LOG.error(_("Failed to create OVS patch port. Cannot have " "tunneling enabled on this agent, since this version " "of OVS does not support tunnels or patch ports. " "Agent terminated!")) exit(1) self.tun_br.remove_all_flows() self.tun_br.add_flow(priority=1, actions="drop")
ofport が 0 より小さい場合はエラー処理になってる。
- Q
- What features are not available in the Open vSwitch kernel datapath that ships as part of the upstream Linux kernel?
Frequently Asked Questions
- A
- The kernel module in upstream Linux 3.3 and later does not include tunnel virtual ports, that is, interfaces with type "gre", "ipsec_gre", "gre64", "ipsec_gre64", "vxlan", or "lisp". It is possible to create tunnels in Linux and attach them to Open vSwitch as system devices. However, they cannot be dynamically created through the OVSDB protocol or set the tunnel ids as a flow action.
Work is in progress in adding tunnel virtual ports to the upstream Linux version of the Open vSwitch kernel module. For now, if you need these features, use the kernel module from the Open vSwitch distribution instead of the upstream Linux kernel module.
The upstream kernel module does not include patch ports, but this only matters for Open vSwitch 1.9 and earlier, because Open vSwitch 1.10 and later implement patch ports without using this kernel feature.
いろいろ回ってたらそれっぽいのを発見。
openvswitch dkms package についての fix が入っているという。この時点ではインストールされていない。
stereocat@oftest03:~$ aptitude search openvswitch i openvswitch-brcompat - Open vSwitch bridge compatibility support p openvswitch-brcompat:i386 - Open vSwitch bridge compatibility support i A openvswitch-common - Open vSwitch common components p openvswitch-common:i386 - Open vSwitch common components i openvswitch-controller - Open vSwitch controller implementation p openvswitch-controller:i386 - Open vSwitch controller implementation p openvswitch-datapath-dkms - Open vSwitch datapath module source - DKMS version i openvswitch-datapath-source - Open vSwitch datapath module source - module-assistant version p openvswitch-dbg - Debug symbols for Open vSwitch packages p openvswitch-dbg:i386 - Debug symbols for Open vSwitch packages p openvswitch-ipsec - Open vSwitch GRE-over-IPsec support p openvswitch-ipsec:i386 - Open vSwitch GRE-over-IPsec support i A openvswitch-pki - Open vSwitch public key infrastructure dependency package i openvswitch-switch - Open vSwitch switch implementations p openvswitch-switch:i386 - Open vSwitch switch implementations p openvswitch-test - Open vSwitch test package p python-openvswitch - Python bindings for Open vSwitch p quantum-plugin-openvswitch - Quantum is a virtual network service for Openstack. (openvswitch plugin) p quantum-plugin-openvswitch-agent - Quantum is a virtual network service for Openstack. (openvswitch plugin agent) stereocat@oftest03:~$
インストールする。
stereocat@oftest03:~$ sudo aptitude install openvswitch-datapath-dkms
[sudo] password for stereocat:
The following NEW packages will be installed:
dkms{a} openvswitch-datapath-dkms
0 packages upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,027 kB of archives. After unpacking 12.8 MB will be used.
Do you want to continue? [Y/n/?]
Get: 1 http://jp.archive.ubuntu.com/ubuntu/ quantal-updates/main dkms all 2.2.0.3-1.1ubuntu1.1 [71.4 kB]
Get: 2 http://jp.archive.ubuntu.com/ubuntu/ quantal-updates/universe openvswitch-datapath-dkms all 1.4.3-0ubuntu2.1 [1,955 kB]
Fetched 2,027 kB in 23s (85.7 kB/s)
Selecting previously unselected package dkms.
(Reading database ... 63865 files and directories currently installed.)
Unpacking dkms (from .../dkms_2.2.0.3-1.1ubuntu1.1_all.deb) ...
Selecting previously unselected package openvswitch-datapath-dkms.
Unpacking openvswitch-datapath-dkms (from .../openvswitch-datapath-dkms_1.4.3-0ubuntu2.1_all.deb) ...
Processing triggers for man-db ...
Setting up dkms (2.2.0.3-1.1ubuntu1.1) ...
Setting up openvswitch-datapath-dkms (1.4.3-0ubuntu2.1) ...
Creating symlink /var/lib/dkms/openvswitch/1.4.3/source ->
/usr/src/openvswitch-1.4.3
DKMS: add completed.
Kernel preparation unnecessary for this kernel. Skipping...
Building module:
cleaning build area....(bad exit status: 2)
./configure --with-linux='/lib/modules/3.5.0-25-generic/build' && make -C datapath/linux...........
cleaning build area....(bad exit status: 2)
DKMS: build completed.
openvswitch:
Running module version sanity check.
- Original module
- Installation
- Installing to /lib/modules/3.5.0-25-generic/updates/dkms/
brcompat.ko:
Running module version sanity check.
- Original module
- Installation
- Installing to /lib/modules/3.5.0-25-generic/updates/dkms/
depmod.........
DKMS: install completed.
stereocat@oftest03:~$
stereocat@oftest03:~$ ls /lib/modules/3.5.0-25-generic/updates/dkms/
brcompat.ko openvswitch.ko
stereocat@oftest03:~$ sudo find / -name openvswitch.ko -type f
/var/lib/dkms/openvswitch/1.4.3/3.5.0-25-generic/x86_64/module/openvswitch.ko
/var/lib/dkms/openvswitch/1.4.3/build/datapath/linux/openvswitch.ko
/lib/modules/3.5.0-25-generic/kernel/net/openvswitch/openvswitch.ko
/lib/modules/3.5.0-25-generic/updates/dkms/openvswitch.ko
/lib/modules/3.5.0-17-generic/kernel/net/openvswitch/openvswitch.ko
stereocat@oftest03:~$カーネルモジュールの入れ替えなので再起動する。
再度 DB を確認。
stereocat@oftest03:~$ sudo ovs-vsctl list interface
(略)
_uuid : 166d972b-fb9d-4a60-8d36-de6b5cac7d37
admin_state : up
cfm_fault : []
cfm_mpid : []
cfm_remote_mpids : []
duplex : []
external_ids : {}
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 0
link_speed : []
link_state : up
mac : []
mtu : []
name : "patch0-1"
ofport : 2
options : {peer="patch1-0"}
other_config : {}
statistics : {collisions=0, rx_bytes=7936, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=124, tx_bytes=7936, tx_dropped=0, tx_errors=0, tx_packets=124}
status : {}
type : patch
(略)Patch Port が有効になっているのがわかる。
再度動作確認
OVS Config を表示してみる。
stereocat@oftest03:~$ sudo ovs-vsctl show
e7f05e4d-46fb-4182-a6ee-d48ec9c232c5
Bridge "ovsbr0"
Controller "tcp:192.168.144.119"
is_connected: true
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port del-controller
Interface del-controller
Port "patch0-1"
Interface "patch0-1"
type: patch
options: {peer="patch1-0"}
Port "eth1"
Interface "eth1"
Port "tap1"
Interface "tap1"
Port "patch0-2"
Interface "patch0-2"
type: patch
options: {peer="patch2-0"}
Bridge "ovsbr1"
Controller "tcp:192.168.144.119"
is_connected: true
Port "ovsbr1"
Interface "ovsbr1"
type: internal
Port "patch1-0"
Interface "patch1-0"
type: patch
options: {peer="patch0-1"}
Port "patch1-2"
Interface "patch1-2"
type: patch
options: {peer="patch2-1"}
Port "tap4"
Interface "tap4"
Bridge "ovsbr2"
Controller "tcp:192.168.144.119"
is_connected: true
Port "patch2-0"
Interface "patch2-0"
type: patch
options: {peer="patch0-2"}
Port "patch2-1"
Interface "patch2-1"
type: patch
options: {peer="patch1-2"}
Port "ovsbr2"
Interface "ovsbr2"
type: internal
Port "tap6"
Interface "tap6"
ovs_version: "1.4.3"
stereocat@oftest03:~$topology-controller でトポロジ確認
stereocat@oftest02:~/trema-orig/ruby_topology$ sudo trema run topology-controller.rb 0x3 (port 1) <-> 0x1 (port 5) topology updated 0x1 (port 2) <-> 0x2 (port 1) 0x3 (port 1) <-> 0x1 (port 5) topology updated 0x1 (port 2) <-> 0x2 (port 1) 0x3 (port 1) <-> 0x1 (port 5) 0x3 (port 2) <-> 0x2 (port 2) topology updated 0x1 (port 2) <-> 0x2 (port 1) 0x1 (port 5) <-> 0x3 (port 1) 0x3 (port 1) <-> 0x1 (port 5) 0x3 (port 2) <-> 0x2 (port 2) topology updated 0x1 (port 2) <-> 0x2 (port 1) 0x1 (port 5) <-> 0x3 (port 1) 0x2 (port 2) <-> 0x3 (port 2) 0x3 (port 1) <-> 0x1 (port 5) 0x3 (port 2) <-> 0x2 (port 2) topology updated 0x1 (port 2) <-> 0x2 (port 1) 0x1 (port 5) <-> 0x3 (port 1) 0x2 (port 1) <-> 0x1 (port 2) 0x2 (port 2) <-> 0x3 (port 2) 0x3 (port 1) <-> 0x1 (port 5) 0x3 (port 2) <-> 0x2 (port 2) topology updated
おわりに
あっさり終わるかなと思いきや結構面倒だった…。OVS Patch Port でもブリッジ間接続ができるというのはわかりました。とりあえず Patch Port 設定して動いた、というところなので、veth 仮想リンク使うのと何か違いがあるのかな? というのは今のところは正直よくわかりません。あ、使う上では、全部 OVS DB に閉じた使い方ができる Patch Port の使い勝手が良さそうだ、というのはあります。(veth 作ってから OVS 設定やるという二度手間が省ける。) openvswitch-datapath-dkms とかカーネル側の対応が必要なようなので、OS側の都合がつけば、ですが。
