How to setup nova-compute for live-migration before Contrail R2.2X

In case of using Openstack included Contrail before Contrail R2.2X, you must configure below.
If you don't do that, you must hit a message below;
ERROR (BadRequest): contrail-07 is not on shared storage: Live migration can not be used without shared storage. (HTTP 400) (Request-ID: req-6e3cf590-13a9-4dd0-8406-af54d1a0f17e)

** Mandatory items
Configure mount point of NFS at /var/lib/nova/instances/global

Configure /etc/nova/nova.conf
state_path=/var/lib/nova
live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE
storage_scope=global

** Step by Step

1. Create shared directory of NFS which user/group name are nova. ( Use common uid/gid on NFS server and Compute node)
2. If different uid/gid is used between NFS server and Compute node, modify them by ''usermod -u XXX nova, groupmod –g XXX nova"
3. Modify /etc/nova.conf. See a Reference below.
4. Modify /etc/libvirt/libvirtd.conf.
• listen_tls = 0
• listen_tcp = 1
• auth_tcp = "none" 
5. Modify  /etc/default/libvirt-bin
• libvirtd_opts=" -d -l"
6. Modify either /etc/hosts or DNS server to resolve IP address of compute-node name
7. Mount shared directory of NFS to /var/lib/nova/instances/global
8. Restart nova-compute and libvirt

Reference)

root@contrail-08:~# mount | grep nfs
172.27.113.201:/lib/nfs/nova on /var/lib/nova/instances/global type nfs (rw,clientaddr=192.168.22.1,addr=172.27.113.201)

/etc/nova/nova.conf

[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lib/nova/tmp
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
verbose=True
ec2_private_dns_show_ip=False
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=ec2,osapi_compute,metadata
neutron_admin_auth_url = http://10.84.50.8:5000/v2.0/
auth_strategy = keystone
libvirt_nonblocking = True
libvirt_inject_partition = -1
rabbit_host = 10.84.50.100
glance_host = 10.84.50.8
neutron_admin_tenant_name = service
neutron_admin_username = neutron
neutron_admin_password = XXXXXXXX
neutron_url = http://10.84.50.100:9696/
neutron_url_timeout = 300
network_api_class = nova.network.neutronv2.api.API
compute_driver = libvirt.LibvirtDriver
novncproxy_base_url = http://172.27.113.201:6080/vnc_auto.html
vncserver_enabled = true
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = 192.168.22.1
security_group_api = neutron
heal_instance_info_cache_interval = 0
image_cache_manager_interval = 0
libvirt_cpu_mode = none
libvirt_vif_driver = nova_contrail_vif.contrailvif.VRouterVIFDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
glance_port = 9292
glance_num_retries = 10
rabbit_port = 5673
rabbit_retry_interval = 1
rabbit_retry_backoff = 2
rabbit_max_retries = 0
rabbit_ha_queues = True
rpc_cast_timeout = 30
rpc_conn_pool_size = 40
rpc_response_timeout = 60
rpc_thread_pool_size = 70
report_interval = 15
novncproxy_port = 6080
vnc_port = 5900
vnc_port_total = 100
resume_guests_state_on_host_boot = True
service_down_time = 300
periodic_fuzzy_delay = 30
disable_process_locking = True
live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE
storage_scope=global

[keystone_authtoken]
admin_tenant_name = service
admin_user = nova
admin_password = XXXXXXXX
auth_host = 10.84.50.8
auth_protocol = http
auth_port = 5000
signing_dir = /tmp/keystone-signing-nova

Compare Contrail TSN with OVSDB-server in ToR Switch

tor_tsn_compare.py compares Contrail TSN with OVSDB-server in ToR switch.
tor_tsn_compare uses contrail_sandeshlibs which outputs useful python dictionary from introspect information in vRouter-agent, Control-node and so on.
I will write how to use contrail_sandeshlibs in other article. Please wait a while.

How to control ToR-Switch by Contrail
Contrail configures Physical-port,VLAN, MAC, Remote-VTEP and so on in ToR Switch by OVSDB messages. OVSDB is one of the standard protocols which was defined by VMware.
OVSDB is used by open vswitch also, you can see the manual at "vtep(5)" in open vswitch.

A pic below shows the OVSDB data flow with Contrail. TSN stands for ToR service node which controls ToR Switch.

1. ToR Switch connects HAProxy by OVSDB over TLS, if High availability is configured. If not, ToR Switch directly connect to TSN.
2. HA Proxy connects one of the TSN by OVSDB over TLS
3. ToR Agent sends some messages such as ff:ff:ff:ff:ff:ff(any-dst) for BUM, IP address of Remote VTEP for Unicast,VLAN,VNI to ToR Switch.
4. ToR Agent also sends some messages such as VTEP info of ToR Switch, MAC address connecting itself to Control-node.
5. Control-node sends received data to other ToR-Agent and vRouter-agent.

（Please see Wiki in Git or Juniper.net to understand Data models）

That is, ToR_Switch receives many Configurations and other data from TSN.
While receiving data, the data might be inconsistent between TSN and ToR Switch due to connection failure or so, in some cases, traffic might be dropped.
To confirm such consistent,  we will have to confirm some tables in TSN and ToR-agent.
There are some sachems for Hardware VTEP in OVSDB-server(More detail Link). We will know lack of data by confirming which data will be synchronized or not with TSN.

For instance,  ToR Switch Ucast_Macs_Remote that ToR Switch receives shows below (Use ovseb-client)

Ucast_Macs_Remote table
MAC                 _uuid                                ipaddr locator                              logical_switch                      
------------------- ------------------------------------ ------ ------------------------------------ ------------------------------------
"02:8f:d5:8e:e6:56" 26a7d358-4c43-4831-a495-a4373042776e ""     331baf5f-6b34-4a7d-8df2-fb2aa2e6eff8 bced0b50-fbaf-4757-bb4c-0ef9532e8eaa
"02:e4:d7:4c:50:cc" a7768131-4d58-4fd0-af67-ced3836011af ""     a113bfee-4156-45ea-8371-f58dc2f1eab5 bced0b50-fbaf-4757-bb4c-0ef9532e8eaa
"22:22:22:00:00:01" 4b514bb4-e28c-49e8-bf27-bfeba71c638b ""     df1e1788-e7a7-449e-be04-4143209b776b 6b1e1249-8d02-4bbf-aac6-da9d9906dd5e
"22:22:22:00:00:01" 8c86ef9a-3a98-4adf-bdb4-cdf087018e59 ""     df1e1788-e7a7-449e-be04-4143209b776b 9a892745-09a4-434a-8bc6-b261668e8019
"22:22:22:00:00:01" b0761188-a077-4869-a6ff-78f0217951cc ""     df1e1788-e7a7-449e-be04-4143209b776b ca899f3d-d5df-44f8-9126-90125f0f52e4
"22:22:22:00:00:01" d88c82db-d8a2-40b5-a59c-8610c7e75320 ""     df1e1788-e7a7-449e-be04-4143209b776b e691aaae-e42f-4b3c-93a6-128b5421de58
"22:22:22:00:00:01" dea6b7d0-69e7-4da3-ab3e-f84e92efc9ed ""     df1e1788-e7a7-449e-be04-4143209b776b 8957219c-ed1a-4a8a-829f-4a4ccd3ef41b
"22:22:22:00:00:01" f1f35b78-aa31-47db-90ae-fb85dee61359 ""     df1e1788-e7a7-449e-be04-4143209b776b 6079900f-5e6a-45d7-b11f-adaab6442663
"22:22:22:00:00:02" 7c70dc8c-7cd7-492a-805a-b5e7519af880 ""     df1e1788-e7a7-449e-be04-4143209b776b 1c6e1fd8-89f0-4752-baec-2859f409b215
"22:22:22:00:00:02" 8b8c086b-b6d0-4519-a620-0da110fc60ff ""     df1e1788-e7a7-449e-be04-4143209b776b 895c18c3-3578-44e9-a0e3-939289b5a473
"22:22:22:00:00:02" 8d5cbf5a-1095-49ed-be28-0c8035762174 ""     df1e1788-e7a7-449e-be04-4143209b776b cdf05c2a-7f0f-4922-a1d8-95d2d38e219f
"22:22:22:00:00:02" 9027f67f-e216-4a6b-9843-a3a40688d261 ""     df1e1788-e7a7-449e-be04-4143209b776b 61f1d0a8-3889-46be-9ca0-ff45794346d5
"22:22:22:00:00:02" 9856a1e9-075e-420f-aa2f-0c45e3bb1b86 ""     df1e1788-e7a7-449e-be04-4143209b776b 827da3e0-b340-4beb-8b8b-77d8a94c1409
"22:22:22:00:00:02" a25fbec4-24a6-454c-81c5-ac6b7e677682 ""     df1e1788-e7a7-449e-be04-4143209b776b c90463d7-42e7-403f-a1ac-acb743d28f12

locator column makes us annoyed.  This locator indicates Remote VTEP, but we have to refer Phsical_locator table to get actual IP address of it.

Physical_Locator table
_uuid                                dst_ip         encapsulation_type
------------------------------------ -------------- ------------------
ea87b2ee-150e-4b60-8374-9792b0cc1172 "10.84.50.4"   "vxlan_over_ipv4" 
353eb19e-6a82-4d5a-9505-e0845b820e86 "10.84.51.72"  "vxlan_over_ipv4" 
df1e1788-e7a7-449e-be04-4143209b776b "10.84.51.73"  "vxlan_over_ipv4" 
331baf5f-6b34-4a7d-8df2-fb2aa2e6eff8 "192.168.21.1" "vxlan_over_ipv4" 
a113bfee-4156-45ea-8371-f58dc2f1eab5 "192.168.22.1" "vxlan_over_ipv4" 

We can see the actual Remote VTEP where MAC address is there by getting two tables together.
Further more, this information refers a part of table named OvsdbUnicastMacRemote in TSN
**OVSDB info in TSN can refer TSN_IPaddress:Port/ovsdb.xml.



UUID of Virtual-network needs to collect table above.
To know which MAC address is allocated in which Virtual-networks needs to confirm UUID of Virtual-network by referring Logical_Switch table from UUID of Ucast_MAC_Remote in Logical_Switch column.
In case of Contrail, UUID of Virtual-netowrk  shows "Contrail-" + UUID

Logical_Switch table
_uuid                                description name                                            tunnel_key
------------------------------------ ----------- ----------------------------------------------- ----------
827da3e0-b340-4beb-8b8b-77d8a94c1409 ""          "Contrail-0c286caa-915e-48e5-a386-50b324c22a70" 2152      
31f1c529-3c00-4066-a589-34ce67fd5e55 ""          "Contrail-220539f5-cee1-44f3-8c79-1a2ad56dc04d" 1809      
e691aaae-e42f-4b3c-93a6-128b5421de58 ""          "Contrail-25fb215b-4819-463f-be6a-0aa13c6ca984" 2041      
1c6e1fd8-89f0-4752-baec-2859f409b215 ""          "Contrail-37b9045c-6411-4fdc-8ec0-62accb6856a3" 2162      
61f1d0a8-3889-46be-9ca0-ff45794346d5 ""          "Contrail-39fb95bc-a339-4d3e-877e-34ba35c1c8a0" 1843      
13f4993a-5eda-46ae-81ff-f9aa6886b74f ""          "Contrail-5818adc2-1a15-4e90-a361-1ff0494f0227" 2359      
cdf05c2a-7f0f-4922-a1d8-95d2d38e219f ""          "Contrail-6cfdaa57-8dea-4c95-885e-3d5b87dd832c" 2008      
895c18c3-3578-44e9-a0e3-939289b5a473 ""          "Contrail-76ac5c31-dda6-4fae-870f-92d48f042c95" 2028      
bced0b50-fbaf-4757-bb4c-0ef9532e8eaa ""          "Contrail-7bab18ee-5eac-46df-9fe6-ccaa4a5dd445" 2336      
9a892745-09a4-434a-8bc6-b261668e8019 ""          "Contrail-8384126e-2ec8-4ab2-8e31-45ba1e87548a" 1878      
6079900f-5e6a-45d7-b11f-adaab6442663 ""          "Contrail-8961314e-afc6-471f-83dc-e32153440897" 2009      
ca899f3d-d5df-44f8-9126-90125f0f52e4 ""          "Contrail-912590b7-cdce-41ae-bc43-b8efff5504c2" 2055      
c90463d7-42e7-403f-a1ac-acb743d28f12 ""          "Contrail-955c489a-9712-4ee0-95ed-03021356bde6" 2046      
8957219c-ed1a-4a8a-829f-4a4ccd3ef41b ""          "Contrail-b573a1b8-e322-4896-85f9-33d7b428411e" 2153      
6b1e1249-8d02-4bbf-aac6-da9d9906dd5e ""          "Contrail-e9291384-666f-4c6a-bc23-54a396b6813f" 1815      

It is so tired to confirm each table, column, value that I created python tor_tsn_compare.py
Please download three files below in https://github.com/nakadaisuke/contrail_utils/tree/master/getnhutils . The programs basically use same libraries in Contrail.
Packages
contrail_sandeshlibs.py
ovsdb_schema_reader.py
tor_tsn_compare.py

Libraries
sys
xmltodict
urllib2
socket
json
argarse
copy

How to use it
tor_tsn_compare.py -t 'IP address of TSN' -o 'IP address of ToR-Switch' -v 'verbose'

For instance, execute command below. In case of multiple TSN in HA, IP address of TSN connects with ",".
python tor_tsn_compare.py -t '172.27.113.205:9011','172.27.113.206:9011' -o 172.27.113.51:9999

With out "-v True"、Synchronized table shows as "Sync all data". Not Synchronized table shows table as is.

python tor_tsn_compare.py -t '172.27.113.205:9011','172.27.113.206:9011' -o 172.27.113.51:9999
-------- Mcast_Macs_Remote Table --------
Sync all data
-------- Ucast_Macs_local Table --------
Sync all data
-------- Mcast_Macs_Local Table --------
Sync all data
-------- Physical_Switch Table --------
Sync all data
-------- Logical_Switch Table --------
Sync all data
-------- Ucast_Macs_Remote Table --------
Sync all data
-------- Physical_Port Table --------
Sync all data

With -v True, all of tales show（Snipped some data）

python tor_tsn_compare.py -t '172.27.113.205:9011','172.27.113.206:9011' -o 172.27.113.51:9999 -v True
-------- Mcast_Macs_Remote Table --------
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'0c286caa-915e-48e5-a386-50b324c22a70', u'10.84.50.4']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'220539f5-cee1-44f3-8c79-1a2ad56dc04d', u'10.84.50.4']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'25fb215b-4819-463f-be6a-0aa13c6ca984', u'10.84.50.4']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'37b9045c-6411-4fdc-8ec0-62accb6856a3', u'10.84.50.4']]
-------- Ucast_Macs_local Table --------
[[u'In sync(4)', u'22:22:22:00:00:01', u'0c286caa-915e-48e5-a386-50b324c22a70', u'10.84.51.72']]
[[u'In sync(4)', u'22:22:22:00:00:01', u'37b9045c-6411-4fdc-8ec0-62accb6856a3', u'10.84.51.72']]
[[u'In sync(4)', u'22:22:22:00:00:02', u'b573a1b8-e322-4896-85f9-33d7b428411e', u'10.84.51.72']]
[[u'In sync(4)', u'22:22:22:00:00:02', u'e9291384-666f-4c6a-bc23-54a396b6813f', u'10.84.51.72']]
-------- Mcast_Macs_Local Table --------
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'0c286caa-915e-48e5-a386-50b324c22a70', u'10.84.51.72']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'220539f5-cee1-44f3-8c79-1a2ad56dc04d', u'10.84.51.72']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'25fb215b-4819-463f-be6a-0aa13c6ca984', u'10.84.51.72']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'37b9045c-6411-4fdc-8ec0-62accb6856a3', u'10.84.51.72']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'39fb95bc-a339-4d3e-877e-34ba35c1c8a0', u'10.84.51.72']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'b573a1b8-e322-4896-85f9-33d7b428411e', u'10.84.51.72']]
[[u'In sync(4)', u'ff:ff:ff:ff:ff:ff', u'e9291384-666f-4c6a-bc23-54a396b6813f', u'10.84.51.72']]
-------- Physical_Switch Table --------
[[u'In sync(4)', u'shinjuku', u'10.84.51.72']]
-------- Logical_Switch Table --------
[[u'In sync(4)', u'0c286caa-915e-48e5-a386-50b324c22a70', u'2152']]
[[u'In sync(4)', u'220539f5-cee1-44f3-8c79-1a2ad56dc04d', u'1809']]
[[u'In sync(4)', u'912590b7-cdce-41ae-bc43-b8efff5504c2', u'2055']]
[[u'In sync(4)', u'b573a1b8-e322-4896-85f9-33d7b428411e', u'2153']]
[[u'In sync(4)', u'e9291384-666f-4c6a-bc23-54a396b6813f', u'1815']]
-------- Ucast_Macs_Remote Table --------
[[u'In sync(4)', u'02:8f:d5:8e:e6:56', u'7bab18ee-5eac-46df-9fe6-ccaa4a5dd445', u'192.168.21.1']]
[[u'In sync(4)', u'02:e4:d7:4c:50:cc', u'7bab18ee-5eac-46df-9fe6-ccaa4a5dd445', u'192.168.22.1']]
[[u'In sync(4)', u'22:22:22:00:00:01', u'8384126e-2ec8-4ab2-8e31-45ba1e87548a', u'10.84.51.73']]
[[u'In sync(4)', u'22:22:22:00:00:01', u'8961314e-afc6-471f-83dc-e32153440897', u'10.84.51.73']]
[[u'In sync(4)', u'22:22:22:00:00:01', u'912590b7-cdce-41ae-bc43-b8efff5504c2', u'10.84.51.73']]
-------- Physical_Port Table --------
[[u'In sync(4)', [[u'4001', u'25fb215b-4819-463f-be6a-0aa13c6ca984']], u'ge-0/0/10']]
[[u'In sync(4)', [[u'4001', u'955c489a-9712-4ee0-95ed-03021356bde6']], u'ge-0/0/11']]
[[u'In sync(4)', [[u'4001', u'912590b7-cdce-41ae-bc43-b8efff5504c2']], u'ge-0/0/12']]
[[u'In sync(4)', [[u'4001', u'0c286caa-915e-48e5-a386-50b324c22a70']], u'ge-0/0/13']]
[[u'In sync(4)', [[u'10', u'5818adc2-1a15-4e90-a361-1ff0494f0227'], [u'100', u'5818adc2-1a15-4e90-a361-1ff0494f0227'], [u'1000', u'220539f5-cee1-44f3-8c79-1a2ad56dc04d']], u'ge-0/0/19']]
[[u'In sync(4)', [[u'0', u'7bab18ee-5eac-46df-9fe6-ccaa4a5dd445']], u'ge-0/0/20']]
[[u'In sync(4)', [], u'ge-0/0/23']]
[[u'In sync(4)', [], u'ge-0/0/24']]
[[u'In sync(4)', [], u'ge-0/0/25']]

If you have any comment or something. Please send me email or post messages to Github.