By Chad Morgenstern and Jenny Yang
Overview
In the blog post Cinder Cheesecake: Things to Consider, we wrote that we would be looking at various failure scenarios. Scenarios that build on one another to give a clear view of what happening in the environment. This blog focuses on scenario 1, namely:
- Your environment encounters an issue where the storage platform has failed yet the Nova compute node has survived. In this scenario, the failure does not affect the root volume. In our studies, we chose to use a persistent root volume, though the scenario is no different with an ephemeral root.
Now we show you how to perform failover and failback when using iSCSI Cinder backends. Specifically, this scenario entails recovery by Cinder failover-host following the loss of an iSCSI backend while the root volumes remain accessible. More blogs will be coming soon covering other disaster recovery scenarios involving Cheesecake, the subject of this sub-series.
If this is the first blog post that you are reading in the series, please STOP and review the blog post Cinder Cheesecake: Things to Consider before proceeding forward. All technologies have strengths and weaknesses to be considered before implementing; Cheesecake is no exception.
Before diving in, here is a summary of the steps detailed in the main body of the blog. Consider it a check list of sorts for when you do this yourself. Please don't be intimidated by the density of this blog. The process is simple:
Configure backends and get replication working from production (site A).
- Configure the production and disaster recovery (or site A and site B) backends in the
cinder.conf
file. - Restart Cinder volume services:
systemctl restart openstack-cinder-volume.services
. - Time passes, users create Cinder volumes, work happens...
- Trigger Cinder failover:
cinder failover-host --backend_id <backend_name> <host@backend>
. - Stop the impacted Nova instances, detach, and reattach Cinder block devices, start Nova instances.
Note: See the Python script found here. It is designed to detach and reattach Cinder block devices while preserving attachment order. The script is unsupported, so please feel free to modify it. Use is, of course, at your own risk. - Re-enable the Cinder backend if you plan to add additional Cinder volumes while failed over.
Make the disaster recovery site (site B) the primary site and replicate back to site A.
- Shutdown Cinder services:
systemctl stop openstack-cinder*
. - Capture a database backup.
- Modify some rows in volumes and
cinder.services
tables. - Modify the
enable_backends
stanza in/etc/cinder.conf
to make the disaster recovery backend active instead of the production backend. - Start up all Cinder services:
systemctl start openstack-cinder[api|volume|scheduler].service
.
Make production (site A) the primary site again and replicate back to site B.
- Shut down all Nova instances that are using "to be failed back" Cinder volumes.
- Manually perform a final SnapMirror update.
- Shut down Cinder services:
systemctl stop openstack-cinder*
. - Capture a database backup.
- Modify some rows in volumes and
cinder.services
tables. - Modify the
enable_backends
stanza in/etc/cinder.conf
to make the production backend active instead of the disaster recovery backend. - Start up all Cinder services:
systemctl start openstack-cinder[api|volume|scheduler].service
. - Detach, and reattach, Cinder block devices, start Nova instances.
Note: See the Python script attached to this blog, it is designed to detach and reattach Cinder block devices while preserving attachment order. The script is unsupported, so please feel free to modify it. Use is, of course, at your own risk.
Getting production ready: preparing for DR
1) Modify the /etc/cinder/cinder.conf to enable replication
Our test environment includes both iSCSI and NFS backends. We setup an NFS backend to store the root device for the Nova instances, as Cheesecake is being evaluated in this blog against the iSCSI backends only. The NFS backend is known by the name loweropenstack1
. All references to backends from this point forward are to the iSCSI backends only.
Please notice in the below /etc/cinder/cinder.conf
that replication is setup in each iSCSI backend stanza. This is to simplify the replicate -> failover -> replicate -> failback process. Note that replication only occurs for the enabled backends.
[DEFAULT] enabled_backends = iprod,loweropenstack1 [iprod] volume_backend_name=iprod volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver netapp_server_hostname=10.63.158.xxx netapp_server_port=80 netapp_storage_protocol=iscsi netapp_storage_family=ontap_cluster netapp_login=admin netapp_password=****** netapp_vserver=iprod replication_device = backend_id:idr netapp_replication_aggregate_map = backend_id:idr,openstack1_01_AggrGroup1_1:openstack2_01_AggrGroup1_1 [idr] volume_backend_name=idr volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver netapp_server_hostname=10.63.158.xxx netapp_server_port=80 netapp_storage_protocol=iscsi netapp_storage_family=ontap_cluster netapp_login=admin netapp_password=****** netapp_vserver=idr replication_device = backend_id:iprod netapp_replication_aggregate_map = backend_id:iprod,openstack2_01_AggrGroup1_1:openstack1_01_AggrGroup1_1
To keep things simple, we:
- Only activated one set of backends at a time, either disaster recovery or production.
- Used the cluster admin user. You may use SVM users. Please make sure to familiarize yourself with the for your OpenStack release to ensure that the SVM user is defined with the appropriate level of permissions.
- Established the cluster peer relationship in advance between the two ONTAP clusters:
openstack2::> cluster peer create -peer-addrs 10.63.158.xxx -address-family ipv4 openstack1::> cluster peer create -peer-addrs 10.63.158.xxx -address-family ipv4
- Established the SVM peer relationships in advance for each set of backends:
openstack1::> vserver peer create -vserver iprod -peer-vserver idr -applications snapmirror -peer-cluster openstack2
2) Enable the Cinder backend and check that the backend is initialized
Please take note, it is possible for the volume services to have a status of 'up' even if the backend failed to properly initialize. While It is important to view the services via cinder service-list --withreplication
, please do not forget to inspect the /var/log/cinder/volume.log
file for initialization errors as well.
[root@cloud1 ~(keystone_admin)]# cinder service-list --withreplication +-----------------------+-----------------------+------+---------+-------+-----------------------------+--------------------+ | Binary | Host | Zone | Status | State | Updated_at | Replication Status | +-----------------------+-----------------------+------+---------+-------+-----------------------------+--------------------+ | cinder-backup | cloud1 | nova | enabled | up | 2017-06-27T18:16:39.000000 | | | cinder-scheduler | cloud1 | nova | enabled | up | 2017-06-27T18:16:36.000000 | | | cinder-volume | cloud1@iprod | nova | enabled | up | 2017-06-27T18:16:39.000000 | enabled | | cinder-volume | cloud1@loweropenstack1| nova | enabled | up | 2017-06-27T18:16:39.000000 | disabled | +-----------------------+-----------------------+------+---------+-------+-----------------------------+--------------------+
3) Ensure that SnapMirror has initialized and is in a healthy state
openstack1::> snapmirror list-destinations -fields source-path,destination-path,status source-path destination-path status ------------------------------------------------------------------------- ----------------------------------------------------------------------- ------ iprod:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1 idr:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1 Idle iprod:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 idr:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 Idle
4) Remove resource limitation for the demo project
Modify the quotas for the demo project, set instance and volume resources to unlimited.
[root@cloud1 cinder(keystone_admin)]# openstack quota set --volumes -1 --ram -1 --cores -1 --gigabytes -1 --ports -1 --instances -1 demo
5) Create Cinder extra-specs and map volume backend names thereon
Use the command cinder get-capabilities
to see the possible Cinder volume attributes. This comes in handy when assigning the extra-specs to volume types too.
[root@cloud1 nova(keystone_admin)]# cinder get-capabilities cloud1@iprod +---------------------------------------+------------------------------------------+ | Volume stats | Value | +---------------------------------------+------------------------------------------+ | description | None | | display_name | None | | driver_version | 1.0.0 | | namespace | OS::Storage::Capabilities::cloud1@iprod | | pool_name | None | | replication_targets | [] | | storage_protocol | iSCSI | | vendor_name | NetApp | | visibility | None | | volume_backend_name | iprod | +---------------------------------------+------------------------------------------+ +-----------------------+---------------------------------------------------------------------------------------------------+ | Backend properties | Value | +-----------------------+---------------------------------------------------------------------------------------------------+ | compression | {u'type': u'boolean', u'description': u'Enables compression.', u'title': u'Compression'} | | qos | {u'type': u'boolean', u'description': u'Enables QoS.', u'title': u'QoS'} | | replication | {u'type': u'boolean', u'description': u'Enables replication.', u'title': u'Replication'} | | thin_provisioning | {u'type': u'boolean', u'description': u'Sets thin provisioning.', u'title': u'Thin Provisioning'} | +-----------------------+---------------------------------------------------------------------------------------------------+ [root@cloud1 ~(keystone_admin)]# cinder type-create netapp_iscsi [root@cloud1 ~(keystone_admin)]# cinder type-create netapp_nfs [root@cloud1 ~(keystone_admin)]# cinder type-key netapp_iscsi set vendor_name=NetApp storage_protocol=iSCSI [root@cloud1 ~(keystone_admin)]# cinder type-key netapp_nfs set vendor_name=NetApp storage_protocol=nfs
6) Create Cinder volumes on their respective backends:
Create the root Cinder volumes using the netapp_nfs volume type.
[root@cloud1 cinder(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 10 --volume-type netapp_nfs --image cirros-nfs-image --display-name cirros-$i; ((i = i + 1 )) ;done
Create the data Cinder volumes using the netapp_iscsi volume type.
[root@cloud1 cinder(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name prefailover-iscsi-$i; ((i = i + 1 )) ;done
7) Create Nova instances and associate the Cinder volumes
Before creating the Nova instances, we need to create a security group (firewall) and keypair.
[root@cloud1 ~(keystone_demo)]# openstack security group create demo && openstack security group rule create --src-ip 0.0.0.0/0 --dst-port 22 demo [root@cloud1 ~(keystone_demo)]# openstack keypair create demo > ~/demo.pem && chmod 400 ~/demo.pem
Create forty-nine Nova instances associated with the forty-nine CirrOS root volumes.
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do openstack server create --nic net-id=f968c291-1f17-4fd2-8fb8-36f3ccac3889 --volume cirros-$i --flavor m1.tiny --security-group admin --key-name demo n$i ; (( i = i + 1 )); done
Connect the pre-failover Cinder volumes to Nova instances.
root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do openstack server add volume n$i prefailover-iscsi-$i; (( i = i + 1 )); done
Review the attachments.
[root@cloud1 nova(keystone_demo)]# cinder list | grep -i pre | sort -k6 +---------------------------------------+---------+---------------------+------+--------------+----------+---------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +-----------------------------------------------------------------------+------+--------------+----------+---------------------------------------+ | 743586fe-5487-406c-9cfb-420c7235a6c5 | in-use | prefailover-iscsi-10 | 1 | netapp_iscsi | false | 7cee3434-8335-4807-8600-1500e6db2aa1 | | cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2 | in-use | prefailover-iscsi-11 | 1 | netapp_iscsi | false | 9d4116ac-490a-419b-9c41-e66faec5cc56 | | 901912f1-1548-469d-986a-9a7371dc3943 | in-use | prefailover-iscsi-1 | 1 | netapp_iscsi | false | f7714ff4-c55e-45f6-bf40-a52f37988bb4 | ... | f3a85f33-771e-4562-ac8f-07ea0b7409fd | in-use | prefailover-iscsi-33 | 1 | netapp_iscsi | false | 0652ebbf-8e48-40e3-8e8b-fed722db2a46 | | 42b54d00-dc6e-4b8c-9b5a-d974fb60a0e2 | in-use | prefailover-iscsi-34 | 1 | netapp_iscsi | false | 8bfd1dfb-98a2-4581-ab5e-ce0f7865b6fc | | 498b0a66-4b6c-40bc-b148-4fc5880d95d2 | in-use | prefailover-iscsi-35 | 1 | netapp_iscsi | false | d080b7d7-b9e3-4e79-848d-8c8a9dc8d1fa | ... | 43b5e897-7f1b-41d1-b7dd-924b26a1bc76 | in-use | prefailover-iscsi-48 | 1 | netapp_iscsi | false | d2074077-be7f-4b5a-a332-af08998f7687 | | c24dadcb-aa39-4c20-b15c-a24a949b0292 | in-use | prefailover-iscsi-49 | 1 | netapp_iscsi | false | f9f7c317-12df-4783-b35c-ade78486ac6b | | 20a00f00-6cda-40ff-b529-460cfa63ecea | in-use | prefailover-iscsi-5 | 1 | netapp_iscsi | false | 461b0075-8ad9-4d51-afb4-e5e58311e542 |
8) Leave breadcrumbs in the Cinder volumes for later review
At every stage of this process Cinder volumes will be formatted and mounted, and a file will be written within each Nova instance. In effect, we are leaving breadcrumbs to prove that the process works at every point.
Connect to the Nova instance to prepare the block device for use.
[root@cloud1 nova(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"sudo mkdir /${name}-prefailover \; sudo /usr/sbin/mkfs.ext4 /dev/vdb \; sudo mount /dev/vdb /${name}-prefailover \; sudo touch /${name}-prefailover/${name}-pre-failover\"; done
Simulate a failure of the production backend
While a true disaster recovery scenario can happen at any moment, planned failovers afford the opportunity to avoid data loss. In the event of an unplanned disaster recovery scenario, only perform step 3.
- (Planned failover only) Power off all instances accessing impacted volumes.
- (Planned failover only) With the instances powered off, issue a SnapMirror update from either NetApp's OnCommand System Manager or via the
snapmirror update
command. In the event of a planned outage, steps 1 and 2 ensure zero data loss. - Issue the
cinder failover-host --backend_id <backend_name> <host@backend>
command.
To simulate an actual failure scenario, we chose to disable iSCSI protocol access on storage virtual machine backing the production backend. Towards that end, we disabled the production SVM.
1) Disable protocol access on the source storage cluster
openstack1::> vserver stop iprod openstack1::> iscsi show -vserver iprod Vserver: iprod, Target Name: iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10, Target Alias: iprod, Administrative Status: down
2) Trigger failover of the production Cinder backend
[root@cloud1 cinder(keystone_admin)]# cinder failover-host --backend_id idr cloud1@iprod
3) Check the status of the Cinder services, ensure that production is failed over
[root@cloud1 cinder(keystone_admin)]# cinder service-list --withreplication +--------------------+------------------------+-------+----------+-------+-------------------------------+-------------+-------------------+-----------------------+ | Binary | Host | State | Status | State | Updated_at | Replication | Active Backend ID | Disabled Reason | +--------------------+------------------------+-------+----------+-------+-------------------------------+-------------+-------------------+-----------------------+ | cinder-backup | cloud1 | nova | enabled | up | 2017-06-27T22:26:35.000000 | | | - | | cinder-scheduler | cloud1 | nova | enabled | up | 2017-06-27T22:26:33.000000 | | | - | | cinder-volume | cloud1@iprod | nova | disabled | up | 2017-06-27T22:26:32.000000 | failed-over | idr | failed-over | | cinder-volume | cloud1@loweropenstack1 | nova | enabled | up | 2017-06-27T22:26:33.000000 | disabled | - | - | +--------------------+------------------------+-------+----------+-------+-------------------------------+-------------+-------------------+-----------------------+
If I/O is performed against the prefailover-iscsi-*
block device attached to any of the Nova instances an I/O error will be triggered. Cinder failover-host
has no immediate effect on the Nova instances, as they are still pointing to the production target (iqn and LUN). A look inside the Nova database will clear things up a bit.
The nova.block_device_mapping
table shows that the Nova instances have not been instructed to disconnect from the production SVM (iprod) and reconnect to the disaster recovery SVM (idr).
The nova.block_device_mapping
table maintains a listing of all block devices attached or formerly attached to each instance. We see for example that the instance f7714ff4-c55e-45f6-bf40-a52f37988bb4
currently sees that a device known to the database as /dev/vdb
is presently attached (deleted = 0
). The connection_info column shows us that Nova is tracking information about the connection behind /dev/vdb
.
The connection_info column of the block_device_mapping
table is a json object. We need to focus specifically on the key:value pairs found within data key. The data key is a hash storing key:value pairs describing the target, or storage device backing the vdb
.
The target_iqn
is the production storage arrays iSCSI Qualified Name.
openstack1::> iscsi show -vserver iprod ( Target Name: iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10)
The target_portal
is the IP address belonging to a logical interface on the production) SVM.
openstack1::> net int show (network interface show) Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home -------- ----------- ----------- ------------------ ----------- ------- ---- Iprod iprod_e0e_1 up/up 192.168.0.7/23 openstack1-01 e0e true
[root@cloud1 cinder(keystone_admin)]# mysql MariaDB [none]> connect nova MariaDB [nova]> select instance_uuid,deleted,device_name,connection_info from block_device_mapping where connection_info like '%netapp:%'; | instance_uuid | f7714ff4-c55e-45f6-bf40-a52f37988bb4 | deleted | 0 | device_name | /dev/vdb | connection_info | { "driver_volume_type": "iscsi", "connector":{}, "serial": "901912f1-1548-469d-986a-9a7371dc3943", "data": { "device_path": "/dev/disk/by-path/ip-192.168.0.7:3260-iscsi-iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10-lun-0", "target_discovered": false, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10", "target_portal": "192.168.0.7:3260", "volume_id": "901912f1-1548-469d-986a-9a7371dc3943", "target_lun": 0, "access_mode": "rw" } }
4) Detach and reattach the Cinder volumes from Nova instances
Show the current state of the Cinder volumes from Cinder's perspective.
[root@cloud1 nova(keystone_demo)]# cinder list | grep –i pre | sort –k6 +---------------------------------------+--------+----------------------+------+--------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +---------------------------------------+--------+----------------------+------+--------------+----------+--------------------------------------+ | 743586fe-5487-406c-9cfb-420c7235a6c5 | in-use | prefailover-iscsi-10 | 1 | netapp_iscsi | false | 7cee3434-8335-4807-8600-1500e6db2aa1 | | cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2 | in-use | prefailover-iscsi-11 | 1 | netapp_iscsi | false | 9d4116ac-490a-419b-9c41-e66faec5cc56 | | 901912f1-1548-469d-986a-9a7371dc3943 | in-use | prefailover-iscsi-1 | 1 | netapp_iscsi | false | f7714ff4-c55e-45f6-bf40-a52f37988bb4 | ... | f3a85f33-771e-4562-ac8f-07ea0b7409fd | in-use | prefailover-iscsi-33 | 1 | netapp_iscsi | false | 0652ebbf-8e48-40e3-8e8b-fed722db2a46 | | 42b54d00-dc6e-4b8c-9b5a-d974fb60a0e2 | in-use | prefailover-iscsi-34 | 1 | netapp_iscsi | false | 8bfd1dfb-98a2-4581-ab5e-ce0f7865b6fc | | 498b0a66-4b6c-40bc-b148-4fc5880d95d2 | in-use | prefailover-iscsi-35 | 1 | netapp_iscsi | false | d080b7d7-b9e3-4e79-848d-8c8a9dc8d1fa | ... | 43b5e897-7f1b-41d1-b7dd-924b26a1bc76 | in-use | prefailover-iscsi-48 | 1 | netapp_iscsi | false | d2074077-be7f-4b5a-a332-af08998f7687 | | c24dadcb-aa39-4c20-b15c-a24a949b0292 | in-use | prefailover-iscsi-49 | 1 | netapp_iscsi | false | f9f7c317-12df-4783-b35c-ade78486ac6b | | 20a00f00-6cda-40ff-b529-460cfa63ecea | in-use | prefailover-iscsi-5 | 1 | netapp_iscsi | false | 461b0075-8ad9-4d51-afb4-e5e58311e542 |
Issue the request to detach the Cinder volumes from the all Nova instances.
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do openstack server remove volume n$i prefailover-iscsi-$i ; (( i = i + 1 )) ; done
Below we see that the detachment request occurred successfully.
[root@cloud1 ~(keystone_demo)]# cinder list | grep pre | sort -k6 +----------------------------------------+-----------+----------------------+------+--------------+----------+ | ID | Status | Name | Size | Volume Type | Bootable | +----------------------------------------+-----------+----------------------+------+--------------+----------+ | 743586fe-5487-406c-9cfb-420c7235a6c5 | available | prefailover-iscsi-10 | 1 | netapp_iscsi | false | | cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2 | available | prefailover-iscsi-11 | 1 | netapp_iscsi | false | ... | 21fd8abf-e63c-4634-a7fa-9fb8344de61d | available | prefailover-iscsi-3 | 1 | netapp_iscsi | false | | a12d5cfb-3aec-42d9-a844-1a50df570077 | available | prefailover-iscsi-32 | 1 | netapp_iscsi | false | | f3a85f33-771e-4562-ac8f-07ea0b7409fd | available | prefailover-iscsi-33 | 1 | netapp_iscsi | false | ... | 2a51b57d-2d91-4bce-9bda-7eebbeb4d39a | available | prefailover-iscsi-47 | 1 | netapp_iscsi | false | | 43b5e897-7f1b-41d1-b7dd-924b26a1bc76 | available | prefailover-iscsi-48 | 1 | netapp_iscsi | false | | c24dadcb-aa39-4c20-b15c-a24a949b0292 | available | prefailover-iscsi-49 | 1 | netapp_iscsi | false |
For academic purposes, let's look once more inside the Nova database.
A similar query of the Nova database's block_device_mapping
table shows that the detach has no impact on information maintained in the connection info field. Notice however that the deleted field is now a positive integer, signifying that Nova is aware that the device has been detached. To make the output easier to read, we collapsed the connector hash.
[root@cloud1 cinder(keystone_admin)]# mysql MariaDB [none]> connect nova MariaDB [nova]> select instance_uuid,deleted,device_name,connection_info from block_device_mapping where connection_info like '%netapp:%'; | instance_uuid | f7714ff4-c55e-45f6-bf40-a52f37988bb4 | deleted | 133 | device_name | /dev/vdb | connection_info | { "driver_volume_type": "iscsi", "connector": {}, "serial": "901912f1-1548-469d-986a-9a7371dc3943", "data": { "device_path": "/dev/disk/by-path/ip-192.168.0.7:3260-iscsi-iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10-lun-0", "target_discovered": false, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10", "target_portal": "192.168.0.7:3260", "volume_id": "901912f1-1548-469d-986a-9a7371dc3943", "target_lun": 0, "access_mode": "rw" } }
Reattach the Cinder volumes to the Nova instances.
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do openstack server add volume n$i prefailover-iscsi-$i ; (( i = i + 1 )) ; done [root@cloud1 nova(keystone_demo)]# cinder list | grep -i pre | sort -k6 +---------------------------------------+--------+----------------------+------+--------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +---------------------------------------+--------+----------------------+------+-------------------------+--------------------------------------+ | 743586fe-5487-406c-9cfb-420c7235a6c5 | in-use | prefailover-iscsi-10 | 1 | netapp_iscsi | false | 7cee3434-8335-4807-8600-1500e6db2aa1 | cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2 | in-use | prefailover-iscsi-11 | 1 | netapp_iscsi | false | 9d4116ac-490a-419b-9c41-e66faec5cc56 | 901912f1-1548-469d-986a-9a7371dc3943 | in-use | prefailover-iscsi-1 | 1 | netapp_iscsi | false | f7714ff4-c55e-45f6-bf40-a52f37988bb4 ... | f3a85f33-771e-4562-ac8f-07ea0b7409fd | in-use | prefailover-iscsi-33 | 1 | netapp_iscsi | false | 0652ebbf-8e48-40e3-8e8b-fed722db2a46 | 42b54d00-dc6e-4b8c-9b5a-d974fb60a0e2 | in-use | prefailover-iscsi-34 | 1 | netapp_iscsi | false | 8bfd1dfb-98a2-4581-ab5e-ce0f7865b6fc | 498b0a66-4b6c-40bc-b148-4fc5880d95d2 | in-use | prefailover-iscsi-35 | 1 | netapp_iscsi | false | d080b7d7-b9e3-4e79-848d-8c8a9dc8d1fa ... | 43b5e897-7f1b-41d1-b7dd-924b26a1bc76 | in-use | prefailover-iscsi-48 | 1 | netapp_iscsi | false | d2074077-be7f-4b5a-a332-af08998f7687 | c24dadcb-aa39-4c20-b15c-a24a949b0292 | in-use | prefailover-iscsi-49 | 1 | netapp_iscsi | false | f9f7c317-12df-4783-b35c-ade78486ac6b | 20a00f00-6cda-40ff-b529-460cfa63ecea | in-use | prefailover-iscsi-5 | 1 | netapp_iscsi | false | 461b0075-8ad9-4d51-afb4-e5e58311e542 ...
Let's look once more inside of the Nova database to see what happens upon reattach.
This time the query returns a second entry backing /dev/vdb
for the f7714ff4-c55e-45f6-bf40-a52f37988bb4
instance in addition to the first shown above. The first is the one we just looked at above, and for clarity's sake it is not shown below. Please notice that this second entry's attachment information comes from the disaster recovery SVM's iSCSI IQN and LIF. This shows us that the connection information is only modified on attach.
openstack2::> iscsi show -vserver idr ( Target Name: iqn.1992-08.com.netapp:sn.3251d9c75a7e11e7841c00a09874b32e:vs.26 MariaDB [none]> connect nova MariaDB [nova]> select instance_uuid,deleted,device_name,connection_info from block_device_mapping where connection_info like '%netapp:%'; | instance_uuid | f7714ff4-c55e-45f6-bf40-a52f37988bb4 | deleted | 0 | device_name | /dev/vdb | connection_info | { "driver_volume_type": "iscsi", "connector": {}, "serial": "901912f1-1548-469d-986a-9a7371dc3943", "data": { "device_path": "/dev/disk/by-path/ip-192.168.0.11:3260-iscsi-iqn.1992-08.com.netapp:sn.3251d9c75a7e11e7841c00a09874b32e:vs.26-lun-0",, "target_discovered": false, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.1992-08.com.netapp:sn.3251d9c75a7e11e7841c00a09874b32e:vs.26", "target_portal": "192.168.0.11:3260", "volume_id": "901912f1-1548-469d-986a-9a7371dc3943", "target_lun": 0, "access_mode": "rw" } }
Detach/reattach left instances the in a funky state, proving the necessity of rebooting the instances.
As we attached the block device without rebooting the instances, each instance sees the reattached device as /dev/vdc
rather than /dev/vdb
which causes problems accessing the filesystems mounted at /dev/vdb
. The following demonstrates the point.
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"lsblk\;df\" ; done NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk `-vda1 253:1 0 10G 0 part / vdc 253:32 0 1G 0 disk Filesystem 1K-blocks Used Available Use% Mounted on /dev/vda1 23797 18447 4122 82% / /dev/vdb 1032088 34052 945608 3% /n1-prefailover ... ... NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk `-vda1 253:1 0 10G 0 part / vdc 253:32 0 1G 0 disk Filesystem 1K-blocks Used Available Use% Mounted on /dev/vda1 23797 18441 4128 82% / /dev/vdb 1032088 34052 945608 3% /n49-prefailover
At this point we reboot the instances to clean up the instances device entries.
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do nova reboot n$i ; (( i = i + 1 )); done
Check access to the volumes had been reset by remounting, then leave breadcrumbs.
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"sudo mount /dev/vdb /${name}-prefailover \; sudo touch /${name}-prefailover/${name}-post-recover-from-failover\"; done
So far so good, breadcrumbs are in place.
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"sudo df \| grep /${name}-prefailover \; sudo ls -l /${name}-prefailover/\"; done /dev/vdb 1032088 34052 945608 3% /n1-prefailover total 16 -rw------- 1 root root 0 Jun 27 23:14 n1-pre-failover -rw------- 1 root root 0 Jun 28 08:00 n1-post-recover-from-failover ... ... /dev/vdb 1032088 34052 945608 3% /n49-prefailover total 16 -rw------- 1 root root 0 Jun 27 23:14 n49-pre-failover -rw------- 1 root root 0 Jun 28 08:01 n49-post-recover-from-failover
5) Enable creation of Cinder volumes in DR
At this point we have confirmed that pre-existing Cinder volumes can be accessed from DR. At this point, we have yet to re-enable replication from disaster recovery to any other site, but this is coming up soon. First, let's test out the creation/attachment of additional Cinder volumes in DR.
Enable the disabled backend so that we can create a new Cinder volume.
[root@cloud1 ~(keystone_admin)]# cinder service-enable cloud1@iprod cinder-volume
The status of the iprod backend is enabled.
[root@cloud1 cinder(keystone_admin)]# cinder service-list --withreplication +------------------+------------------------+------+---------+-------+-----------------------------+--------------------+-------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Replication Status | Active Backend ID | Disabled Reason | +------------------+------------------------+------+---------+-------+-----------------------------+--------------------+-------------------+-----------------+ | cinder-scheduler | cloud1 | nova | enabled | up | 2017-06-28T07:10:23.000000 | | | - | | cinder-volume | cloud1@iprod | nova | enabled | up | 2017-06-28T07:10:22.000000 | failed-over | idr |failed-over | | cinder-volume | cloud1@loweropenstack1 | nova | enabled | up | 2017-06-28T07:10:15.000000 | disabled | - | - | +------------------+------------------------+------+---------+-------+-----------------------------+--------------------+-------------------+-----------------+
6) Create post-failover Cinder volumes, then attach the volumes to their respective instances
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name postfailover-iscsi-$i; openstack server add volume n$i postfailover-iscsi-$i; ((i = i + 1 )) ;done [root@cloud1 ~(keystone_demo)]# openstack volume list | grep post | sort -k6 | 41c44567-1718-4ad0-9dd1-e47fac3ab250 | postfailover-iscsi-10 | in-use | 1 | Attached to n10 on /dev/vdc | | 798c8bb4-43d8-4cb2-aa8c-e0221d9299d5 | postfailover-iscsi-11 | in-use | 1 | Attached to n11 on /dev/vdc | | 3a21d3e8-b6ed-4c28-8a00-e23ef4b7dc01 | postfailover-iscsi-12 | in-use | 1 | Attached to n12 on /dev/vdc | ... | cb77b0a0-5a2d-4656-a01f-9672ad961507 | postfailover-iscsi-5 | in-use | 1 | Attached to n5 on /dev/vdc | | b1ea3c35-1ec6-415a-8f18-0f89990740f8 | postfailover-iscsi-6 | in-use | 1 | Attached to n6 on /dev/vdc | | 879cc859-b950-43e0-bcdc-a960031622d0 | postfailover-iscsi-7 | in-use | 1 | Attached to n7 on /dev/vdc | | b33dd2bb-da80-4375-a034-01a74c25cf41 | postfailover-iscsi-8 | in-use | 1 | Attached to n8 on /dev/vdc | | eba5a235-4a11-4578-8c94-6e6734ac14e8 | postfailover-iscsi-9 | in-use | 1 | Attached to n9 on /dev/vdc |
7) Create more breadcrumbs, this time on the /dev/vdc device
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"sudo mkdir /${name}-postfailover \; sudo /usr/sbin/mkfs.ext4 /dev/vdc \; sudo mount /dev/vdc /${name}-postfailover \; sudo touch /${name}-postfailover/${name}-post-failover\"; done
Prepare idr for production needs and eventual failback
Though the disaster recovery site is serving in the place of production, it is not yet the source of replication. Additionally, you may eventually wish to fail back to the original production site. The steps in this section accomplish both needs:
Keep the following in mind:
- Cheesecake only enables replication on active backends. At this point, the idr backend is only active by reference of iprod being failed over. The steps in this section enable replication. Replication will occur between idr and the backend configured in the
/etc/cinder/cinder.conf
. - When the iprod environment is brought back online (eventually), you may wish to fail back to iprod. This section walks through making that possible.
Before triggering the failback
1) Prepare for fail back by enabling idr backend and resyncing backwards from idr to iprod
Modify /etc/cinder/cinder.conf
placing idr in the list of enabled_backends instead of iprod.
[root@cloud1 ~(keystone_demo)]# vim /etc/cinder/cinder.conf #enabled_backends = iprod,loweropenstack1 enabled_backends = idr,loweropenstack1
2) Shutdown all Cinder services, as we are about to modify the Cinder database; this is non-disruptive to client I/O
[root@cloud1 ~(keystone_admin)]# systemctl stop openstack-cinder*
3) Create breadcrumbs
The OpenStack control plane has no impact on I/O, even with openstack-cinder-*.services
disabled, client I/O continues. To prove that, we created breadcrumbs here.
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"sudo touch /${name}-prefailover/${name}-down-services \; sudo touch /${name}-postfailover/${name}-down-services\"; done
Backup your database:
The instructions to follow will take you through modifications of the tables(s) within the Cinder database. Modification of OpenStack database(s) is inherently dangerous, as such it is imperative to preserve the database against the eventuality that something should go wrong. Numerous database engines can be used in OpenStack, so please consult the users guide for your database flavor regarding database backups/dumps.
4) Modify cinder.volumes and cinder.services tables using the following SQL code
Before making changes to the cinder database, we need to understand what the tables that we are about to change look like. @iprod will be replaced with @idr when done.
[root@cloud1 ~(keystone_admin)]# mysql MariaDB > connect cinder MariaDB [cinder]> select id,display_name,host,provider_location from volumes where deleted = 0 and host like '%BLOCK%' order by display_name; | id | 55174f08-2d2e-470f-84a0-e5abaa382e63 | display_name | postfailover-iscsi-1 | host | cloud1@iprod#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 | provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-55174f08-2d2e-470f-84a0-e5abaa382e63 ... | id | 41c44567-1718-4ad0-9dd1-e47fac3ab250 | display_name | postfailover-iscsi-10 | host | cloud1@iprod#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 | provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-41c44567-1718-4ad0-9dd1-e47fac3ab250 ... | Id | 0c38db73-6d74-42f7-8708-420ba9b4207a | display_name | prefailover-iscsi-7 | host | cloud1@iprod#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1 | provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1/volume-0c38db73-6d74-42f7-8708-420ba9b4207a
Replace iprod with idr in the host column of the volumes table. Post failover, all Cinder volumes on the iprod backend had been accessed via the idr backend. From this point forward, the Cinder volumes will get accessed directly from idr without reference of iprod.
MariaDB [cinder]> update volumes set host = REPLACE(host,'iprod', 'idr') where host like '%iprod%'; MariaDB [cinder]> select id,display_name,host,provider_location from volumes where deleted = 0 and host like '%BLOCK%' order by display_name; | id | 55174f08-2d2e-470f-84a0-e5abaa382e63 | display_name | postfailover-iscsi-1 | host | cloud1@idr#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 | provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-55174f08-2d2e-470f-84a0-e5abaa382e63 | id | 41c44567-1718-4ad0-9dd1-e47fac3ab250 | display_name | postfailover-iscsi-10 | host | cloud1@idr#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 | provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-41c44567-1718-4ad0-9dd1-e47fac3ab250 ... | id | 0c38db73-6d74-42f7-8708-420ba9b4207a | display_name | prefailover-iscsi-7 | host | cloud1@idr#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1 | provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1/volume-0c38db73-6d74-42f7-8708-420ba9b4207a
The next set of commands remove the failed-over
flags from all volumes and services, and the active-backend
field is made NULL
. Once this is done, two benefits are derived:
- Cheesecake will be able to resync from idr to iprod. Otherwise, though Cheesecake will setup the SnapMirror relationship, it will not re-establish the transfers between the two sites.
- iprod, when the time comes, will be able to actively serve data once again.
Before making the change, pay attention to the active_backend_id
, disabled_reason
and replication status
fields.
MariaDB [cinder]> select id,host,topic,disabled,disabled_reason,replication_status,active_backend_id from services; +-----+-------------------------+-------------------+----------+-----------------+--------------------+-------------------+ | id | host | topic | disabled | disabled_reason | replication_status | active_backend_id | +-----+-------------------------+-------------------+----------+-----------------+--------------------+-------------------+ | 6 | cloud1 | cinder-scheduler | 0 | NULL | not-capable | NULL | | 8 | cloud1@iprod | cinder-volume | 0 | failed-over | failed-over | idr | | 11 | cloud1@loweropenstack1 | cinder-volume | 0 | NULL | disabled | NULL | +-----+-------------------------+-------------------+----------+-----------------+--------------------+-------------------+
MariaDB [cinder]> update services set disabled_reason= NULL where disabled_reason = 'failed-over';
NOTE: The above command will have no effect, as you already enabled the service. However, the command is included here just in case you had not done so previously.
MariaDB [cinder]> update services set replication_status = 'disabled' where replication_status = 'failed-over'; MariaDB [cinder]> update services set active_backend_id = NULL where active_backend_id = 'idr'; MariaDB [cinder]> select id,host,topic,disabled,disabled_reason,replication_status,active_backend_id from services; +-----+-------------------------+-------------------+-----------+-----------------+---------------------+-------------------+ | id | host | topic | disabled | disabled_reason | replication_status | active_backend_id | +-----+-------------------------+-------------------+-----------+-----------------+---------------------+-------------------+ | 6 | cloud1 | cinder-scheduler | 0 | NULL | not-capable | NULL | | 8 | cloud1@iprod | cinder-volume | 0 | NULL | NULL | NULL | | 11 | cloud1@loweropenstack1 | cinder-volume | 0 | NULL | disabled | NULL | +-----+-------------------------+-------------------+-----------+-----------------+---------------------+-------------------+
Update the replication_status
fields for affected volumes.
MariaDB [cinder]> update volumes set replication_status = 'disabled' where replication_status = 'failed-over';
5) Start up the openstack-cinder services
[root@cloud1 ~(keystone_demo)]# systemctl start openstack-cinder-{api,volume,scheduler}.service
Check the status of the Cinder services as well as the replication status. The idr host no longer shows as failed-over
, and idr is shown to be active. Just as cool, replication is now occurring between site idr and iprod.
[root@cloud1 cinder(keystone_admin)]# cinder service-list --withreplication +------------------+------------------------+------+---------+-------+-----------------------------+--------------------+ | Binary | Host | Zone | Status | State | Updated_at | Replication Status | +------------------+------------------------+------+---------+-------+-----------------------------+--------------------+ | cinder-scheduler | cloud1 | nova | enabled | up | 2017-06-28T08:01:50.000000 | | | cinder-volume | cloud1@idr | nova | enabled | up | 2017-06-28T08:01:53.000000 | enabled | | cinder-volume | cloud1@iprod | nova | enabled | down | 2017-06-28T07:41:42.000000 | disabled | | cinder-volume | cloud1@loweropenstack1 | nova | enabled | up | 2017-06-28T08:01:54.000000 | disabled | +------------------+------------------------+------+---------+-------+-----------------------------+--------------------+
6) Create and attach one more set of Cinder volumes at the disaster recovery site
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name prefailback-iscsi-$i; openstack server add volume n$i prefailback-iscsi-$i; ((i = i + 1 )) ;done
7) Once more, we leave breadcrumbs
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"sudo mkdir /${name}-prefailback \; sudo /usr/sbin/mkfs.ext4 /dev/vdd \; sudo mount /dev/vdd /${name}-prefailback \; sudo touch /${name}-prefailback/${name}-pre-failback \"; done
Failback from idr to iprod
Failing back from the disaster recovery site to the production site is a planned event. It is important that you shutdown your Nova instances, manually initiate a final SnapMirror transfer, and then fail back. This section details the process of failing back from the DR backend to the production backend.
Before triggering the failback
1) Enable the SVM on site 1
Recovering the production environment may be more complicated than enabling the production SVM.
openstack1::> vserver start iprod
2) Shut down the instances prior to failing back
Shutting down the instances prevents further I/O and ensures no data loss.
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do nova stop n$i ; (( i = i + 1 )); done
Confirm that the instances have shut down.
[root@cloud1 ~(keystone_demo)]# nova list +----------------------------------------+------+----------+------------+-------------+ | ID | Name | Status | Task State | Power State | +----------------------------------------+------+----------+------------+-------------+ | f7714ff4-c55e-45f6-bf40-a52f37988bb4 | n1 | SHUTOFF | - | Shutdown | ... | f9f7c317-12df-4783-b35c-ade78486ac6b | n49 | SHUTOFF | - | Shutdown | ... | b8f57eab-b150-41f6-9a3f-bd5803bfc784 | n9 | SHUTOFF | - | Shutdown | +----------------------------------------+------+----------+------------+-------------+
4) Initiate a manual SnapMirror update
SnapMirror transfers have been occurring between idr and iprod, perhaps for some time. After shutting down the Nova instances, trigger a final SnapMirror transfer to pick up the last set of data.
openstack1::> snapmirror update -destination-path *
5) Trigger the failback
Notice that idr is now listed as failed over to iprod.
[root@cloud1 ~(keystone_admin)]# cinder failover-host --backend_id iprod cloud1@idr [root@cloud1 ~(keystone_admin)]# cinder service-list --withreplication +------------------+------------------------+------+----------+-------+----------------------------+--------------------+-------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Replication Status | Active Backend ID | Disabled Reason | +------------------+------------------------+------+----------+-------+----------------------------+--------------------+-------------------+-----------------+ | cinder-scheduler | cloud1 | nova | enabled | up | 2017-06-28T08:24:20.000000 | | | - | | cinder-volume | cloud1@idr | nova | disabled | up | 2017-06-28T08:24:23.000000 | failed-over | iprod | failed-over | | cinder-volume | cloud1@iprod | nova | enabled | down | 2017-06-28T07:41:42.000000 | disabled | - | - | | cinder-volume | cloud1@loweropenstack1 | nova | enabled | up | 2017-06-28T08:24:25.000000 | disabled | - | - | +------------------+------------------------+------+----------+-------+----------------------------+--------------------+-------------------+-----------------+
Make the production site production again
1) Modify /etc/cinder/cinder.conf by placing iprod in the list of enabled_backends instead of idr
[root@cloud1 ~(keystone_demo)]# vim /etc/cinder/cinder.conf enabled_backends = iprod,loweropenstack1 #enabled_backends = idr,loweropenstack1
2) Shut down all Cinder services, as we are about to modify the Cinder database
[root@cloud1 ~(keystone_admin)]# systemctl stop openstack-cinder*
3) Modify the Cinder database, replacing idr with iprod
NOTE: Get a MySQL database backup before performing this step.
[root@cloud1 ~(keystone_admin)]# mysql MariaDB [none] > connect cinder MariaDB [cinder]> update volumes set host = REPLACE(host,'idr', 'iprod') where host like '%idr%'; MariaDB [cinder]> update volumes set replication_status = 'disabled' where replication_status = 'failed-over'; MariaDB [cinder]> update services set replication_status = 'disabled' where replication_status = 'failed-over'; MariaDB [cinder]> update services set disabled_reason= NULL where disabled_reason = 'failed-over'; MariaDB [cinder]> update services set active_backend_id = NULL where active_backend_id = 'iprod';
4) Restart Cinder services
[root@scsor0013409001 ~(keystone_demo)]# systemctl start openstack-cinder-{api,volume,scheduler}.service
5) Re-enable the idr backend so it's ready to go in case you want to fail over again in the future
[root@cloud1 ~(keystone_admin)]# cinder service-enable cloud1@idr cinder-volume
6) Using automation, detach, and reattach the Cinder volumes to the Nova instances
Reattaching (detaching and reattaching) multiple block devices from a Nova instance is more complicated than reattaching a single block device because of attachment order issues. If you want to maintain the attachment order, the devices must all be detached from an instance before any are reattached. Also, the devices must be reattached in the same order each time.
To simplify this process, I have written and attached to this blog a short Python script called recinder.py
. The script is currently only intended for use with Nova instances that are booted from persistent volumes. In its current form, it will only detach and reattach non-root disks.
How it works:
recinder.py
makes an ordered list of all:
- Cinder volumes attached to Nova instances
- The Nova instance names
- The current device names associated with the Cinder volumes according to Cinder. For example
/dev/vdb
,/dev/vdc
, etc...
Using this ordered list, recinder.py
detaches all non-root devices (anything other than /dev/vda
) from all instances and then reattaches the Cinder volumes in the original order of device_name, i.e. starting at /dev/vdb
, then /dev/vdc
, etc... In this version, the root device is not considered as root devices cannot be detached. Future versions may have the ability to destroy the Nova instances associated with persistent root volumes and then redeploy and reattach all associated Cinder volumes.
[root@cloud1 ~(keystone_demo)]# recinder.py openstack server remove volume n12 c2df6ec2-9833-4ab8-bdab-6dba9a255467 openstack server remove volume n12 3a21d3e8-b6ed-4c28-8a00-e23ef4b7dc01 openstack server remove volume n12 a49b524f-2abe-472b-87c5-dd0945dcd813 openstack server remove volume n12 a2b16bc2-af58-46e4-96fc-61a6c04d572b ... openstack server add volume n49 9e278a46-e49c-4148-84a9-c4ad715c6ce7 openstack server add volume n49 e567c262-9772-44e5-b7d1-6bc576a6df62 openstack server add volume n49 7d57be2e-45a3-454a-9066-aa568b61a634 openstack server add volume n49 eaeac0b1-411d-4d25-b875-547ed5717263
7) Create and attach a final set of Cinder volumes
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name postfailback-iscsi-$i ; openstack server add volume n$i postfailback-iscsi-$i ; ((i = i + 1 )) ;done
View all post-failback Cinder volumes. Let's focus on the total attachment list for a couple of Nova instances.
[root@cloud1 ~(keystone_demo)]# openstack volume list | egrep "Attached to n1 |Attached to n49|Attached to n22" | sort -k12 | 931554bd-82c3-4f11-9f73-5aa125fbc96f | cirros-1 | in-use | 10 | Attached to n1 on /dev/vda | | 901912f1-1548-469d-986a-9a7371dc3943 | prefailover-iscsi-1 | in-use | 1 | Attached to n1 on /dev/vdb | | 55174f08-2d2e-470f-84a0-e5abaa382e63 | postfailover-iscsi-1 | in-use | 1 | Attached to n1 on /dev/vdc | | 83cb90b0-767b-4464-aa35-bafd25f53386 | prefailback-iscsi-1 | in-use | 1 | Attached to n1 on /dev/vdd | | a9f91719-e93b-4013-a6e9-8260d86948cd | postfailback-iscsi-1 | in-use | 1 | Attached to n1 on /dev/vde | ... | 55a1cf30-bd5d-4d02-bb9f-a46f3a3c5d76 | cirros-22 | in-use | 10 | Attached to n22 on /dev/vda | | 3cad8d03-015b-472a-9e99-43574b8fb684 | prefailover-iscsi-22 | in-use | 1 | Attached to n22 on /dev/vdb | | 69909909-d03d-4561-af12-bfc3617ea796 | postfailover-iscsi-22 | in-use | 1 | Attached to n22 on /dev/vdc | | e8603cb2-199d-4491-913c-8511c6b0064a | prefailback-iscsi-22 | in-use | 1 | Attached to n22 on /dev/vdd | | 2dbf505e-2a58-4a6f-a749-be49cddc9cb4 | postfailback-iscsi-22 | in-use | 1 | Attached to n22 on /dev/vde | ... | 348bccc3-7b81-4871-93ad-12d13f6a3f45 | cirros-49 | in-use | 10 | Attached to n49 on /dev/vda | | c24dadcb-aa39-4c20-b15c-a24a949b0292 | prefailover-iscsi-49 | in-use | 1 | Attached to n49 on /dev/vdb | | 5a78efb4-ba66-451f-95b1-c91c3ea69a65 | postfailover-iscsi-49 | in-use | 1 | Attached to n49 on /dev/vdc | | 4039e49a-1c8e-4135-a8d2-f1eae8a935e1 | prefailback-iscsi-49 | in-use | 1 | Attached to n49 on /dev/vdd | | 1abfe06b-424c-4c01-83d5-652ce19f2ef4 | postfailback-iscsi-49 | in-use | 1 | Attached to n49 on /dev/vde |
8) Start up the Nova instances
[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do nova start n$i ; ((i = i + 1 )) ;done [root@cloud1 ~(keystone_demo)]# nova list +---------------------------------------+------+--------+------------+-------------+ | ID | Name | Status | Task State | Power State | +---------------------------------------+------+--------+------------+-------------+ | f7714ff4-c55e-45f6-bf40-a52f37988bb4 | n1 | ACTIVE | - | Running | | 7cee3434-8335-4807-8600-1500e6db2aa1 | n10 | ACTIVE | - | Running | ... | 8593bda5-39b4-44c4-a461-ba0a17b40dfc | n47 | ACTIVE | - | Running | | d2074077-be7f-4b5a-a332-af08998f7687 | n48 | ACTIVE | - | Running | | f9f7c317-12df-4783-b35c-ade78486ac6b | n49 | ACTIVE | - | Running | | 461b0075-8ad9-4d51-afb4-e5e58311e542 | n5 | ACTIVE | - | Running | | 2b890f68-f0b8-4239-87da-972e4cfa984b | n6 | ACTIVE | - | Running | | d11e09c6-9974-4c67-8cc5-408546ddfbee | n7 | ACTIVE | - | Running | | d0f9bf29-c71d-4212-8ea0-e965155c5887 | n8 | ACTIVE | - | Running | | b8f57eab-b150-41f6-9a3f-bd5803bfc784 | n9 | ACTIVE | - | Running | +---------------------------------------+------+--------+------------+-------------+
9) Create more breadcrumbs
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"sudo mount /dev/vdb /${name}-prefailover \; sudo mount /dev/vdc /${name}-postfailover \; sudo mount /dev/vdd /${name}-prefailback \; sudo mkdir /${name}-postfailback \; sudo /usr/sbin/mkfs.ext4 /dev/vde \; sudo mount /dev/vde /${name}-postfailback \; sudo touch /${name}-postfailback/${name}-post-failback\"; done
10) Check for breadcrumbs
We left breadcrumbs at every stage of the test. These breadcrumbs help us to confirm that filesystems residing on the cinder block devices attached to each nova instance were usable at the end of every stage. Rather than clutter an already wordy blog post, we delayed showing the breadcrumbs until this final point.
Confirm that all filesystems exist, that they contain the correct number of files, and that the files are named as expected.
[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3 | sed "s/private1=//g" | while read name ip; do echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889 ssh -i ~/demo.pem cirros@$ip \"hostname \; df \| grep fail \; x; done $ ls -ltr /netapp_iscsi*
Picking a sample of instances to examine:
- Notice that each instance shows four filesystems called "failover".
- Notice that the mapping of the filesystems matches what we expect:
- Prefailover -> vdb
- Postfailover -> vdc
- Prefailback -> vdd
- Postfailback -> vde
- The files inside each filesystem are as such, and each file name starts with the corresponding instance name.
- Prefailover -> 3 files [*pre-failover,*post-recover-from-failover,*down-services]
- Postfailover -> 2 files [*post-failover, *down-services]
- Prefailback -> 1 file [*pre-failback]
- Postfailback -> 1 file [*post-failback]
... ### n3 ### /dev/vde 1032088 34052 945608 3% /n3-postfailback /dev/vdb 1032088 34052 945608 3% /n3-prefailover /dev/vdc 1032088 34052 945608 3% /n3-postfailover /dev/vdd 1032088 34052 945608 3% /n3-prefailback /n3-prefailover/: total 16 -rw------- 1 root root 0 Jun 27 23:14 n3-pre-failover -rw------- 1 root root 0 Jun 28 08:00 n3-post-recover-from-failover -rw------- 1 root root 0 Jun 28 08:50 n3-down-services /n3-postfailover/: total 16 -rw------- 1 root root 0 Jun 28 08:48 n3-post-failover -rw------- 1 root root 0 Jun 28 08:50 n3-down-services /n3-prefailback/: total 16 -rw------- 1 root root 0 Jun 28 09:15 n3-pre-failback /n3-postfailback/: total 16 -rw------- 1 root root 0 Jun 28 10:03 n3-post-failback ... ### n49 ### /dev/vde 1032088 34052 945608 3% /n49-postfailback /dev/vdb 1032088 34052 945608 3% /n49-prefailover /dev/vdc 1032088 34052 945608 3% /n49-postfailover /dev/vdd 1032088 34052 945608 3% /n49-prefailback /n49-prefailover/: total 16 -rw------- 1 root root 0 Jun 27 23:14 n49-pre-failover -rw------- 1 root root 0 Jun 28 08:01 n49-post-recover-from-failover -rw------- 1 root root 0 Jun 28 08:50 n49-down-services /n49-postfailover/: total 16 -rw------- 1 root root 0 Jun 28 08:48 n49-post-failover -rw------- 1 root root 0 Jun 28 08:50 n49-down-services /n49-prefailback/: total 16 -rw------- 1 root root 0 Jun 28 09:15 n49-pre-failback /n49-postfailback/: total 16 -rw------- 1 root root 0 Jun 28 10:03 n49-post-failback ... ### n9 ### /dev/vde 1032088 34052 945608 3% /n9-postfailback /dev/vdb 1032088 34052 945608 3% /n9-prefailover /dev/vdc 1032088 34052 945608 3% /n9-postfailover /dev/vdd 1032088 34052 945608 3% /n9-prefailback /n9-prefailover/: total 16 -rw------- 1 root root 0 Jun 27 23:14 n9-pre-failover -rw------- 1 root root 0 Jun 28 08:01 n9-post-recover-from-failover -rw------- 1 root root 0 Jun 28 08:50 n9-down-services /n9-postfailover/: total 16 -rw------- 1 root root 0 Jun 28 08:48 n9-post-failover -rw------- 1 root root 0 Jun 28 08:50 n9-down-services /n9-prefailback/: total 16 -rw------- 1 root root 0 Jun 28 09:15 n9-pre-failback /n9-postfailback/: total 16 -rw------- 1 root root 0 Jun 28 10:03 n9-post-failback
Summary
Failures and disasters happen, so plan for them and test your plans. We hope that this blog has helped you become more comfortable with this one mechanism for performing disaster recovery. There are many more scenarios to be considered and reviewed. We covered only one in this blog, namely:
- Recovery by cinder failover-host following the loss of an iSCSI backend while the root volumes remain accessible
The procedures in this blog have demonstrated that Cinder's built-in (with support for CDOT as of Newton) disaster recovery mechanism is a viable part of your disaster recovery repertoire. Together we have overcome the main obstacle presented by Cinder Cheesecake—a lack of native failback. Remember, no disaster recovery plan can be called such unless it not only can be tested, but is tested, and tested, and tested again.