Cinder Cheesecake & iSCSI Secondary Devices

by | Aug 23, 2017 | OpenStack

By Chad Morgenstern and Jenny Yang

Overview

In the blog post Cinder Cheesecake: Things to Consider, we wrote that we would be looking at various failure scenarios. Scenarios that build on one another to give a clear view of what happening in the environment.  This blog focuses on scenario 1, namely:

  • Your environment encounters an issue where the storage platform has failed yet the Nova compute node has survived.  In this scenario, the failure does not affect the root volume. In our studies, we chose to use a persistent root volume, though the scenario is no different with an ephemeral root.

Now we show you how to perform failover and failback when using iSCSI Cinder backends. Specifically,  this scenario entails recovery by Cinder failover-host following the loss of an iSCSI backend while the root volumes remain accessible. More blogs will be coming soon covering other disaster recovery scenarios involving Cheesecake, the subject of this sub-series.

If this is the first blog post that you are reading in the series, please STOP and review the blog post Cinder Cheesecake: Things to Consider before proceeding forward.  All technologies have strengths and weaknesses to be considered before implementing; Cheesecake is no exception.

Before diving in, here is a summary of the steps detailed in the main body of the blog.  Consider it a check list of sorts for when you do this yourself.  Please don't be intimidated by the density of this blog. The process is simple:

Configure backends and get replication working from production (site A).

  1. Configure the production and disaster recovery (or site A and site B) backends in the cinder.conf file.
  2. Restart Cinder volume services: systemctl restart openstack-cinder-volume.services.
  3. Time passes, users create Cinder volumes, work happens...
  4. Trigger Cinder failover: cinder failover-host --backend_id <backend_name> <host@backend>.
  5. Stop the impacted Nova instances, detach, and reattach Cinder block devices, start Nova instances.
    Note: See the Python script found here. It is designed to detach and reattach Cinder block devices while preserving attachment order. The script is unsupported, so please feel free to modify it. Use is, of course, at your own risk.
  6. Re-enable the Cinder backend if you plan to add additional Cinder volumes while failed over.

Make the disaster recovery site (site B) the primary site and replicate back to site A.

  1. Shutdown Cinder services: systemctl stop openstack-cinder*.
  2. Capture a database backup.
  3. Modify some rows in volumes and cinder.services tables.
  4. Modify the enable_backends stanza in /etc/cinder.conf to make the disaster recovery backend active instead of the production backend.
  5. Start up all Cinder services: systemctl start openstack-cinder[api|volume|scheduler].service.

Make production (site A) the primary site again and replicate back to site B.

  1. Shut down all Nova instances that are using "to be failed back" Cinder volumes.
  2. Manually perform a final SnapMirror update.
  3. Shut down Cinder services: systemctl stop openstack-cinder*.
  4. Capture a database backup.
  5. Modify some rows in volumes and cinder.services tables.
  6. Modify the enable_backends stanza in /etc/cinder.conf to make the production backend active instead of the disaster recovery backend.
  7. Start up all Cinder services: systemctl start openstack-cinder[api|volume|scheduler].service.
  8. Detach, and reattach, Cinder block devices, start Nova instances.
    Note: See the Python script attached to this blog, it is designed to detach and reattach Cinder block devices while preserving attachment order. The script is unsupported, so please feel free to modify it. Use is, of course, at your own risk.

Getting production ready: preparing for DR

1) Modify the /etc/cinder/cinder.conf to enable replication

Our test environment includes both iSCSI and NFS backends. We setup an NFS backend to store the root device for the Nova instances, as Cheesecake is being evaluated in this blog against the iSCSI backends only.  The NFS backend is known by the name loweropenstack1.  All references to backends from this point forward are to the iSCSI backends only.

Please notice in the below /etc/cinder/cinder.conf that replication is setup in each iSCSI backend stanza.  This is to simplify the replicate -> failover -> replicate -> failback process. Note that replication only occurs for the enabled backends.

[DEFAULT] 
enabled_backends = iprod,loweropenstack1 
 
[iprod] 
volume_backend_name=iprod 
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver 
netapp_server_hostname=10.63.158.xxx
netapp_server_port=80 
netapp_storage_protocol=iscsi 
netapp_storage_family=ontap_cluster 
netapp_login=admin 
netapp_password=******
netapp_vserver=iprod 
replication_device = backend_id:idr 
netapp_replication_aggregate_map =   backend_id:idr,openstack1_01_AggrGroup1_1:openstack2_01_AggrGroup1_1 
 
[idr] 
volume_backend_name=idr 
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver 
netapp_server_hostname=10.63.158.xxx 
netapp_server_port=80 
netapp_storage_protocol=iscsi 
netapp_storage_family=ontap_cluster 
netapp_login=admin 
netapp_password=******
netapp_vserver=idr 
replication_device = backend_id:iprod 
netapp_replication_aggregate_map = backend_id:iprod,openstack2_01_AggrGroup1_1:openstack1_01_AggrGroup1_1 

To keep things simple, we:

  • Only activated one set of backends at a time, either disaster recovery or production.
  • Used the cluster admin user. You may use SVM users.  Please make sure to familiarize yourself with the for your OpenStack release to ensure that the SVM user is defined with the appropriate level of permissions.
  • Established the cluster peer relationship in advance between the two ONTAP clusters: 
    openstack2::> cluster peer create -peer-addrs 10.63.158.xxx -address-family ipv4  
openstack1::> cluster peer create -peer-addrs 10.63.158.xxx -address-family ipv4
  • Established the SVM peer relationships in advance for each set of backends: 
    openstack1::> vserver peer create -vserver iprod -peer-vserver idr -applications snapmirror -peer-cluster openstack2

2) Enable the Cinder backend and check that the backend is initialized

Please take note, it is possible for the volume services to have a status of 'up' even if the backend failed to properly initialize.  While It is important to view the services via cinder service-list --withreplication, please do not forget to inspect the /var/log/cinder/volume.log file for initialization errors as well.

[root@cloud1 ~(keystone_admin)]# cinder service-list --withreplication
+-----------------------+-----------------------+------+---------+-------+-----------------------------+--------------------+
| Binary                | Host                  | Zone | Status  | State | Updated_at                  | Replication Status |
+-----------------------+-----------------------+------+---------+-------+-----------------------------+--------------------+
| cinder-backup         | cloud1                | nova | enabled | up    | 2017-06-27T18:16:39.000000  |                    |
| cinder-scheduler      | cloud1                | nova | enabled | up    | 2017-06-27T18:16:36.000000  |                    |
| cinder-volume         | cloud1@iprod          | nova | enabled | up    | 2017-06-27T18:16:39.000000  | enabled            |
| cinder-volume         | cloud1@loweropenstack1| nova | enabled | up    | 2017-06-27T18:16:39.000000  | disabled           |
+-----------------------+-----------------------+------+---------+-------+-----------------------------+--------------------+

3) Ensure that SnapMirror has initialized and is in a healthy state

openstack1::> snapmirror list-destinations -fields source-path,destination-path,status 
source-path                                                               destination-path                                                        status 
------------------------------------------------------------------------- ----------------------------------------------------------------------- ------
iprod:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1 idr:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1 Idle 
iprod:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 idr:OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2 Idle 

4) Remove resource limitation for the demo project

Modify the quotas for the demo project, set instance and volume resources to unlimited.

[root@cloud1 cinder(keystone_admin)]# openstack quota set --volumes -1 --ram -1 --cores -1 --gigabytes -1 --ports -1 --instances -1 demo

5) Create Cinder extra-specs and map volume backend names thereon

Use the command cinder get-capabilities to see the possible Cinder volume attributes.  This comes in handy when assigning the extra-specs to volume types too.

[root@cloud1 nova(keystone_admin)]# cinder get-capabilities  cloud1@iprod 
+---------------------------------------+------------------------------------------+ 
| Volume stats                          | Value                                    | 
+---------------------------------------+------------------------------------------+ 
| description                           | None                                     | 
| display_name                          | None                                     | 
| driver_version                        | 1.0.0                                    | 
| namespace                             | OS::Storage::Capabilities::cloud1@iprod  | 
| pool_name                             | None                                     | 
| replication_targets                   | []                                       | 
| storage_protocol                      | iSCSI                                    | 
| vendor_name                           | NetApp                                   | 
| visibility                            | None                                     | 
| volume_backend_name                   | iprod                                    | 
+---------------------------------------+------------------------------------------+ 
+-----------------------+---------------------------------------------------------------------------------------------------+ 
| Backend properties    | Value                                                                                             | 
+-----------------------+---------------------------------------------------------------------------------------------------+ 
| compression           | {u'type': u'boolean', u'description': u'Enables compression.', u'title': u'Compression'}          | 
| qos                   | {u'type': u'boolean', u'description': u'Enables QoS.', u'title': u'QoS'}                          | 
| replication           | {u'type': u'boolean', u'description': u'Enables replication.', u'title': u'Replication'}          |    
| thin_provisioning     | {u'type': u'boolean', u'description': u'Sets thin provisioning.', u'title': u'Thin Provisioning'} | 
+-----------------------+---------------------------------------------------------------------------------------------------+ 
 
[root@cloud1 ~(keystone_admin)]# cinder type-create netapp_iscsi
[root@cloud1 ~(keystone_admin)]# cinder type-create netapp_nfs
[root@cloud1 ~(keystone_admin)]# cinder type-key netapp_iscsi set vendor_name=NetApp storage_protocol=iSCSI
[root@cloud1 ~(keystone_admin)]# cinder type-key netapp_nfs set vendor_name=NetApp storage_protocol=nfs

6) Create Cinder volumes on their respective backends:

Create the root Cinder volumes using the netapp_nfs volume type.

[root@cloud1 cinder(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 10 --volume-type netapp_nfs --image cirros-nfs-image --display-name cirros-$i; ((i = i + 1 )) ;done

Create the data Cinder volumes using the netapp_iscsi volume type.

[root@cloud1 cinder(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name prefailover-iscsi-$i; ((i = i + 1 )) ;done

7) Create Nova instances and associate the Cinder volumes

Before creating the Nova instances, we need to create a security group (firewall) and keypair.

[root@cloud1 ~(keystone_demo)]# openstack security group create demo &&  openstack security group rule create --src-ip 0.0.0.0/0 --dst-port 22 demo
[root@cloud1 ~(keystone_demo)]# openstack keypair create demo > ~/demo.pem && chmod 400 ~/demo.pem

Create forty-nine Nova instances associated with the forty-nine CirrOS root volumes.

[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do openstack server create --nic net-id=f968c291-1f17-4fd2-8fb8-36f3ccac3889  --volume cirros-$i --flavor m1.tiny --security-group admin --key-name demo n$i ; (( i = i + 1 )); done

Connect the pre-failover Cinder volumes to Nova instances.

root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do openstack server add volume n$i prefailover-iscsi-$i; (( i = i + 1 )); done

Review the attachments.

[root@cloud1 nova(keystone_demo)]# cinder list | grep -i pre | sort -k6
+---------------------------------------+---------+---------------------+------+--------------+----------+---------------------------------------+
| ID                                    | Status | Name                 | Size | Volume Type  | Bootable | Attached to                           |
+-----------------------------------------------------------------------+------+--------------+----------+---------------------------------------+
| 743586fe-5487-406c-9cfb-420c7235a6c5  | in-use | prefailover-iscsi-10 | 1    | netapp_iscsi | false    | 7cee3434-8335-4807-8600-1500e6db2aa1  |
| cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2  | in-use | prefailover-iscsi-11 | 1    | netapp_iscsi | false    | 9d4116ac-490a-419b-9c41-e66faec5cc56  |
| 901912f1-1548-469d-986a-9a7371dc3943  | in-use | prefailover-iscsi-1  | 1    | netapp_iscsi | false    | f7714ff4-c55e-45f6-bf40-a52f37988bb4  |
...
| f3a85f33-771e-4562-ac8f-07ea0b7409fd  | in-use | prefailover-iscsi-33 | 1    | netapp_iscsi | false    | 0652ebbf-8e48-40e3-8e8b-fed722db2a46  |
| 42b54d00-dc6e-4b8c-9b5a-d974fb60a0e2  | in-use | prefailover-iscsi-34 | 1    | netapp_iscsi | false    | 8bfd1dfb-98a2-4581-ab5e-ce0f7865b6fc  |
| 498b0a66-4b6c-40bc-b148-4fc5880d95d2  | in-use | prefailover-iscsi-35 | 1    | netapp_iscsi | false    | d080b7d7-b9e3-4e79-848d-8c8a9dc8d1fa  |
...
| 43b5e897-7f1b-41d1-b7dd-924b26a1bc76  | in-use | prefailover-iscsi-48 | 1    | netapp_iscsi | false    | d2074077-be7f-4b5a-a332-af08998f7687  |
| c24dadcb-aa39-4c20-b15c-a24a949b0292  | in-use | prefailover-iscsi-49 | 1    | netapp_iscsi | false    | f9f7c317-12df-4783-b35c-ade78486ac6b  |
| 20a00f00-6cda-40ff-b529-460cfa63ecea  | in-use | prefailover-iscsi-5  | 1    | netapp_iscsi | false    | 461b0075-8ad9-4d51-afb4-e5e58311e542  |

8) Leave breadcrumbs in the Cinder volumes for later review

At every stage of this process Cinder volumes will be formatted and mounted, and a file will be written within each Nova instance. In effect, we are leaving breadcrumbs to prove that the process works at every point.

Connect to the Nova instance to prepare the block device for use.

[root@cloud1 nova(keystone_demo)]#   nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"sudo mkdir /${name}-prefailover \; sudo /usr/sbin/mkfs.ext4 /dev/vdb \;  sudo mount /dev/vdb  /${name}-prefailover \; sudo touch /${name}-prefailover/${name}-pre-failover\"; done

Simulate a failure of the production backend

While a true disaster recovery scenario can happen at any moment, planned failovers afford the opportunity to avoid data loss.  In the event of an unplanned disaster recovery scenario, only perform step 3.

  1. (Planned failover only) Power off all instances accessing impacted volumes.
  2. (Planned failover only) With the instances powered off, issue a SnapMirror update from either NetApp's OnCommand System Manager or via the snapmirror update command. In the event of a planned outage, steps 1 and 2 ensure zero data loss.
  3. Issue the cinder failover-host --backend_id <backend_name> <host@backend> command.

To simulate an actual failure scenario, we chose to disable iSCSI protocol access on storage virtual machine backing the production backend. Towards that end, we disabled the production SVM.

1) Disable protocol access on the source storage cluster

openstack1::> vserver stop iprod
openstack1::> iscsi show -vserver iprod
Vserver: iprod, Target Name: iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10, Target Alias: iprod, Administrative Status: down

2) Trigger failover of the production Cinder backend

[root@cloud1 cinder(keystone_admin)]# cinder failover-host --backend_id idr cloud1@iprod

3) Check the status of the Cinder services, ensure that production is failed over

 [root@cloud1 cinder(keystone_admin)]# cinder service-list --withreplication
+--------------------+------------------------+-------+----------+-------+-------------------------------+-------------+-------------------+-----------------------+
| Binary             | Host                   | State | Status   | State | Updated_at                    | Replication | Active Backend ID | Disabled Reason       |
+--------------------+------------------------+-------+----------+-------+-------------------------------+-------------+-------------------+-----------------------+
| cinder-backup      | cloud1                 | nova  | enabled  | up    | 2017-06-27T22:26:35.000000    |             |                   | -                     |
| cinder-scheduler   | cloud1                 | nova  | enabled  | up    | 2017-06-27T22:26:33.000000    |             |                   | -                     |
| cinder-volume      | cloud1@iprod           | nova  | disabled | up    | 2017-06-27T22:26:32.000000    | failed-over | idr               | failed-over           |
| cinder-volume      | cloud1@loweropenstack1 | nova  | enabled  | up    | 2017-06-27T22:26:33.000000    | disabled    | -                 | -                     |
+--------------------+------------------------+-------+----------+-------+-------------------------------+-------------+-------------------+-----------------------+

If I/O is performed against the prefailover-iscsi-* block device attached to any of the Nova instances an I/O error will be triggered. Cinder failover-host has no immediate effect on the Nova instances, as they are still pointing to the production target (iqn and LUN). A look inside the Nova database will clear things up a bit.

The nova.block_device_mapping table shows that the Nova instances have not been instructed to disconnect from the production SVM (iprod) and reconnect to the disaster recovery SVM (idr).

The nova.block_device_mapping table maintains a listing of all block devices attached or formerly attached to each instance. We see for example that the instance f7714ff4-c55e-45f6-bf40-a52f37988bb4 currently sees that a device known to the database as /dev/vdb is presently attached (deleted = 0). The connection_info column shows us that Nova is tracking information about the connection behind /dev/vdb.

The connection_info column of the block_device_mapping table is a json object. We need to focus specifically on the key:value pairs found within data key. The data key is a hash storing key:value pairs describing the target, or storage device backing the vdb.

The target_iqn is the production storage arrays iSCSI Qualified Name.

openstack1::> iscsi show -vserver iprod    ( Target Name: iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10)

The target_portal is the IP address belonging to a logical interface on the production) SVM.

openstack1::> net int show
(network interface show)
                Logical          Status           Network                Current            Current     Is
Vserver         Interface        Admin/Oper       Address/Mask           Node               Port        Home
--------        -----------      -----------      ------------------     -----------        -------     ----
Iprod           iprod_e0e_1      up/up            192.168.0.7/23         openstack1-01      e0e         true

[root@cloud1 cinder(keystone_admin)]# mysql
MariaDB [none]> connect nova
MariaDB [nova]> select instance_uuid,deleted,device_name,connection_info from block_device_mapping where connection_info like '%netapp:%';
| instance_uuid         | f7714ff4-c55e-45f6-bf40-a52f37988bb4
| deleted               | 0
| device_name           | /dev/vdb
| connection_info       |
{
    "driver_volume_type": "iscsi",
    "connector":{},
    "serial": "901912f1-1548-469d-986a-9a7371dc3943",
    "data":
    {
        "device_path": "/dev/disk/by-path/ip-192.168.0.7:3260-iscsi-iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10-lun-0",
        "target_discovered": false,
        "encrypted": false,
        "qos_specs": null,
        "target_iqn": "iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10",
        "target_portal": "192.168.0.7:3260",
        "volume_id": "901912f1-1548-469d-986a-9a7371dc3943",
        "target_lun": 0,
        "access_mode": "rw"
    }
}

4) Detach and reattach the Cinder volumes from Nova instances

Show the current state of the Cinder volumes from Cinder's perspective.

[root@cloud1 nova(keystone_demo)]# cinder list | grep –i pre | sort –k6
+---------------------------------------+--------+----------------------+------+--------------+----------+--------------------------------------+
| ID                                    | Status | Name                 | Size | Volume Type  | Bootable | Attached to                          |
+---------------------------------------+--------+----------------------+------+--------------+----------+--------------------------------------+
| 743586fe-5487-406c-9cfb-420c7235a6c5  | in-use | prefailover-iscsi-10 | 1    | netapp_iscsi | false    | 7cee3434-8335-4807-8600-1500e6db2aa1 |
| cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2  | in-use | prefailover-iscsi-11 | 1    | netapp_iscsi | false    | 9d4116ac-490a-419b-9c41-e66faec5cc56 |
| 901912f1-1548-469d-986a-9a7371dc3943  | in-use | prefailover-iscsi-1  | 1    | netapp_iscsi | false    | f7714ff4-c55e-45f6-bf40-a52f37988bb4 |
...
| f3a85f33-771e-4562-ac8f-07ea0b7409fd  | in-use | prefailover-iscsi-33 | 1    | netapp_iscsi | false    | 0652ebbf-8e48-40e3-8e8b-fed722db2a46 |
| 42b54d00-dc6e-4b8c-9b5a-d974fb60a0e2  | in-use | prefailover-iscsi-34 | 1    | netapp_iscsi | false    | 8bfd1dfb-98a2-4581-ab5e-ce0f7865b6fc |
| 498b0a66-4b6c-40bc-b148-4fc5880d95d2  | in-use | prefailover-iscsi-35 | 1    | netapp_iscsi | false    | d080b7d7-b9e3-4e79-848d-8c8a9dc8d1fa |
...
| 43b5e897-7f1b-41d1-b7dd-924b26a1bc76  | in-use | prefailover-iscsi-48 | 1    | netapp_iscsi | false    | d2074077-be7f-4b5a-a332-af08998f7687 |
| c24dadcb-aa39-4c20-b15c-a24a949b0292  | in-use | prefailover-iscsi-49 | 1    | netapp_iscsi | false    | f9f7c317-12df-4783-b35c-ade78486ac6b |
| 20a00f00-6cda-40ff-b529-460cfa63ecea  | in-use | prefailover-iscsi-5  | 1    | netapp_iscsi | false    | 461b0075-8ad9-4d51-afb4-e5e58311e542 |

Issue the request to detach the Cinder volumes from the all Nova instances.

[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do openstack server remove volume n$i prefailover-iscsi-$i ; (( i = i + 1 )) ; done

Below we see that the detachment request occurred successfully.

[root@cloud1 ~(keystone_demo)]# cinder list | grep pre | sort -k6
+----------------------------------------+-----------+----------------------+------+--------------+----------+
| ID                                     | Status    | Name                 | Size | Volume Type  | Bootable |
+----------------------------------------+-----------+----------------------+------+--------------+----------+
| 743586fe-5487-406c-9cfb-420c7235a6c5   | available | prefailover-iscsi-10 | 1    | netapp_iscsi | false    |
| cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2   | available | prefailover-iscsi-11 | 1    | netapp_iscsi | false    |
...
| 21fd8abf-e63c-4634-a7fa-9fb8344de61d   | available | prefailover-iscsi-3  | 1    | netapp_iscsi | false    |
| a12d5cfb-3aec-42d9-a844-1a50df570077   | available | prefailover-iscsi-32 | 1    | netapp_iscsi | false    |
| f3a85f33-771e-4562-ac8f-07ea0b7409fd   | available | prefailover-iscsi-33 | 1    | netapp_iscsi | false    |
...
| 2a51b57d-2d91-4bce-9bda-7eebbeb4d39a   | available | prefailover-iscsi-47 | 1    | netapp_iscsi | false    |
| 43b5e897-7f1b-41d1-b7dd-924b26a1bc76   | available | prefailover-iscsi-48 | 1    | netapp_iscsi | false    |
| c24dadcb-aa39-4c20-b15c-a24a949b0292   | available | prefailover-iscsi-49 | 1    | netapp_iscsi | false    |

For academic purposes, let's look once more inside the Nova database.

A similar query of the Nova database's block_device_mapping table shows that the detach has no impact on information maintained in the connection info field. Notice however that the deleted field is now a positive integer, signifying that Nova is aware that the device has been detached. To make the output easier to read, we collapsed the connector hash.

[root@cloud1 cinder(keystone_admin)]# mysql
MariaDB [none]> connect nova
MariaDB [nova]> select instance_uuid,deleted,device_name,connection_info from block_device_mapping where connection_info like '%netapp:%';
| instance_uuid         | f7714ff4-c55e-45f6-bf40-a52f37988bb4
| deleted               | 133
| device_name           | /dev/vdb
| connection_info       |
{
    "driver_volume_type": "iscsi",
    "connector":   {},
    "serial": "901912f1-1548-469d-986a-9a7371dc3943",
    "data":
    {
        "device_path": "/dev/disk/by-path/ip-192.168.0.7:3260-iscsi-iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10-lun-0",
        "target_discovered": false,
        "encrypted": false,
        "qos_specs": null,
        "target_iqn": "iqn.1992-08.com.netapp:sn.2d6605375a7e11e7874400a09874b7f2:vs.10",
        "target_portal": "192.168.0.7:3260",
        "volume_id": "901912f1-1548-469d-986a-9a7371dc3943",
        "target_lun": 0,
        "access_mode": "rw"
    }
}

Reattach the Cinder volumes to the Nova instances.

[root@cloud1 ~(keystone_demo)]#   i=1; while [[ $i -le 49 ]]; do openstack server add volume n$i prefailover-iscsi-$i ; (( i = i + 1 )) ; done

[root@cloud1 nova(keystone_demo)]# cinder list | grep -i pre | sort -k6
+---------------------------------------+--------+----------------------+------+--------------+----------+--------------------------------------+
| ID                                    | Status | Name                 | Size | Volume Type  | Bootable | Attached to                          |
+---------------------------------------+--------+----------------------+------+-------------------------+--------------------------------------+
| 743586fe-5487-406c-9cfb-420c7235a6c5  | in-use | prefailover-iscsi-10 | 1    | netapp_iscsi | false    | 7cee3434-8335-4807-8600-1500e6db2aa1
| cad3e524-d424-4cb1-b7dc-bc3b2f50d7d2  | in-use | prefailover-iscsi-11 | 1    | netapp_iscsi | false    | 9d4116ac-490a-419b-9c41-e66faec5cc56
| 901912f1-1548-469d-986a-9a7371dc3943  | in-use | prefailover-iscsi-1  | 1    | netapp_iscsi | false    | f7714ff4-c55e-45f6-bf40-a52f37988bb4
...
| f3a85f33-771e-4562-ac8f-07ea0b7409fd  | in-use | prefailover-iscsi-33 | 1    | netapp_iscsi | false    | 0652ebbf-8e48-40e3-8e8b-fed722db2a46
| 42b54d00-dc6e-4b8c-9b5a-d974fb60a0e2  | in-use | prefailover-iscsi-34 | 1    | netapp_iscsi | false    | 8bfd1dfb-98a2-4581-ab5e-ce0f7865b6fc
| 498b0a66-4b6c-40bc-b148-4fc5880d95d2  | in-use | prefailover-iscsi-35 | 1    | netapp_iscsi | false    | d080b7d7-b9e3-4e79-848d-8c8a9dc8d1fa
...
| 43b5e897-7f1b-41d1-b7dd-924b26a1bc76  | in-use | prefailover-iscsi-48 | 1    | netapp_iscsi | false    | d2074077-be7f-4b5a-a332-af08998f7687
| c24dadcb-aa39-4c20-b15c-a24a949b0292  | in-use | prefailover-iscsi-49 | 1    | netapp_iscsi | false    | f9f7c317-12df-4783-b35c-ade78486ac6b
| 20a00f00-6cda-40ff-b529-460cfa63ecea  | in-use | prefailover-iscsi-5  | 1    | netapp_iscsi | false    | 461b0075-8ad9-4d51-afb4-e5e58311e542
...

Let's look once more inside of the Nova database to see what happens upon reattach.

This time the query returns a second entry backing /dev/vdb for the f7714ff4-c55e-45f6-bf40-a52f37988bb4 instance in addition to the first shown above. The first is the one we just looked at above, and for clarity's sake it is not shown below. Please notice that this second entry's attachment information comes from the disaster recovery SVM's iSCSI IQN and LIF. This shows us that the connection information is only modified on attach.

openstack2::> iscsi show -vserver idr    ( Target Name: iqn.1992-08.com.netapp:sn.3251d9c75a7e11e7841c00a09874b32e:vs.26

MariaDB [none]> connect nova
MariaDB [nova]> select instance_uuid,deleted,device_name,connection_info from block_device_mapping where connection_info like '%netapp:%';

| instance_uuid         | f7714ff4-c55e-45f6-bf40-a52f37988bb4
| deleted               | 0
| device_name           | /dev/vdb
| connection_info       |
{
    "driver_volume_type": "iscsi",
    "connector":   {},
    "serial": "901912f1-1548-469d-986a-9a7371dc3943",
    "data":
    {
        "device_path": "/dev/disk/by-path/ip-192.168.0.11:3260-iscsi-iqn.1992-08.com.netapp:sn.3251d9c75a7e11e7841c00a09874b32e:vs.26-lun-0",,
        "target_discovered": false,
        "encrypted": false,
        "qos_specs": null,
        "target_iqn": "iqn.1992-08.com.netapp:sn.3251d9c75a7e11e7841c00a09874b32e:vs.26",
        "target_portal": "192.168.0.11:3260",
        "volume_id": "901912f1-1548-469d-986a-9a7371dc3943",
        "target_lun": 0,
        "access_mode": "rw"
    }
}

Detach/reattach left instances the in a funky state, proving the necessity of rebooting the instances.

As we attached the block device without rebooting the instances, each instance sees the reattached device as /dev/vdc rather than /dev/vdb which causes problems accessing the filesystems mounted at /dev/vdb. The following demonstrates the point.

[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"lsblk\;df\" ; done

NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  10G  0 disk
`-vda1 253:1    0  10G  0 part /
vdc    253:32   0   1G  0 disk
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vda1                23797     18447      4122  82% /
/dev/vdb               1032088     34052    945608   3% /n1-prefailover
...
...
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  10G  0 disk
`-vda1 253:1    0  10G  0 part /
vdc    253:32   0   1G  0 disk
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vda1                23797     18441      4128  82% /
/dev/vdb               1032088     34052    945608   3% /n49-prefailover

At this point we reboot the instances to clean up the instances device entries.

[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do nova reboot n$i ; (( i = i + 1 )); done

Check access to the volumes had been reset by remounting, then leave breadcrumbs.

[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"sudo mount /dev/vdb  /${name}-prefailover \; sudo touch /${name}-prefailover/${name}-post-recover-from-failover\"; done

So far so good, breadcrumbs are in place.

[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"sudo df \| grep /${name}-prefailover \; sudo ls -l  /${name}-prefailover/\"; done

/dev/vdb               1032088     34052    945608   3% /n1-prefailover
total 16
-rw-------    1 root     root             0 Jun 27 23:14 n1-pre-failover
-rw-------    1 root     root             0 Jun 28 08:00 n1-post-recover-from-failover
...
...
/dev/vdb               1032088     34052    945608   3% /n49-prefailover
total 16
-rw-------    1 root     root             0 Jun 27 23:14 n49-pre-failover
-rw-------    1 root     root             0 Jun 28 08:01 n49-post-recover-from-failover

5) Enable creation of Cinder volumes in DR

At this point we have confirmed that pre-existing Cinder volumes can be accessed from DR. At this point, we have yet to re-enable replication from disaster recovery to any other site, but this is coming up soon. First, let's test out the creation/attachment of additional Cinder volumes in DR.

Enable the disabled backend so that we can create a new Cinder volume.

[root@cloud1 ~(keystone_admin)]# cinder service-enable cloud1@iprod cinder-volume

The status of the iprod backend is enabled.

[root@cloud1 cinder(keystone_admin)]# cinder service-list --withreplication
+------------------+------------------------+------+---------+-------+-----------------------------+--------------------+-------------------+-----------------+
| Binary           | Host                   | Zone | Status  | State |  Updated_at                 | Replication Status | Active Backend ID | Disabled Reason |
+------------------+------------------------+------+---------+-------+-----------------------------+--------------------+-------------------+-----------------+
| cinder-scheduler | cloud1                 | nova | enabled | up    | 2017-06-28T07:10:23.000000  |                    |                   | -               |
| cinder-volume    | cloud1@iprod           | nova | enabled | up    | 2017-06-28T07:10:22.000000  | failed-over        | idr               |failed-over      |
| cinder-volume    | cloud1@loweropenstack1 | nova | enabled | up    | 2017-06-28T07:10:15.000000  | disabled           | -                 | -               |
+------------------+------------------------+------+---------+-------+-----------------------------+--------------------+-------------------+-----------------+

6) Create post-failover Cinder volumes, then attach the volumes to their respective instances

[root@cloud1 ~(keystone_demo)]#  i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name postfailover-iscsi-$i; openstack server add volume n$i postfailover-iscsi-$i;   ((i = i + 1 )) ;done

[root@cloud1 ~(keystone_demo)]# openstack volume list | grep post | sort -k6
| 41c44567-1718-4ad0-9dd1-e47fac3ab250 | postfailover-iscsi-10  | in-use |    1 | Attached to n10 on /dev/vdc  |
| 798c8bb4-43d8-4cb2-aa8c-e0221d9299d5 | postfailover-iscsi-11  | in-use |    1 | Attached to n11 on /dev/vdc  |
| 3a21d3e8-b6ed-4c28-8a00-e23ef4b7dc01 | postfailover-iscsi-12  | in-use |    1 | Attached to n12 on /dev/vdc  |
...
| cb77b0a0-5a2d-4656-a01f-9672ad961507 | postfailover-iscsi-5   | in-use |    1 | Attached to n5 on /dev/vdc   |
| b1ea3c35-1ec6-415a-8f18-0f89990740f8 | postfailover-iscsi-6   | in-use |    1 | Attached to n6 on /dev/vdc   |
| 879cc859-b950-43e0-bcdc-a960031622d0 | postfailover-iscsi-7   | in-use |    1 | Attached to n7 on /dev/vdc   |
| b33dd2bb-da80-4375-a034-01a74c25cf41 | postfailover-iscsi-8   | in-use |    1 | Attached to n8 on /dev/vdc   |
| eba5a235-4a11-4578-8c94-6e6734ac14e8 | postfailover-iscsi-9   | in-use |    1 | Attached to n9 on /dev/vdc   |

7) Create more breadcrumbs, this time on the /dev/vdc device

[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"sudo mkdir /${name}-postfailover \; sudo /usr/sbin/mkfs.ext4 /dev/vdc \;  sudo mount /dev/vdc  /${name}-postfailover \; sudo touch /${name}-postfailover/${name}-post-failover\"; done

Prepare idr for production needs and eventual failback

Though the disaster recovery site is serving in the place of production, it is not yet the source of replication. Additionally, you may eventually wish to fail back to the original production site. The steps in this section accomplish both needs:

Keep the following in mind:

  • Cheesecake only enables replication on active backends. At this point, the idr backend is only active by reference of iprod being failed over. The steps in this section enable replication. Replication will occur between idr and the backend configured in the /etc/cinder/cinder.conf.
  • When the iprod environment is brought back online (eventually), you may wish to fail back to iprod. This section walks through making that possible.

Before triggering the failback

1) Prepare for fail back by enabling idr backend and resyncing backwards from idr to iprod

Modify /etc/cinder/cinder.conf placing idr in the list of enabled_backends instead of iprod.

[root@cloud1 ~(keystone_demo)]# vim /etc/cinder/cinder.conf
#enabled_backends = iprod,loweropenstack1
enabled_backends = idr,loweropenstack1

2) Shutdown all Cinder services, as we are about to modify the Cinder database; this is non-disruptive to client I/O

[root@cloud1 ~(keystone_admin)]#  systemctl stop openstack-cinder*

3) Create breadcrumbs

The OpenStack control plane has no impact on I/O, even with openstack-cinder-*.services disabled, client I/O continues. To prove that, we created breadcrumbs here.

[root@cloud1 ~(keystone_demo)]#  nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"sudo touch /${name}-prefailover/${name}-down-services \; sudo touch /${name}-postfailover/${name}-down-services\"; done

Backup your database:

The instructions to follow will take you through modifications of the tables(s) within the Cinder database. Modification of OpenStack database(s) is inherently dangerous, as such it is imperative to preserve the database against the eventuality that something should go wrong. Numerous database engines can be used in OpenStack, so please consult the users guide for your database flavor regarding database backups/dumps.

4) Modify cinder.volumes and cinder.services tables using the following SQL code

Before making changes to the cinder database, we need to understand what the tables that we are about to change look like. @iprod will be replaced with @idr when done.

[root@cloud1 ~(keystone_admin)]# mysql
MariaDB > connect cinder
MariaDB [cinder]> select id,display_name,host,provider_location from volumes where deleted = 0 and host like '%BLOCK%' order by display_name;
| id                | 55174f08-2d2e-470f-84a0-e5abaa382e63
| display_name      | postfailover-iscsi-1
| host              | cloud1@iprod#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2
| provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-55174f08-2d2e-470f-84a0-e5abaa382e63
...
| id                | 41c44567-1718-4ad0-9dd1-e47fac3ab250
| display_name      | postfailover-iscsi-10
| host              | cloud1@iprod#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2
| provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-41c44567-1718-4ad0-9dd1-e47fac3ab250
...
| Id                | 0c38db73-6d74-42f7-8708-420ba9b4207a
| display_name      | prefailover-iscsi-7
| host              | cloud1@iprod#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1
| provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1/volume-0c38db73-6d74-42f7-8708-420ba9b4207a

Replace iprod with idr in the host column of the volumes table. Post failover, all Cinder volumes on the iprod backend had been accessed via the idr backend. From this point forward, the Cinder volumes will get accessed directly from idr without reference of iprod.

MariaDB [cinder]> update  volumes set  host = REPLACE(host,'iprod', 'idr') where host like '%iprod%';
MariaDB [cinder]> select id,display_name,host,provider_location from volumes where deleted = 0 and host like '%BLOCK%' order by display_name;
| id                | 55174f08-2d2e-470f-84a0-e5abaa382e63
| display_name      | postfailover-iscsi-1
| host              | cloud1@idr#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2
| provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-55174f08-2d2e-470f-84a0-e5abaa382e63

| id                | 41c44567-1718-4ad0-9dd1-e47fac3ab250
| display_name      | postfailover-iscsi-10
| host              | cloud1@idr#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2
| provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK2/volume-41c44567-1718-4ad0-9dd1-e47fac3ab250
...
| id                | 0c38db73-6d74-42f7-8708-420ba9b4207a
| display_name      | prefailover-iscsi-7
| host              | cloud1@idr#OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1
| provider_location | idr:/vol/OPENSTACK_RDO_openstack1_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_BLOCK1/volume-0c38db73-6d74-42f7-8708-420ba9b4207a

The next set of commands remove the failed-over flags from all volumes and services, and the active-backend field is made NULL. Once this is done, two benefits are derived:

  • Cheesecake will be able to resync from idr to iprod. Otherwise, though Cheesecake will setup the SnapMirror relationship, it will not re-establish the transfers between the two sites.
  • iprod, when the time comes, will be able to actively serve data once again.

Before making the change, pay attention to the active_backend_id, disabled_reason and replication status fields.

MariaDB [cinder]> select id,host,topic,disabled,disabled_reason,replication_status,active_backend_id from services;
+-----+-------------------------+-------------------+----------+-----------------+--------------------+-------------------+
| id  | host                    | topic             | disabled | disabled_reason | replication_status | active_backend_id |
+-----+-------------------------+-------------------+----------+-----------------+--------------------+-------------------+
|  6  | cloud1                  | cinder-scheduler  |        0 | NULL            | not-capable        | NULL              |
|  8  | cloud1@iprod            | cinder-volume     |        0 | failed-over     | failed-over        | idr               |
| 11  | cloud1@loweropenstack1  | cinder-volume     |        0 | NULL            | disabled           | NULL              |
+-----+-------------------------+-------------------+----------+-----------------+--------------------+-------------------+

MariaDB  [cinder]> update  services set disabled_reason= NULL where disabled_reason = 'failed-over';

NOTE: The above command will have no effect, as you already enabled the service. However, the command is included here just in case you had not done so previously.

MariaDB [cinder]> update  services set replication_status = 'disabled' where replication_status = 'failed-over';
MariaDB [cinder]> update  services set active_backend_id = NULL where active_backend_id = 'idr';

MariaDB [cinder]>  select id,host,topic,disabled,disabled_reason,replication_status,active_backend_id from services;
+-----+-------------------------+-------------------+-----------+-----------------+---------------------+-------------------+
| id  | host                    | topic             | disabled  | disabled_reason | replication_status  | active_backend_id |
+-----+-------------------------+-------------------+-----------+-----------------+---------------------+-------------------+
|  6  | cloud1                  | cinder-scheduler  |        0  | NULL            | not-capable         | NULL              |
|  8  | cloud1@iprod            | cinder-volume     |        0  | NULL            | NULL                | NULL              |
| 11  | cloud1@loweropenstack1  | cinder-volume     |        0  | NULL            | disabled            | NULL              |
+-----+-------------------------+-------------------+-----------+-----------------+---------------------+-------------------+

Update the replication_status fields for affected volumes.

MariaDB [cinder]> update  volumes set replication_status = 'disabled' where replication_status = 'failed-over';

5) Start up the openstack-cinder services

[root@cloud1 ~(keystone_demo)]# systemctl start openstack-cinder-{api,volume,scheduler}.service

Check the status of the Cinder services as well as the replication status. The idr host no longer shows as failed-over, and idr is shown to be active. Just as cool, replication is now occurring between site idr and iprod.

[root@cloud1 cinder(keystone_admin)]# cinder service-list --withreplication
+------------------+------------------------+------+---------+-------+-----------------------------+--------------------+
| Binary           | Host                   | Zone | Status  | State | Updated_at                  | Replication Status |
+------------------+------------------------+------+---------+-------+-----------------------------+--------------------+
| cinder-scheduler | cloud1                 | nova | enabled | up    | 2017-06-28T08:01:50.000000  |                    |
| cinder-volume    | cloud1@idr             | nova | enabled | up    | 2017-06-28T08:01:53.000000  | enabled            |
| cinder-volume    | cloud1@iprod           | nova | enabled | down  | 2017-06-28T07:41:42.000000  | disabled           |
| cinder-volume    | cloud1@loweropenstack1 | nova | enabled | up    | 2017-06-28T08:01:54.000000  | disabled           |
+------------------+------------------------+------+---------+-------+-----------------------------+--------------------+

6) Create and attach one more set of Cinder volumes at the disaster recovery site

[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name prefailback-iscsi-$i; openstack server add volume n$i prefailback-iscsi-$i;   ((i = i + 1 )) ;done

7) Once more, we leave breadcrumbs

[root@cloud1 ~(keystone_demo)]#  nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"sudo mkdir /${name}-prefailback \; sudo /usr/sbin/mkfs.ext4 /dev/vdd \;  sudo mount /dev/vdd /${name}-prefailback \; sudo touch /${name}-prefailback/${name}-pre-failback \"; done

Failback from idr to iprod

Failing back from the disaster recovery site to the production site is a planned event.  It is important that you shutdown your Nova instances, manually initiate a final SnapMirror transfer, and then fail back.  This section details the process of failing back from the DR backend to the production backend.

Before triggering the failback

1) Enable the SVM on site 1

Recovering the production environment may be more complicated than enabling the production SVM.

openstack1::> vserver start iprod

2) Shut down the instances prior to failing back

Shutting down the instances prevents further I/O and ensures no data loss.

[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do nova stop n$i ; (( i = i + 1 )); done

Confirm that the instances have shut down.

[root@cloud1 ~(keystone_demo)]# nova list
+----------------------------------------+------+----------+------------+-------------+
| ID                                     | Name | Status   | Task State | Power State |
+----------------------------------------+------+----------+------------+-------------+
| f7714ff4-c55e-45f6-bf40-a52f37988bb4   | n1   | SHUTOFF  | -          | Shutdown    |
...
| f9f7c317-12df-4783-b35c-ade78486ac6b   | n49  | SHUTOFF  | -          | Shutdown    |
...
| b8f57eab-b150-41f6-9a3f-bd5803bfc784   | n9   | SHUTOFF  | -          | Shutdown    |
+----------------------------------------+------+----------+------------+-------------+

4) Initiate a manual SnapMirror update

SnapMirror transfers have been occurring between idr and iprod, perhaps for some time. After shutting down the Nova instances, trigger a final SnapMirror transfer to pick up the last set of data.

openstack1::> snapmirror update -destination-path  *

5) Trigger the failback

Notice that idr is now listed as failed over to iprod.

[root@cloud1 ~(keystone_admin)]# cinder failover-host --backend_id iprod cloud1@idr
[root@cloud1 ~(keystone_admin)]# cinder service-list --withreplication
+------------------+------------------------+------+----------+-------+----------------------------+--------------------+-------------------+-----------------+
| Binary           | Host               | Zone | Status   | State | Updated_at             | Replication Status | Active Backend ID | Disabled Reason |
+------------------+------------------------+------+----------+-------+----------------------------+--------------------+-------------------+-----------------+
| cinder-scheduler | cloud1                 | nova | enabled  | up    | 2017-06-28T08:24:20.000000 |                    |                   | -               |
| cinder-volume    | cloud1@idr             | nova | disabled | up    | 2017-06-28T08:24:23.000000 | failed-over        | iprod             | failed-over     |
| cinder-volume    | cloud1@iprod           | nova | enabled  | down  | 2017-06-28T07:41:42.000000 | disabled           | -                 | -               |
| cinder-volume    | cloud1@loweropenstack1 | nova | enabled  | up    | 2017-06-28T08:24:25.000000 | disabled           | -                 | -               |
+------------------+------------------------+------+----------+-------+----------------------------+--------------------+-------------------+-----------------+

Make the production site production again

1) Modify /etc/cinder/cinder.conf by placing iprod in the list of enabled_backends instead of idr

[root@cloud1 ~(keystone_demo)]# vim /etc/cinder/cinder.conf
enabled_backends = iprod,loweropenstack1
#enabled_backends = idr,loweropenstack1

2) Shut down all Cinder services, as we are about to modify the Cinder database

[root@cloud1 ~(keystone_admin)]# systemctl stop openstack-cinder*

3) Modify the Cinder database, replacing idr with iprod

NOTE: Get a MySQL database backup before performing this step.

[root@cloud1 ~(keystone_admin)]# mysql
MariaDB [none] > connect cinder
MariaDB [cinder]>  update  volumes set  host = REPLACE(host,'idr', 'iprod') where host like '%idr%';
MariaDB [cinder]>  update  volumes set replication_status = 'disabled' where replication_status = 'failed-over';
MariaDB [cinder]>  update  services set replication_status = 'disabled' where replication_status = 'failed-over';
MariaDB [cinder]>  update  services set disabled_reason= NULL where disabled_reason = 'failed-over';
MariaDB [cinder]>  update  services set active_backend_id = NULL where active_backend_id = 'iprod';

4) Restart Cinder services

[root@scsor0013409001 ~(keystone_demo)]# systemctl start openstack-cinder-{api,volume,scheduler}.service

5) Re-enable the idr backend so it's ready to go in case you want to fail over again in the future

[root@cloud1 ~(keystone_admin)]# cinder service-enable cloud1@idr cinder-volume

6) Using automation, detach, and reattach the Cinder volumes to the Nova instances

Reattaching (detaching and reattaching) multiple block devices from a Nova instance is more complicated than reattaching a single block device because of attachment order issues.  If you want to maintain the attachment order, the devices must all be detached from an instance before any are reattached.  Also, the devices must be reattached in the same order each time.

To simplify this process, I have written and attached to this blog a short Python script called recinder.py. The script is currently only intended for use with Nova instances that are booted from persistent volumes.  In its current form, it will only detach and reattach non-root disks.

How it works:

recinder.py makes an ordered list of all:

  • Cinder volumes attached to Nova instances
  • The Nova instance names
  • The current device names associated with the Cinder volumes according to Cinder. For example /dev/vdb, /dev/vdc, etc...

Using this ordered list, recinder.py detaches all non-root devices (anything other than /dev/vda) from all instances and then reattaches the Cinder volumes in the original order of device_name, i.e. starting at /dev/vdb, then /dev/vdc, etc... In this version, the root device is not considered as root devices cannot be detached. Future versions may have the ability to destroy the Nova instances associated with persistent root volumes and then redeploy and reattach all associated Cinder volumes.

[root@cloud1 ~(keystone_demo)]# recinder.py
openstack server remove volume n12 c2df6ec2-9833-4ab8-bdab-6dba9a255467
openstack server remove volume n12 3a21d3e8-b6ed-4c28-8a00-e23ef4b7dc01
openstack server remove volume n12 a49b524f-2abe-472b-87c5-dd0945dcd813
openstack server remove volume n12 a2b16bc2-af58-46e4-96fc-61a6c04d572b
...
openstack server add volume n49 9e278a46-e49c-4148-84a9-c4ad715c6ce7
openstack server add volume n49 e567c262-9772-44e5-b7d1-6bc576a6df62
openstack server add volume n49 7d57be2e-45a3-454a-9066-aa568b61a634
openstack server add volume n49 eaeac0b1-411d-4d25-b875-547ed5717263

7) Create and attach a final set of Cinder volumes

[root@cloud1 ~(keystone_demo)]# i=1; while [[ $i -le 49 ]]; do cinder create 1 --volume-type netapp_iscsi --display-name postfailback-iscsi-$i ; openstack server add volume n$i postfailback-iscsi-$i ;   ((i = i + 1 )) ;done

View all post-failback Cinder volumes. Let's focus on the total attachment list for a couple of Nova instances.

[root@cloud1 ~(keystone_demo)]# openstack volume list | egrep "Attached to n1 |Attached to n49|Attached to n22" | sort -k12
| 931554bd-82c3-4f11-9f73-5aa125fbc96f  | cirros-1              | in-use |   10 | Attached to n1 on /dev/vda   |
| 901912f1-1548-469d-986a-9a7371dc3943  | prefailover-iscsi-1   | in-use |    1 | Attached to n1 on /dev/vdb   |
| 55174f08-2d2e-470f-84a0-e5abaa382e63  | postfailover-iscsi-1  | in-use |    1 | Attached to n1 on /dev/vdc   |
| 83cb90b0-767b-4464-aa35-bafd25f53386  | prefailback-iscsi-1   | in-use |    1 | Attached to n1 on /dev/vdd   |
| a9f91719-e93b-4013-a6e9-8260d86948cd  | postfailback-iscsi-1  | in-use |    1 | Attached to n1 on /dev/vde   |
...
| 55a1cf30-bd5d-4d02-bb9f-a46f3a3c5d76  | cirros-22             | in-use |   10 | Attached to n22 on /dev/vda  |
| 3cad8d03-015b-472a-9e99-43574b8fb684  | prefailover-iscsi-22  | in-use |    1 | Attached to n22 on /dev/vdb  |
| 69909909-d03d-4561-af12-bfc3617ea796  | postfailover-iscsi-22 | in-use |    1 | Attached to n22 on /dev/vdc  |
| e8603cb2-199d-4491-913c-8511c6b0064a  | prefailback-iscsi-22  | in-use |    1 | Attached to n22 on /dev/vdd  |
| 2dbf505e-2a58-4a6f-a749-be49cddc9cb4  | postfailback-iscsi-22 | in-use |    1 | Attached to n22 on /dev/vde  |
...
| 348bccc3-7b81-4871-93ad-12d13f6a3f45  | cirros-49             | in-use |   10 | Attached to n49 on /dev/vda  |
| c24dadcb-aa39-4c20-b15c-a24a949b0292  | prefailover-iscsi-49  | in-use |    1 | Attached to n49 on /dev/vdb  |
| 5a78efb4-ba66-451f-95b1-c91c3ea69a65  | postfailover-iscsi-49 | in-use |    1 | Attached to n49 on /dev/vdc  |
| 4039e49a-1c8e-4135-a8d2-f1eae8a935e1  | prefailback-iscsi-49  | in-use |    1 | Attached to n49 on /dev/vdd  |
| 1abfe06b-424c-4c01-83d5-652ce19f2ef4  | postfailback-iscsi-49 | in-use |    1 | Attached to n49 on /dev/vde  |

8) Start up the Nova instances

[root@cloud1 ~(keystone_demo)]#  i=1; while [[ $i -le 49 ]]; do  nova start n$i ;   ((i = i + 1 )) ;done
[root@cloud1 ~(keystone_demo)]# nova list
+---------------------------------------+------+--------+------------+-------------+
| ID                                    | Name | Status | Task State | Power State |
+---------------------------------------+------+--------+------------+-------------+
| f7714ff4-c55e-45f6-bf40-a52f37988bb4  | n1   | ACTIVE | -          | Running     |
| 7cee3434-8335-4807-8600-1500e6db2aa1  | n10  | ACTIVE | -          | Running     |
...
| 8593bda5-39b4-44c4-a461-ba0a17b40dfc  | n47  | ACTIVE | -          | Running     |
| d2074077-be7f-4b5a-a332-af08998f7687  | n48  | ACTIVE | -          | Running     |
| f9f7c317-12df-4783-b35c-ade78486ac6b  | n49  | ACTIVE | -          | Running     |
| 461b0075-8ad9-4d51-afb4-e5e58311e542  | n5   | ACTIVE | -          | Running     |
| 2b890f68-f0b8-4239-87da-972e4cfa984b  | n6   | ACTIVE | -          | Running     |
| d11e09c6-9974-4c67-8cc5-408546ddfbee  | n7   | ACTIVE | -          | Running     |
| d0f9bf29-c71d-4212-8ea0-e965155c5887  | n8   | ACTIVE | -          | Running     |
| b8f57eab-b150-41f6-9a3f-bd5803bfc784  | n9   | ACTIVE | -          | Running     |
+---------------------------------------+------+--------+------------+-------------+

9) Create more breadcrumbs

[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"sudo mount /dev/vdb /${name}-prefailover \; sudo mount /dev/vdc /${name}-postfailover \; sudo mount /dev/vdd /${name}-prefailback \; sudo mkdir /${name}-postfailback \; sudo /usr/sbin/mkfs.ext4 /dev/vde \;  sudo mount /dev/vde /${name}-postfailback \; sudo touch /${name}-postfailback/${name}-post-failback\"; done

10) Check for breadcrumbs

We left breadcrumbs at every stage of the test. These breadcrumbs help us to confirm that filesystems residing on the cinder block devices attached to each nova instance were usable at the end of every stage. Rather than clutter an already wordy blog post, we delayed showing the breadcrumbs until this final point.

Confirm that all filesystems exist, that they contain the correct number of files, and that the files are named as expected.

[root@cloud1 ~(keystone_demo)]# nova list | awk '{print $4,$12}' | grep 10.3  | sed "s/private1=//g" | while read name ip; do   echo ip netns exec qdhcp-f968c291-1f17-4fd2-8fb8-36f3ccac3889  ssh -i ~/demo.pem cirros@$ip  \"hostname \; df \| grep fail \; x; done $ ls -ltr /netapp_iscsi*

Picking a sample of instances to examine:

  • Notice that each instance shows four filesystems called "failover".
  • Notice that the mapping of the filesystems matches what we expect:
    • Prefailover -> vdb
    • Postfailover -> vdc
    • Prefailback -> vdd
    • Postfailback -> vde
  • The files inside each filesystem are as such, and each file name starts with the corresponding instance name.
    • Prefailover -> 3 files [*pre-failover,*post-recover-from-failover,*down-services]
    • Postfailover -> 2 files [*post-failover, *down-services]
    • Prefailback -> 1 file [*pre-failback]
    • Postfailback -> 1 file [*post-failback]
...
### n3 ###
/dev/vde               1032088     34052    945608   3% /n3-postfailback
/dev/vdb               1032088     34052    945608   3% /n3-prefailover
/dev/vdc               1032088     34052    945608   3% /n3-postfailover
/dev/vdd               1032088     34052    945608   3% /n3-prefailback
/n3-prefailover/:
total 16
-rw-------    1 root     root             0 Jun 27 23:14 n3-pre-failover
-rw-------    1 root     root             0 Jun 28 08:00 n3-post-recover-from-failover
-rw-------    1 root     root             0 Jun 28 08:50 n3-down-services

/n3-postfailover/:
total 16
-rw-------    1 root     root             0 Jun 28 08:48 n3-post-failover
-rw-------    1 root     root             0 Jun 28 08:50 n3-down-services

/n3-prefailback/:
total 16
-rw-------    1 root     root             0 Jun 28 09:15 n3-pre-failback

/n3-postfailback/:
total 16
-rw-------    1 root     root             0 Jun 28 10:03 n3-post-failback




...
### n49 ###
/dev/vde               1032088     34052    945608   3% /n49-postfailback
/dev/vdb               1032088     34052    945608   3% /n49-prefailover
/dev/vdc               1032088     34052    945608   3% /n49-postfailover
/dev/vdd               1032088     34052    945608   3% /n49-prefailback
/n49-prefailover/:
total 16
-rw-------    1 root     root             0 Jun 27 23:14 n49-pre-failover
-rw-------    1 root     root             0 Jun 28 08:01 n49-post-recover-from-failover
-rw-------    1 root     root             0 Jun 28 08:50 n49-down-services

/n49-postfailover/:
total 16
-rw-------    1 root     root             0 Jun 28 08:48 n49-post-failover
-rw-------    1 root     root             0 Jun 28 08:50 n49-down-services

/n49-prefailback/:
total 16
-rw-------    1 root     root             0 Jun 28 09:15 n49-pre-failback

/n49-postfailback/:
total 16
-rw-------    1 root     root             0 Jun 28 10:03 n49-post-failback

...
### n9 ###
/dev/vde               1032088     34052    945608   3% /n9-postfailback
/dev/vdb               1032088     34052    945608   3% /n9-prefailover
/dev/vdc               1032088     34052    945608   3% /n9-postfailover
/dev/vdd               1032088     34052    945608   3% /n9-prefailback
/n9-prefailover/:
total 16
-rw-------    1 root     root             0 Jun 27 23:14 n9-pre-failover
-rw-------    1 root     root             0 Jun 28 08:01 n9-post-recover-from-failover
-rw-------    1 root     root             0 Jun 28 08:50 n9-down-services

/n9-postfailover/:
total 16
-rw-------    1 root     root             0 Jun 28 08:48 n9-post-failover
-rw-------    1 root     root             0 Jun 28 08:50 n9-down-services

/n9-prefailback/:
total 16
-rw-------    1 root     root             0 Jun 28 09:15 n9-pre-failback

/n9-postfailback/:
total 16
-rw-------    1 root     root             0 Jun 28 10:03 n9-post-failback

Summary

Failures and disasters happen, so plan for them and test your plans. We hope that this blog has helped you become more comfortable with this one mechanism for performing disaster recovery.  There are many more scenarios to be considered and reviewed. We covered only one in this blog, namely:

  • Recovery by cinder failover-host following the loss of an iSCSI backend while the root volumes remain accessible

The procedures in this blog have demonstrated that Cinder's built-in (with support for CDOT as of Newton) disaster recovery mechanism is a viable part of your disaster recovery repertoire. Together we have overcome the main obstacle presented by Cinder Cheesecake—a lack of native failback. Remember, no disaster recovery plan can be called such unless it not only can be tested, but is tested, and tested, and tested again.

About Chad Morgenstern

Chad is a 10 -year veteran at NetApp having held positions In both escalations, reference architecture teams, and most recently as part of the workload performance team where he contributed to significant understandings of AFF performance for things such as VDI environments, working on the SPC-1 benchmark, and more. In addition, Chad spent time building automated tools and frameworks for performance testing with a few of them based on opensource technologies such as cloud and containers.

Chad is happily married and is the father of four daughters, and soon to be owner of two ferrets.

View all posts by Chad Morgenstern

Pin It on Pinterest