Welcome, Guest

TOPIC:

VDI Not available on second server 10 years 7 months ago #95

christ neeskens
Topic Author
Offline
Posts: 18

Pulse Supply wrote: Was the host rebooted or was a crash simulated before this happened? We have seen this happen many times in various crash scenarios with XenServer (even without HA-lizard).

We have noticed that a XenServer host booting after a simulated crash may not attach the SR at all on a host. This appears to be related to the order in which services start after a crash. So, we introduced auto-plug-pbd in iscsi-ha to deal with this automatically. If an SR is disconnected we connect it. this is likely not what caused your issue, but worth mentioning.

The second scenario we see is that an SR is attached and "appears" ok, but the logical volume for the SR is not read by LVM. This has appeared for us occasionally after simulated crashes (with and without HA-Lizard). SO, we introduced replug-pbd in the latest version of iscsi-ha. You can run this tool by hand should this happen again. It may solve the problem. This is active in the latest version of iscsi-ha (1.2.16). You may be running a previous version. If so, download the latest tarball, extract files and move into the /iscsi-ha/scripts/ folder. In there you will see a script named "replug_pbd". The script simply checks for a logical volume for each iscsi SR. If the logical volume is not found it will unplug and plug the SR.

Lastly - you may want to check to ensure there are no stale cache files in LVM. This would cause some weird behavior. The latest version of the how-to has instructions for doing so. halizard.com/images/pdf/iscsi-ha_2-node_...to_1.2.14a_final.pdf

Because it happend again i thought I better create a new topic to keep things a bit more sorted.

Now that DRBD is working again like it should, i have an issue again with the second server.
I can't start any VM on it, or migrate anything to it.
The last time i "simply removed the server out of the pool, and added it back again. Sometimes I had to do this several times, but in the end the server was working again.

This time i gave replug_pdb a try, but that didn't solve the issue, so hence this topic.

The console output of replug_pdb

[root@xenserver2 scripts]# ./replug_pbd
  No volume groups found
replug_pbd Version 1.1: Replugs a PBD where a VG is not scanned. Attempts to resolve
XenServer errors: SR_BACKEND_FAILURE_40 or SR_BACKEND_FAILURE_46 or VDI is not available
Currently works with iSCSI SRs.


Hostname Found:          xenserver2
Host UUID Found:         130c6a0b-14d5-4144-a382-09c0f1b9dd96
iSCSI SR(s) Found:       2e56fc62-c9d7-28f8-c1aa-da6734138f16
Volume Groups Found:

Scanning for Volume Group -> iscsi-sr: 2e56fc62-c9d7-28f8-c1aa-da6734138f16

If you want any more log files, or want me to try anything just ask, and I'll do what I can to assist.
Just keep in mind it's already in a production environment.

Please Log in or Create an account to join the conversation.

VDI Not available on second server 10 years 7 months ago #97

Pulse Supply
Offline
Posts: 157

In looking at the output of replug_pbd - it looks like the host did not properly scan the VG for the iscsi SR (hence the common "VDI not available error" in xenserver)

You can verify this with "vgscan". When all is working normally, there whould be a VG with the same UUID as the iscsi SR. This issue seems to be somewhat common with xenserver when hosts are rebooted or recover from a crash (irrespective of HA-Lizard).

You can try the steps manually.

- Find the PBD of the iscsi SR (maybe "xe pbd-list sr-uuid=<SR UUID>" )

- once you have the UUID of the PBD

- xe pbd-unplug uuid=<PBD UUID>

- xe pbd-plug uuid=<PBD UUID>

plugging the PBD again usually results in xenserver properly scanning for the VG and things return to normal.

The following user(s) said Thank You: christ neeskens

Please Log in or Create an account to join the conversation.

VDI Not available on second server 10 years 7 months ago #99

christ neeskens
Topic Author
Offline
Posts: 18

Thanks, this did the trick!

Vgscan resulted in nothing found.
Step by step what I did to resolve this:

Find the uuid of the host
xe host-list
Use the corresponding uuid to find the pdb uuid

XE pbd-list host-uuid=host-uuid

This results in a short list. In here is one device-config showing the iSCSI detais.
Use this uuid for the plug and unplug commands

xe pbd-unplug uuid=pbd-uuid
Now it's gone and xencenter shows this as well.

Now replug it
xe pbd-plug uuid=pbd-uuid

Verify it is working again using vgscan.
This now shows a VG_XenStorage entry

Please Log in or Create an account to join the conversation.

Last edit: by Salvatore Costantino. Reason: mark as resolved