Nov 4 15:45:58 xcp-ng-secondary ha-lizard: 17947 get_pool_ip_list: returned 192.168.10.1 Nov 4 15:45:58 xcp-ng-secondary ha-lizard: 17947 get_pool_ip_list: returned 192.168.10.1 192.168.10.2 Nov 4 15:45:58 xcp-ng-secondary ha-lizard: 17947 write_pool_state: Host IP List = 192.168.10.1 192.168.10.2 Nov 4 15:45:58 xcp-ng-secondary ha-lizard: 17947 write_status_report: Writing status report Nov 4 15:46:07 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 LOG_TERMINAL = [false] Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 This iteration is count 23 Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 Checking if this host is a Pool Master or Slave Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 This host's pool status = slave:192.168.10.1 Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 update_global_conf_params: Successfully updated global poo l configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONIT OR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/h a-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_ IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_RE BOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@l ocalhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT _LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE =10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SM TP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'# 012XE_TIMEOUT=10 Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 master_ip: Pool Master IP Address = 192.168.10.1 Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 Validating master is still a master Nov 4 15:46:10 xcp-ng-secondary ha-lizard: 18819 [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/ host_is_slave 192.168.10.1 ] Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 This slave - xcp-ng-secondary: f543502f-1445-429e-8220-b36 0cd2a6946 selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1 Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 check_xapi: Pool Host 192.168.10.1 xapi status = 7 Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 validate_this_host_vm_states: MY HOST UUID = [ f543502f-14 45-429e-8220-b360cd2a6946 ] - validated Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 validate_this_host_vm_states: VM [ 2909b2a3-dc9a-42fc-b55f -525edde498ac ] validaton start Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 validate_this_host_vm_states: VM [ 2909b2a3-dc9a-42fc-b55f -525edde498ac ] reported as running on host [ f543502f-1445-429e-8220-b360cd2a6946 ] Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 validate_this_host_vm_states: VM [ 2909b2a3-dc9a-42fc-b55f -525edde498ac ] Validation passed Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 Pool Master is OK - calling function check_ha_enabled - up dating local status Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 check_ha_enabled: Checking if ha-lizard is enabled for poo l: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 check_ha_enabled: ha-lizard is enabled Nov 4 15:46:11 xcp-ng-secondary ha-lizard: 18819 check_ha_enabled: checking whether maintenance mode enable d Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 Checking state file for status if ha-lizard is enabled Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: ch ecking if ha-lizard is enabled Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 ha-lizard is enabled - continuing Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 check_xs_ha: Checking XenServer HA status Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 Calling Function write_pool_state - updating local state f iles Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: MASTER UUID found: b6dcd4f7-860c-4dfc-8e 7c-22c00ef08b33 Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: MASTER UUID: b6dcd4f7-860c-4dfc-8e7c-22c 00ef08b33 written to local state storage Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: Calling function get_vms_on_host for UUI D: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 get_vms_on_host: Returned 312abe84-9704-a079-3dc2-02cb08a1 bf1f Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: Writing VM array to local state file hos t.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: Calling function get_vms_on_host for UUI D: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 get_vms_on_host: No VMs found on host: f543502f-1445-429e- 8220-b360cd2a6946 Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: Writing VM array to local state file hos t.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 pool autopromote_uuid = [f543502f-1445-429e-8220-b360cd2a6 946] Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: Pool autopromote_uuid=f543502f-1445-429e -8220-b360cd2a6946 Nov 4 15:46:12 xcp-ng-secondary ha-lizard: 18819 write_pool_state: autopromote_uuid unchanged - not updatin g Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 check_ha_enabled: Checking if ha-lizard is enabled for poo l: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 check_ha_enabled: ha-lizard is enabled Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 check_ha_enabled: checking whether maintenance mode enable d Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 write_pool_state: Pool contains 2 hosts. Writing to /etc/h a-lizard/state/pool_num_hosts Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 get_pool_host_list: enabled flag set - returning only host s with enabled=true Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00 ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 get_pool_ip_list: returned 192.168.10.1 Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 get_pool_ip_list: returned 192.168.10.1 192.168.10.2 Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 write_pool_state: Host IP List = 192.168.10.1 192.168.10.2 Nov 4 15:46:13 xcp-ng-secondary ha-lizard: 18819 write_status_report: Writing status report Nov 4 15:46:17 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 19512 LOG_TERMINAL = [false] Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 19512 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 19512 This iteration is count 24 Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 19512 Checking if this host is a Pool Master or Slave Nov 4 15:46:25 xcp-ng-secondary ha-lizard: 19512 This host's pool status = slave:192.168.10.1 Nov 4 15:46:27 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 update_global_conf_params: Failed to update global pool co nfiguration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg - type f -mmin +60 -delete] Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 check_email_enabled: Email enabled for update_global_conf_ params Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = update_g lobal_conf_params: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 master_ip: Pool Master IP Address = 192.168.10.1 Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 Validating master is still a master Nov 4 15:46:31 xcp-ng-secondary ha-lizard: 19512 [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/ host_is_slave 192.168.10.1 ] Nov 4 15:46:32 xcp-ng-secondary ha-lizard: 19512 This slave - xcp-ng-secondary: f543502f-1445-429e-8220-b36 0cd2a6946 selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1 Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 check_xapi: Pool Host 192.168.10.1 failed to respond to IC MP ping Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg - type f -mmin +60 -delete] Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 check_email_enabled: Email enabled for check_xapi Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = check_xa pi: Pool Host on Server: 192.168.10.1 not responding to ICMP ping - manual intervention may be required Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Pool Master NOT OK - Checking if ha-lizard is enabled in l atest state file Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Checking if ha-lizard is enabled Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: ch ecking if ha-lizard is enabled Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 ha-lizard is enabled - continuing Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Pool Master Monitor = Failed Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg - type f -mmin +60 -delete] Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = Server x cp-ng-secondary: Failed to contact pool master - manual intervention may be required Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Retry Count set to 2. Retrying 2 times in 10 second interv als.. Nov 4 15:46:33 xcp-ng-secondary ha-lizard: 19512 Attempt 0: Checking Pool Master Status Nov 4 15:46:37 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:46:40 xcp-ng-secondary ha-lizard: 31214 ha-lizard already running: Attempt 1 on PIDS: 19512 Nov 4 15:46:44 xcp-ng-secondary ha-lizard: 19512 check_xapi: Pool Host 192.168.10.1 failed to respond to IC MP ping Nov 4 15:46:45 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:45 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg - type f -mmin +60 -delete] Nov 4 15:46:45 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:46:45 xcp-ng-secondary ha-lizard: 19512 check_email_enabled: Email enabled for check_xapi Nov 4 15:46:45 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = check_xa pi: Pool Host on Server: 192.168.10.1 not responding to ICMP ping - manual intervention may be required Nov 4 15:46:45 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:46:45 xcp-ng-secondary ha-lizard: 19512 Attempt 1: Checking Pool Master Status Nov 4 15:46:47 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:46:50 xcp-ng-secondary ha-lizard: 31214 ha-lizard already running: Attempt 2 on PIDS: 19512 Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 check_xapi: Pool Host 192.168.10.1 failed to respond to ICMP ping Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 check_email_enabled: Email enabled for check_xapi Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = check_xapi: Pool Host on Server: 192.168.10.1 not responding to ICMP ping - manual intervention may be required Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 Retrieving number of hosts in pool. Setting NUM_HOSTS = 2 Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 Failed to reach Pool Master - Checking if this host promotes to Master.. Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 State file MASTER UUID = b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 fence_host: fence_host called to fencehost: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 with action: stop and method: POOL. Check Pool! Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 check_email_enabled: Email enabled for fence_host Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = fence_host: fence_host called to fencehost: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 with action: stop and method: POOL. Check Pool! Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:46:56 xcp-ng-secondary ha-lizard: 19512 check_quorum: Checking host IPs: 192.168.10.1 192.168.10.2 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: HOST IP: 192.168.10.1 Response = Fail Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: LIVE HOSTs = 1 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: Successful Replies = 1 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 check_quorum: 2 Hosts found. Minimum needed = 1 + 1. Fencing allowed Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 fence_host: Fencing enabled attempting to fence host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 with Fence Method: POOL and Action: stop Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 fence_host: Fence b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33, Action stop: POOL fencing, returning exit status 0 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 Function fence_host returned status 0 Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 Master: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 successfully fenced, attempting to start any failed VMs Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 19512 Promote Slave enabled for this host. Calling promote_slave - attempt to become pool master Nov 4 15:46:57 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:47:00 xcp-ng-secondary ha-lizard: 31214 ha-lizard already running: Attempt 3 on PIDS: 20229 20228 19512 Nov 4 15:47:00 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Host agent will restart and transition to master in 10.000 seconds... Nov 4 15:47:00 xcp-ng-secondary ha-lizard: 19512 promote_slave: xe pool-emergency-transition-to-master successful Nov 4 15:47:07 xcp-ng-secondary ha-lizard: 20424 xe_raise_alert: Alert level [1] has been passed in Nov 4 15:47:07 xcp-ng-secondary ha-lizard: 20424 xe_raise_alert: Raising system alert [HA-Lizard Storage Controller Notice] with message [check_drbd_resource_state: DRBD Resource: [iscsi1] in Secondary mode, expected in [Primary] mode] severity [1] Nov 4 15:47:07 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:47:08 xcp-ng-secondary ha-lizard: 20641 xe_raise_alert: Alert level [4] has been passed in Nov 4 15:47:08 xcp-ng-secondary ha-lizard: 20641 xe_raise_alert: Raising system alert [HA-Lizard Storage Controller Notice] with message [DRBD Resource: [iscsi1] successfully transitioned to Primary] severity [4] Nov 4 15:47:10 xcp-ng-secondary ha-lizard: 31214 ha-lizard already running: Attempt 4 on PIDS: 20229 20228 19512 Nov 4 15:47:15 xcp-ng-secondary ha-lizard: 21407 xe_raise_alert: Alert level [1] has been passed in Nov 4 15:47:15 xcp-ng-secondary ha-lizard: 21407 xe_raise_alert: Raising system alert [HA-Lizard Storage Controller Notice] with message [check_ip_health: [10.10.10.3] response = FAIL] severity [1] Nov 4 15:47:15 xcp-ng-secondary ha-lizard: 21524 xe_raise_alert: Alert level [1] has been passed in Nov 4 15:47:15 xcp-ng-secondary ha-lizard: 21524 xe_raise_alert: Raising system alert [HA-Lizard Storage Controller Notice] with message [Host: xcp-ng-secondary Failed to add [10.10.10.3] on local interface [xapi1]] severity [1] Nov 4 15:47:17 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 promote_slave: xe pool-recover-slaves successfully executed Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 check_email_enabled: Email enabled for promote_slave Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = promote_slave: Server: xcp-ng-secondary: Transitioned to pool master Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 New Master ha_enabled check Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Retrieving list of VMs on failed master from local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Resetting Power State for VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Power State for uuid: 312abe84-9704-a079-3dc2-02cb08a1bf1f set to: halted Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 7e58a3b1-3e41-f035-2a19-5f0967129d79 on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 7e58a3b1-3e41-f035-2a19-5f0967129d79 Success! Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 5ee9437b-16c0-c311-5991-3cb061322634 on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 5ee9437b-16c0-c311-5991-3cb061322634 Success! Nov 4 15:47:19 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 08e1ee24-de07-1132-380c-42cab363bfd2 on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 08e1ee24-de07-1132-380c-42cab363bfd2 Success! Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: cab7592d-696c-8e2d-16c5-f977e404a8cc on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: cab7592d-696c-8e2d-16c5-f977e404a8cc Success! Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 760d5ed8-9609-f9bb-c69a-fb7a9f92a19e on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 22067 xe_raise_alert: Alert level [1] has been passed in Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 22067 xe_raise_alert: Raising system alert [HA-Lizard Storage Controller Notice] with message [check_ip_health: [10.10.10.3] response = FAIL] severity [1] Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 31214 ha-lizard already running: Attempt 5 on PIDS: 20229 20228 19512 Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 19512 Resetting VDI: 760d5ed8-9609-f9bb-c69a-fb7a9f92a19e Success! Nov 4 15:47:20 xcp-ng-secondary ha-lizard: 19512 Slave HA is ON, Master is unreachable - Checking for VMs or appliances to start in this pool Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: Removing Control Domains from VM list Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: 1 Eligible Halted VMs found Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: Attempting to start VMs in halted state Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:47:25 xcp-ng-secondary ha-lizard: 19512 FLUSH_MAIL_EXEC returned [0] Nov 4 15:47:26 xcp-ng-secondary ha-lizard: 19512 check_email_enabled: Email enabled for vm_mon Nov 4 15:47:26 xcp-ng-secondary ha-lizard: 19512 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:26 xcp-ng-secondary ha-lizard: 19512 email: Message barred for 60 minutes Nov 4 15:47:26 xcp-ng-secondary ha-lizard: 19512 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:47:26 xcp-ng-secondary ha-lizard: 19512 vm_mon: No host available to start VM Nov 4 15:47:27 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 LOG_TERMINAL = [false] Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 This iteration is count 25 Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 Checking if this host is a Pool Master or Slave Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 This host's pool status = master Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 Checking if ha-lizard is enabled for this pool Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: ha-lizard is enabled Nov 4 15:47:30 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Host is in maintenance mode Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:47:31 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 ha-lizard is enabled Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_xs_ha: Checking XenServer HA status Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_master_mgt_link_state: Checking management interface link state Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 Master management link OK - checking prior link state Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 This host detected as pool Master Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 Found 2 hosts in pool Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 validate_vm_ha_state: Validating VM HA-state Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 Calling function write_pool_state Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 Calling function autoselect_slave Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 Calling function check_slave_status Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_master_mgt_link_state: Checking management interface link state Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_slave_status: Management link OK - continue Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 No slaves available to become pool master Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_slave_status: Quorum check called Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 check_quorum: Checking host IPs: 192.168.10.1 192.168.10.2 Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 pool autopromote_uuid = [none_available] Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:47:31 xcp-ng-secondary ha-lizard: 22408 write_pool_state: autopromote_uuid has changed - updating state file: /etc/ha-lizard/state/autopromote_uuid with uuid: none_available Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: ha-lizard is enabled Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Host is in maintenance mode Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:47:32 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 write_pool_state: Pool contains 0 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 get_pool_host_list: found no host machines in pool - major error Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 FLUSH_MAIL_EXEC returned [0] Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_email_enabled: Email enabled for get_pool_host_list Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 email: Duplicate message - not sending. Content = get_pool_host_list: Failed to get list of hosts in this pool Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 email: Message barred for 60 minutes Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 write_pool_state: Host IP List = Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 write_status_report: Writing status report Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: HOST IP: 192.168.10.1 Response = Fail Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: LIVE HOSTs = 1 Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: Successful Replies = 1 Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1 Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_quorum: 2 Hosts found. Minimum needed = 1 + 1. Fencing allowed Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_slave_status: Failed slave count = 0 Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 check_slave_status: No Failed slaves detected Nov 4 15:47:32 xcp-ng-secondary ha-lizard: 22408 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: Removing Control Domains from VM list Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: 1 Eligible Halted VMs found Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: Attempting to start VMs in halted state Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 FLUSH_MAIL_EXEC returned [0] Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 check_email_enabled: Email enabled for vm_mon Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 email: Message barred for 60 minutes Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:47:33 xcp-ng-secondary ha-lizard: 22408 vm_mon: No host available to start VM Nov 4 15:47:37 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:47:45 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:47:45 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:47:45 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:47:45 xcp-ng-secondary ha-lizard: 23518 LOG_TERMINAL = [false] Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 This iteration is count 26 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Checking if this host is a Pool Master or Slave Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 This host's pool status = master Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Checking if ha-lizard is enabled for this pool Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: ha-lizard is enabled Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Host is in maintenance mode Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:47:46 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 ha-lizard is enabled Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_xs_ha: Checking XenServer HA status Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_master_mgt_link_state: Checking management interface link state Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Master management link OK - checking prior link state Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 This host detected as pool Master Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Found 2 hosts in pool Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 validate_vm_ha_state: Validating VM HA-state Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Calling function write_pool_state Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Calling function autoselect_slave Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 Calling function check_slave_status Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_master_mgt_link_state: Checking management interface link state Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_slave_status: Management link OK - continue Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 No slaves available to become pool master Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:47:46 xcp-ng-secondary ha-lizard: 23518 check_slave_status: Quorum check called Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_quorum: No host IPs detected - Do Not Fence - exiting now Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 pool autopromote_uuid = [none_available] Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: ha-lizard is enabled Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Host is in maintenance mode Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:47:47 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 write_pool_state: Pool contains 0 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 get_pool_host_list: found no host machines in pool - major error Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 FLUSH_MAIL_EXEC returned [0] Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 check_email_enabled: Email enabled for get_pool_host_list Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 email: Duplicate message - not sending. Content = get_pool_host_list: Failed to get list of hosts in this pool Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 email: Message barred for 60 minutes Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 write_pool_state: Host IP List = Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 23518 write_status_report: Writing status report Nov 4 15:47:47 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:47:57 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:48:00 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:48:00 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 LOG_TERMINAL = [false] Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 This iteration is count 27 Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Checking if this host is a Pool Master or Slave Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 This host's pool status = master Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Checking if ha-lizard is enabled for this pool Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:01 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 ha-lizard is enabled Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_xs_ha: Checking XenServer HA status Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Master management link OK - checking prior link state Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 This host detected as pool Master Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Found 2 hosts in pool Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 validate_vm_ha_state: Validating VM HA-state Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Calling function write_pool_state Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Calling function autoselect_slave Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 Calling function check_slave_status Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:01 xcp-ng-secondary ha-lizard: 24487 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_slave_status: Management link OK - continue Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 No slaves available to become pool master Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_slave_status: Quorum check called Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_quorum: No host IPs detected - Do Not Fence - exiting now Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 pool autopromote_uuid = [none_available] Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:02 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_pool_state: Pool contains 0 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 get_pool_host_list: found no host machines in pool - major error Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 FLUSH_MAIL_EXEC returned [0] Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 check_email_enabled: Email enabled for get_pool_host_list Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 email: Duplicate message - not sending. Content = get_pool_host_list: Failed to get list of hosts in this pool Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 email: Message barred for 60 minutes Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_pool_state: Host IP List = Nov 4 15:48:02 xcp-ng-secondary ha-lizard: 24487 write_status_report: Writing status report Nov 4 15:48:07 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 LOG_TERMINAL = [false] Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 This iteration is count 28 Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 Checking if this host is a Pool Master or Slave Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 This host's pool status = master Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 Checking if ha-lizard is enabled for this pool Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:16 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 ha-lizard is enabled Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_xs_ha: Checking XenServer HA status Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 Master management link OK - checking prior link state Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 This host detected as pool Master Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 Found 2 hosts in pool Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 validate_vm_ha_state: Validating VM HA-state Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 Calling function write_pool_state Nov 4 15:48:16 xcp-ng-secondary ha-lizard: 25460 Calling function autoselect_slave Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 Calling function check_slave_status Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_slave_status: Management link OK - continue Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 No slaves available to become pool master Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_slave_status: Quorum check called Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_quorum: No host IPs detected - Do Not Fence - exiting now Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 pool autopromote_uuid = [none_available] Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:17 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: Pool contains 0 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 get_pool_host_list: found no host machines in pool - major error Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 FLUSH_MAIL_EXEC returned [0] Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 check_email_enabled: Email enabled for get_pool_host_list Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 email: Duplicate message - not sending. Content = get_pool_host_list: Failed to get list of hosts in this pool Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 email: Message barred for 60 minutes Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_pool_state: Host IP List = Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 25460 write_status_report: Writing status report Nov 4 15:48:17 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:48:27 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 LOG_TERMINAL = [false] Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 This iteration is count 29 Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 Checking if this host is a Pool Master or Slave Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 This host's pool status = master Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 Checking if ha-lizard is enabled for this pool Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:31 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 ha-lizard is enabled Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_xs_ha: Checking XenServer HA status Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 Master management link OK - checking prior link state Nov 4 15:48:31 xcp-ng-secondary ha-lizard: 26430 This host detected as pool Master Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 Found 2 hosts in pool Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 validate_vm_ha_state: Validating VM HA-state Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 Calling function write_pool_state Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 Calling function autoselect_slave Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 Calling function check_slave_status Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_slave_status: Management link OK - continue Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 No slaves available to become pool master Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_slave_status: Quorum check called Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_quorum: No host IPs detected - Do Not Fence - exiting now Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 pool autopromote_uuid = [none_available] Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:32 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 write_pool_state: Pool contains 0 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 get_pool_host_list: found no host machines in pool - major error Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 FLUSH_MAIL_EXEC returned [0] Nov 4 15:48:32 xcp-ng-secondary ha-lizard: 26430 check_email_enabled: Email enabled for get_pool_host_list Nov 4 15:48:33 xcp-ng-secondary ha-lizard: 26430 email: Duplicate message - not sending. Content = get_pool_host_list: Failed to get list of hosts in this pool Nov 4 15:48:33 xcp-ng-secondary ha-lizard: 26430 email: Message barred for 60 minutes Nov 4 15:48:33 xcp-ng-secondary ha-lizard: 26430 write_pool_state: Host IP List = Nov 4 15:48:33 xcp-ng-secondary ha-lizard: 26430 write_status_report: Writing status report Nov 4 15:48:37 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 LOG_TERMINAL = [false] Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 This iteration is count 30 Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 Checking if this host is a Pool Master or Slave Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 This host's pool status = master Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 Checking if ha-lizard is enabled for this pool Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:46 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 ha-lizard is enabled Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 check_xs_ha: Checking XenServer HA status Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:48:46 xcp-ng-secondary ha-lizard: 27395 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 Master management link OK - checking prior link state Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 This host detected as pool Master Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 Found 2 hosts in pool Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 validate_vm_ha_state: Validating VM HA-state Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 Calling function write_pool_state Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 Calling function autoselect_slave Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 Calling function check_slave_status Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_master_mgt_link_state: Checking management interface link state Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_slave_status: Management link OK - continue Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 No slaves available to become pool master Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_slave_status: Quorum check called Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_quorum: No host IPs detected - Do Not Fence - exiting now Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 pool autopromote_uuid = [none_available] Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: ha-lizard is enabled Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Host is in maintenance mode Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Master cannot be in maintenance mode - enabling host Nov 4 15:48:47 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait. Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_ha_enabled: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] has been enabled Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 write_pool_state: Pool contains 0 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 get_pool_host_list: found no host machines in pool - major error Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 FLUSH_MAIL_EXEC returned [0] Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 27395 check_email_enabled: Email enabled for get_pool_host_list Nov 4 15:48:47 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:48:48 xcp-ng-secondary ha-lizard: 27395 email: Duplicate message - not sending. Content = get_pool_host_list: Failed to get list of hosts in this pool Nov 4 15:48:48 xcp-ng-secondary ha-lizard: 27395 email: Message barred for 60 minutes Nov 4 15:48:48 xcp-ng-secondary ha-lizard: 27395 write_pool_state: Host IP List = Nov 4 15:48:48 xcp-ng-secondary ha-lizard: 27395 write_status_report: Writing status report Nov 4 15:48:58 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 LOG_TERMINAL = [false] Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 This iteration is count 31 Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 Checking if this host is a Pool Master or Slave Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 This host's pool status = master Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 Checking if ha-lizard is enabled for this pool Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 ha-lizard is enabled Nov 4 15:49:01 xcp-ng-secondary ha-lizard: 28964 check_xs_ha: Checking XenServer HA status Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 Master management link OK - checking prior link state Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 This host detected as pool Master Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 Found 2 hosts in pool Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 validate_vm_ha_state: Validating VM HA-state Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 Calling function write_pool_state Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 Calling function autoselect_slave Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 Calling function check_slave_status Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_slave_status: Management link OK - continue Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 No slaves available to become pool master Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_slave_status: Quorum check called Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_quorum: No host IPs detected - Do Not Fence - exiting now Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 pool autopromote_uuid = [none_available] Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:49:02 xcp-ng-secondary ha-lizard: 28964 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:03 xcp-ng-secondary ha-lizard: 28964 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:49:03 xcp-ng-secondary ha-lizard: 28964 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:49:03 xcp-ng-secondary ha-lizard: 28964 write_status_report: Writing status report Nov 4 15:49:08 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 LOG_TERMINAL = [false] Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 This iteration is count 32 Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 Checking if this host is a Pool Master or Slave Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 This host's pool status = master Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 Checking if ha-lizard is enabled for this pool Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:16 xcp-ng-secondary ha-lizard: 30010 ha-lizard is enabled Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_xs_ha: Checking XenServer HA status Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 Master management link OK - checking prior link state Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 This host detected as pool Master Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 Found 2 hosts in pool Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 validate_vm_ha_state: Validating VM HA-state Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 Calling function write_pool_state Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 Calling function autoselect_slave Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 Calling function check_slave_status Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_slave_status: Management link OK - continue Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 No slaves available to become pool master Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_slave_status: Quorum check called Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 pool autopromote_uuid = [none_available] Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: LIVE HOSTs = 1 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Successful Replies = 1 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_slave_status: Failed slave count = 0 Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_slave_status: No Failed slaves detected Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 vm_mon: Removing Control Domains from VM list Nov 4 15:49:17 xcp-ng-secondary ha-lizard: 30010 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: 1 Eligible Halted VMs found Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: Attempting to start VMs in halted state Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 write_status_report: Writing status report Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 FLUSH_MAIL_EXEC returned [0] Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 check_email_enabled: Email enabled for vm_mon Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 email: Message barred for 60 minutes Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 1144 ] Nov 4 15:49:18 xcp-ng-secondary ha-lizard: 30010 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 vm_mon: VM start exit result = [ 1 ] Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:49:21 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:49:21 xcp-ng-secondary ha-lizard: 30010 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:49:24 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:49:24 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:49:24 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:49:24 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:49:24 xcp-ng-secondary ha-lizard: 30010 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:24 xcp-ng-secondary ha-lizard: 30010 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:24 xcp-ng-secondary ha-lizard: 30010 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:49:24 xcp-ng-secondary ha-lizard: 30010 FLUSH_MAIL_EXEC returned [0] Nov 4 15:49:24 xcp-ng-secondary ha-lizard: 30010 check_email_enabled: Email enabled for vm_mon Nov 4 15:49:24 xcp-ng-secondary ha-lizard: 30010 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:24 xcp-ng-secondary ha-lizard: 30010 email: Message barred for 60 minutes Nov 4 15:49:28 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31757 LOG_TERMINAL = [false] Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31757 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31757 This iteration is count 33 Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31757 Checking if this host is a Pool Master or Slave Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31757 This host's pool status = master Nov 4 15:49:31 xcp-ng-secondary ha-lizard: 31757 Checking if ha-lizard is enabled for this pool Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 ha-lizard is enabled Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_xs_ha: Checking XenServer HA status Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 Master management link OK - checking prior link state Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 This host detected as pool Master Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 Found 2 hosts in pool Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 validate_vm_ha_state: Validating VM HA-state Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 Calling function write_pool_state Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 Calling function autoselect_slave Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 Calling function check_slave_status Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_slave_status: Management link OK - continue Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 No slaves available to become pool master Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_slave_status: Quorum check called Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 pool autopromote_uuid = [none_available] Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_quorum: LIVE HOSTs = 1 Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:49:32 xcp-ng-secondary ha-lizard: 31757 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_quorum: Successful Replies = 1 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_slave_status: Failed slave count = 0 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_slave_status: No Failed slaves detected Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: Removing Control Domains from VM list Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: 1 Eligible Halted VMs found Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: Attempting to start VMs in halted state Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 write_status_report: Writing status report Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 FLUSH_MAIL_EXEC returned [0] Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 check_email_enabled: Email enabled for vm_mon Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 email: Message barred for 60 minutes Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 89 ] Nov 4 15:49:33 xcp-ng-secondary ha-lizard: 31757 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 vm_mon: VM start exit result = [ 1 ] Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:49:36 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:49:36 xcp-ng-secondary ha-lizard: 31757 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:49:38 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:49:39 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:49:39 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:49:39 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:49:39 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:49:39 xcp-ng-secondary ha-lizard: 31757 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:39 xcp-ng-secondary ha-lizard: 31757 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:39 xcp-ng-secondary ha-lizard: 31757 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:49:39 xcp-ng-secondary ha-lizard: 31757 FLUSH_MAIL_EXEC returned [0] Nov 4 15:49:39 xcp-ng-secondary ha-lizard: 31757 check_email_enabled: Email enabled for vm_mon Nov 4 15:49:39 xcp-ng-secondary ha-lizard: 31757 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:39 xcp-ng-secondary ha-lizard: 31757 email: Message barred for 60 minutes Nov 4 15:49:46 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:49:46 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:49:46 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 LOG_TERMINAL = [false] Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 This iteration is count 34 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Checking if this host is a Pool Master or Slave Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 This host's pool status = master Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Checking if ha-lizard is enabled for this pool Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 ha-lizard is enabled Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_xs_ha: Checking XenServer HA status Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Master management link OK - checking prior link state Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 This host detected as pool Master Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Found 2 hosts in pool Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 validate_vm_ha_state: Validating VM HA-state Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Calling function write_pool_state Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Calling function autoselect_slave Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 Calling function check_slave_status Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_master_mgt_link_state: Checking management interface link state Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 check_slave_status: Management link OK - continue Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 No slaves available to become pool master Nov 4 15:49:47 xcp-ng-secondary ha-lizard: 1030 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_slave_status: Quorum check called Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 pool autopromote_uuid = [none_available] Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: LIVE HOSTs = 1 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Successful Replies = 1 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_slave_status: Failed slave count = 0 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_slave_status: No Failed slaves detected Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_ha_enabled: ha-lizard is enabled Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: Removing Control Domains from VM list Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: 1 Eligible Halted VMs found Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: Attempting to start VMs in halted state Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 write_status_report: Writing status report Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 FLUSH_MAIL_EXEC returned [0] Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 check_email_enabled: Email enabled for vm_mon Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 email: Message barred for 60 minutes Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 751 ] Nov 4 15:49:48 xcp-ng-secondary ha-lizard: 1030 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 vm_mon: VM start exit result = [ 1 ] Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:49:51 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:49:51 xcp-ng-secondary ha-lizard: 1030 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:49:54 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:49:54 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:49:54 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:49:54 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:49:54 xcp-ng-secondary ha-lizard: 1030 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:54 xcp-ng-secondary ha-lizard: 1030 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:49:54 xcp-ng-secondary ha-lizard: 1030 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:49:54 xcp-ng-secondary ha-lizard: 1030 FLUSH_MAIL_EXEC returned [0] Nov 4 15:49:54 xcp-ng-secondary ha-lizard: 1030 check_email_enabled: Email enabled for vm_mon Nov 4 15:49:55 xcp-ng-secondary ha-lizard: 1030 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:49:55 xcp-ng-secondary ha-lizard: 1030 email: Message barred for 60 minutes Nov 4 15:49:58 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 LOG_TERMINAL = [false] Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 This iteration is count 35 Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Checking if this host is a Pool Master or Slave Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 This host's pool status = master Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Checking if ha-lizard is enabled for this pool Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 ha-lizard is enabled Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_xs_ha: Checking XenServer HA status Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Master management link OK - checking prior link state Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 This host detected as pool Master Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Found 2 hosts in pool Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 validate_vm_ha_state: Validating VM HA-state Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Calling function write_pool_state Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Calling function autoselect_slave Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 Calling function check_slave_status Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:02 xcp-ng-secondary ha-lizard: 3190 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_slave_status: Management link OK - continue Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 No slaves available to become pool master Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_slave_status: Quorum check called Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 pool autopromote_uuid = [none_available] Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: LIVE HOSTs = 1 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Successful Replies = 1 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_slave_status: Failed slave count = 0 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_slave_status: No Failed slaves detected Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: Removing Control Domains from VM list Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: 1 Eligible Halted VMs found Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: Attempting to start VMs in halted state Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 write_status_report: Writing status report Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 email: Message barred for 60 minutes Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 496 ] Nov 4 15:50:03 xcp-ng-secondary ha-lizard: 3190 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:50:06 xcp-ng-secondary ha-lizard: 3190 vm_mon: VM start exit result = [ 1 ] Nov 4 15:50:06 xcp-ng-secondary ha-lizard: 3190 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:50:06 xcp-ng-secondary ha-lizard: 3190 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:06 xcp-ng-secondary ha-lizard: 3190 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:50:06 xcp-ng-secondary ha-lizard: 3190 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:50:07 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:50:07 xcp-ng-secondary ha-lizard: 3190 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:50:07 xcp-ng-secondary ha-lizard: 3190 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:50:07 xcp-ng-secondary ha-lizard: 3190 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:50:08 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:50:09 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:50:09 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:50:09 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:50:09 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:09 xcp-ng-secondary ha-lizard: 3190 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:09 xcp-ng-secondary ha-lizard: 3190 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:10 xcp-ng-secondary ha-lizard: 3190 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:10 xcp-ng-secondary ha-lizard: 3190 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:10 xcp-ng-secondary ha-lizard: 3190 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:10 xcp-ng-secondary ha-lizard: 3190 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:10 xcp-ng-secondary ha-lizard: 3190 email: Message barred for 60 minutes Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 LOG_TERMINAL = [false] Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 This iteration is count 36 Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 Checking if this host is a Pool Master or Slave Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 This host's pool status = master Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 Checking if ha-lizard is enabled for this pool Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 ha-lizard is enabled Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 check_xs_ha: Checking XenServer HA status Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 Master management link OK - checking prior link state Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 This host detected as pool Master Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 Found 2 hosts in pool Nov 4 15:50:17 xcp-ng-secondary ha-lizard: 4905 validate_vm_ha_state: Validating VM HA-state Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 Calling function write_pool_state Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 Calling function autoselect_slave Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 Calling function check_slave_status Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_slave_status: Management link OK - continue Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 No slaves available to become pool master Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_slave_status: Quorum check called Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 pool autopromote_uuid = [none_available] Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: LIVE HOSTs = 1 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Successful Replies = 1 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_slave_status: Failed slave count = 0 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_slave_status: No Failed slaves detected Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: Removing Control Domains from VM list Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: 1 Eligible Halted VMs found Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: Attempting to start VMs in halted state Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 write_status_report: Writing status report Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 email: Message barred for 60 minutes Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:50:18 xcp-ng-secondary ha-lizard: 4905 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 2860 ] Nov 4 15:50:19 xcp-ng-secondary ha-lizard: 4905 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:50:21 xcp-ng-secondary ha-lizard: 4905 vm_mon: VM start exit result = [ 1 ] Nov 4 15:50:21 xcp-ng-secondary ha-lizard: 4905 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:50:21 xcp-ng-secondary ha-lizard: 4905 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:21 xcp-ng-secondary ha-lizard: 4905 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:50:21 xcp-ng-secondary ha-lizard: 4905 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:50:22 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:50:22 xcp-ng-secondary ha-lizard: 4905 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:50:22 xcp-ng-secondary ha-lizard: 4905 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:50:22 xcp-ng-secondary ha-lizard: 4905 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:50:25 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:50:25 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:50:25 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:50:25 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:25 xcp-ng-secondary ha-lizard: 4905 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:25 xcp-ng-secondary ha-lizard: 4905 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:25 xcp-ng-secondary ha-lizard: 4905 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:25 xcp-ng-secondary ha-lizard: 4905 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:25 xcp-ng-secondary ha-lizard: 4905 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:25 xcp-ng-secondary ha-lizard: 4905 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:25 xcp-ng-secondary ha-lizard: 4905 email: Message barred for 60 minutes Nov 4 15:50:28 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 LOG_TERMINAL = [false] Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 This iteration is count 37 Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 Checking if this host is a Pool Master or Slave Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 This host's pool status = master Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 Checking if ha-lizard is enabled for this pool Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 ha-lizard is enabled Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 check_xs_ha: Checking XenServer HA status Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:32 xcp-ng-secondary ha-lizard: 6658 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 Master management link OK - checking prior link state Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 This host detected as pool Master Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 Found 2 hosts in pool Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 validate_vm_ha_state: Validating VM HA-state Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 Calling function write_pool_state Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 Calling function autoselect_slave Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 Calling function check_slave_status Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_slave_status: Management link OK - continue Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 No slaves available to become pool master Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_slave_status: Quorum check called Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 pool autopromote_uuid = [none_available] Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: LIVE HOSTs = 1 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Successful Replies = 1 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_slave_status: Failed slave count = 0 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_slave_status: No Failed slaves detected Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: Removing Control Domains from VM list Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: 1 Eligible Halted VMs found Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: Attempting to start VMs in halted state Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 write_status_report: Writing status report Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:33 xcp-ng-secondary ha-lizard: 6658 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:34 xcp-ng-secondary ha-lizard: 6658 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:34 xcp-ng-secondary ha-lizard: 6658 email: Message barred for 60 minutes Nov 4 15:50:34 xcp-ng-secondary ha-lizard: 6658 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:50:34 xcp-ng-secondary ha-lizard: 6658 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 3758 ] Nov 4 15:50:34 xcp-ng-secondary ha-lizard: 6658 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:50:36 xcp-ng-secondary ha-lizard: 6658 vm_mon: VM start exit result = [ 1 ] Nov 4 15:50:36 xcp-ng-secondary ha-lizard: 6658 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:50:37 xcp-ng-secondary ha-lizard: 6658 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:37 xcp-ng-secondary ha-lizard: 6658 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:50:37 xcp-ng-secondary ha-lizard: 6658 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:50:37 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:50:37 xcp-ng-secondary ha-lizard: 6658 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:50:37 xcp-ng-secondary ha-lizard: 6658 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:50:37 xcp-ng-secondary ha-lizard: 6658 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:50:38 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:50:40 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:50:40 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:50:40 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:50:40 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:40 xcp-ng-secondary ha-lizard: 6658 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:40 xcp-ng-secondary ha-lizard: 6658 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:40 xcp-ng-secondary ha-lizard: 6658 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:40 xcp-ng-secondary ha-lizard: 6658 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:40 xcp-ng-secondary ha-lizard: 6658 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:40 xcp-ng-secondary ha-lizard: 6658 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:40 xcp-ng-secondary ha-lizard: 6658 email: Message barred for 60 minutes Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 LOG_TERMINAL = [false] Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 This iteration is count 38 Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 Checking if this host is a Pool Master or Slave Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 This host's pool status = master Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 Checking if ha-lizard is enabled for this pool Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 ha-lizard is enabled Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 check_xs_ha: Checking XenServer HA status Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:50:47 xcp-ng-secondary ha-lizard: 8351 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 Master management link OK - checking prior link state Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 This host detected as pool Master Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 Found 2 hosts in pool Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 validate_vm_ha_state: Validating VM HA-state Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 Calling function write_pool_state Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 Calling function autoselect_slave Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 Calling function check_slave_status Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_master_mgt_link_state: Checking management interface link state Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_slave_status: Management link OK - continue Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 No slaves available to become pool master Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_slave_status: Quorum check called Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 pool autopromote_uuid = [none_available] Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: LIVE HOSTs = 1 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Successful Replies = 1 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_slave_status: Failed slave count = 0 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_slave_status: No Failed slaves detected Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_ha_enabled: ha-lizard is enabled Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: Removing Control Domains from VM list Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: 1 Eligible Halted VMs found Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: Attempting to start VMs in halted state Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 write_status_report: Writing status report Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:48 xcp-ng-secondary ha-lizard: 8351 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:49 xcp-ng-secondary ha-lizard: 8351 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:49 xcp-ng-secondary ha-lizard: 8351 email: Message barred for 60 minutes Nov 4 15:50:49 xcp-ng-secondary ha-lizard: 8351 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:50:49 xcp-ng-secondary ha-lizard: 8351 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 1143 ] Nov 4 15:50:49 xcp-ng-secondary ha-lizard: 8351 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 vm_mon: VM start exit result = [ 1 ] Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:50:52 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:50:52 xcp-ng-secondary ha-lizard: 8351 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:50:55 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:50:55 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:50:55 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:50:55 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:50:55 xcp-ng-secondary ha-lizard: 8351 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:55 xcp-ng-secondary ha-lizard: 8351 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:50:55 xcp-ng-secondary ha-lizard: 8351 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:50:55 xcp-ng-secondary ha-lizard: 8351 FLUSH_MAIL_EXEC returned [0] Nov 4 15:50:55 xcp-ng-secondary ha-lizard: 8351 check_email_enabled: Email enabled for vm_mon Nov 4 15:50:55 xcp-ng-secondary ha-lizard: 8351 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:50:55 xcp-ng-secondary ha-lizard: 8351 email: Message barred for 60 minutes Nov 4 15:50:58 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 31214 Spawning new instance of ha-lizard Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: Checking logger processes Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 31214 check_logger_processes: No processes to clear Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 LOG_TERMINAL = [false] Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 This iteration is count 39 Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 Checking if this host is a Pool Master or Slave Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 This host's pool status = master Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 Checking if ha-lizard is enabled for this pool Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 check_ha_enabled: ha-lizard is enabled Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 ha-lizard is enabled Nov 4 15:51:02 xcp-ng-secondary ha-lizard: 10078 check_xs_ha: Checking XenServer HA status Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=192.168.255.254#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_master_mgt_link_state: Checking management interface link state Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 Master management link OK - checking prior link state Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 This host detected as pool Master Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 Found 2 hosts in pool Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 validate_vm_ha_state: Validating VM HA-state Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 validate_vm_ha_state: VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] state [ false ] = OK Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 Calling function write_pool_state Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 Calling function autoselect_slave Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 Calling function check_slave_status Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_master_mgt_link_state: Checking management interface link state Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: MASTER UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 autoselect_slave: This host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: MASTER UUID: f543502f-1445-429e-8220-b360cd2a6946 written to local state storage Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 autoselect_slave: MASTER host UUID found: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: Calling function get_vms_on_host for UUID: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 192.168.10.2 ] Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_master_mgt_link_state: Link [ xenbr0 ] state UP Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_slave_status: Management link OK - continue Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 get_vms_on_host: No VMs found on host: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: Writing VM array to local state file host.b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33.vmlist.uuid_array Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 autoselect_slave: f543502f-1445-429e-8220-b360cd2a6946 is Master UUID - excluding from list of available slaves Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 autoselect_slave: 0 available Slave UUIDs found: Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: Calling function get_vms_on_host for UUID: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 get_pool_host_list: returned b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33#012f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 get_vms_on_host: No VMs found on host: f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: Writing VM array to local state file host.f543502f-1445-429e-8220-b360cd2a6946.vmlist.uuid_array Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 No slaves available to become pool master Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_slave_status: Removing Slave UUID from list of Hosts - Slave: b6dcd4f7-860c-4dfc-8e7c-22c00ef08b33 is disabled or in maintenance mode Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_slave_status: Removing Master UUID from list of Hosts Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_slave_status: Host IP Address check Status Array for Slaves = () Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_slave_status: Quorum check called Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Checking host IPs: 192.168.10.2 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 pool autopromote_uuid = [none_available] Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: Pool autopromote_uuid=none_available Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Host IP: 192.168.10.2 Response = OK Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: autopromote_uuid unchanged - not updating Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: LIVE HOSTs = 1 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Using network points: 192.168.255.254 as possible additional vote Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Heuristic IP: 192.168.255.254 Response = OK Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Successful Replies = 1 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_slave_status: Failed slave count = 0 Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_slave_status: No Failed slaves detected Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_ha_enabled: Checking if ha-lizard is enabled for pool: e43e608e-4ead-c0df-8f26-9d52f628752a Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 Function check_slave_status reported no failures: calling vm_mon Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_ha_enabled: ha-lizard is enabled Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 check_ha_enabled: checking whether maintenance mode enabled Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: Retrived list of VMs for this poll: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: Removing Control Domains from VM list Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: VM list returned = 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_state: Machine state for 312abe84-9704-a079-3dc2-02cb08a1bf1f returned: halted Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: VM 312abe84-9704-a079-3dc2-02cb08a1bf1f state = halted Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 312abe84-9704-a079-3dc2-02cb08a1bf1f to list of failed VMs on this run. Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: 1 Eligible Halted VMs found Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: Halted VMs found: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 vm_mon: Attempting to start VMs in halted state Nov 4 15:51:03 xcp-ng-secondary ha-lizard: 10078 write_pool_state: Pool contains 1 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 validate_vm_safe_to_start_here: VM [312abe84-9704-a079-3dc2-02cb08a1bf1f] home pool validated [e43e608e-4ead-c0df-8f26-9d52f628752a] - safe to start here Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 get_pool_host_list: returned f543502f-1445-429e-8220-b360cd2a6946 Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 get_pool_ip_list: returned 192.168.10.2 Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 write_pool_state: Host IP List = 192.168.10.2 Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 write_status_report: Writing status report Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 FLUSH_MAIL_EXEC returned [0] Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 check_email_enabled: Email enabled for vm_mon Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 email: Duplicate message - not sending. Content = vm_mon: Starting VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 email: Message barred for 60 minutes Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 vm_mon: This host [ f543502f-1445-429e-8220-b360cd2a6946 ] start on serial [ 970 ] Nov 4 15:51:04 xcp-ng-secondary ha-lizard: 10078 vm_mon: Host [ f543502f-1445-429e-8220-b360cd2a6946 ] health status = [ master ] Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 vm_mon: VM start exit result = [ 1 ] Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 vm_mon: VM start returned messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], ] Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address#012 Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found#012 Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 reset_vm_vdi: Resetting VDI(s) for VM [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 reset_vm_vdi: Found VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] Nov 4 15:51:07 xcp-ng-secondary ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: VDI 85a74300-0ce0-4a65-b21d-34184c4a2e8b is not marked as attached anywhere, nothing to do Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 reset_vm_vdi: VDI [ 85a74300-0ce0-4a65-b21d-34184c4a2e8b ] reset Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 reset_vm_vdi: No VDI found for VBD [ 27bc9965-8a28-ed69-b3c5-bf9a9179f879 ] Nov 4 15:51:07 xcp-ng-secondary ha-lizard: 10078 vm_mon: Reattempting vm [ 312abe84-9704-a079-3dc2-02cb08a1bf1f ] start Nov 4 15:51:08 xcp-ng-secondary ha-lizard: 2547 ha-lizard Watchdog: ha-lizard running - OK Nov 4 15:51:10 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error code: SR_BACKEND_FAILURE_46 Nov 4 15:51:10 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error parameters: , The VDI is not available [opterr=Command ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff'] failed (5): /dev/sdc: open failed: No such device or address Nov 4 15:51:10 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Volume group "VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff" not found Nov 4 15:51:10 xcp-ng-secondary ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Cannot process volume group VG_XenStorage-3aeb126f-7d32-39ac-1626-a334dc5404ff], Nov 4 15:51:10 xcp-ng-secondary ha-lizard: 10078 vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:51:10 xcp-ng-secondary ha-lizard: 10078 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 4 15:51:10 xcp-ng-secondary ha-lizard: 10078 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Nov 4 15:51:10 xcp-ng-secondary ha-lizard: 10078 FLUSH_MAIL_EXEC returned [0] Nov 4 15:51:10 xcp-ng-secondary ha-lizard: 10078 check_email_enabled: Email enabled for vm_mon Nov 4 15:51:10 xcp-ng-secondary ha-lizard: 10078 email: Duplicate message - not sending. Content = vm_mon: Error starting failed VM: Windows 10 (64-bit) (1) UUID: 312abe84-9704-a079-3dc2-02cb08a1bf1f Nov 4 15:51:10 xcp-ng-secondary ha-lizard: 10078 email: Message barred for 60 minutes