Hi All,
i have a question for ESXI host, today the host suddenly disconnect, client cannot VMs Server, i try to ping iSCSI storage, VM Server and ESXI host
i found ESXI host and VM Server cannot ping, but it restore few minute, i found the hostd.log and kernal log, can anyone help to check the problem?
2015-03-12T05:15:47.624Z [5F040B70 info 'Vimsvc.ha-eventmgr'] Event 1575 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:16:22.119Z [5F681B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2015-03-12T05:16:39.668Z [5F681B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 572,
--> chainId = 572,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3726",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "46565789",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:16:39.669Z [5F681B70 info 'Vimsvc.ha-eventmgr'] Event 1576 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3726 microseconds to 46565789 microseconds.
2015-03-12T05:16:39.669Z [60680B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 0,
--> chainId = 1612902776,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:16:39.669Z [60680B70 info 'Vimsvc.ha-eventmgr'] Event 1577 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:16:40.008Z [5F6C2B70 warning 'Statssvc.vim.PerformanceManager'] Calculated read I/O size 675384 for scsi0:0 is out of range -- 675384,prevBytes = 788582736384 curBytes = 788588814848 prevCommands = 21522047curCommands = 21522056
2015-03-12T05:16:40.008Z [5F6C2B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 1052672 for scsi0:0 is out of range -- 1052672,prevBytes = 28093455360 curBytes = 28097666048 prevCommands = 979379curCommands = 979383
2015-03-12T05:16:40.103Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1449489007,
--> chainId = 1935822669,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.clear.scsi.device.io.latency.improved",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "46565789",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "9252611",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:16:40.104Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1578 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has improved. I/O latency reduced from 46565789 microseconds to 9252611 microseconds.
2015-03-12T05:16:40.456Z [5F040B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 41,
--> chainId = 1,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.clear.scsi.device.io.latency.improved",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "9252611",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "1819113",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:16:40.456Z [5F040B70 info 'Vimsvc.ha-eventmgr'] Event 1579 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has improved. I/O latency reduced from 9252611 microseconds to 1819113 microseconds.
2015-03-12T05:16:40.811Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] CheckHeartbeatMonitor: Reviving the heartbeat monitor in 30 seconds
2015-03-12T05:16:40.811Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are ready
2015-03-12T05:16:40.909Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are ready
2015-03-12T05:16:40.925Z [606C1B70 verbose 'Default' opID=11214005-000008D9 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:16:41.439Z [5F040B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1579,
--> chainId = 1579,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.clear.scsi.device.io.latency.improved",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "1819113",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "362911",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:16:41.439Z [5F040B70 info 'Vimsvc.ha-eventmgr'] Event 1580 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has improved. I/O latency reduced from 1819113 microseconds to 362911 microseconds.
2015-03-12T05:16:41.558Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are auto-upgrade capable
2015-03-12T05:16:41.558Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:16:41.561Z [60680B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are set-resolution capable
2015-03-12T05:16:41.920Z [60680B70 verbose 'Default' opID=11214005-000008DA user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:16:42.919Z [60680B70 verbose 'Default' opID=11214005-000008DB user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:17:00.010Z [60680B70 warning 'Statssvc.vim.PerformanceManager'] Calculated read I/O size 789543 for scsi0:0 is out of range -- 789543,prevBytes = 788588814848 curBytes = 788785411072 prevCommands = 21522056curCommands = 21522305
2015-03-12T05:17:00.010Z [60680B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 1003441 for scsi0:0 is out of range -- 1003441,prevBytes = 28097666048 curBytes = 28169913856 prevCommands = 979383curCommands = 979455
2015-03-12T05:17:08.180Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are not ready
2015-03-12T05:17:08.180Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are not ready
2015-03-12T05:17:08.180Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not auto-upgrade capable
2015-03-12T05:17:08.180Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:17:08.181Z [5EE55B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not set-resolution capable
2015-03-12T05:17:08.184Z [606C1B70 verbose 'Default' opID=11214005-000008DC user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:17:09.183Z [5EE55B70 verbose 'Default' opID=11214005-000008DD user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:17:10.813Z [5F6C2B70 info 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Turning off heartbeat checker
2015-03-12T05:17:20.758Z [5F6C2B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12908MB to 9248MHz/12906MB
2015-03-12T05:17:48.462Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1612917588,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:17:48.463Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1581 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:17:48.463Z [606C1B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1604814860,
--> chainId = 1604815000,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:17:48.463Z [606C1B70 info 'Vimsvc.ha-eventmgr'] Event 1582 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:17:49.509Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are ready
2015-03-12T05:17:49.509Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] CheckHeartbeatMonitor: Reviving the heartbeat monitor in 30 seconds
2015-03-12T05:17:49.524Z [5EE55B70 verbose 'Default' opID=11214005-000008DE user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:17:53.440Z [5F640B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2015-03-12T05:18:00.006Z [5F6C2B70 warning 'Statssvc.vim.PerformanceManager'] Calculated read I/O size 528384 for scsi0:0 is out of range -- 528384,prevBytes = 788785411072 curBytes = 788787524608 prevCommands = 21522305curCommands = 21522309
2015-03-12T05:18:00.006Z [5F6C2B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 938194 for scsi0:0 is out of range -- 938194,prevBytes = 28169913856 curBytes = 28185863168 prevCommands = 979455curCommands = 979472
2015-03-12T05:18:19.512Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: green
2015-03-12T05:18:19.523Z [5F6C2B70 verbose 'Default' opID=11214005-000008DF user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:18:20.760Z [5F6C2B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12906MB to 9248MHz/12904MB
2015-03-12T05:18:52.787Z [606C1B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1608972620,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:18:52.788Z [606C1B70 info 'Vimsvc.ha-eventmgr'] Event 1583 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:18:52.788Z [606C1B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1608972620,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:18:52.788Z [606C1B70 info 'Vimsvc.ha-eventmgr'] Event 1584 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:19:00.007Z [5F640B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 1052672 for scsi0:0 is out of range -- 1052672,prevBytes = 28185863168 curBytes = 28186915840 prevCommands = 979472curCommands = 979473
2015-03-12T05:19:04.860Z [5F640B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1604868380,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:19:04.860Z [5F640B70 info 'Vimsvc.ha-eventmgr'] Event 1585 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:19:12.211Z [606C1B70 verbose 'Hostsvc.DvsManager'] PersistAllDvsInfo called
2015-03-12T05:19:20.762Z [5F6C2B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12904MB to 9248MHz/12903MB
2015-03-12T05:19:24.760Z [606C1B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2015-03-12T05:19:31.955Z [5F640B70 verbose 'Default' opID=11214005-000008E0 user=root] AdapterServer: target='vmodl.query.PropertyCollector.Filter:session[9f09af08-5d81-abde-3d35-1db914f6d4aa]52144248-fbcc-49f2-e13d-6b63c90f4c7f', method='destroy'
2015-03-12T05:19:31.962Z [5F640B70 verbose 'Default' opID=11214005-000008E2 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='createFilter'
2015-03-12T05:19:31.964Z [5F6C2B70 verbose 'Default' opID=11214005-000008E1 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='createFilter'
2015-03-12T05:19:31.967Z [5F640B70 verbose 'Default' opID=11214005-000008E3 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:19:32.344Z [5F640B70 verbose 'Default' opID=11214005-000008E4 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='retrieveContents'
2015-03-12T05:19:32.358Z [5F640B70 verbose 'Default' opID=11214005-000008E5 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='createFilter'
2015-03-12T05:19:32.359Z [5F6C2B70 verbose 'Default' opID=11214005-000008E6 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:19:32.359Z [FFC70B70 verbose 'Default' opID=11214005-000008E7 user=root] AdapterServer: target='vmodl.query.PropertyCollector.Filter:session[9f09af08-5d81-abde-3d35-1db914f6d4aa]52f953c7-f2da-33d8-2a70-a9e80312d7b6', method='destroy'
2015-03-12T05:19:32.366Z [FFC70B70 verbose 'Default' opID=11214005-000008E8 user=root] AdapterServer: target='vim.SessionManager:ha-sessionmgr', method='acquireCloneTicket'
2015-03-12T05:19:32.969Z [5EE55B70 verbose 'Default' opID=11214005-000008E9 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:19:33.720Z [FFC0D920 info 'Vimsvc.ha-eventmgr' opID=hostd-16f4] Event 1586 : User root@192.168.9.7 logged in as VMware-client
2015-03-12T05:19:33.881Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.Datastore.UpdateVirtualMachineMetadata.label' expected in module 'auth'.
2015-03-12T05:19:33.881Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.Datastore.UpdateVirtualMachineMetadata.summary' expected in module 'auth'.
2015-03-12T05:19:33.883Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.VmCheckTestletManager.Config.label' expected in module 'auth'.
2015-03-12T05:19:33.883Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.VmCheckTestletManager.Config.summary' expected in module 'auth'.
2015-03-12T05:19:33.883Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.VmCheckTestletManager.View.label' expected in module 'auth'.
2015-03-12T05:19:33.883Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.VmCheckTestletManager.View.summary' expected in module 'auth'.
2015-03-12T05:19:33.883Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.Certificate.Manage.label' expected in module 'auth'.
2015-03-12T05:19:33.883Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.Certificate.Manage.summary' expected in module 'auth'.
2015-03-12T05:19:33.884Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.VmCheckTestletManager.label' expected in module 'auth'.
2015-03-12T05:19:33.884Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.VmCheckTestletManager.summary' expected in module 'auth'.
2015-03-12T05:19:33.884Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.Certificate.label' expected in module 'auth'.
2015-03-12T05:19:33.884Z [FFC70B70 verbose 'Locale' opID=hostd-133f] Default resource used for 'privilege.Certificate.summary' expected in module 'auth'.
2015-03-12T05:19:34.173Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.OperationCleanupManager.createEntry.label' expected in module 'task'.
2015-03-12T05:19:34.173Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.OperationCleanupManager.createEntry.summary' expected in module 'task'.
2015-03-12T05:19:34.173Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.OperationCleanupManager.updateEntry.label' expected in module 'task'.
2015-03-12T05:19:34.173Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.OperationCleanupManager.updateEntry.summary' expected in module 'task'.
2015-03-12T05:19:34.173Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.OperationCleanupManager.queryEntry.label' expected in module 'task'.
2015-03-12T05:19:34.173Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.OperationCleanupManager.queryEntry.summary' expected in module 'task'.
2015-03-12T05:19:34.174Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.prov.ProvisioningManager.queryVmLocation.label' expected in module 'task'.
2015-03-12T05:19:34.174Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.prov.ProvisioningManager.queryVmLocation.summary' expected in module 'task'.
2015-03-12T05:19:34.174Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.prov.ProvisioningManager.createShadowVm.label' expected in module 'task'.
2015-03-12T05:19:34.174Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.prov.ProvisioningManager.createShadowVm.summary' expected in module 'task'.
2015-03-12T05:19:34.174Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.NetworkSystem.performHostOpaqueNetworkDataOperation.label' expected in module 'task'.
2015-03-12T05:19:34.174Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.NetworkSystem.performHostOpaqueNetworkDataOperation.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.check.TestletManager.registerTestProvider.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.check.TestletManager.registerTestProvider.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.check.TestletManager.unregisterTestProvider.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.check.TestletManager.unregisterTestProvider.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.HostProfile.validate.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.HostProfile.validate.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'CertificateManager.refreshCACertificatesAndCRLs.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'CertificateManager.refreshCACertificatesAndCRLs.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.setCustomValue.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.setCustomValue.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.reload.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.reload.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.rename.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.rename.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.destroy.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.destroy.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.addTag.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.addTag.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.removeTag.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.removeTag.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.destroyNetwork.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'OpaqueNetwork.destroyNetwork.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'DistributedVirtualSwitch.reconfigureVmVnicNetworkResourcePool.summary' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.check.StaleObjectChecker.checkStaleObject.label' expected in module 'task'.
2015-03-12T05:19:34.175Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.check.StaleObjectChecker.checkStaleObject.summary' expected in module 'task'.
2015-03-12T05:19:34.176Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.profileEngine.HostProfileManager.generateTaskList.label' expected in module 'task'.
2015-03-12T05:19:34.176Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.profileEngine.HostProfileManager.generateTaskList.summary' expected in module 'task'.
2015-03-12T05:19:34.176Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.generateCertificateSigningRequest.label' expected in module 'task'.
2015-03-12T05:19:34.176Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.generateCertificateSigningRequest.summary' expected in module 'task'.
2015-03-12T05:19:34.176Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.installServerCertificate.label' expected in module 'task'.
2015-03-12T05:19:34.176Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.installServerCertificate.summary' expected in module 'task'.
2015-03-12T05:19:34.176Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.replaceCACertificatesAndCRLs.label' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.replaceCACertificatesAndCRLs.summary' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.listCACertificates.label' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.listCACertificates.summary' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.listCACertificateRevocationLists.label' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.CertificateManager.listCACertificateRevocationLists.summary' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.ProfileManager.retrieveAnswerFileForProfile.label' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.ProfileManager.retrieveAnswerFileForProfile.summary' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.ProfileManager.exportCustomizations.label' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.ProfileManager.exportCustomizations.summary' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.ProfileManager.importCustomizations.label' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'profile.host.ProfileManager.importCustomizations.summary' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.VsanInternalSystem.deleteVsanObjects.label' expected in module 'task'.
2015-03-12T05:19:34.177Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.VsanInternalSystem.deleteVsanObjects.summary' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.StorageSystem.resolveMultipleUnresolvedVmfsVolumesEx.label' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'host.StorageSystem.resolveMultipleUnresolvedVmfsVolumesEx.summary' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.checkReconfigureDasVmcp.label' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.checkReconfigureDasVmcp.summary' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.queryFaultToleranceCompatibleHost.label' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.queryFaultToleranceCompatibleHost.summary' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.queryFaultToleranceCompatibleDatastore.label' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.queryFaultToleranceCompatibleDatastore.summary' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.verifyFaultToleranceConfigSpec.label' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.verifyFaultToleranceConfigSpec.summary' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.placeVm.label' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'ClusterComputeResource.placeVm.summary' expected in module 'task'.
2015-03-12T05:19:34.178Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.retrieveConfigValue.label' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.retrieveConfigValue.summary' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.updateConfigValue.label' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.updateConfigValue.summary' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.initiateProductInfoQuery.label' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.initiateProductInfoQuery.summary' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.initiateProductInstall.label' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.initiateProductInstall.summary' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.retrieveJobStatus.label' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.retrieveJobStatus.summary' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.retrieveJobResults.label' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'vm.guest.GuestUpdateAgent.retrieveJobResults.summary' expected in module 'task'.
2015-03-12T05:19:34.179Z [FFC70B70 verbose 'Locale' opID=hostd-f4dc user=root] Default resource used for 'dvs.VmwareDistributedVirtualSwitch.reconfigureVmVnicNetworkResourcePool.summary' expected in module 'task'.
2015-03-12T05:19:35.411Z [FFC70B70 info 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx' opID=hostd-e0a3 user=root] Ticket issued for mks service to user: root
2015-03-12T05:19:35.653Z [5F040B70 verbose 'Default' opID=11214005-000008EA user=root] AdapterServer: target='vim.event.EventManager:ha-eventmgr', method='postEvent'
2015-03-12T05:19:56.866Z [5F040B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 24,
--> chainId = 121,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:19:56.866Z [5F040B70 info 'Vimsvc.ha-eventmgr'] Event 1587 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:19:57.068Z [FFC70B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are ready
2015-03-12T05:19:57.103Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are set-resolution capable
2015-03-12T05:19:57.103Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are auto-upgrade capable
2015-03-12T05:19:57.103Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:19:57.119Z [5F640B70 verbose 'Default' opID=11214005-000008EB user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:19:58.347Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] New MKS connection count: 1
2015-03-12T05:19:58.354Z [5F681B70 verbose 'Default' opID=11214005-000008EC user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:19:59.354Z [5F681B70 verbose 'Default' opID=11214005-000008ED user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:20:00.006Z [5F040B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 790666 for scsi0:0 is out of range -- 790666,prevBytes = 28186915840 curBytes = 28257285120 prevCommands = 979473curCommands = 979562
2015-03-12T05:20:02.054Z [FFC0D920 verbose 'SoapAdapter'] Responded to service state request
2015-03-12T05:20:20.010Z [606C1B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 881834 for scsi0:0 is out of range -- 881834,prevBytes = 28257285120 curBytes = 28291676672 prevCommands = 979562curCommands = 979601
2015-03-12T05:20:20.764Z [5F681B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12903MB to 9248MHz/12906MB
2015-03-12T05:20:23.208Z [60680B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are not ready
2015-03-12T05:20:23.208Z [60680B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are not ready
2015-03-12T05:20:23.208Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not auto-upgrade capable
2015-03-12T05:20:23.208Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:20:23.208Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not set-resolution capable
2015-03-12T05:20:23.214Z [606C1B70 verbose 'Default' opID=11214005-000008EE user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:20:49.523Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: red
2015-03-12T05:20:49.523Z [5F040B70 info 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Turning off heartbeat checker
2015-03-12T05:20:49.532Z [5F6C2B70 verbose 'Default' opID=11214005-000008EF user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:20:56.094Z [5F6C2B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2015-03-12T05:21:05.571Z [FFC0D920 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1617959300,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:21:05.572Z [FFC0D920 info 'Vimsvc.ha-eventmgr'] Event 1588 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:21:05.572Z [FFC0D920 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1617959300,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:21:05.573Z [FFC0D920 info 'Vimsvc.ha-eventmgr'] Event 1589 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:21:07.121Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are ready
2015-03-12T05:21:07.121Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] CheckHeartbeatMonitor: Reviving the heartbeat monitor in 30 seconds
2015-03-12T05:21:07.133Z [5F040B70 verbose 'Default' opID=11214005-000008F0 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:21:07.166Z [5EE55B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are ready
2015-03-12T05:21:07.227Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are auto-upgrade capable
2015-03-12T05:21:07.227Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:21:07.229Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are set-resolution capable
2015-03-12T05:21:08.133Z [5F6C2B70 verbose 'Default' opID=11214005-000008F1 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:21:08.778Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1768712546,
--> chainId = 1932425588,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.clear.scsi.device.io.latency.improved",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "362911",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "71091",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:21:08.778Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1590 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has improved. I/O latency reduced from 362911 microseconds to 71091 microseconds.
2015-03-12T05:21:09.131Z [FFC70B70 verbose 'Default' opID=11214005-000008F2 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:21:20.010Z [5F040B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 532522 for scsi0:0 is out of range -- 532522,prevBytes = 28291676672 curBytes = 28420014592 prevCommands = 979601curCommands = 979842
2015-03-12T05:21:20.766Z [5F6C2B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12906MB to 9248MHz/12898MB
2015-03-12T05:21:21.382Z [FFC70B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 525137824,
--> chainId = 524,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:21:21.383Z [FFC70B70 info 'Vimsvc.ha-eventmgr'] Event 1591 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:21:37.124Z [606C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: green
2015-03-12T05:21:37.130Z [FFC70B70 verbose 'Default' opID=11214005-000008F3 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:21:38.218Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are not ready
2015-03-12T05:21:38.218Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are not ready
2015-03-12T05:21:38.219Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not auto-upgrade capable
2015-03-12T05:21:38.219Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:21:38.220Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not set-resolution capable
2015-03-12T05:21:38.222Z [606C1B70 verbose 'Default' opID=11214005-000008F4 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:21:39.222Z [5EE55B70 verbose 'Default' opID=11214005-000008F5 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:22:07.126Z [5EE55B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: red
2015-03-12T05:22:07.126Z [5EE55B70 info 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Turning off heartbeat checker
2015-03-12T05:22:07.131Z [606C1B70 verbose 'Default' opID=11214005-000008F6 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:22:16.975Z [FFC0D920 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 778922358,
--> chainId = 1953722216,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:22:16.975Z [FFC0D920 info 'Vimsvc.ha-eventmgr'] Event 1592 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:22:18.011Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are ready
2015-03-12T05:22:18.011Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] CheckHeartbeatMonitor: Reviving the heartbeat monitor in 30 seconds
2015-03-12T05:22:18.011Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are ready
2015-03-12T05:22:18.023Z [60680B70 verbose 'Default' opID=11214005-000008F7 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:22:18.026Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are set-resolution capable
2015-03-12T05:22:18.026Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are auto-upgrade capable
2015-03-12T05:22:18.026Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:22:19.021Z [5F040B70 verbose 'Default' opID=11214005-000008F8 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:22:20.007Z [5F640B70 warning 'Statssvc.vim.PerformanceManager'] Calculated read I/O size 548074 for scsi0:0 is out of range -- 548074,prevBytes = 789040922624 curBytes = 789100662784 prevCommands = 21522868curCommands = 21522977
2015-03-12T05:22:20.007Z [5F640B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 911881 for scsi0:0 is out of range -- 911881,prevBytes = 28420014592 curBytes = 28468344320 prevCommands = 979842curCommands = 979895
2015-03-12T05:22:20.767Z [5F6C2B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12898MB to 9248MHz/12902MB
2015-03-12T05:22:23.691Z [5F040B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1608945444,
--> chainId = 1615873576,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3759",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "82233",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:22:23.692Z [5F040B70 info 'Vimsvc.ha-eventmgr'] Event 1593 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3759 microseconds to 82233 microseconds.
2015-03-12T05:22:26.592Z [5F040B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1608945444,
--> chainId = 1615873576,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3759",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "605387",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:22:26.592Z [5F040B70 info 'Vimsvc.ha-eventmgr'] Event 1594 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3759 microseconds to 605387 microseconds.
2015-03-12T05:22:26.651Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1701344288,
--> chainId = 1818326560,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3759",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "1217153",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:22:26.651Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1595 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3759 microseconds to 1217153 microseconds.
2015-03-12T05:22:27.408Z [5F040B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2015-03-12T05:22:29.095Z [5F040B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1635196973,
--> chainId = 1866888297,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.clear.scsi.device.io.latency.improved",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "1217153",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "242917",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:22:29.095Z [5F040B70 info 'Vimsvc.ha-eventmgr'] Event 1596 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has improved. I/O latency reduced from 1217153 microseconds to 242917 microseconds.
2015-03-12T05:22:40.005Z [5F040B70 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 615347 for scsi0:0 is out of range -- 615347,prevBytes = 28468344320 curBytes = 28661563392 prevCommands = 979895curCommands = 980209
2015-03-12T05:22:45.382Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 41,
--> chainId = 1,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:22:45.382Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1597 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:22:48.013Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: green
2015-03-12T05:22:48.019Z [5F6C2B70 verbose 'Default' opID=11214005-000008F9 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:23:08.221Z [60680B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are not ready
2015-03-12T05:23:08.221Z [60680B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are not ready
2015-03-12T05:23:08.223Z [606C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not auto-upgrade capable
2015-03-12T05:23:08.223Z [606C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:23:08.223Z [5F6C2B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not set-resolution capable
2015-03-12T05:23:08.227Z [606C1B70 verbose 'Default' opID=11214005-000008FA user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:23:09.225Z [5F681B70 verbose 'Default' opID=11214005-000008FB user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:23:18.014Z [606C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: red
2015-03-12T05:23:18.014Z [606C1B70 info 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Turning off heartbeat checker
2015-03-12T05:23:18.020Z [5F040B70 verbose 'Default' opID=11214005-000008FC user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:23:20.769Z [5F681B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12902MB to 9248MHz/12901MB
2015-03-12T05:23:40.550Z [5F681B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1348822881,
--> chainId = 761820257,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:23:40.550Z [5F681B70 info 'Vimsvc.ha-eventmgr'] Event 1598 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:23:41.646Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] CheckHeartbeatMonitor: Reviving the heartbeat monitor in 30 seconds
2015-03-12T05:23:41.646Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are ready
2015-03-12T05:23:41.647Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are ready
2015-03-12T05:23:41.656Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are auto-upgrade capable
2015-03-12T05:23:41.656Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:23:41.657Z [60680B70 verbose 'Default' opID=11214005-000008FD user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:23:41.658Z [FFC70B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are set-resolution capable
2015-03-12T05:23:42.661Z [5F040B70 verbose 'Default' opID=11214005-000008FE user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:23:43.661Z [FFC0D920 verbose 'Default' opID=11214005-000008FF user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:23:58.735Z [5F640B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2015-03-12T05:24:01.142Z [FFC0D920 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1702259060,
--> chainId = 1701669204,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.clear.scsi.device.io.latency.improved",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "242917",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "47070",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:24:01.143Z [FFC0D920 info 'Vimsvc.ha-eventmgr'] Event 1599 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has improved. I/O latency reduced from 242917 microseconds to 47070 microseconds.
2015-03-12T05:24:10.567Z [FFC0D920 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 0,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3773",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "85947",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:24:10.567Z [FFC0D920 info 'Vimsvc.ha-eventmgr'] Event 1600 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3773 microseconds to 85947 microseconds.
2015-03-12T05:24:11.647Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: green
2015-03-12T05:24:11.656Z [606C1B70 verbose 'Default' opID=11214005-00000900 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:24:12.213Z [5F6C2B70 verbose 'Hostsvc.DvsManager'] PersistAllDvsInfo called
2015-03-12T05:24:13.084Z [FFC0D920 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1615066380,
--> chainId = 1621175520,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3773",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "176361",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:24:13.084Z [FFC0D920 info 'Vimsvc.ha-eventmgr'] Event 1601 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3773 microseconds to 176361 microseconds.
2015-03-12T05:24:20.772Z [5F6C2B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12901MB to 9248MHz/12905MB
2015-03-12T05:24:31.501Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 5194/5194 bytes
2015-03-12T05:24:31.501Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD / (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.512Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 241/241 bytes
2015-03-12T05:24:31.512Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /default.js (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.538Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 4451/4451 bytes
2015-03-12T05:24:31.538Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /en/welcomeRes.js (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.543Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 4898/4898 bytes
2015-03-12T05:24:31.543Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /default.css (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.545Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 2984/2984 bytes
2015-03-12T05:24:31.545Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /watermark.js (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.551Z [5F6C2B70 verbose 'WelcomePageCustomizer'] HandleGetDynData invoked for path '/dyndata.js'
2015-03-12T05:24:31.551Z [5F6C2B70 verbose 'WelcomePageCustomizer'] HandleGetDynData sending data '
-->
--> '
2015-03-12T05:24:31.551Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 2/2 bytes
2015-03-12T05:24:31.558Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 793/793 bytes
2015-03-12T05:24:31.558Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /print.css (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.564Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 544/544 bytes
2015-03-12T05:24:31.564Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /background.jpeg (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.577Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 2572/2572 bytes
2015-03-12T05:24:31.577Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /banner.png (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.583Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 109/109 bytes
2015-03-12T05:24:31.583Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /bullet.png (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.585Z [5F6C2B70 verbose 'Solo.HttpSvc.HTTPService'] HTTP Response: Auto-completing at 54275/54275 bytes
2015-03-12T05:24:31.585Z [5F6C2B70 verbose 'HTTP server'] Sent response for HEAD /watermark.png (from /usr/lib/vmware/hostd/docroot/)
2015-03-12T05:24:31.643Z [5F6C2B70 verbose 'HTTP server'] Sent NotFound response for GET /favicon.ico
2015-03-12T05:24:53.228Z [606C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are not ready
2015-03-12T05:24:53.228Z [606C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are not ready
2015-03-12T05:24:53.230Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not auto-upgrade capable
2015-03-12T05:24:53.230Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:24:53.230Z [60680B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are not set-resolution capable
2015-03-12T05:24:53.233Z [5F681B70 verbose 'Default' opID=11214005-00000901 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:24:54.234Z [606C1B70 verbose 'Default' opID=11214005-00000902 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:25:02.076Z [5EE55B70 verbose 'SoapAdapter'] Responded to service state request
2015-03-12T05:25:11.650Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: red
2015-03-12T05:25:11.650Z [FFC0D920 info 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Turning off heartbeat checker
2015-03-12T05:25:11.655Z [5EE55B70 verbose 'Default' opID=11214005-00000903 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:25:20.774Z [5EE55B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12905MB to 9248MHz/12903MB
2015-03-12T05:25:26.314Z [5EE55B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 825110831,
--> chainId = 808464928,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:25:26.314Z [5EE55B70 info 'Vimsvc.ha-eventmgr'] Event 1602 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:25:26.315Z [5F681B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1768712546,
--> chainId = 1932425588,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3774",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "2379314",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:25:26.315Z [5F681B70 info 'Vimsvc.ha-eventmgr'] Event 1603 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3774 microseconds to 2379314 microseconds.
2015-03-12T05:25:26.315Z [5EE55B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1602,
--> chainId = 1602,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:25:26.315Z [5EE55B70 info 'Vimsvc.ha-eventmgr'] Event 1604 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:25:26.413Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] guest operations are ready
2015-03-12T05:25:26.413Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] CheckHeartbeatMonitor: Reviving the heartbeat monitor in 30 seconds
2015-03-12T05:25:26.419Z [60680B70 verbose 'Default' opID=11214005-00000904 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:25:26.493Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] interactive guest operations are ready
2015-03-12T05:25:30.016Z [5F640B70 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2015-03-12T05:25:38.386Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1592911720,
--> chainId = 1592911720,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.timedout",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = (vim.event.DatastoreEventArgument) {
--> dynamicType = <unset>,
--> name = "Server",
--> datastore = 'vim.Datastore:54f5e9f8-c35be4c2-38e8-40f2e9d5ae78',
--> },
--> }
--> ],
--> objectId = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> objectType = "vim.Datastore",
--> objectName = "Server",
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:25:38.387Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1605 : Lost access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2015-03-12T05:25:56.415Z [5F640B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: green
2015-03-12T05:25:56.426Z [FFC0D920 verbose 'Default' opID=11214005-00000905 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:26:15.830Z [FFC70B70 verbose 'Default' opID=11214005-00000907 user=root] AdapterServer: target='vmodl.query.PropertyCollector:session[9f09af08-5d81-abde-3d35-1db914f6d4aa]52b1f2af-319c-ec01-8d79-2e246248c89c', method='waitForUpdates'
2015-03-12T05:26:15.832Z [5F6C2B70 verbose 'Default' opID=11214005-00000906 user=root] AdapterServer: target='vmodl.query.PropertyCollector:session[9f09af08-5d81-abde-3d35-1db914f6d4aa]52b1f2af-319c-ec01-8d79-2e246248c89c', method='cancelWaitForUpdates'
2015-03-12T05:26:15.832Z [FFC70B70 info 'Solo.Vmomi' opID=11214005-000008C9 user=root] Activation [N5Vmomi10ActivationE:0x60538bd0] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:session[9f09af08-5d81-abde-3d35-1db914f6d4aa]52b1f2af-319c-ec01-8d79-2e246248c89c]
2015-03-12T05:26:15.832Z [FFC70B70 verbose 'Solo.Vmomi' opID=11214005-000008C9 user=root] Arg version:
--> "1"
2015-03-12T05:26:15.832Z [FFC70B70 info 'Solo.Vmomi' opID=11214005-000008C9 user=root] Throw vmodl.fault.RequestCanceled
2015-03-12T05:26:15.832Z [FFC70B70 info 'Solo.Vmomi' opID=11214005-000008C9 user=root] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2015-03-12T05:26:15.832Z [FFC70B70 info 'Solo.Vmomi' opID=11214005-00000907 user=root] Activation [N5Vmomi10ActivationE:0x6024a558] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:session[9f09af08-5d81-abde-3d35-1db914f6d4aa]52b1f2af-319c-ec01-8d79-2e246248c89c]
2015-03-12T05:26:15.832Z [FFC70B70 verbose 'Solo.Vmomi' opID=11214005-00000907 user=root] Arg version:
--> "1"
2015-03-12T05:26:15.832Z [FFC70B70 info 'Solo.Vmomi' opID=11214005-00000907 user=root] Throw vmodl.fault.RequestCanceled
2015-03-12T05:26:15.832Z [FFC70B70 info 'Solo.Vmomi' opID=11214005-00000907 user=root] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2015-03-12T05:26:15.833Z [5F681B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x6036ab60, h:49, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2015-03-12T05:26:15.837Z [5F681B70 verbose 'Default' opID=11214005-00000908 user=root] AdapterServer: target='vmodl.query.PropertyCollector:session[9f09af08-5d81-abde-3d35-1db914f6d4aa]52b1f2af-319c-ec01-8d79-2e246248c89c', method='waitForUpdates'
2015-03-12T05:26:20.776Z [FFC70B70 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 9248MHz/12903MB to 9248MHz/12899MB
2015-03-12T05:26:26.417Z [5F681B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Updating current heartbeatStatus: red
2015-03-12T05:26:26.423Z [5F640B70 verbose 'Default' opID=11214005-00000909 user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:26:30.606Z [FFC70B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1619133472,
--> chainId = 1619133472,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3781",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "6534829",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:26:30.606Z [FFC70B70 info 'Vimsvc.ha-eventmgr'] Event 1606 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3781 microseconds to 6534829 microseconds.
2015-03-12T05:26:30.607Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1768712546,
--> chainId = 1932425588,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3781",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "23072056",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:26:30.607Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1607 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3781 microseconds to 23072056 microseconds.
2015-03-12T05:26:30.607Z [FFC70B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1619133472,
--> chainId = 1619133472,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.scsi.device.io.latency.high",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "naa.60014059f3135acd7798d31dbda95fd2",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "3787",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "3",
--> value = "48259310",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:26:30.607Z [FFC70B70 info 'Vimsvc.ha-eventmgr'] Event 1608 : Device naa.60014059f3135acd7798d31dbda95fd2 performance has deteriorated. I/O latency increased from average value of 3787 microseconds to 48259310 microseconds.
2015-03-12T05:26:30.726Z [5F6C2B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
--> dynamicType = <unset>,
--> key = 1870016371,
--> chainId = 1701672300,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> dynamicType = <unset>,
--> name = "localhost.localdomain",
--> host = 'vim.HostSystem:ha-host',
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.vmfs.heartbeat.recovered",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "1",
--> value = "54f5e9f8-c35be4c2-38e8-40f2e9d5ae78",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "2",
--> value = "Server",
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null,
--> }
2015-03-12T05:26:30.726Z [5F6C2B70 info 'Vimsvc.ha-eventmgr'] Event 1609 : Successfully restored access to volume 54f5e9f8-c35be4c2-38e8-40f2e9d5ae78 (Server) following connectivity issues.
2015-03-12T05:26:31.316Z [FFC0D920 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are set-resolution capable
2015-03-12T05:26:31.316Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Tools are auto-upgrade capable
2015-03-12T05:26:31.316Z [5F040B70 verbose 'Vmsvc.vm:/vmfs/volumes/54f5e9f8-c35be4c2-38e8-40f2e9d5ae78/Server/Server.vmx'] Skipped tools ManifestInfo update.
2015-03-12T05:26:31.323Z [60680B70 verbose 'Default' opID=11214005-0000090A user=root] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='waitForUpdates'
2015-03-12T05:26:31.630Z [5EE55B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
-->