NSX Installation – XXX eam.agent.enable.label not found XXX

This post was originally published on this site

Problem Today I ran into the following error message while I was trying to prepare the ESXi hosts for VXLAN (installing VIBs) on a production cluster. The install would immediately fail with no real good error messages tasks and events displayed “Cannot complete the operation. See the event log for details.” Task: XXX eam.agent.enable.label not found XXX.

from the perspective of the “Installation and Upgrade”, “Host Preparation” looks like the image below

I already tried to put into maintenance mode the Host ESXi involved, reboot and click on RESOLVE in ACTIONS several times but without success. I have checked NTP, TIME, DNS and so on, on the entire environment, but everything seems fine and properly configured. Then under the Networking area I saw that the vDS at the corresponding ESXi hosts was “Out of sync

I also tried with Rectify vSphere Distribuited Switch on Host but didn’t solve so I tried to:- Remove the host from the cluster- Remove the host from the vCenter server- Remove NSX DV_Switch- Add the host to vCenter- Add NSX DV_Switch- Add the host to the cluster.But the result doesn’t change …… After digging a little bit deeper into NSX Manager logs and vCenter logs, I was able to find the following error message in my eam.log.



2019-12-12T11:50:48.052Z | ERROR | agent-4 | AuditedJob.java | 75 | JOB FAILED: [#1094199235] EnableDisableAgentJob(AgentImpl(ID:'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03')), com.vmware.eam.EamException: VibInstallationFailed
2019-12-12T13:43:20.165Z | INFO | vlsi | AgencyIssueHandler.java | 93 | Resolving AgencyImpl(ID:'Agency:adf028a8-f028-4c48-974a-0fcd60082b28:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03'): issues=int[] [
25
], unknown=null
2019-12-12T13:43:20.166Z | INFO | vlsi | AgentIssueHandler.java | 591 | Resolving AgentImpl(ID:'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03') issues: issues=[I@337b2457, unknown=null
2019-12-12T13:43:20.166Z | INFO | vlsi | IssueHandler.java | 197 | Issue removed: AgentIssueHandler:AgentImpl(ID:'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03')
eam.issue.VibNotInstalled {
description = '
99
'NoneType' object has no attribute 'Copy'
',
time = 2019-12-12 11:50:48,024,
key = 25,
agency = 'Agency:adf028a8-f028-4c48-974a-0fcd60082b28:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
solutionId = 'VSPHERE.LOCALAdministrator',
agencyName = '_NSX_146_Cluster-LAB_VMware Network Fabric',
solutionName = 'Administrator vsphere.local',
agentName = 'VMware Network Fabric (6)',
agent = 'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
host = 'HostSystem:host-10:BAAA27E9-8AB1-4C50-A69D-1B7F9EDB480D',
hostName = 'r730n4.local.lab',
}
Location: AgentIssueHandler.java:resolve:687
2019-12-12T13:43:20.166Z | INFO | vlsi | IssueHandler.java | 121 | Updating issues:
New issues:
[]
Removed issues: [
eam.issue.VibNotInstalled {
description = '
99
'NoneType' object has no attribute 'Copy'
',
time = 2019-12-12 11:50:48,024,
key = 25,
agency = 'Agency:adf028a8-f028-4c48-974a-0fcd60082b28:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
solutionId = 'VSPHERE.LOCALAdministrator',
agencyName = '_NSX_146_Cluster-LAB_VMware Network Fabric',
solutionName = 'Administrator vsphere.local',
agentName = 'VMware Network Fabric (6)',
agent = 'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
host = 'HostSystem:host-10:BAAA27E9-8AB1-4C50-A69D-1B7F9EDB480D',
hostName = 'r730n4.local.lab',
}
]
2019-12-12T13:43:20.188Z | INFO | vlsi | IssueHandler.java | 121 | Updating issues:
New issues:
[]
Removed issues: [
eam.issue.VibNotInstalled {
description = '
99
'NoneType' object has no attribute 'Copy'
',
time = 2019-12-12 11:50:48,024,
key = 25,
agency = 'Agency:adf028a8-f028-4c48-974a-0fcd60082b28:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
solutionId = 'VSPHERE.LOCALAdministrator',
agencyName = '_NSX_146_Cluster-LAB_VMware Network Fabric',
solutionName = 'Administrator vsphere.local',
agentName = 'VMware Network Fabric (6)',
agent = 'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
host = 'HostSystem:host-10:BAAA27E9-8AB1-4C50-A69D-1B7F9EDB480D',
hostName = 'r730n4.local.lab',
}
]
2019-12-12T13:43:30.204Z | INFO | compute-res-0 | ClusterVibJob.java | 192 | No hosts needs maintenance mode in compute resource: domain-c7
2019-12-12T13:43:33.762Z | INFO | host-10-4 | VcPatchManager.java | 356 | Scan result on VcHostSystem(ID: host-10):
ScanResult {
errorCode = 99,
errorMessage = '
99
'NoneType' object has no attribute 'Copy'
',
success = false,
vibUrl = https://10.254.7.241/bin/vdn/vibs-6.4.6/6.0-14762108/vxlan.zip,
pendingReboot = false,
requiresReboot = false,
requiresMaintenanceMode = false,
requiresHostdRestart = false,
installed = false,
bulletins = [Vib [VibHeader [id=VMware_bootbank_esx-nsxv_6.0.0-0.0.14762108, VibId [name=esx-nsxv, version=6.0.0-0.0.14762108], vendor=VMware, summary=NSX datapath and host tools, release date=Mon Sep 30 12:14:44 UTC 2019], payload=https://vcenter.local.lab:443/eam/vib?id=6c4f942e-7e0c-41df-803f-4869809d4320]],
}
2019-12-12T13:43:33.762Z | ERROR | host-10-4 | VcPatchManager.java | 358 | PatchManager operation failed:

1.50

VMware_locker_tools-light_6.0.0-2.43.4192238
installed
tools-light
6.0.0-2.43.4192238
VMware
2017-01-12 10:37:55.871855+00:00
True
False

esx-version
False


False


False
False



False
False


False
False


False



VMware_bootbank_esx-nsxv_6.0.0-0.0.14762108
uninstalled
esx-nsxv
6.0.0-0.0.14762108
VMware

True
False

esx-base
False


vmkapi_2_3_0_0
False


nsx-api
False


False


False
False



False
True


False
False


False



False
False


99
'NoneType' object has no attribute 'Copy'



2019-12-12T13:43:33.763Z | ERROR | host-10-4 | VibJob.java | 730 | Unhandled response code: 99
2019-12-12T13:43:33.763Z | ERROR | host-10-4 | VibJob.java | 736 | PatchManager operation failed with error code: 99
With VibUrl: https://10.254.7.241/bin/vdn/vibs-6.4.6/6.0-14762108/vxlan.zip
2019-12-12T13:43:33.763Z | INFO | host-10-4 | IssueHandler.java | 121 | Updating issues:
New issues:
[
eam.issue.VibNotInstalled {
description = 'XXX uninitialized',
time = 2019-12-12 13:43:33,763,
key = 27,
agency = 'Agency:adf028a8-f028-4c48-974a-0fcd60082b28:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
solutionId = 'VSPHERE.LOCALAdministrator',
agencyName = '_NSX_146_Cluster-LAB_VMware Network Fabric',
solutionName = 'Administrator vsphere.local',
agentName = 'VMware Network Fabric (6)',
agent = 'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
host = 'HostSystem:host-10:BAAA27E9-8AB1-4C50-A69D-1B7F9EDB480D',
hostName = 'r730n4.local.lab',
}
]
Removed issues: []
2019-12-12T13:43:33.763Z | INFO | host-10-4 | IssueHandler.java | 121 | Updating issues:
New issues:
[
eam.issue.VibNotInstalled {
description = 'XXX uninitialized',
time = 2019-12-12 13:43:33,763,
key = 27,
agency = 'Agency:adf028a8-f028-4c48-974a-0fcd60082b28:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
solutionId = 'VSPHERE.LOCALAdministrator',
agencyName = '_NSX_146_Cluster-LAB_VMware Network Fabric',
solutionName = 'Administrator vsphere.local',
agentName = 'VMware Network Fabric (6)',
agent = 'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03',
host = 'HostSystem:host-10:BAAA27E9-8AB1-4C50-A69D-1B7F9EDB480D',
hostName = 'r730n4.local.lab',
}
]
Removed issues: []
2019-12-12T13:43:33.767Z | ERROR | host-10-4 | AgentImpl.java | 2413 | AgentImpl(ID:'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03') VIB installation failed: https://10.254.7.241/bin/vdn/vibs-6.4.6/6.0-14762108/vxlan.zip
2019-12-12T13:43:33.767Z | INFO | host-10-4 | VcComputeResource.java | 505 | VcClusterComputeResource(ID: domain-c7) setting required agent count in VC to 0
2019-12-12T13:43:33.786Z | ERROR | agent-0 | AuditedJob.java | 75 | JOB FAILED: [#1885632038] EnableDisableAgentJob(AgentImpl(ID:'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03')), com.vmware.eam.EamException: VibInstallationFailed
2019-12-12T13:43:33.792Z | INFO | host-10-4 | VibJob.java | 241 | VIBs installed at AgentImpl(ID:'Agent:c05fcd62-1865-475c-a63e-ea552503b4ad:c4aa57e6-93f2-4bbd-af53-8a296ebe1e03') (VcHostSystem(ID: host-10))
2019-12-12T13:43:33.792Z | INFO | host-10-4 | VibJob.java | 245 | All agents checked for pending VIB actions.
2019-12-12T13:43:33.792Z | INFO | compute-res-0 | ClusterVibJob.java | 433 | [] are to be rebooted.

and looking inside the ESXi host I observed.. that no image profile was present … [InstallationError] No image profile is found on the host or image profile is empty. An image profile is required to install or remove VIBs. To install an image profile, use the esxcli image profile install command. Please refer to the log file for more details.



[root@r7XXXX:/vmfs/volumes/5ca5de41-90ef28e5-5a93-149ecf45a06f/VIBs] esxcli software vib list
Name Version Vendor Acceptance Level Install Date
----------- ------------------ ------ ---------------- ------------
tools-light 6.0.0-2.43.4192238 VMware VMwareCertified 2017-01-12
[root@r7XXXX:/vmfs/volumes/5ca5de41-90ef28e5-5a93-149ecf45a06f/VIBs]

then I followed the procedure showed in this site, describe step-by-step below here.

Solution This can be fixed by replacing the corrupt image file and replacing with a known good one from another host. Down here, how to do it.

  1. On the working ESXi host, copy the following image file: imgdb.tgzcp /bootbank/imgdb.tgz /vmfs/volumes/
  2. On the corrupt host, copy the file imgdb.tgz from the working host to /tmp:cp /vmfs/volumes//imgdb.tgz /tmp
  3. Change Directories to /tmpcd /tmp
  4. Extract file you just copiedtar -xzf imgdb.tgz
  5. Copy the working profile files to the profile directorycp /tmp/var/db/esximg/profiles/* /var/db/esximg/profiles/
  6. Copy the working VIBs to the VIB repositorycp /tmp/var/db/esximg/vibs/* /var/db/esximg/vibs/
  7. Remove the corrupt imgdb.tgz from the bootbankrm /bootbank/imgdb.tgz
  8. Move the working copy of imgdb.tgz into the bootbankcp /tmp/imgdb.tgz /bootbank/
  9. Make Config Backup/sbin/auto-backup.sh
  10. Reboot the hostreboot

When the ESXi host is up and running again, the system install the missed eam.agent and VIBs.

Clicking Resolve in the ACTIONS menu and the system automatically detects the VIBs.

That’s it.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.