VRA 8.1 Web interface not loading after networking issue

This post was originally published on this site

We are currently experiencing an issue with our vra 8.1 appliance.  We had a network issue over the weekend that caused the vsan node that our appliance was running on to lose connectivity which seems to have caused a file system corruption on the appliance.  We were able to run fsck on the appliance and get the appliance back on the network by following this kb article https://kb.vmware.com/s/article/2149838.

 

However, we are still unable to get the web service interface to respond when we try to access it using a web browser.  Unfortunately, we are also having difficulty finding any documentation/kb articles etc. that might be helpful in our efforts to restore our services.  (and no we don’t have a backup of the appliance because it is a poc environment..)

 

I’m not sure exactly what I should be looking for/at. 

 

I am getting these results from various kube commands.

 

root@vraa [ ~ ]# kubectl -n prelude get pods

The connection to the server localhost:8080 was refused – did you specify the right host or port?

root@vraa [ ~ ]#

 

 

root@vraa [ ~ ]# ps -ef |grep kube

root      9608     1  2 13:46 ?        00:01:53 /usr/bin/kubelet –bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf –kubeconfig=/etc/kubernetes/kubel                   et.conf –config=/var/lib/kubelet/config.yaml –allowed-unsafe-sysctls=net.* –cgroup-driver=cgroupfs –network-plugin=cni –pod-infra-container-image=vmware/pau                   se:3.1 –resolv-conf=/run/systemd/resolve/resolv.conf –tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE                   _ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 –allowed-unsafe                   -sysctls=net.ipv4.tcp_keepalive_time,net.ipv4.tcp_keepalive_intvl,net.ipv4.tcp_keepalive_probes

root     10007  9956  0 13:46 ?        00:00:03 kube-controller-manager –allocate-node-cidrs=true –authentication-kubeconfig=/etc/kubernetes/controller-manager                   .conf –authorization-kubeconfig=/etc/kubernetes/controller-manager.conf –bind-address=127.0.0.1 –client-ca-file=/etc/kubernetes/pki/ca.crt –cluster-cidr=10.2                   44.0.0/22 –cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt –cluster-signing-key-file=/etc/kubernetes/pki/ca.key –controllers=*,bootstrapsigner,tokenclean                   er –kubeconfig=/etc/kubernetes/controller-manager.conf –leader-elect=true –node-cidr-mask-size=24 –node-monitor-grace-period=20s –node-monitor-period=5s –p                   od-eviction-timeout=30s –requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt –root-ca-file=/etc/kubernetes/pki/ca.crt –service-account-private                   -key-file=/etc/kubernetes/pki/sa.key –terminated-pod-gc-threshold=5 –tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_S                   HA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 –                   -use-service-account-credentials=true

root     10015  9957  0 13:46 ?        00:00:42 kube-scheduler –bind-address=127.0.0.1 –kubeconfig=/etc/kubernetes/scheduler.conf –leader-elect=true

root     28893 28682  0 15:14 pts/0    00:00:00 grep –color=auto kube

root@vraa [ ~ ]# docker ps | grep kube-apiserver

54624cbca2e3        vmware/pause:3.1    “/pause”                 2 hours ago         Up 2 hours                              k8s_POD_kube-apiserver-vraa.mathwork                   s.com_kube-system_28839991b8fc3b74a6425edef23a542e_0

 

root@vraa [ ~ ]# docker ps -a

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES

4029351efb47        b102b1c1814a        “etcd –advertise-cl…”   4 minutes ago       Exited (1) 4 minutes ago                       k8s_etcd_etcd-vraa.mathworks.com_kube-system_20595ace0b8d3581907f0866be350161_23

6913147d1efc        30e0ba87cac4        “kube-apiserver –ad…”   4 minutes ago       Exited (1) 4 minutes ago                       k8s_kube-apiserver_kube-apiserver-vraa.mathworks.com_kube-system_28839991b8fc3b74a6425edef23a542e_23

d97bb9f54541        dfd6cf8c8a40        “kube-controller-man…”   2 hours ago         Up 2 hours                                     k8s_kube-controller-manager_kube-controller-manager-vraa.mathworks.com_kube-system_f1fcf01628f38c636bd150fddffff503_0

eb64fdbde524        1c8152ca81c7        “kube-scheduler –bi…”   2 hours ago         Up 2 hours                                     k8s_kube-scheduler_kube-scheduler-vraa.mathworks.com_kube-system_7c41b41a1094ea7470703cdc2c891adc_0

df4607cd2814        vmware/pause:3.1    “/pause”                 2 hours ago         Up 2 hours                                     k8s_POD_kube-scheduler-vraa.mathworks.com_kube-system_7c41b41a1094ea7470703cdc2c891adc_0

b29c38f2663f        vmware/pause:3.1    “/pause”                 2 hours ago         Up 2 hours                                     k8s_POD_kube-controller-manager-vraa.mathworks.com_kube-system_f1fcf01628f38c636bd150fddffff503_0

54624cbca2e3        vmware/pause:3.1    “/pause”                 2 hours ago         Up 2 hours                                     k8s_POD_kube-apiserver-vraa.mathworks.com_kube-system_28839991b8fc3b74a6425edef23a542e_0

f77f7f82cef5        vmware/pause:3.1    “/pause”                 2 hours ago         Up 2 hours                                     k8s_POD_etcd-vraa.mathworks.com_kube-system_20595ace0b8d3581907f0866be350161_0

root@vraa [ ~ ]#

 

Any thoughts and advice would be much appreciated.

 

Thanks

Dan

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.