vROPS Notifications list empty??

Today i ran into an issue where i found that the notifications list in vROPS was coming up empty with an error message that dissapeared very quikly :

I made a support request with vmware and after some troubles with the root passwords and expiration issues we quickly went to troubleshooting the actual issue.

We first made snapshots of all vrops nodes (do not forget this!) and then we connected with SSH to the MASTER node. To cleanup the alert definition we first needed to find the identifier for which we used this command :

$VMWARE_PYTHON_BIN $ALIVE_BASE/cassandra/apache-cassandra-*/bin/cqlsh.py --ssl --cqlshrc $ALIVE_BASE/user/conf/cassandra/cqlshrc -e 'PAGING OFF; SELECT * from globalpersistence.notificationplugin' > /tmp/notifications.txt

This will put the complete list of notifications in a txt file which you can copy out with WinSCP and open locally with Notepad++ to easily search and find the identifier which in my case was this :

You need the part which starts with NotificationRule. First we tried deleting the culprit with a curl command for which we first needed to enable basic authentication following this KB article :

https://kb.vmware.com/s/article/77271?lang=en_US&queryTerm=how%20to%20enable%20basic%20authentication%20in%20vrops

open up the config file :

vi /usr/lib/vmware-vcops/user/conf/api-conf.properties

service api restart

Now we can went ahead and tried to delete the notification with this command but unfortunately it failed.

curl -v -X DELETE -k --user admin https://IPADDRESS/suite-api/api/notifications/rules/NotificationRule.c837603b-139e-42a0-8cca-20c10d001b78

Our second attempt at removing the rule is with a Cassandra query, first we opened up the Cassandra command line

$VMWARE_PYTHON_BIN $ALIVE_BASE/cassandra/apache-cassandra-*/bin/cqlsh --ssl --cqlshrc $ALIVE_BASE/user/conf/cassandra/cqlshrc

Then we ran the query (offcourse with our rule identifier we looked up before!):

DELETE from globalpersistence.notificationplugin where namespace='notificationplugin' and classtype='notificationplugin' and key='NotificationRule.c837603b-139e-42a0-8cca-20c10d001b78';

This made sure the rule was deleted and we could hop back into the vRops UI to see that we can actually get the list of notifications again! Hope this helps others who also run into this weird “bug”.

The original article was posted on: www.hollebollevsan.nl