IBM Application Performance Management - Why no start ?

We saw an interesting issue with a pair of IBM APM agents today, each on a different AIX LPAR.

Each time we tried to start the agent: -

/opt/IBM/ITM/bin/itmcmd agent start yn

( where yn is a particular agent - ITCAM Agent for WebSphere Applications )

we saw: -

Processing. Please wait...
Starting ITCAM Agent for WebSphere Applications ...
KCIIN0198E Unable to start agent. Please, check log file.

When I dug further, I found this: -

java.lang.UnsatisfiedLinkError: net (A file or directory in the path name does not exist.)

in one of the logs in /opt/IBM/ITM/logs.

This most closely resembled this IBM Technote: -

ITCAM for WebSphere agent error

which suggested that the Java Runtime Environment (JRE) might be corrupted.

Acting on this suggestion, I took a backup of a working JRE from another box running the same version of the yn agent: -

cd /opt/IBM/ITM/JRE/
/opt/freeware/bin/tar cvzf /tmp/itm_java.tar.gz aix523/

scp /tmp/itm_java.tar.gz wasadmin@bpmlpar:/tmp

and restored it in place of the allegedly borked JRE: -

cd /opt/IBM/ITM/JRE/
mv aix523 foobar
/opt/freeware/bin/tar xvzf /tmp/itm_java.tar.gz

and, guess what ..... it worked :-)

/opt/IBM/ITM/bin/itmcmd agent start yn

Processing. Please wait...
Starting ITCAM Agent for WebSphere Applications ...
ITCAM Agent for WebSphere Applications started

/opt/IBM/ITM/bin/cinfo -r

*********** Tue Jul  7 11:53:04 BST 2015 ******************
User: wasadmin Groups: wasadmin staff
Host name : bpm101  Installer Lvl:06.23.03.00
CandleHome: /opt/IBM/ITM
***********************************************************

Host         Prod  PID       Owner     Start     ID    ..Status
bpm101       d4    10027226  wasadmin  11:48:54  None  ...running

bpm101       th    9896038   wasadmin  11:49:12  None  ...running

bpm101       yn    14286896  wasadmin  11:49:22  None  ...running

It goes without saying that (a) your mileage may well vary and (b) if in doubt, please raise a PMR with IBM Support before trying this