IBM UrbanCode Deploy - Adding Agents - Or, Waiting for provisioned node

Continuing on my voyage of discovery with IBM UrbanCode Deploy (UCD), I was having all sorts of fun and games adding a new agent to a "clean" installation of Red Hat Enterprise Linux 6.6.

I chose to add the agent via Resources > Agents > Add New Agent, manually entering details such as hostname, SSH port, user ID and password: -

validating the Agent Installation Properties: -

Note that I'd previously installed IBM Java 1.7 as validated here: -

 /opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470_27sr2-20141026_01(SR2))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64 Compressed References 20141017_217728 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR2_20141017_1632_B217728
JIT  - tr.r13.java_20141003_74587.01
GC   - R27_Java727_SR2_20141017_1632_B217728_CMPRSS
J9CL - 20141017_217728)
JCL - 20141004_01 based on Oracle 7u71-b13

I verified that the installation had completed successfully by checking the appropriate log on the UCD server: -

view /opt/ibm-ucd/server/var/log/agent-install.out

...

install-non-interactive:
    [unzip] Expanding: /tmp/agent-upgrade/overlay.zip into /tmp/agent-install-4316336539585453701.tmp
     [echo]
     [echo] Installing IBM UrbanCode Deploy Agent (non-interactive)
     [echo] The specified directory does not exist. Do you want to create it? Y,n [Default: Y]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: Y
     [echo]
     [echo] Installing Agent to: /opt/ibm-ucd/agent
     [echo] JAVA_HOME: /opt/IBM/Java/jre
     [echo] Enter the hostname or address of the server the agent will connect to. [Default: localhost]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: localhost
     [echo] Enter the agent communication port for the server. [Default: 7918]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: 7918
     [echo] Do you want to configure another failover server connection? y,N [Default: N]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: N
     [echo] The agent can be added to one or more teams when it first connects to the server. Changing this setting after initial connection to the server will not have any effect.
     [echo] Enter teams to add this agent to, separated by commas. [Default: None]
     [echo] Warning: Installer prompting for input in non-interactive mode.
     [copy] Copying 2 files to /opt/ibm-ucd/agent/conf
     [copy] Copying 1 file to /opt/ibm-ucd/agent/properties
[propertyfile] Creating new property file: /opt/ibm-ucd/agent/conf/agent/installed.properties
[propertyfile] Updating property file: /opt/ibm-ucd/agent/conf/agent/installed.properties
     [copy] Copying 56 files to /opt/ibm-ucd/agent/lib
     [copy] Copying 4 files to /opt/ibm-ucd/agent/monitor
     [copy] Copying 1621 files to /opt/ibm-ucd/agent/opt
     [copy] Copying 2 files to /opt/ibm-ucd/agent/opt/udclient
    [mkdir] Created dir: /opt/ibm-ucd/agent/bin/init
     [copy] Copying 1 file to /opt/ibm-ucd/agent/bin
     [copy] Copying 3 files to /opt/ibm-ucd/agent/bin
     [copy] Copying 2 files to /opt/ibm-ucd/agent/bin
     [echo] Installed version 6.1.1.0.608443
     [echo] Installer Complete. (press return to exit installer)
     [echo] Warning: Installer prompting for input in non-interactive mode.
   [delete] Deleting directory /tmp/agent-install-4316336539585453701.tmp

BUILD SUCCESSFUL
Total time: 7 seconds

...

Having installed the agent, I was expecting to find the agent up and running, and ready to receive.

Instead, I saw this: -

( Waiting for provisioned node )

When I checked, the agent appeared to be running: -

cat /opt/ibm-ucd/agent/var/log/agent.out 

2014-12-30 15:31:00,943 INFO  com.urbancode.air.agent.AgentWorker - Logging configured
2014-12-30 15:31:00,990 INFO  com.urbancode.air.agent.AgentWorker - Agent version: 6.1.1.0.608443

ps auxw | grep -i java

wasadmin  23880  0.0  1.5 1729324 28996 ?       Sl   15:30   0:00 /opt/IBM/Java/jre/bin/java -Dfile.encoding=UTF-8 -jar /opt/ibm-ucd/agent/monitor/air-monitor.jar /opt/ibm-ucd/agent /opt/ibm-ucd/agent/bin/worker-args.conf 7000 -Djava.io.tmpdir=/opt/ibm-ucd/agent/var/temp
wasadmin  23897  0.5  2.2 1461488 42272 ?       Sl   15:30   0:03 /opt/IBM/Java/jre/bin/java -Dcom.urbancode.air.mw.common.Monitor.port=46470 -Djava.io.tmpdir=/opt/ibm-ucd/agent/var/temp -Xmx256m -Dfile.encoding=UTF-8 -Dconsole.encoding=UTF-8 -Djava.security.properties=/opt/ibm-ucd/agent/conf/agent/java.security -Djava.io.tmpdir=/opt/ibm-ucd/agent/var/temp -jar /opt/ibm-ucd/agent/monitor/air-worker.jar /opt/ibm-ucd/agent/bin/classpath.conf 5000 com.urbancode.air.agent.AgentWorker

and yet nothing is listening on port 7918 : -

netstat -aon | grep 7918

<NOTHING RETURNED>

I dug further into the UCD agent configuration: -

cat /opt/ibm-ucd/agent/conf/agent/installed.properties

#Tue Dec 30 15:31:01 GMT 2014
IBM\ UrbanCode\ Deploy/java.home=/opt/IBM/Java/jre
agent.HttpFailoverHandler.disabled=null
locked/agent.brokerUrl=failover\:(ah3\://was855.uk.ibm.com\:7918,ah3\://localhost\:7918)
locked/agent.home=/opt/ibm-ucd/agent
locked/agent.http.proxy.host=
locked/agent.http.proxy.port=
locked/agent.id=ShbCxatf4gd13mrtGL7P
locked/agent.jms.remote.host=was855.uk.ibm.com
locked/agent.jms.remote.port=7918
locked/agent.keystore=../conf/agent.keystore
locked/agent.keystore.pwd=pbe{Y1NGSlNWQFuHSkFNYFwpTGmxmeT6djKzSAp/m8zIHSI\=}
locked/agent.mutual_auth=false
locked/agent.name=was855.uk.ibm.com
system.default.encoding=UTF-8

and compared this to the configuration of the agent on the actual UCD server: -

cat /opt/ibm-ucd/agent/conf/agent/installed.properties

 
#Tue Dec 23 21:00:27 GMT 2014
IBM\ UrbanCode\ Deploy/java.home=/opt/IBM/Java/jre
agent.HttpFailoverHandler.disabled=null
locked/agent.brokerUrl=failover\:(ah3\://ucd61.uk.ibm.com\:7918,ah3\://localhost\:7918)
locked/agent.home=/opt/ibm-ucd/agent
locked/agent.http.proxy.host=
locked/agent.http.proxy.port=
locked/agent.id=ZzloWWuEcF3FGHkfe6HJ
locked/agent.jms.remote.host=ucd61.uk.ibm.com
locked/agent.jms.remote.port=7918
locked/agent.keystore=../conf/agent.keystore
locked/agent.keystore.pwd=pbe{rGO6c95brUqvVAJ6/myFWKT3M/a3fLNUSTgUNeAQLUI\=}
locked/agent.mutual_auth=false
locked/agent.name=ucd61
system.default.encoding=UTF-8

Can you spot the problem ?

The properties agent.brokerURL and agent.ms.remote.host are, on the UCD server, pointing at  ..... the UCD server, as the agent AND server are on the same host-name.

However, on the standalone agent, they're also pointing at .... the hostname of the agent itself :-(

I changed the properties of the standalone agent: -

locked/agent.brokerUrl=failover\:(ah3\://ucd61.uk.ibm.com\:7918)
locked/agent.jms.remote.host=ucd61.uk.ibm.com

and restarted it: -

/opt/ibm-ucd/agent/bin/agent stop
/opt/ibm-ucd/agent/bin/agent start

and checked the logs: -

cat /opt/ibm-ucd/agent/var/log/agent.out

2014-12-30 16:16:46,105 INFO  com.urbancode.air.agent.AgentWorker - Logging configured
2014-12-30 16:16:46,134 INFO  com.urbancode.air.agent.AgentWorker - Agent version: 6.1.1.0.608443
2014-12-30 16:16:51,608 INFO  com.urbancode.air.devilfish.common.SaContainer - Starting SaContainer...
2014-12-30 16:16:51,727 INFO  com.urbancode.air.devilfish.apps.plugin.PluginRuntimeServer - Plugin Recovery ...
2014-12-30 16:16:51,728 INFO  com.urbancode.air.devilfish.common.SaContainer - Started SaContainer
2014-12-30 16:16:51,728 INFO  com.urbancode.air.agent.AgentWorker - Agent started
2014-12-30 16:16:52,542 INFO  com.urbancode.air.devilfish.apps.agentcontrol.PropertiesCollector - Checking connection to ucd61.uk.ibm.com:8443...
2014-12-30 16:16:52,591 INFO  com.urbancode.air.devilfish.apps.agentcontrol.PropertiesCollector - HTTP connection successful.

and checked that port 7918 is now listening: -

netstat -aon | grep 7918

tcp        0      0 ::ffff:192.168.1.74:38701   ::ffff:192.168.1.71:7918    ESTABLISHED off (0.00/0/0)

and finally verified by the UCD server itself: -

Hmm, interesting or what ?