IT Security - what do I know ?

Well, I'm constantly adding to the list of things that I know ( in full or in part )

To quote Donald Rumsfeld: -

"...because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know…"

Source: Wikipedia

These are some of my sources: -

https://www.feistyduck.com/

https://www.securemac.com/topics/checklist

https://risky.biz/

https://twit.tv/shows/security-now

and these are those who I choose to follow on Twitter: -

Patrick Gray https://twitter.com/riskybusiness

Adam Boileau https://twitter.com/metlstorm?lang=en

Steve Gibson https://twitter.com/sggrc?lang=en

Ivan Ristic https://twitter.com/ivanristic

Scott Helme https://twitter.com/scott_helme?lang=en

Swift On Security https://twitter.com/SwiftOnSecurity

Duo Security https://twitter.com/duosec

etc.

IBM Cloud Private - 502 Bad Gateway

I saw this: -

after resuming three VMs that host my IBM Cloud Private (ICP) cluster.

Assuming that the VM sleep had borked things, I followed this: -

What is the proper way to shut down ICp?

to shut down and restart Docker and Kubelet: -

sudo systemctl stop kubelet

sudo systemctl stop docker

sudo systemctl start kubelet

sudo systemctl start docker

but to no avail.

I even rebooted the VMs ( having again stopped Docker and Kubelet ), but still no dice :-(

Running this: -

sudo journalctl -e -u kubelet

on the Boot node ( VM ) showed me this: -

Feb 28 09:13:54 icpboot.uk.ibm.com hyperkube[856]: W0228 09:13:54.072049     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: I0228 09:14:04.003480     856 kuberuntime_manager.go:500] Container {Name:iam-policy-decision Image:ibmcom/iam-policy-decision:2.1.0.1 Command:[] Args:
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: n:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: I0228 09:14:04.004425     856 kuberuntime_manager.go:739] checking backoff for container "iam-policy-decision" in pod "auth-pdp-bpk7h_kube-system(b5ddc
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:04.190524     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:04.193761     856 kuberuntime_container.go:191] Non-root verification doesn't support non-numeric user (iam)
Feb 28 09:14:14 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:14.313716     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:24 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:24.350819     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:34 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:34.484130     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:44 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:44.564230     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:54 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:54.641818     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:04.763905     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:15 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:15.138129     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:25 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:25.268208     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:35 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:35.303902     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:45 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:45.377637     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:55 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:55.508744     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available

However, it may have just been timing, because, after ~10 minutes, things started to improve.

I again hit the URL: -

https://192.168.1.100:8443/console/

and got a login page and, post-login, this: -

Patience is the key !

WebSphere Application Server to Oracle - Listener Says No

I saw this: -

The test connection operation failed for data source BPM Business Space data source on server nodeagent at node Node1 with the following exception: java.sql.SQLException: IO Error: The Network Adapter could not establish the connection DSRA0010E: SQL State = 08006, Error Code = 17,002. View JVM logs for further details.

whilst testing an IBM BPM database connection to Oracle 12c.

This was my WAS JDBC Datasource configuration: -

jdbc:oracle:thin:@//bpm856.uk.ibm.com:1521/orcl.uk.ibm.com

In the WAS Node Agent log, I saw this: -

[23/02/18 21:13:33:393 GMT] 00000066 DataSourceCon E   DSRA8040I: Failed to connect to the DataSource "".  Encountered java.sql.SQLException: IO Error: The Network Adapter could not establish the connection DSRA0010E: SQL State = 08006, Error Code = 17,002
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:673)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:30)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:564)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:303)
at oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:500)
at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:174)
at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:143)
at com.ibm.ws.rsadapter.DSConfigHelper$1.run(DSConfigHelper.java:1267)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.rsadapter.spi.ServerFunction$6.run(ServerFunction.java:567)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.rsadapter.DSConfigHelper.getPooledConnection(DSConfigHelper.java:1282)
at com.ibm.ws.rsadapter.DSConfigHelper.getPooledConnection(DSConfigHelper.java:1190)
at com.ibm.ws.rsadapter.DSConfigurationHelper.getConnectionFromDSOrPooledDS(DSConfigurationHelper.java:2075)
at com.ibm.ws.rsadapter.DSConfigurationHelper.getConnectionFromDSOrPooledDS(DSConfigurationHelper.java:1951)
at com.ibm.ws.rsadapter.DSConfigurationHelper.testConnectionForGUI(DSConfigurationHelper.java:2819)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at com.ibm.ws.management.DataSourceConfigHelperMBean.testConnectionToDataSource2(DataSourceConfigHelperMBean.java:556)
at com.ibm.ws.management.DataSourceConfigHelperMBean.testConnection(DataSourceConfigHelperMBean.java:484)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:88)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:292)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1261)
at java.security.AccessController.doPrivileged(AccessController.java:311)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:88)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1255)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1093)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:832)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:814)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1335)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1228)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:181)
at com.ibm.ws.management.connector.ipc.CallRouter.route(CallRouter.java:247)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink.doWork(IPCConnectorInboundLink.java:360)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink$IPCConnectorReadCallback.complete(IPCConnectorInboundLink.java:602)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Caused by: java.lang.Exception: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:445)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:464)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:594)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:229)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
... 56 more
Caused by: java.net.ConnectException: Connection refused
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:370)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:231)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:213)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:404)
at java.net.Socket.connect(Socket.java:643)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:162)
at oracle.net.nt.ConnOption.connect(ConnOption.java:133)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:411)
... 61 more

From an Oracle perspective, I checked the Listener configuration: -

lsnrctl start LISTENER

  LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 23-FEB-2018 21:16:14

  Copyright (c) 1991, 2016, Oracle.  All rights reserved.

  Starting /home/oracle/app/oracle/product/12.2.0/dbhome_1/bin/tnslsnr: please wait...

  TNSLSNR for Linux: Version 12.2.0.1.0 - Production
  System parameter file is /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
  Log messages written to /home/oracle/app/oracle/diag/tnslsnr/bpm856/listener/alert/log.xml
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))
  STATUS of the LISTENER
  ------------------------
  Alias                     LISTENER
  Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
  Start Date                23-FEB-2018 21:16:14
  Uptime                    0 days 0 hr. 0 min. 0 sec
  Trace Level               off
  Security                  ON: Local OS Authentication
  SNMP                      OFF
  Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
  Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/bpm856/listener/alert/log.xml
  Listening Endpoints Summary...
    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
    (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  The listener supports no services
  The command completed successfully

lsnrctl status listener

  LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 23-FEB-2018 21:16:17

  Copyright (c) 1991, 2016, Oracle.  All rights reserved.

  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))
  STATUS of the LISTENER
  ------------------------
  Alias                     LISTENER
  Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
  Start Date                23-FEB-2018 21:16:14
  Uptime                    0 days 0 hr. 0 min. 3 sec
  Trace Level               off
  Security                  ON: Local OS Authentication
  SNMP                      OFF
  Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
  Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/bpm856/listener/alert/log.xml
  Listening Endpoints Summary...
    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
    (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  The listener supports no services
  The command completed successfully

I also validated the Service Name: -

sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Fri Feb 23 21:19:24 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>  select value from v$parameter where name='service_names';

VALUE
--------------------------------------------------------------------------------
orcl.uk.ibm.com

SQL>  exit

Yep, that was the problem - I was trying to connect to the Listener using the hostname bpm856.uk.ibm.com whereas the Listener was configured to use127.0.0.1.

Once I updated the listener: -

vi /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora

from: -

# listener.ora Network Configuration File: /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

to: -

# listener.ora Network Configuration File: /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = bpm856.uk.ibm.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

and restarted the Listener: -

lsnrctl start LISTENER

lsnrctl start LISTENER

and re-validated the configuration: -

lsnrctl status listener

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 23-FEB-2018 21:33:20

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=bpm856.uk.ibm.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                23-FEB-2018 21:18:08
Uptime                    0 days 0 hr. 15 min. 11 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/bpm86/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bpm86.uk.ibm.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "orcl.uk.ibm.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.uk.ibm.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully

we were good to go :-)

Oracle 12c on Linux - Fonts Not So Good

Here we go again ….

Installing Oracle 12c on a Red Hat Enterprise Linux 7.4 box: -

/tmp/database/runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 13067 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 2076 MB    Passed
Checking monitor: must be configured to display at least 256 colors
    >>> Could not execute auto check for display colors using command /usr/bin/xdpyinfo. Check if the DISPLAY variable is set.    Failed <<<<

Some requirement checks failed. You must fulfill these requirements before

continuing with the installation,

Continue? (y/n) [n] y

>>> Ignoring required pre-requisite failures. Continuing...
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2018-02-23_07-10-09PM. Please wait ...[oracle@bpm86 ~]$ Fontconfig error: "local.conf", line 2: XML or text declaration not at start of entity
Exception in thread "main" java.lang.ExceptionInInitializerError
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at javax.swing.UIDefaults.getUI(UIDefaults.java:769)
at javax.swing.UIManager.getUI(UIManager.java:1016)
at javax.swing.JComboBox.updateUI(JComboBox.java:266)
at javax.swing.JComboBox.init(JComboBox.java:231)
at javax.swing.JComboBox.<init>(JComboBox.java:183)
at oracle.help.DefaultNavigatorPanel$MinimumSizedComboBox.<init>(Unknown Source)
at oracle.help.DefaultNavigatorPanel.<init>(Unknown Source)
at oracle.help.Help._initHelpSystem(Unknown Source)
at oracle.help.Help.<init>(Unknown Source)
at oracle.help.Help.<init>(Unknown Source)
at oracle.help.Help.<init>(Unknown Source)
at oracle.install.commons.util.HelpManager.loadHelp(HelpManager.java:230)
at oracle.install.commons.util.Application.startup(Application.java:943)
at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:181)
at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:198)
at oracle.install.commons.base.driver.common.Installer.startup(Installer.java:566)
at oracle.install.ivw.db.driver.DBInstaller.startup(DBInstaller.java:127)
at oracle.install.ivw.db.driver.DBInstaller.main(DBInstaller.java:165)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75)
at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)
at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)
at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113)
at javax.swing.JComponent.getFontMetrics(JComponent.java:1626)
at javax.swing.text.PlainView.calculateLongestLine(PlainView.java:639)
at javax.swing.text.PlainView.updateMetrics(PlainView.java:209)
at javax.swing.text.PlainView.updateDamage(PlainView.java:527)
at javax.swing.text.PlainView.insertUpdate(PlainView.java:451)
at javax.swing.text.FieldView.insertUpdate(FieldView.java:293)
at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610)
at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869)
at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201)
at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748)
at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707)
at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130)
at javax.swing.text.AbstractDocument.replace(AbstractDocument.java:669)
at javax.swing.text.JTextComponent.setText(JTextComponent.java:1669)
at javax.swing.JTextField.<init>(JTextField.java:243)
at javax.swing.JTextField.<init>(JTextField.java:183)
at com.jgoodies.looks.plastic.PlasticComboBoxUI.<clinit>(PlasticComboBoxUI.java:88)
... 27 more

As ever, Google had the answer ( Google is my friend ) ….

Oracle Database 12 install crashes on Oracle Linux 7u4

The TL;DR; was: -

su -

vi /etc/fonts/local.conf

<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
  <alias>
    <family>serif</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>sans-serif</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>monospace</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>dialog</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>dialoginput</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
</fontconfig>

Job done :-)

Aide Memoire - Windows Server 2012 and the Missing Network

Not 100% sure why this occurred, but I had an issue today whereby a VM, hosted on my Mac using VMware Fusion 10.1.1, seemingly lost it's mind or, to be more accurate, it's network connection.

The VM is Windows Server 2012, and it's been running happily for some weeks, since first I built it last month.

Even more weirdly, it was working today, until lunchtime - perhaps that's what changed ?

Anyway, the symptom was that, suddenly, the client that was attempted to connect to the VM ( which is another VM, running on the same Mac, containing Red Hat Enterprise Linux ) could no longer connect.

This was a major PITA as the "client" VM was a WAS/BPM box that I was attempting to connect to Windows Server 2012 via LDAP ( provided by Active Directory ).

Both VMs were configured to use Network Address Translation (NAT), which meant that they were sharing the Mac's network interface.

The Mac itself is using my client's guest WiFi, and I have no other network connection from the Mac itself ( no wired, no Bluetooth etc. ).

As far as Windows was concerned, there was NO network, using IP v4 or IP v6.

VMware was acting as the DHCP server, which usually works …..

I disabled / enabled the network adapter within Windows, rebooted the VM etc. but to no avail.

I also shut down the VM and used VMware to change the virtual Media Access Control (MAC) address for the adapter, but again no dice.

I even deleted and recreated the VMware Network Adapter, but still no joy.

Finally, in desperation, I changed the IPv4 configuration within Windows to use a static IP address, in the same subnet as the "client" RHEL VM ( to 192.168.153.200 ) …..

And it started working ….

So I can only assume that something borked the Mac's DHCP server ( part of VMware Fusion ), but …..

Ah well, it's good to know for next time ….

IBM Installation Manager - Reporting

I had a need to find out where IBM Installation Manager (IIM) was keeping it's Shared Resources: -

Shared resources location for a silent install of Installation Manager

com.ibm.cic.common.core.preferences.eclipseCache

for an existing installation.

Thankfully, IIM has the -verbose switch on some of its commands, including listInstalledPackages as per this: -

/opt/ibm/InstallationManager/eclipse/tools/imcl listInstalledPackages -verbose

[Shared]
Shared resources directory: /opt/ibm/IMShared

[Package group]
Name: IBM Installation Manager
Installation directory: /opt/ibm/InstallationManager/eclipse
Translations: Croatian (hr), Simplified Chinese (zh), Turkish (tr), Brazilian Portuguese (pt_BR), Norwegian (no), Hungarian (hu), Thai (th), German (de), Finnish (fi), Traditional Chinese (zh_TW), French (fr), Swedish (sv), Slovenian (sl), Slovak (sk), Danish (da), Italian (it), Hebrew (iw), Korean (ko), Arabic (ar), Traditional Chinese (zh_HK), Czech (cs), Greek (el), Polish (pl), English (en), Russian (ru), Spanish (es), Dutch (nl), Japanese (ja)
Architecture: 64-bit

[Package]
Name: IBM® Installation Manager (com.ibm.cic.agent) 
Version: 1.8.8 (1.8.8000.20171130_1105) 
Repository: /mnt/IIM/
Features:
    None
Fixes:
    None
Rollback versions:
    None

[Package group]
Name: IBM WebSphere Application Server V8.5
Installation directory: /opt/ibm/WebSphere/AppServer
Translations: 

[Package]
Name: IBM® Business Process Manager Server (com.ibm.bpm.ADV.v85) 
Version: 8.6.0 CF2017.12 (8.6.201712.20171211_1726) 
Repository: /mnt/BPM86/Fixpack/CF201712
Features:
    Business Process Manager Server Production License (BPMServer.Production) 
Fixes:
    None
Rollback versions:
    None

[Package]
Name: IBM WebSphere Application Server Network Deployment (com.ibm.websphere.ND.v85) 
Version: 8.5.5.12 (8.5.5012.20170627_1018) 
Repository: /mnt/BPM86/Product/repository/repos_64bit
Features:
    IBM 64-bit WebSphere SDK for Java (com.ibm.sdk.6_64bit) 
    EJBDeploy tool for pre-EJB 3.0 modules (ejbdeploy) 
    Embeddable EJB container (embeddablecontainer) 
    Stand-alone thin clients and resource adapters (thinclient) 
Fixes:
    8.5.5.12-WS-WASND-IFPI82630 (8.5.5.12-WS-WASND-IFPI82630) 
Rollback versions:
    None

Enhance your Kubernetes Experience with UrbanCode Deploy

Saw this on Slack today: -

Kubernetes is a powerful platform for working with application containers. However, as application complexity increases and containers change and evolve, managing your application becomes challenging. UrbanCode Deploy works with Kubernetes to simplify your application deployment. With UrbanCode Deploy, you may easily change the versions of the containers used in your application without the need of editing a YAML file or using the command line. You gain a visual representation of which container versions are deployed where. You may easily roll back container versions in your application without editing a YAML file or using the command line. You may even alter properties of your application by using UrbanCode Deploy with Kubernetes.

In this article, we will go through a simple tutorial showing how UrbanCode Deploy enhances your Kubernetes experience.

Enhance your Kubernetes Experience with UrbanCode Deploy

Installing IBM Cloud Private - Now what did I forget ?

I tried, and failed, to install IBM Cloud Private (ICP) several times: -

docker run --net=host -t -e LICENSE=accept -v $(pwd):/installer/cluster ibmcom/icp-inception:2.1.0.1-ee install

PLAY [Checking Python interpreter] ***********************************************************************************************************************************************************************

TASK [Checking Python interpreter] ***********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

PLAY [Checking prerequisites] ****************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.102]
ok: [192.168.1.101]

TASK [check : Getting DNS server] ************************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Checking Hostname is resolvable] ***********************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [check : Getting the lock file] *********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [check : Validating ports] **************************************************************************************************************************************************************************
ok: [192.168.1.100] => (item=3000)
ok: [192.168.1.102] => (item=3000)
ok: [192.168.1.101] => (item=3000)
ok: [192.168.1.100] => (item=3100)
ok: [192.168.1.101] => (item=3100)
ok: [192.168.1.102] => (item=3100)
ok: [192.168.1.100] => (item=4300)
ok: [192.168.1.101] => (item=4300)
ok: [192.168.1.102] => (item=4300)
ok: [192.168.1.100] => (item=4500)
ok: [192.168.1.101] => (item=4500)
ok: [192.168.1.102] => (item=4500)
ok: [192.168.1.100] => (item=9080)
ok: [192.168.1.101] => (item=9080)
ok: [192.168.1.102] => (item=9080)
ok: [192.168.1.100] => (item=9443)
ok: [192.168.1.102] => (item=9443)
ok: [192.168.1.101] => (item=9443)
ok: [192.168.1.100] => (item=10443)
ok: [192.168.1.102] => (item=10443)
ok: [192.168.1.101] => (item=10443)
ok: [192.168.1.100] => (item=39001)
ok: [192.168.1.102] => (item=39001)
ok: [192.168.1.101] => (item=39001)
ok: [192.168.1.100] => (item=7998)
ok: [192.168.1.102] => (item=7998)
ok: [192.168.1.101] => (item=7998)
ok: [192.168.1.100] => (item=4000)
ok: [192.168.1.102] => (item=4000)
ok: [192.168.1.101] => (item=4000)
ok: [192.168.1.100] => (item=4001)
ok: [192.168.1.102] => (item=4001)
ok: [192.168.1.101] => (item=4001)
ok: [192.168.1.100] => (item=2380)
ok: [192.168.1.102] => (item=2380)
ok: [192.168.1.101] => (item=2380)
ok: [192.168.1.100] => (item=3306)
ok: [192.168.1.102] => (item=3306)
ok: [192.168.1.101] => (item=3306)
ok: [192.168.1.100] => (item=4567)
ok: [192.168.1.102] => (item=4567)
ok: [192.168.1.101] => (item=4567)
ok: [192.168.1.100] => (item=4568)
ok: [192.168.1.101] => (item=4568)
ok: [192.168.1.102] => (item=4568)
ok: [192.168.1.100] => (item=4444)
ok: [192.168.1.100] => (item=8500)
ok: [192.168.1.102] => (item=4444)
ok: [192.168.1.101] => (item=4444)
ok: [192.168.1.100] => (item=8600)
ok: [192.168.1.102] => (item=8500)
ok: [192.168.1.101] => (item=8500)
ok: [192.168.1.100] => (item=80)
ok: [192.168.1.102] => (item=8600)
ok: [192.168.1.101] => (item=8600)
ok: [192.168.1.100] => (item=443)
ok: [192.168.1.102] => (item=80)
ok: [192.168.1.101] => (item=80)
ok: [192.168.1.100] => (item=8080)
ok: [192.168.1.102] => (item=443)
ok: [192.168.1.101] => (item=443)
ok: [192.168.1.100] => (item=8443)
ok: [192.168.1.102] => (item=8080)
ok: [192.168.1.101] => (item=8080)
ok: [192.168.1.100] => (item=8743)
ok: [192.168.1.101] => (item=8443)
ok: [192.168.1.102] => (item=8443)
ok: [192.168.1.100] => (item=5044)
ok: [192.168.1.102] => (item=8743)
ok: [192.168.1.101] => (item=8743)
ok: [192.168.1.100] => (item=5046)
ok: [192.168.1.102] => (item=5044)
ok: [192.168.1.101] => (item=5044)
ok: [192.168.1.100] => (item=9200)
ok: [192.168.1.102] => (item=5046)
ok: [192.168.1.101] => (item=5046)
ok: [192.168.1.100] => (item=9300)
ok: [192.168.1.102] => (item=9200)
ok: [192.168.1.101] => (item=9200)
ok: [192.168.1.100] => (item=8001)
ok: [192.168.1.102] => (item=9300)
ok: [192.168.1.101] => (item=9300)
ok: [192.168.1.100] => (item=8888)
ok: [192.168.1.102] => (item=8001)
ok: [192.168.1.100] => (item=28443)
ok: [192.168.1.101] => (item=8001)
ok: [192.168.1.102] => (item=8888)
ok: [192.168.1.100] => (item=4194)
ok: [192.168.1.101] => (item=8888)
ok: [192.168.1.102] => (item=28443)
ok: [192.168.1.100] => (item=10250)
ok: [192.168.1.101] => (item=28443)
ok: [192.168.1.100] => (item=10251)
ok: [192.168.1.102] => (item=4194)
ok: [192.168.1.101] => (item=4194)
ok: [192.168.1.100] => (item=10252)
ok: [192.168.1.102] => (item=10250)
ok: [192.168.1.101] => (item=10250)
ok: [192.168.1.100] => (item=179)
ok: [192.168.1.102] => (item=10251)
ok: [192.168.1.101] => (item=10251)
ok: [192.168.1.100] => (item=9099)
ok: [192.168.1.102] => (item=10252)
ok: [192.168.1.101] => (item=10252)
ok: [192.168.1.100] => (item=6969)
ok: [192.168.1.102] => (item=179)
ok: [192.168.1.101] => (item=179)
ok: [192.168.1.100] => (item=4242)
ok: [192.168.1.102] => (item=9099)
ok: [192.168.1.101] => (item=9099)
ok: [192.168.1.100] => (item=500)
ok: [192.168.1.102] => (item=6969)
ok: [192.168.1.101] => (item=6969)
ok: [192.168.1.100] => (item=31030)
ok: [192.168.1.102] => (item=4242)
ok: [192.168.1.101] => (item=4242)
ok: [192.168.1.100] => (item=31031)
ok: [192.168.1.102] => (item=500)
ok: [192.168.1.101] => (item=500)
ok: [192.168.1.100] => (item=3130)
ok: [192.168.1.102] => (item=31030)
ok: [192.168.1.101] => (item=31030)
ok: [192.168.1.102] => (item=31031)
ok: [192.168.1.101] => (item=31031)
ok: [192.168.1.102] => (item=3130)
ok: [192.168.1.101] => (item=3130)

TASK [check : Getting data files] ************************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Creating lock file directory] **************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Creating the lock file] ********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [check : Copying License files] *********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Initialize MTU correction] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Initializing interface list to be verified] ****************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Finding Interface when autodetection_method is first-found] ************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Getting list of interfaces to be excluded] *****************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Getting list of interfaces for mtu validation] *************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Finding MTU for the detected Interface(s)] *****************************************************************************************************************************
ok: [192.168.1.101] => (item=[u'ens32', u'ens32'])
ok: [192.168.1.100] => (item=[u'ens32', u'ens32'])
ok: [192.168.1.102] => (item=[u'ens32', u'ens32'])

TASK [check : Calico Validation - Updating MTU correction(20 bytes) when IP-in-IP is enabled] ************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Preparing list of required kernel modules] *****************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Appending list of required IP tunnel kernel modules] *******************************************************************************************************************
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.101] => (item=tunnel4)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.102] => (item=tunnel4)

TASK [check : Calico Validation - Get builtin kernel module list] ****************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Calico Validation - Initializing empty list to filter out builtin modules] *****************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Filtering out builtin kernel modules] **********************************************************************************************************************************
ok: [192.168.1.101] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip_tables)
ok: [192.168.1.101] => (item=ip6_tables)
ok: [192.168.1.102] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip6_tables)
ok: [192.168.1.101] => (item=ip_set)
ok: [192.168.1.102] => (item=ip6_tables)
ok: [192.168.1.100] => (item=ip_set)
ok: [192.168.1.101] => (item=x_tables)
ok: [192.168.1.102] => (item=ip_set)
ok: [192.168.1.100] => (item=x_tables)
ok: [192.168.1.101] => (item=xt_mark)
ok: [192.168.1.102] => (item=x_tables)
ok: [192.168.1.100] => (item=xt_mark)
ok: [192.168.1.101] => (item=xt_addrtype)
ok: [192.168.1.102] => (item=xt_mark)
ok: [192.168.1.100] => (item=xt_addrtype)
ok: [192.168.1.101] => (item=xt_multiport)
ok: [192.168.1.102] => (item=xt_addrtype)
ok: [192.168.1.100] => (item=xt_multiport)
ok: [192.168.1.101] => (item=nfnetlink)
ok: [192.168.1.102] => (item=xt_multiport)
ok: [192.168.1.100] => (item=nfnetlink)
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.102] => (item=nfnetlink)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=tunnel4)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.102] => (item=tunnel4)

TASK [check : Calico Validation - Loading kernel modules] ************************************************************************************************************************************************
ok: [192.168.1.100] => (item=ip_tables)
ok: [192.168.1.102] => (item=ip_tables)
ok: [192.168.1.101] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip6_tables)
ok: [192.168.1.102] => (item=ip6_tables)
ok: [192.168.1.101] => (item=ip6_tables)
ok: [192.168.1.100] => (item=ip_set)
ok: [192.168.1.101] => (item=ip_set)
ok: [192.168.1.102] => (item=ip_set)
ok: [192.168.1.100] => (item=x_tables)
ok: [192.168.1.101] => (item=x_tables)
ok: [192.168.1.102] => (item=x_tables)
ok: [192.168.1.100] => (item=xt_mark)
ok: [192.168.1.101] => (item=xt_mark)
ok: [192.168.1.102] => (item=xt_mark)
ok: [192.168.1.100] => (item=xt_addrtype)
ok: [192.168.1.101] => (item=xt_addrtype)
ok: [192.168.1.102] => (item=xt_addrtype)
ok: [192.168.1.100] => (item=xt_multiport)
ok: [192.168.1.102] => (item=xt_multiport)
ok: [192.168.1.101] => (item=xt_multiport)
ok: [192.168.1.100] => (item=nfnetlink)
ok: [192.168.1.102] => (item=nfnetlink)
ok: [192.168.1.101] => (item=nfnetlink)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.102] => (item=tunnel4)
ok: [192.168.1.101] => (item=tunnel4)

TASK [check : Calico Validation - Making kernel modules persistent across reboot] ************************************************************************************************************************
ok: [192.168.1.100] => (item=ip_tables)
ok: [192.168.1.102] => (item=ip_tables)
ok: [192.168.1.101] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip6_tables)
ok: [192.168.1.102] => (item=ip6_tables)
ok: [192.168.1.101] => (item=ip6_tables)
ok: [192.168.1.100] => (item=ip_set)
ok: [192.168.1.102] => (item=ip_set)
ok: [192.168.1.101] => (item=ip_set)
ok: [192.168.1.100] => (item=x_tables)
ok: [192.168.1.102] => (item=x_tables)
ok: [192.168.1.101] => (item=x_tables)
ok: [192.168.1.100] => (item=xt_mark)
ok: [192.168.1.102] => (item=xt_mark)
ok: [192.168.1.100] => (item=xt_addrtype)
ok: [192.168.1.101] => (item=xt_mark)
ok: [192.168.1.102] => (item=xt_addrtype)
ok: [192.168.1.100] => (item=xt_multiport)
ok: [192.168.1.101] => (item=xt_addrtype)
ok: [192.168.1.102] => (item=xt_multiport)
ok: [192.168.1.100] => (item=nfnetlink)
ok: [192.168.1.102] => (item=nfnetlink)
ok: [192.168.1.101] => (item=xt_multiport)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=nfnetlink)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.102] => (item=tunnel4)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=tunnel4)

TASK [check : Getting ufw status] ************************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

PLAY [Installing Docker Runtime] *************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [docker-engine : Getting Docker engine version] *****************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [docker-engine : Setting docker version variable] ***************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

PLAY [Checking master node] ******************************************************************************************************************************************************************************

TASK [Fetching kubelet service status] *******************************************************************************************************************************************************************
ok: [192.168.1.100]

PLAY [Checking local work directory] *********************************************************************************************************************************************************************

TASK [Getting installer directory] ***********************************************************************************************************************************************************************
ok: [localhost]

TASK [Checking work directory] ***************************************************************************************************************************************************************************
changed: [localhost]

PLAY [Getting images] ************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [pull-image : Checking image package] ***************************************************************************************************************************************************************
changed: [192.168.1.101 -> localhost]
changed: [192.168.1.100 -> localhost]
changed: [192.168.1.102 -> localhost]

TASK [pull-image : Checking Kubernetes image] ************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

PLAY [Making certificates for Kubernetes] ****************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [localhost]

TASK [Ensuring that the cluster/cfc-keys directory exist] ************************************************************************************************************************************************
changed: [localhost]

TASK [Generating ssl certificates] ***********************************************************************************************************************************************************************
changed: [localhost]

TASK [Generating auth certificates] **********************************************************************************************************************************************************************
changed: [localhost]

TASK [Generating registry certificates] ******************************************************************************************************************************************************************
changed: [localhost]

TASK [Generating platform auth certificates] *************************************************************************************************************************************************************
changed: [localhost]

PLAY [Deploying kubelet and kube-proxy on all nodes] *****************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [iptables : Load ip6tables kernel module] ***********************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [iptables : Updating iptables policy to always accept package forward] ******************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [iptables : Enabling IP forward] ********************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [kubelet : Checking if Cgroup Driver of docker is set to systemd] ***********************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [kubelet : Getting cgdriver flag] *******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [kubelet : Enabling Docker service] *****************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [kubelet : Adding cluster_CA_domain to /etc/hosts file] *********************************************************************************************************************************************
changed: [192.168.1.102]
changed: [192.168.1.101]
changed: [192.168.1.100]

TASK [kubelet : Ensuring that cert directory for the private docker registry exist] **********************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring that the image registry cacert file exist] **************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [kubelet : Ensuring that the /etc/cfc/pods directory exist] *****************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring that the /etc/cfc/conf directory exist] *****************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring that the kube-proxy PodSpec exist] **********************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Creating kubelet directory] **************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Copying CA to /var/lib/kubelet directory] ************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Copying kubelet and kube-proxy's certs to /var/lib/kubelet directory] ********************************************************************************************************************
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubecfg.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubecfg.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubecfg.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-proxy.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-proxy.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-proxy.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet-client.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet-client.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet-client.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubecfg.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubecfg.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubecfg.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-controller-manager.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-controller-manager.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-controller-manager.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-scheduler.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-scheduler.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-scheduler.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-scheduler.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet-client.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-scheduler.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-scheduler.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-controller-manager.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet-client.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet-client.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-proxy.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-controller-manager.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-controller-manager.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-proxy.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-proxy.key)

TASK [kubelet : Creating kubeconfig file for kubelet] ****************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Creating kubeconfig file for kube-proxy] *************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [kubelet : Copying volume_collect.sh to /var/lib/kubelet directory] *********************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Executing volume collection by a shell script] *******************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring kubelet install dir exists] *****************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]
FAILED - RETRYING: Copying hyperkube onto operating system (3 retries left).

TASK [kubelet : Copying hyperkube onto operating system] *************************************************************************************************************************************************
changed: [192.168.1.100]
FAILED - RETRYING: Copying hyperkube onto operating system (3 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (2 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (2 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (1 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (1 retries left).
fatal: [192.168.1.102]: FAILED! => {"attempts": 3, "changed": true, "cmd": "docker run --rm -v /opt/kubernetes/:/data ibmcom/kubernetes:v1.8.3-ee sh -c 'cp -f /hyperkube /data/'", "delta": "0:00:01.667348", "end": "2018-02-19 12:52:37.782362", "failed": true, "rc": 125, "start": "2018-02-19 12:52:36.115014", "stderr": "Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally\ndocker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.\nSee 'docker run --help'.", "stderr_lines": ["Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally", "docker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.", "See 'docker run --help'."], "stdout": "", "stdout_lines": []}
fatal: [192.168.1.101]: FAILED! => {"attempts": 3, "changed": true, "cmd": "docker run --rm -v /opt/kubernetes/:/data ibmcom/kubernetes:v1.8.3-ee sh -c 'cp -f /hyperkube /data/'", "delta": "0:00:01.740711", "end": "2018-02-19 12:52:38.525134", "failed": true, "rc": 125, "start": "2018-02-19 12:52:36.784423", "stderr": "Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally\ndocker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.\nSee 'docker run --help'.", "stderr_lines": ["Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally", "docker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.", "See 'docker run --help'."], "stdout": "", "stdout_lines": []}

TASK [kubelet : Check that nsenter exists] ***************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [kubelet : Check that socat exists] *****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [kubelet : Installing kubelet service] **************************************************************************************************************************************************************
changed: [192.168.1.100]

TASK [kubelet : Starting kubelet] ************************************************************************************************************************************************************************
changed: [192.168.1.100]

RUNNING HANDLER [kubelet : reload systemd] ***************************************************************************************************************************************************************
changed: [192.168.1.100]

RUNNING HANDLER [kubelet : restart daemons] **************************************************************************************************************************************************************
changed: [192.168.1.100]

RUNNING HANDLER [kubelet : restart kubelet] **************************************************************************************************************************************************************
changed: [192.168.1.100]

PLAY [Deploying master node] *****************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the Platform Auth Client ID Secret file exists] *****************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Enabling Docker service] ******************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Getting Etcd name] ************************************************************************************************************************************************************************
ok: [192.168.1.100] => (item=(0, u'192.168.1.100'))

TASK [master : Ensuring that the /etc/cfc/conf directory exist] ******************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the /etc/cfc/pods directory exist] ******************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the /etc/cfc/scripts directory exist] ***************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that the /var/log/audit directory exist] *****************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Generating etcd certificates] *************************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating elasticsearch certificates] ****************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating logstash beats certificates] ***************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating front proxy requestheader certificates for K8S API SERVER] *********************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating certificates for service catalog apiserver for aggregation] ********************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating monitoring certificates] *******************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating cloudant certificates] *********************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Copying certificate files to /etc/cfc/conf directory] *************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Copying ca files to /etc/cfc/conf directory] **********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that Kubernetes configuration file for Kubernetes controller manager exist] ******************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that Kubernetes configuration file for Kubernetes scheduler exist] ***************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that Kubernetes audit log policy file for Kubenretes apiserver exist] ************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that pod spec of PV recycler exist] **********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that the Etcd PodSpec file exist] ************************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : include] **********************************************************************************************************************************************************************************

TASK [master : Getting MariaDB start command] ************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the MariaDB start.sh file exist] ********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that the MariaDB PodSpec file exist] *********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Creating Kubernetes pods directory] *******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Waiting for Etcd to start] ****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Creating Kubernetes services file] ********************************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Waiting for Kubernetes to start] **********************************************************************************************************************************************************
ok: [192.168.1.100]

PLAY [Deploying proxy node] ******************************************************************************************************************************************************************************

PLAY [Deploying Kubernetes addon services] ***************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [Setting install type when fresh install] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [localhost]

TASK [config-mgtsvc-common : Getting components directory] ***********************************************************************************************************************************************
ok: [localhost]

TASK [config-mgtsvc-common : Ensuring that the cluster/cfc-components directory exist] *******************************************************************************************************************
changed: [localhost]

TASK [config-mgtsvc-common : Ensuring that directory /opt/ibm/cfc exist] *********************************************************************************************************************************
changed: [localhost]

TASK [config-mgtsvc-common : Getting Etcd certs info for calico, platform api and image manager services] ************************************************************************************************
ok: [localhost]

TASK [config-kubectl : Configuring kubectl] **************************************************************************************************************************************************************
ok: [localhost]

TASK [config-k8s-roles : Ensuring that the roles directory exist] ****************************************************************************************************************************************
changed: [localhost]

TASK [config-k8s-roles : Copying role files to roles] ****************************************************************************************************************************************************
changed: [localhost]

TASK [config-k8s-roles : Creating rbac roles] ************************************************************************************************************************************************************
changed: [localhost]

TASK [config-k8s-nodes : Adding label to master nodes] ***************************************************************************************************************************************************
changed: [localhost] => (item=192.168.1.100)

TASK [config-k8s-nodes : Adding label to proxy nodes] ****************************************************************************************************************************************************
failed: [localhost] (item=192.168.1.102) => {"changed": true, "cmd": "index=0\n retries=60\n while true; do\n sleep 5\n index=$(( index + 1 ))\n if [[ $index -eq $retries ]]; then\n echo \"Failed to add label to node\"\n exit 1\n fi\n kubectl get nodes 192.168.1.102 && kubectl label nodes 192.168.1.102 proxy=true --overwrite=true && break || continue\n done", "delta": "0:05:12.618903", "end": "2018-02-19 13:03:00.102208", "failed": true, "item": "192.168.1.102", "rc": 1, "start": "2018-02-19 12:57:47.483305", "stderr": "Error from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found"], "stdout": "Failed to add label to node", "stdout_lines": ["Failed to add label to node"]}

PLAY RECAP ***********************************************************************************************************************************************************************************************
192.168.1.100              : ok=175  changed=54   unreachable=0    failed=0   
192.168.1.101              : ok=108  changed=29   unreachable=0    failed=1   
192.168.1.102              : ok=108  changed=29   unreachable=0    failed=1   
localhost                  : ok=75   changed=12   unreachable=0    failed=1   

Playbook run took 0 days, 0 hours, 15 minutes, 41 seconds

What was really strange was that the ICP Docker images were NOT being deployed to the other components within my cluster, the Worker and Proxy nodes.

I validated this via the command: -

docker images

on both nodes; in both cases, there were NO images available.

This helped explain the root exception: -

fatal: [192.168.1.102]: FAILED! => {"attempts": 3, "changed": true, "cmd": "docker run --rm -v /opt/kubernetes/:/data ibmcom/kubernetes:v1.8.3-ee sh -c 'cp -f /hyperkube /data/'", "delta": "0:00:01.667348", "end": "2018-02-19 12:52:37.782362", "failed": true, "rc": 125, "start": "2018-02-19 12:52:36.115014", "stderr": "Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally\ndocker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.\nSee 'docker run --help'.", "stderr_lines": ["Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally", "docker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.", "See 'docker run --help'."], "stdout": "", "stdout_lines": []}

It took me a while, and then I re-read the Knowledge Centre: -

Installing a IBM® Cloud Private Cloud Native environment

and realised what I'd missed.

Looking back at my original installation instructions: -

Move the downloaded image

…

mkdir -p /opt/icp2101/cluster/images
mv /tmp/ibm-cloud-private-x86_64-2.1.0.1.tar.gz /opt/icp2101/cluster/images
…

that was what I'd missed.

I'm now re-running the installation, having put the .TAR.GZ file in the right place …………

So far, so good ……