Installing IBM Cloud Private - Now what did I forget ?

I tried, and failed, to install IBM Cloud Private (ICP) several times: -

docker run --net=host -t -e LICENSE=accept -v $(pwd):/installer/cluster ibmcom/icp-inception:2.1.0.1-ee install

PLAY [Checking Python interpreter] ***********************************************************************************************************************************************************************

TASK [Checking Python interpreter] ***********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

PLAY [Checking prerequisites] ****************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.102]
ok: [192.168.1.101]

TASK [check : Getting DNS server] ************************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Checking Hostname is resolvable] ***********************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [check : Getting the lock file] *********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [check : Validating ports] **************************************************************************************************************************************************************************
ok: [192.168.1.100] => (item=3000)
ok: [192.168.1.102] => (item=3000)
ok: [192.168.1.101] => (item=3000)
ok: [192.168.1.100] => (item=3100)
ok: [192.168.1.101] => (item=3100)
ok: [192.168.1.102] => (item=3100)
ok: [192.168.1.100] => (item=4300)
ok: [192.168.1.101] => (item=4300)
ok: [192.168.1.102] => (item=4300)
ok: [192.168.1.100] => (item=4500)
ok: [192.168.1.101] => (item=4500)
ok: [192.168.1.102] => (item=4500)
ok: [192.168.1.100] => (item=9080)
ok: [192.168.1.101] => (item=9080)
ok: [192.168.1.102] => (item=9080)
ok: [192.168.1.100] => (item=9443)
ok: [192.168.1.102] => (item=9443)
ok: [192.168.1.101] => (item=9443)
ok: [192.168.1.100] => (item=10443)
ok: [192.168.1.102] => (item=10443)
ok: [192.168.1.101] => (item=10443)
ok: [192.168.1.100] => (item=39001)
ok: [192.168.1.102] => (item=39001)
ok: [192.168.1.101] => (item=39001)
ok: [192.168.1.100] => (item=7998)
ok: [192.168.1.102] => (item=7998)
ok: [192.168.1.101] => (item=7998)
ok: [192.168.1.100] => (item=4000)
ok: [192.168.1.102] => (item=4000)
ok: [192.168.1.101] => (item=4000)
ok: [192.168.1.100] => (item=4001)
ok: [192.168.1.102] => (item=4001)
ok: [192.168.1.101] => (item=4001)
ok: [192.168.1.100] => (item=2380)
ok: [192.168.1.102] => (item=2380)
ok: [192.168.1.101] => (item=2380)
ok: [192.168.1.100] => (item=3306)
ok: [192.168.1.102] => (item=3306)
ok: [192.168.1.101] => (item=3306)
ok: [192.168.1.100] => (item=4567)
ok: [192.168.1.102] => (item=4567)
ok: [192.168.1.101] => (item=4567)
ok: [192.168.1.100] => (item=4568)
ok: [192.168.1.101] => (item=4568)
ok: [192.168.1.102] => (item=4568)
ok: [192.168.1.100] => (item=4444)
ok: [192.168.1.100] => (item=8500)
ok: [192.168.1.102] => (item=4444)
ok: [192.168.1.101] => (item=4444)
ok: [192.168.1.100] => (item=8600)
ok: [192.168.1.102] => (item=8500)
ok: [192.168.1.101] => (item=8500)
ok: [192.168.1.100] => (item=80)
ok: [192.168.1.102] => (item=8600)
ok: [192.168.1.101] => (item=8600)
ok: [192.168.1.100] => (item=443)
ok: [192.168.1.102] => (item=80)
ok: [192.168.1.101] => (item=80)
ok: [192.168.1.100] => (item=8080)
ok: [192.168.1.102] => (item=443)
ok: [192.168.1.101] => (item=443)
ok: [192.168.1.100] => (item=8443)
ok: [192.168.1.102] => (item=8080)
ok: [192.168.1.101] => (item=8080)
ok: [192.168.1.100] => (item=8743)
ok: [192.168.1.101] => (item=8443)
ok: [192.168.1.102] => (item=8443)
ok: [192.168.1.100] => (item=5044)
ok: [192.168.1.102] => (item=8743)
ok: [192.168.1.101] => (item=8743)
ok: [192.168.1.100] => (item=5046)
ok: [192.168.1.102] => (item=5044)
ok: [192.168.1.101] => (item=5044)
ok: [192.168.1.100] => (item=9200)
ok: [192.168.1.102] => (item=5046)
ok: [192.168.1.101] => (item=5046)
ok: [192.168.1.100] => (item=9300)
ok: [192.168.1.102] => (item=9200)
ok: [192.168.1.101] => (item=9200)
ok: [192.168.1.100] => (item=8001)
ok: [192.168.1.102] => (item=9300)
ok: [192.168.1.101] => (item=9300)
ok: [192.168.1.100] => (item=8888)
ok: [192.168.1.102] => (item=8001)
ok: [192.168.1.100] => (item=28443)
ok: [192.168.1.101] => (item=8001)
ok: [192.168.1.102] => (item=8888)
ok: [192.168.1.100] => (item=4194)
ok: [192.168.1.101] => (item=8888)
ok: [192.168.1.102] => (item=28443)
ok: [192.168.1.100] => (item=10250)
ok: [192.168.1.101] => (item=28443)
ok: [192.168.1.100] => (item=10251)
ok: [192.168.1.102] => (item=4194)
ok: [192.168.1.101] => (item=4194)
ok: [192.168.1.100] => (item=10252)
ok: [192.168.1.102] => (item=10250)
ok: [192.168.1.101] => (item=10250)
ok: [192.168.1.100] => (item=179)
ok: [192.168.1.102] => (item=10251)
ok: [192.168.1.101] => (item=10251)
ok: [192.168.1.100] => (item=9099)
ok: [192.168.1.102] => (item=10252)
ok: [192.168.1.101] => (item=10252)
ok: [192.168.1.100] => (item=6969)
ok: [192.168.1.102] => (item=179)
ok: [192.168.1.101] => (item=179)
ok: [192.168.1.100] => (item=4242)
ok: [192.168.1.102] => (item=9099)
ok: [192.168.1.101] => (item=9099)
ok: [192.168.1.100] => (item=500)
ok: [192.168.1.102] => (item=6969)
ok: [192.168.1.101] => (item=6969)
ok: [192.168.1.100] => (item=31030)
ok: [192.168.1.102] => (item=4242)
ok: [192.168.1.101] => (item=4242)
ok: [192.168.1.100] => (item=31031)
ok: [192.168.1.102] => (item=500)
ok: [192.168.1.101] => (item=500)
ok: [192.168.1.100] => (item=3130)
ok: [192.168.1.102] => (item=31030)
ok: [192.168.1.101] => (item=31030)
ok: [192.168.1.102] => (item=31031)
ok: [192.168.1.101] => (item=31031)
ok: [192.168.1.102] => (item=3130)
ok: [192.168.1.101] => (item=3130)

TASK [check : Getting data files] ************************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Creating lock file directory] **************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Creating the lock file] ********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [check : Copying License files] *********************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Initialize MTU correction] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Initializing interface list to be verified] ****************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Finding Interface when autodetection_method is first-found] ************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Getting list of interfaces to be excluded] *****************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Getting list of interfaces for mtu validation] *************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Finding MTU for the detected Interface(s)] *****************************************************************************************************************************
ok: [192.168.1.101] => (item=[u'ens32', u'ens32'])
ok: [192.168.1.100] => (item=[u'ens32', u'ens32'])
ok: [192.168.1.102] => (item=[u'ens32', u'ens32'])

TASK [check : Calico Validation - Updating MTU correction(20 bytes) when IP-in-IP is enabled] ************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Preparing list of required kernel modules] *****************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Appending list of required IP tunnel kernel modules] *******************************************************************************************************************
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.101] => (item=tunnel4)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.102] => (item=tunnel4)

TASK [check : Calico Validation - Get builtin kernel module list] ****************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [check : Calico Validation - Initializing empty list to filter out builtin modules] *****************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [check : Calico Validation - Filtering out builtin kernel modules] **********************************************************************************************************************************
ok: [192.168.1.101] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip_tables)
ok: [192.168.1.101] => (item=ip6_tables)
ok: [192.168.1.102] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip6_tables)
ok: [192.168.1.101] => (item=ip_set)
ok: [192.168.1.102] => (item=ip6_tables)
ok: [192.168.1.100] => (item=ip_set)
ok: [192.168.1.101] => (item=x_tables)
ok: [192.168.1.102] => (item=ip_set)
ok: [192.168.1.100] => (item=x_tables)
ok: [192.168.1.101] => (item=xt_mark)
ok: [192.168.1.102] => (item=x_tables)
ok: [192.168.1.100] => (item=xt_mark)
ok: [192.168.1.101] => (item=xt_addrtype)
ok: [192.168.1.102] => (item=xt_mark)
ok: [192.168.1.100] => (item=xt_addrtype)
ok: [192.168.1.101] => (item=xt_multiport)
ok: [192.168.1.102] => (item=xt_addrtype)
ok: [192.168.1.100] => (item=xt_multiport)
ok: [192.168.1.101] => (item=nfnetlink)
ok: [192.168.1.102] => (item=xt_multiport)
ok: [192.168.1.100] => (item=nfnetlink)
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.102] => (item=nfnetlink)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=tunnel4)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.102] => (item=tunnel4)

TASK [check : Calico Validation - Loading kernel modules] ************************************************************************************************************************************************
ok: [192.168.1.100] => (item=ip_tables)
ok: [192.168.1.102] => (item=ip_tables)
ok: [192.168.1.101] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip6_tables)
ok: [192.168.1.102] => (item=ip6_tables)
ok: [192.168.1.101] => (item=ip6_tables)
ok: [192.168.1.100] => (item=ip_set)
ok: [192.168.1.101] => (item=ip_set)
ok: [192.168.1.102] => (item=ip_set)
ok: [192.168.1.100] => (item=x_tables)
ok: [192.168.1.101] => (item=x_tables)
ok: [192.168.1.102] => (item=x_tables)
ok: [192.168.1.100] => (item=xt_mark)
ok: [192.168.1.101] => (item=xt_mark)
ok: [192.168.1.102] => (item=xt_mark)
ok: [192.168.1.100] => (item=xt_addrtype)
ok: [192.168.1.101] => (item=xt_addrtype)
ok: [192.168.1.102] => (item=xt_addrtype)
ok: [192.168.1.100] => (item=xt_multiport)
ok: [192.168.1.102] => (item=xt_multiport)
ok: [192.168.1.101] => (item=xt_multiport)
ok: [192.168.1.100] => (item=nfnetlink)
ok: [192.168.1.102] => (item=nfnetlink)
ok: [192.168.1.101] => (item=nfnetlink)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.102] => (item=tunnel4)
ok: [192.168.1.101] => (item=tunnel4)

TASK [check : Calico Validation - Making kernel modules persistent across reboot] ************************************************************************************************************************
ok: [192.168.1.100] => (item=ip_tables)
ok: [192.168.1.102] => (item=ip_tables)
ok: [192.168.1.101] => (item=ip_tables)
ok: [192.168.1.100] => (item=ip6_tables)
ok: [192.168.1.102] => (item=ip6_tables)
ok: [192.168.1.101] => (item=ip6_tables)
ok: [192.168.1.100] => (item=ip_set)
ok: [192.168.1.102] => (item=ip_set)
ok: [192.168.1.101] => (item=ip_set)
ok: [192.168.1.100] => (item=x_tables)
ok: [192.168.1.102] => (item=x_tables)
ok: [192.168.1.101] => (item=x_tables)
ok: [192.168.1.100] => (item=xt_mark)
ok: [192.168.1.102] => (item=xt_mark)
ok: [192.168.1.100] => (item=xt_addrtype)
ok: [192.168.1.101] => (item=xt_mark)
ok: [192.168.1.102] => (item=xt_addrtype)
ok: [192.168.1.100] => (item=xt_multiport)
ok: [192.168.1.101] => (item=xt_addrtype)
ok: [192.168.1.102] => (item=xt_multiport)
ok: [192.168.1.100] => (item=nfnetlink)
ok: [192.168.1.102] => (item=nfnetlink)
ok: [192.168.1.101] => (item=xt_multiport)
ok: [192.168.1.100] => (item=ipip)
ok: [192.168.1.102] => (item=ipip)
ok: [192.168.1.100] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=nfnetlink)
ok: [192.168.1.102] => (item=ip_tunnel)
ok: [192.168.1.100] => (item=tunnel4)
ok: [192.168.1.101] => (item=ipip)
ok: [192.168.1.102] => (item=tunnel4)
ok: [192.168.1.101] => (item=ip_tunnel)
ok: [192.168.1.101] => (item=tunnel4)

TASK [check : Getting ufw status] ************************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

PLAY [Installing Docker Runtime] *************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [docker-engine : Getting Docker engine version] *****************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [docker-engine : Setting docker version variable] ***************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

PLAY [Checking master node] ******************************************************************************************************************************************************************************

TASK [Fetching kubelet service status] *******************************************************************************************************************************************************************
ok: [192.168.1.100]

PLAY [Checking local work directory] *********************************************************************************************************************************************************************

TASK [Getting installer directory] ***********************************************************************************************************************************************************************
ok: [localhost]

TASK [Checking work directory] ***************************************************************************************************************************************************************************
changed: [localhost]

PLAY [Getting images] ************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [pull-image : Checking image package] ***************************************************************************************************************************************************************
changed: [192.168.1.101 -> localhost]
changed: [192.168.1.100 -> localhost]
changed: [192.168.1.102 -> localhost]

TASK [pull-image : Checking Kubernetes image] ************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

PLAY [Making certificates for Kubernetes] ****************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [localhost]

TASK [Ensuring that the cluster/cfc-keys directory exist] ************************************************************************************************************************************************
changed: [localhost]

TASK [Generating ssl certificates] ***********************************************************************************************************************************************************************
changed: [localhost]

TASK [Generating auth certificates] **********************************************************************************************************************************************************************
changed: [localhost]

TASK [Generating registry certificates] ******************************************************************************************************************************************************************
changed: [localhost]

TASK [Generating platform auth certificates] *************************************************************************************************************************************************************
changed: [localhost]

PLAY [Deploying kubelet and kube-proxy on all nodes] *****************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [iptables : Load ip6tables kernel module] ***********************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [iptables : Updating iptables policy to always accept package forward] ******************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [iptables : Enabling IP forward] ********************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [kubelet : Checking if Cgroup Driver of docker is set to systemd] ***********************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [kubelet : Getting cgdriver flag] *******************************************************************************************************************************************************************
ok: [192.168.1.101]
ok: [192.168.1.100]
ok: [192.168.1.102]

TASK [kubelet : Enabling Docker service] *****************************************************************************************************************************************************************
ok: [192.168.1.100]
ok: [192.168.1.101]
ok: [192.168.1.102]

TASK [kubelet : Adding cluster_CA_domain to /etc/hosts file] *********************************************************************************************************************************************
changed: [192.168.1.102]
changed: [192.168.1.101]
changed: [192.168.1.100]

TASK [kubelet : Ensuring that cert directory for the private docker registry exist] **********************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring that the image registry cacert file exist] **************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [kubelet : Ensuring that the /etc/cfc/pods directory exist] *****************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring that the /etc/cfc/conf directory exist] *****************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring that the kube-proxy PodSpec exist] **********************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Creating kubelet directory] **************************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Copying CA to /var/lib/kubelet directory] ************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Copying kubelet and kube-proxy's certs to /var/lib/kubelet directory] ********************************************************************************************************************
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubecfg.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubecfg.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubecfg.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-proxy.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-proxy.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-proxy.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet-client.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet-client.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet-client.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubecfg.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubecfg.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubecfg.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-controller-manager.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-controller-manager.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-controller-manager.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-scheduler.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-scheduler.key)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-scheduler.key)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-scheduler.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kubelet-client.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-scheduler.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-scheduler.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-controller-manager.crt)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kubelet-client.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kubelet-client.crt)
changed: [192.168.1.100] => (item=/installer/cluster/cfc-certs/kube-proxy.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-controller-manager.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-controller-manager.crt)
changed: [192.168.1.101] => (item=/installer/cluster/cfc-certs/kube-proxy.key)
changed: [192.168.1.102] => (item=/installer/cluster/cfc-certs/kube-proxy.key)

TASK [kubelet : Creating kubeconfig file for kubelet] ****************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Creating kubeconfig file for kube-proxy] *************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]

TASK [kubelet : Copying volume_collect.sh to /var/lib/kubelet directory] *********************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Executing volume collection by a shell script] *******************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.101]
changed: [192.168.1.102]

TASK [kubelet : Ensuring kubelet install dir exists] *****************************************************************************************************************************************************
changed: [192.168.1.100]
changed: [192.168.1.102]
changed: [192.168.1.101]
FAILED - RETRYING: Copying hyperkube onto operating system (3 retries left).

TASK [kubelet : Copying hyperkube onto operating system] *************************************************************************************************************************************************
changed: [192.168.1.100]
FAILED - RETRYING: Copying hyperkube onto operating system (3 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (2 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (2 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (1 retries left).
FAILED - RETRYING: Copying hyperkube onto operating system (1 retries left).
fatal: [192.168.1.102]: FAILED! => {"attempts": 3, "changed": true, "cmd": "docker run --rm -v /opt/kubernetes/:/data ibmcom/kubernetes:v1.8.3-ee sh -c 'cp -f /hyperkube /data/'", "delta": "0:00:01.667348", "end": "2018-02-19 12:52:37.782362", "failed": true, "rc": 125, "start": "2018-02-19 12:52:36.115014", "stderr": "Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally\ndocker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.\nSee 'docker run --help'.", "stderr_lines": ["Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally", "docker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.", "See 'docker run --help'."], "stdout": "", "stdout_lines": []}
fatal: [192.168.1.101]: FAILED! => {"attempts": 3, "changed": true, "cmd": "docker run --rm -v /opt/kubernetes/:/data ibmcom/kubernetes:v1.8.3-ee sh -c 'cp -f /hyperkube /data/'", "delta": "0:00:01.740711", "end": "2018-02-19 12:52:38.525134", "failed": true, "rc": 125, "start": "2018-02-19 12:52:36.784423", "stderr": "Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally\ndocker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.\nSee 'docker run --help'.", "stderr_lines": ["Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally", "docker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.", "See 'docker run --help'."], "stdout": "", "stdout_lines": []}

TASK [kubelet : Check that nsenter exists] ***************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [kubelet : Check that socat exists] *****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [kubelet : Installing kubelet service] **************************************************************************************************************************************************************
changed: [192.168.1.100]

TASK [kubelet : Starting kubelet] ************************************************************************************************************************************************************************
changed: [192.168.1.100]

RUNNING HANDLER [kubelet : reload systemd] ***************************************************************************************************************************************************************
changed: [192.168.1.100]

RUNNING HANDLER [kubelet : restart daemons] **************************************************************************************************************************************************************
changed: [192.168.1.100]

RUNNING HANDLER [kubelet : restart kubelet] **************************************************************************************************************************************************************
changed: [192.168.1.100]

PLAY [Deploying master node] *****************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the Platform Auth Client ID Secret file exists] *****************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Enabling Docker service] ******************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Getting Etcd name] ************************************************************************************************************************************************************************
ok: [192.168.1.100] => (item=(0, u'192.168.1.100'))

TASK [master : Ensuring that the /etc/cfc/conf directory exist] ******************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the /etc/cfc/pods directory exist] ******************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the /etc/cfc/scripts directory exist] ***************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that the /var/log/audit directory exist] *****************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Generating etcd certificates] *************************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating elasticsearch certificates] ****************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating logstash beats certificates] ***************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating front proxy requestheader certificates for K8S API SERVER] *********************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating certificates for service catalog apiserver for aggregation] ********************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating monitoring certificates] *******************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Generating cloudant certificates] *********************************************************************************************************************************************************
changed: [192.168.1.100 -> localhost]

TASK [master : Copying certificate files to /etc/cfc/conf directory] *************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Copying ca files to /etc/cfc/conf directory] **********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that Kubernetes configuration file for Kubernetes controller manager exist] ******************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that Kubernetes configuration file for Kubernetes scheduler exist] ***************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that Kubernetes audit log policy file for Kubenretes apiserver exist] ************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that pod spec of PV recycler exist] **********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that the Etcd PodSpec file exist] ************************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : include] **********************************************************************************************************************************************************************************

TASK [master : Getting MariaDB start command] ************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Ensuring that the MariaDB start.sh file exist] ********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Ensuring that the MariaDB PodSpec file exist] *********************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Creating Kubernetes pods directory] *******************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Waiting for Etcd to start] ****************************************************************************************************************************************************************
ok: [192.168.1.100]

TASK [master : Creating Kubernetes services file] ********************************************************************************************************************************************************
changed: [192.168.1.100]

TASK [master : Waiting for Kubernetes to start] **********************************************************************************************************************************************************
ok: [192.168.1.100]

PLAY [Deploying proxy node] ******************************************************************************************************************************************************************************

PLAY [Deploying Kubernetes addon services] ***************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [Setting install type when fresh install] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting master nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting worker nodes] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting proxy nodes] **********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting management nodes] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Number of Master nodes] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Proxy nodes] ************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting number of Management nodes] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the master node architecture] *****************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the management node architecture] *************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting the proxy node architecture] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting installer dir] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting ha_enabled flag] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes master host] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Kubernetes apiserver url] *********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting UI Access IP] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Auth service url] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Etcd info] ************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch master quorum] ******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Elasticsearch node list] **********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Initialize gfs_storage_class facts] *******************************************************************************************************************************************************
ok: [localhost]

TASK [common : Getting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Proxy IP] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting cloud provider flags] *************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting Cloudant DB URL] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting MariaDB info] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client ID] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP Client Secret] ****************************************************************************************************************************************************************
ok: [localhost]

TASK [common : Setting WLP OAuth2 Client Registration Password] ******************************************************************************************************************************************
ok: [localhost]

TASK [config-mgtsvc-common : Getting components directory] ***********************************************************************************************************************************************
ok: [localhost]

TASK [config-mgtsvc-common : Ensuring that the cluster/cfc-components directory exist] *******************************************************************************************************************
changed: [localhost]

TASK [config-mgtsvc-common : Ensuring that directory /opt/ibm/cfc exist] *********************************************************************************************************************************
changed: [localhost]

TASK [config-mgtsvc-common : Getting Etcd certs info for calico, platform api and image manager services] ************************************************************************************************
ok: [localhost]

TASK [config-kubectl : Configuring kubectl] **************************************************************************************************************************************************************
ok: [localhost]

TASK [config-k8s-roles : Ensuring that the roles directory exist] ****************************************************************************************************************************************
changed: [localhost]

TASK [config-k8s-roles : Copying role files to roles] ****************************************************************************************************************************************************
changed: [localhost]

TASK [config-k8s-roles : Creating rbac roles] ************************************************************************************************************************************************************
changed: [localhost]

TASK [config-k8s-nodes : Adding label to master nodes] ***************************************************************************************************************************************************
changed: [localhost] => (item=192.168.1.100)

TASK [config-k8s-nodes : Adding label to proxy nodes] ****************************************************************************************************************************************************
failed: [localhost] (item=192.168.1.102) => {"changed": true, "cmd": "index=0\n retries=60\n while true; do\n sleep 5\n index=$(( index + 1 ))\n if [[ $index -eq $retries ]]; then\n echo \"Failed to add label to node\"\n exit 1\n fi\n kubectl get nodes 192.168.1.102 && kubectl label nodes 192.168.1.102 proxy=true --overwrite=true && break || continue\n done", "delta": "0:05:12.618903", "end": "2018-02-19 13:03:00.102208", "failed": true, "item": "192.168.1.102", "rc": 1, "start": "2018-02-19 12:57:47.483305", "stderr": "Error from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found\nError from server (NotFound): nodes \"192.168.1.102\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found", "Error from server (NotFound): nodes \"192.168.1.102\" not found"], "stdout": "Failed to add label to node", "stdout_lines": ["Failed to add label to node"]}

PLAY RECAP ***********************************************************************************************************************************************************************************************
192.168.1.100              : ok=175  changed=54   unreachable=0    failed=0   
192.168.1.101              : ok=108  changed=29   unreachable=0    failed=1   
192.168.1.102              : ok=108  changed=29   unreachable=0    failed=1   
localhost                  : ok=75   changed=12   unreachable=0    failed=1   

Playbook run took 0 days, 0 hours, 15 minutes, 41 seconds

What was really strange was that the ICP Docker images were NOT being deployed to the other components within my cluster, the Worker and Proxy nodes.

I validated this via the command: -

docker images

on both nodes; in both cases, there were NO images available.

This helped explain the root exception: -

fatal: [192.168.1.102]: FAILED! => {"attempts": 3, "changed": true, "cmd": "docker run --rm -v /opt/kubernetes/:/data ibmcom/kubernetes:v1.8.3-ee sh -c 'cp -f /hyperkube /data/'", "delta": "0:00:01.667348", "end": "2018-02-19 12:52:37.782362", "failed": true, "rc": 125, "start": "2018-02-19 12:52:36.115014", "stderr": "Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally\ndocker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.\nSee 'docker run --help'.", "stderr_lines": ["Unable to find image 'ibmcom/kubernetes:v1.8.3-ee' locally", "docker: Error response from daemon: manifest for ibmcom/kubernetes:v1.8.3-ee not found.", "See 'docker run --help'."], "stdout": "", "stdout_lines": []}

It took me a while, and then I re-read the Knowledge Centre: -

Installing a IBM® Cloud Private Cloud Native environment

and realised what I'd missed.

Looking back at my original installation instructions: -

Move the downloaded image

…

mkdir -p /opt/icp2101/cluster/images
mv /tmp/ibm-cloud-private-x86_64-2.1.0.1.tar.gz /opt/icp2101/cluster/images
…

that was what I'd missed.

I'm now re-running the installation, having put the .TAR.GZ file in the right place …………

So far, so good ……