Sharing - WebSphere Application Server Liberty Profile and Docker

I was tinkering with Docker this evening, and found this from an IBM colleague, David Currie

WebSphere Application Server Liberty Profile and Docker [PDF]

and, thanks to this, I'm now installing WebSphere Application Server Liberty Profile ( aka WAS Liberty Profile ) on Docker: -

docker run -e LICENSE=accept websphere-liberty

Unable to find image 'websphere-liberty:latest' locally
latest: Pulling from websphere-liberty
2332d8973c93: Downloading [===============>                                   ]    20 MB/65.67 MB
ea358092da77: Download complete 
a467a7c6794f: Download complete 
ca4d7b1b9a51: Download complete 
3e14d1c2650c: Download complete 
e3bc32d50851: Download complete 
745d07bf2ad2: Download complete 
495673ce32e6: Downloading [=========>                                         ] 20.53 MB/108.6 MB
050c02b24496: Download complete 
ee8038061e05: Download complete 
64f4a9295383: Download complete 
76917cbdc083: Download complete 
a9eaaa439e4f: Download complete 
a07b053d7434: Download complete 
d023c7386610: Download complete 
180d4e84d457: Download complete 
d14c17ab414e: Download complete 
6db629706d49: Download complete 
ff829fb7e487: Download complete 
35d2d9594700: Download complete 
cd570f0ee8f5: Download complete 
db13fa4f809e: Downloading [===============================>                   ] 19.99 MB/31.62 MB
6696612e0538: Download complete 
8c33ed5c2443: Downloading [==========================>                        ] 20.47 MB/39.36 MB
37a6052584a4: Download complete 
3f56975cb181: Downloading [=======================>                           ] 20.02 MB/42.28 MB

using Boot2Docker on the Mac, as per a previous post here: -

Getting up to speed with Docker and IBM Bluemix

Back the past, again that's why I blog ...

I had cause to refer back to an old post: -

VMware - Where the X is my X ?

this afternoon, when I saw this exception, when using IBM Installation Manager : -

/opt/ibm/InstallationManager/eclipse/IBMIM 

java: cairo-misc.c:380: _cairo_operator_bounded_by_source: Assertion `NOT_REACHED' failed.
JVMDUMP039I Processing dump event "abort", detail "" at 2015/11/28 12:56:58 - please wait.
JVMDUMP032I JVM requested System dump using '/home/wasadmin/core.20151128.125658.17779.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.18146.

JVMDUMP010I System dump written to /home/wasadmin/core.20151128.125658.17779.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/wasadmin/javacore.20151128.125658.17779.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/wasadmin/javacore.20151128.125658.17779.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/wasadmin/Snap.20151128.125658.17779.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/wasadmin/Snap.20151128.125658.17779.0003.trc
JVMDUMP013I Processed dump event "abort", detail "".

I'm using IBM Installation Manager 1.8.1 : -

/opt/ibm/InstallationManager/eclipse/tools/imcl -version

Installation Manager (installed)
Version: 1.8.1
Internal Version: 1.8.1000.20141126_2002
Architecture: 64-bit

As before, the answer was to add: -

-Dorg.eclipse.swt.internal.gtk.cairoGraphics=false
-Dorg.eclipse.swt.internal.gtk.useCairo=false

to /opt/ibm/InstallationManager/eclipse/IBMIM.ini and retry.

For the record, I also saw the same problem earlier this year: -

WebSphere MQ Explorer v8 and the IBM Integration Bus 9 Explorer

Ask the Experts Replay: Understanding IBM HTTP Server (IHS) Administration using the WebSphere Admin console

Abstract

This Ask the Experts will discuss Information about Web Server Adminisrtion thru WebSphere Administraiton Console.

On October 13, 2015, IBM is hosting a panel discussion on understanding IHS Administration using the WebSphere Admin console.

The Panel of Experts have 5 questions that will answer at this session.

• What is 'Web Server Administration'?

• How does the WCT affect the configuration of the web server admin?

• What are the components of the web server admin?

• What is the difference between DMGR console and standalone WAS console?

• What are the common problems?

Ask the Experts Replay: Understanding IBM HTTP Server (IHS) Administration using the WebSphere Admin console

Hmmm, USB and  Mac Mini - Not BFFs

I had a wee problem connecting a StarTech USB enclosure ( containing a 512 GB SSD drive ) to my Mac Mini this evening.

The drive never showed up in Finder, in Terminal ( under /Volumes ) or in Disk Utility.

Eventually, the enclosed would throw up a connection error.

In the logs ( sudo dmesg ) I saw: -

...
1642046.822691 EzLockDown@14900000: AppleUSBDevice::waitForInterfacesGated: timeout waiting for _interfacesMatched
1642046.822777 EzLockDown@14900000: AppleUSBDevice::waitForInterfacesGated: timeout waiting for _interfacesMatched
vmioplug: Warning: com_vmware_kext_UsbPortArbiter_14_1_4[fffffffffc5c018f]::updateDeviceByPort(14900000) resetDevice() failed: e00002bc
USBMSC Identifier (non-unique): 0000000000014905 0x4c5 0x2028 0x1, 3
1642116.059756 EzLockDown@14900000: AppleUSBDevice::waitForInterfacesGated: timeout waiting for _interfacesMatched
1642116.060507 EzLockDown@14900000: AppleUSBDevice::waitForInterfacesGated: timeout waiting for _interfacesMatched
vmioplug: Warning: com_vmware_kext_UsbPortArbiter_14_1_4[fffffffffc5c018f]::updateDeviceByPort(14900000) resetDevice() failed: e00002bc
…

I tried various things, including suspending a running VM and shutting down VMware Fusion 8, but to no avail.

In the end, I moved the drive from one USB port to another.

( the arrow indicates the from, the connected USB cable indicates the to )

Strange but true ;-)

IBM AIX Support Center Tools

I saw this earlier: -

AIX Support Center Tools provides information about widely used data gathering tools and recommendation tools that are used by AIX system administrators in conjunction with the IBM support center team. The gathering tools help reduce the amount of time spent during initial problem determination. The recommendation tools help provide health check reports or cross-product compatibility information.

Data gathering tools

zsnap

The zsnap utility delivers all the benefits of the standard snap command and captures additional debugging data in an easily understood format. IBM's world-class AIX support centers have teamed up to deliver a more fully integrated tool for data gathering, one that has already been proven to reduce the time needed to resolve customer problems. IBM recommends that you install zsnap before you call IBM technical support.

devscan

The devscan tool facilitates the debugging of storage problems by rapidly gathering a great deal of information about the SAN. It then displays the information in an easy-to-understand manner. You can run devscan from any AIX host, including VIO clients, or from a VIOS.

perfpmr

The perfpmr tool is used extensively by the AIX technical support centers. This package contains a set of tools and instructions for collecting the data needed to analyze performance problems. IBM may ask you to download and use this tool.

pdump

The pdump script extracts information from the running process using kdb command and other AIX tools.

snap

The snap command is included with the operating system. Snap captures system configuration information for AIX and PowerHA. Product technical support centers regularly request snap output to identify and resolve problems. Snap output can be delivered to IBM in portable archive exchange format (pax).

AIX Support Center Tools

Ask the Experts Replay: Understanding HA Manager, WLM, and ORB in WebSphere Application Server

The High Availability Manager (HA), Work Load Management (WLM), and Object Request Broker (ORB) component provide several core features in WebSphere Application Server. This session is open to discuss the concepts, issues, and best practices of each component.

Ask the Experts Replay: Understanding HA Manager, WLM, and ORB in WebSphere Application Server

Bash - Using Variables in Sed

Set a variable

export NAME=DAVID

Validate the variable

echo $NAME

DAVID

Initialise a file

echo "DAVE" > foobar.txt

Validate the file contents

cat foobar.txt

DAVE

Replace the contents of the file with the contents of the variable

sed -i'' "s/DAVE/$NAME/g" foobar.txt

Validate the file contents

cat foobar.txt

DAVID

This works with Bash 4.1.2(1)-release on RHEL 6.6.

How To Create Multiple Deployment Environments Within One Cell

Definitely worth a read; -

How To Create Multiple Deployment Environments Within One Cell

IBM ODM Rules 8.7 - Can you say "Doh" ?

I saw this exception today: -

The initialization of the model failed.
A resource provider error occurred during the loading.
Failed to load the repository.
null

ilog.rules.res.console.IlrConsoleException: The initialization of the model failed.
at ilog.rules.res.console.util.IlrModelManager.createRepository(IlrModelManager.java:546)
at ilog.rules.res.console.util.IlrModelManager.init(IlrModelManager.java:181)
at ilog.rules.res.console.util.IlrConsoleInitializer.consoleInitialized(IlrConsoleInitializer.java:86)
at ilog.rules.res.console.jsf.IlrRequestFilter.updateState(IlrRequestFilter.java:413)
at ilog.rules.res.console.jsf.IlrRequestFilter.doFilter(IlrRequestFilter.java:234)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:967)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1107)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:919)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Caused by: ilog.rules.res.model.IlrResourceRuntimeException: A resource provider error occurred during the loading.
at ilog.rules.res.model.internal.IlrRepositoryFactoryImpl.loadRepository(IlrRepositoryFactoryImpl.java:179)
at ilog.rules.res.model.mbean.IlrJMXRepositoryFactoryImpl.createRepository(IlrJMXRepositoryFactoryImpl.java:58)
at ilog.rules.res.console.util.IlrModelManager.createRepository(IlrModelManager.java:544)
... 25 more
Caused by: ilog.rules.res.persistence.IlrDAOException: Failed to load the repository.
at ilog.rules.res.persistence.impl.IlrDAOLocalization.newIlrDAOException(IlrDAOLocalization.java:35)
at ilog.rules.res.persistence.impl.jdbc.IlrGenericRepositoryDAO.load(IlrGenericRepositoryDAO.java:127)
at ilog.rules.res.model.internal.IlrRepositoryFactoryImpl.loadRepository(IlrRepositoryFactoryImpl.java:177)
... 27 more
Caused by: java.lang.NullPointerException
at ilog.rules.res.persistence.impl.jdbc.helper.IlrRulesetsTable.load(IlrRulesetsTable.java:50)
at ilog.rules.res.persistence.impl.jdbc.IlrGenericRepositoryDAO.load(IlrGenericRepositoryDAO.java:115)
... 28 more

...

[18/11/15 15:26:45:078 GMT] 000000a7 sql           W   An error occurred when executing the SQL query.
                                 com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=DB2INST1.RULEAPPS, DRIVER=3.68.61
...

com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2;-204;42704;DB2INST1.RULEAPPS, DRIVER=3.68.61
...

com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2;-204;42704;DB2INST1.RULEAPPS, DRIVER=3.68.61
...

[18/11/15 15:26:45:094 GMT] 000000a7 sql           W   An error occurred when executing the SQL query.
                                 com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=DB2INST1.RULESETS, DRIVER=3.68.61
...

com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2;-204;42704;DB2INST1.RULESETS, DRIVER=3.68.61
...

I realised that I was missing a table or two ( RULEAPPS and RULESETS to name but two ), but I was 100% sure I'd created everything.

I checked my notes: -

db2 connect to RESDB
db2 create bufferpool BP32K size 8000 automatic pagesize 32K
db2 -tvf C:\IBM\ODM87\executionserver\databases\trace_db2.sql
db2 -tvf C:\IBM\ODM87\executionserver\databases\repository_db2.sql
db2 -tvf C:\IBM\ODM87\executionserver\databases\xomrepository_db2.sql
db2 -tvf C:\IBM\ODM87\executionserver\databases\decisionrunner_db2.sql

( For the record, I'm installing ODM Rules 8.7.0.0 on Windows Server 2008 R2 )

Can you see what I did wrong ?

Yes, when I connected to DB2, I did not specify a user - on the Windows platform, the command will use the ID that's currently logged in - Administrator in my case - which means that all objects are created with the schema of ADMINISTRATOR rather than, as required, DB2INST1.

Once I realised what I'd done wrong, I dropped the DB: -

db2 drop db RESDB

and recreated everything BUT with the correct schema: -

db2 connect to RESDB user DB2INST1
db2 create bufferpool BP32K size 8000 automatic pagesize 32K
db2 -tvf C:\IBM\ODM87\executionserver\databases\trace_db2.sql
db2 -tvf C:\IBM\ODM87\executionserver\databases\repository_db2.sql
db2 -tvf C:\IBM\ODM87\executionserver\databases\xomrepository_db2.sql
db2 -tvf C:\IBM\ODM87\executionserver\databases\decisionrunner_db2.sql

I'm sure I could have also used this: -

db2 set current schema DB2INST1

but that's not important right now.

Windows Server 2008 and Data Execution Prevention

I was trying to install a SQL Server component onto a Windows Server 2008 R2 VM, for self-enablement, but kept hitting blockers, most of which were due to missing dependencies, including the Microsoft .NET Framework.

I finally obtained an up-do-date version of the latter here: -

Microsoft .NET Framework 4.5.2 (Offline Installer) for Windows Vista SP2, Windows 7 SP1, Windows 8, Windows 8.1, Windows Server 2008 SP2, Windows Server 2008 R2 SP1, Windows Server 2012 and Windows Server 2012 R2

specifically NDP452-KB2901907-x86-x64-AllOS-ENU.exe.

Having downloaded this, when I attempted to install, Windows kept throwing up exceptions similar to: -

dotNetFx40_Full_x86_x64 setup has encountered a problem and needs to close. We are sorry for the inconvenience.

with detailed traces such as: -

Problem signature:
  Problem Event Name: APPCRASH
  Application Name: NDP452-KB2901907-x86-x64-AllOS-ENU.exe
  Application Version: 4.5.51209.34209
  Application Timestamp: 52bb74a6
  Fault Module Name: StackHash_0a9e
  Fault Module Version: 0.0.0.0
  Fault Module Timestamp: 00000000
  Exception Code: c0000008
  Exception Offset: 7695c0ea
  OS Version: 6.1.7600.2.0.0.274.10
  Locale ID: 2057
  Additional Information 1: 0a9e
  Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
  Additional Information 3: 0a9e
  Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

Read our privacy statement online:
  http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409

If the online privacy statement is not available, please read our privacy statement offline:
  C:\Windows\system32\en-US\erofflps.txt


Problem signature:
  Problem Event Name: VSSetup
  Problem Signature 01: Microsoft .NET Framework 4 Setup
  Problem Signature 02: 4.0.30319
  Problem Signature 03: 10.0.30319.1
  Problem Signature 04: 1
  Problem Signature 05: unknown
  Problem Signature 06: None_UI_Interactive_Crash
  Problem Signature 07: 0xc0000005
  Problem Signature 08: 0
  Problem Signature 09: unknown
  OS Version: 6.1.7600.2.0.0.274.10
  Locale ID: 2057

Read our privacy statement online:
  http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409

If the online privacy statement is not available, please read our privacy statement offline:
  C:\Windows\system32\en-US\erofflps.txt

From reading, it appeared that Data Execution Prevention (DEP) was getting in the way.

I read various posts online, and finally found this: -

How to turn off Data Execution Prevention (DEP)

which described how to disable DEP at boot-time: -

Once I did this, and rebooted, the .NET Framework installed without problems, which is nice.

IBM BPM Advanced 8.5.6 - Sorting out the JDBC Data Sources

Following on from a much earlier post: -

IBM Business Process Manager 8.5 - The (long) story of the missing WebSphere Variables and why my JDBC datasources don't fully work

I had a requirement to sort out the JDBC data sources / variables that lead to this: -

with an IBM BPM Advanced 8.5.6 installation on Linux on IBM Z ( aka LinuxONE )

Here's the Jython script I used: -

cellID = AdminControl.getCell()

AdminConfig.create('VariableSubstitutionEntry', '(cells/'+cellID+'|variables.xml#VariableMap_1)', '[[symbolicName "DB2_JCC_DRIVER_PATH"] [description ""] [value "${WAS_INSTALL_ROOT}/jdbcdrivers/DB2"]]')

AdminConfig.create('VariableSubstitutionEntry', '(cells/'+cellID+'|variables.xml#VariableMap_1)', '[[symbolicName "WAS_INSTALL_ROOT"] [description ""] [value "/opt/IBM/WebSphere/AppServer"]]')

AdminConfig.create('VariableSubstitutionEntry', '(cells/'+cellID+'|variables.xml#VariableMap_1)', '[[symbolicName "UNIVERSAL_JDBC_DRIVER_PATH"] [description ""] [value "${WAS_INSTALL_ROOT}/jdbcdrivers/DB2"]]')

AdminConfig.create('VariableSubstitutionEntry', '(cells/'+cellID+'|variables.xml#VariableMap_1)', '[[symbolicName "PUREQUERY_PATH"] [description ""] [value ""]]')

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

and here's the result: -

Tab Previews in Firefox - Who Knew ?

Firefox - Enable Tab preview

IBM HTTP Server and the Strange Story of Line Numbers

One of my colleagues saw an interesting issue with IBM HTTP Server today.

She's using IHS 8.5.5.0 on Linux.

In essence, whilst she could connect to IHS on port 8080 ( non-SSL port ), she couldn't connect on port 8443 ( SSL port ).

There were no obvious exceptions in the IHS error log, so I asked her to enable SSL tracing ( via the SSLTrace directive in httpd.conf ).

This she did, and this is what we saw in the logs: -

[Thu Nov 12 20:17:34 2015] [debug] ssl_getpwd() entry
[Thu Nov 12 20:17:34 2015] [debug] ssl_getpwd processing :0, ssl flag [0] prompt flag [0]
[Thu Nov 12 20:17:34 2015] [info] mod_unique_id: using ip addr 192.168.1.80
[Thu Nov 12 20:17:35 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8080
[Thu Nov 12 20:17:35 2015] [notice] Using GSKit version 8.0.14.9
[Thu Nov 12 20:17:35 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8080
[Thu Nov 12 20:17:35 2015] [info] mod_unique_id: using ip addr 192.168.1.80
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property StrictSecurity
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property MarkBusyDown
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property KillWebServerStartUpOnParseErr
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property IISDisableFlushFlag
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: WebSphere Plugins loaded.
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: --------------------Plugin Information-----------------------
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: Bld version: 8.5.0
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: Bld date: Apr 24 2012, 15:17:46
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: Webserver: IBM_HTTP_Server
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: --------------------------------------------------------------
[Thu Nov 12 20:17:36 2015] [notice] Using config file /opt/ibm/HTTPServer/conf/httpd.conf 
[Thu Nov 12 20:17:36 2015] [notice] IBM_HTTP_Server/8.5.0.0 (Unix) configured -- resuming normal operations
[Thu Nov 12 20:17:36 2015] [info] Server built: Mar  7 2012 18:25:14
[Thu Nov 12 20:17:36 2015] [debug] worker.c(1859): AcceptMutex: sysvsem (default: sysvsem)
[Thu Nov 12 20:17:36 2015] [notice] Core file limit is 0; core dumps will be not be written for server crashes
[Thu Nov 12 20:17:36 2015] [debug] mod_mpmstats.c(211): mpmstats daemon started (pid 6494)

which was somewhat strange.

This suggested that IHS was trying to bring up SSL on port 8080 rather than 8443.

Also, I noticed that there was almost no SSL debug appearing in the log, implying that SSL was NOT coming up properly / at all.

This is what she had in httpd.conf: -

Listen 8080
ServerName webserver.uk.ibm.com:8080
LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
SSLProtocolDisable SSLv2 SSLv3
SSLEnable
</VirtualHost>
KeyFile /opt/IBM/HttpServer/ssl/keystore.kdb
SSLDisable
SSLTrace

which looks OK although I wouldn't normally expect to see the Listen and ServerName directives immediately next to the SSL configuration block.

Note the inclusion of SSLTrace at my request.

I asked her to enable line number in vi in order to see where precisely the SSL configuration block occurs: -

106 #
107 # Listen: Allows you to bind the web server to specific IP addresses
108 # and/or ports, in addition to the default. See also the <VirtualHost>
109 # directive.
110 #
111 # Change this to Listen on specific IP addresses as shown below to
112 # prevent the web server from accepting connections on all interfaces
113 # (0.0.0.0)
114 #
115 # Change this to "Listen 0.0.0.0:port" to restrict the server to
116 # IPv4.
117 #
118 #Listen 12.34.56.78:80
119 Listen 8080
120 ServerName webserver.uk.ibm.com:8080
121 LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
122 Listen 8443
123 <VirtualHost *:8443>
124 SSLProtocolDisable SSLv2 SSLv3
125 SSLEnable
126 </VirtualHost>
127 KeyFile /opt/IBM/HttpServer/ssl/keystore.kdb
128 SSLDisable
129 SSLTrace

I then compared this to a vanilla IHS configuration: -

106 #
107 # Listen: Allows you to bind the web server to specific IP addresses 
108 # and/or ports, in addition to the default. See also the <VirtualHost>
109 # directive.
110 #
111 # Change this to Listen on specific IP addresses as shown below to 
112 # prevent the web server from accepting connections on all interfaces
113 # (0.0.0.0)
114 #
115 # Change this to "Listen 0.0.0.0:port" to restrict the server to
116 # IPv4.
117 #
118 #Listen 12.34.56.78:80
119 Listen 8080

This made me think "Aha, I wonder if the various blocks need to be on specific lines?".

Again, I looked at the vanilla configuration, in order to locate the SSL block: -

848 # Example SSL configuration which supports SSLv3 and TLSv1
849 # To enable this support:
850 #   1) Create a key database with ikeyman
851 #   2) Update the KeyFile directive below to point to that key database
852 #   3) Uncomment the directives up through the end of the example
853 #
854 #LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
855 #Listen 443
856 #<VirtualHost *:443>
857 #SSLEnable
858 #</VirtualHost>
859 #KeyFile /opt/ibm/HTTPServer/ihsserverkey.kdb
860 #SSLDisable
861 # End of example SSL configuration

I don't know for sure, but I'm guessing that's the root cause.

I had her move the Listen and ServerName directives further up the file, and restart ....

As expected, this did the trick, and IHS came up properly on both port 8080 ( HTTP ) and 8443 ( HTTPS ).

Equally importantly, the debug appeared as expected: -

...

[Thu Nov 12 20:26:19 2015] [info] mod_unique_id: using ip addr 192.168.1.80
[Thu Nov 12 20:26:20 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8080
[Thu Nov 12 20:26:20 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8443
[Thu Nov 12 20:26:20 2015] [debug] mod_ibm_ssl.c(1604): Accelerator device has not been enabled (0)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC4_128_WITH_MD5(21)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC4_128_EXPORT40_WITH_MD5(22)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC2_CBC_128_CBC_WITH_MD5(23)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC2_CBC_128_CBC_EXPORT40_WITH_MD5(24)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_DES_64_CBC_WITH_MD5(26)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_DES_192_EDE3_CBC_WITH_MD5(27)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_AES_128_CBC_SHA(2F)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_NULL_NULL(30)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_NULL_MD5(31)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_NULL_SHA(32)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_EXPORT_WITH_RC4_40_MD5(33)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_RC4_128_MD5(34)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_RC4_128_SHA(35)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_AES_256_CBC_SHA(35b)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5(36)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_DES_CBC_SHA(39)
...

Having recreated the problem on my own VM, I confirmed that the problem is definitely related to positioning.

Once I moved the SSL block down to line ~848, it just worked.

I need to think about WHY, but the moral of the story is the positioning of Listen, ServerName and the SSL block is obviously more important than I had realised ...

IBM HTTP Server and the WebSphere Plugin - ws_config_parser: handleLogEnd: Failed to open log file: ' /opt/IBM/Websphere/plugins/logs/webserver2/http_plugin.log', OS Err: 2

I saw this issue this evening: -

[Thu Nov 12 20:07:47 2015] [error] ws_config_parser: handleLogEnd: Failed to open log file: ' /opt/IBM/Websphere/plugins/logs/webserver2/http_plugin.log', OS Err: 2

when starting IBM HTTP Server 8.5.5.0.

I checked my IHS configuration file - httpd.conf - which contained: -

WebSpherePluginConfig /opt/IBM/Websphere/plugins/config/webserver2/plugin-cfg.xml

and checked the Plugin configuration file itself: -

/opt/IBM/Websphere/plugins/config/webserver2/plugin-cfg.xml

which contained: -

   <Log LogLevel="Error" Name=" /opt/IBM/Websphere/plugins/logs/webserver2/http_plugin.log"/>

Note the obvious ?

Yes, there's a space between the quote and the leading /: -

   <Log LogLevel="Error" Name=" /opt/IBM/Websphere/plugins/logs/webserver2/http_plugin.log"/>

Once I removed it: -

   <Log LogLevel="Error" Name="/opt/IBM/Websphere/plugins/logs/webserver2/http_plugin.log"/>

IHS started without problems.

Using -Xgc:preferredHeapBase with -Xcompressedrefs

Saw this IBM Technote via Twitter: -

Using -Xgc:preferredHeapBase with -Xcompressedrefs

...

"Why does the JVM report a native out-of-memory (NOOM) when using compressed references? I am using a 64bit JVM and I clearly have plenty of memory left. How can I resolve this problem?"

The IBM JVM will automatically use compressed references when using a maximum heap size less than 25GB. This automated behavior was introduced in Java 626 SR5 and Java 7 SR4*. Compressed references (CR) decreases the size of Java objects making better use of available memory space. This better use of space results in improved JVM performance. *(Java 7 SR1 and later uses compressed references by default on z/OS)

...

Worth a read

AMQ9631: The CipherSpec negotiated during the SSL handshake does not match the required CipherSpec for channel

Following on from my earlier post: -

AMQ9660: SSL key repository: password stash file absent or unusable.

I also saw this: -

11/11/15 20:31:03 - Process(39916.7) User(mqm) Program(amqrmppa)
                    Host(nemdemo.uk.ibm.com) Installation(Installation1)
                    VRMF(8.0.0.0) QMgr(TESTQM)

AMQ9631: The CipherSpec negotiated during the SSL handshake does not match the
required CipherSpec for channel 'TEST.QMGR.SVRCONN'.

EXPLANATION:
There is a mismatch between the CipherSpecs on the local and remote ends of
channel 'TEST.QMGR.SVRCONN'. The channel will not run until this mismatch is
resolved. The CipherSpec required in the local channel definition is
'TLS_RSA_WITH_AES_128_CBC_SHA256'. The name of the CipherSpec negotiated during
the SSL handshake is 'TLS_RSA_WITH_AES_256_CBC_SHA256'. A code is displayed if
the name of the negotiated CipherSpec cannot be determined.
ACTION:
Change the channel definitions for 'TEST.QMGR.SVRCONN' so the two ends have
matching CipherSpecs and restart the channel. If the certificate in use by one
end of the channel is a Global Server Certificate, then the negotiated
CipherSpec may not match that specified on either end of the channel. This is
because the SSL protocol allows a Global Server Certificate to automatically
negotiate a higher level of encryption. In these cases specify a CipherSpec
which meets the requirements of the Global Server Certificate.

in my MQ log - /var/mqm/qmgrs/TESTQM/errors/AMQERR01.LOG.

Again, this happened when I attempted to start the Message Driven Bean (MDB) in WAS, which is the "client" to the MQ "server".

This time around, the message was more meaningful.

The connection from WAS is using a SSL Configuration that asserts a specific Cipher Specification - SSL_RSA_WITH_AES_256_CBC_SHA256 : -

AdminTask.createSSLConfig('[-alias WAS_to_WMQ -type JSSE -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+'  -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers SSL_RSA_WITH_AES_256_CBC_SHA256 ]')

whereas the MQ Channel was expecting a slightly different one: -

echo "DIS CHANNEL("TEST.QMGR.SVRCONN") SSLCIPH" | runmqsc TESTQM

5724-H72 (C) Copyright IBM Corp. 1994, 2014.
Starting MQSC for queue manager TESTQM.


     1 : DIS CHANNEL(TEST.QMGR.SVRCONN) SSLCIPH
AMQ8414: Display Channel details.
   CHANNEL(TEST.QMGR.SVRCONN)              CHLTYPE(SVRCONN)
   SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.

Once I updated the Channel to use the same Cipher Specification as WAS: -

echo "ALTER CHANNEL("TEST.QMGR.SVRCONN") CHLTYPE(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_256_CBC_SHA256)" | runmqsc TESTQM

the MDB started without problems.

AMQ9660: SSL key repository: password stash file absent or unusable.

I've spent a few happy hours trying to resolve this one.

When attempting to read a message from a WebSphere MQ 8 Queue from WebSphere Application Server (WAS), using a Message Driven Bean (MDB), a JMS Activation Specification and a JMS Queue, I kept seeing this: -

-------------------------------------------------------------------------------
11/11/15 19:57:29 - Process(39916.4) User(mqm) Program(amqrmppa)
                    Host(nemdemo.uk.ibm.com) Installation(Installation1)
                    VRMF(8.0.0.0) QMgr(TESTQM)
                   
AMQ9660: SSL key repository: password stash file absent or unusable.

EXPLANATION:
The SSL key repository cannot be used because MQ cannot obtain a password to
access it. Reasons giving rise to this error include: 
(a) the key database file and password stash file are not present in the
  location configured for the key repository, 
(b) the key database file exists in the correct place but that no password
  stash file has been created for it, 
(c) the files are present in the correct place but the userid under which MQ is
  running does not have permission to read them, 
(d) one or both of the files are corrupt. 

The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The channel did not start.
ACTION:
Ensure that the key repository variable is set to where the key database file
is. Ensure that a password stash file has been associated with the key database
file in the same directory, and that the userid under which MQ is running has
read access to both files. If both are already present and readable in the
correct place, delete and recreate them. Restart the channel.

I also noticed that WAS returned: -

     Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').

     Caused by [2] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host 'nemdemo.uk.ibm.com(1420)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host closed connection during handshake],3=nemdemo.uk.ibm.com/192.168.1.113:1420 (nemdemo.uk.ibm.com),4=SSLSocket.startHandshake,5=default]],3=nemdemo.uk.ibm.com(1420),5=RemoteTCPConnection.protocolConnect]

     Caused by [3] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host closed connection during handshake],3=nemdemo.uk.ibm.com/192.168.1.113:1420 (nemdemo.uk.ibm.com),4=SSLSocket.startHandshake,5=default]

     Caused by [4] --> Message : javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

     Caused by [5] --> Message : java.io.EOFException: SSL peer shut down incorrectly

This all occurred as soon as I started the MDB.

The connectivity between WAS and MQ is encrypted using a CA-signed certificate, over a TLS 1.2 connection.

I had previously set up my Queue Manager to use TLS: -

echo "ALTER QMGR SSLKEYR('"/home/mqm/SSL/keystore.kdb"')" | runmqsc TESTQM
echo "DIS QMGR SSLKEYR" | runmqsc TESTQM
echo "REFRESH SECURITY TYPE(SSL)" | runmqsc TESTQM

and my Server Connection Channel to use a specific cipher: -

echo "DEFINE CHANNEL("TEST.QMGR.SVRCONN") CHLTYPE(SVRCONN) SSLCIPH("TLS_RSA_WITH_AES_128_CBC_SHA256") REPLACE" | runmqsc TESTQM
echo "ALTER CHANNEL("TEST.QMGR.SVRCONN") CHLTYPE(SVRCONN) SSLCAUTH(OPTIONAL)" | runmqsc TESTQM

Reading the message explanation above, I spent time looking at the key store / stashed password etc.

ls -al /home/mqm/SSL/

total 28
drwxrwxr-x  2 mqm mqm  4096 Nov 11 13:02 .
drwx------. 4 mqm mqm  4096 Nov 11 19:02 ..
-rwxrwxrwx  1 mqm mqm 10080 Nov 11 15:11 keystore.kdb
-rwxrwxrwx  1 mqm mqm    80 Nov 11 15:11 keystore.rdb
-rwxrwxrwx  1 mqm mqm   129 Nov 11 15:11 keystore.sth

checking that the MQ user mqm had full access to the key store.

I also checked that the stashed password worked OK: -

runmqakm -cert -list -db /home/mqm/SSL/keystore.kdb -stashed

Certificates found
* default, - personal, ! trusted, # secret key
! "CN=uk-WIN-AJ9S32NP29C-CA, DC=uk, DC=ibm, DC=com"
*- nemdemo.uk.ibm.com

so that ruled that one out.

I then read a few IBM PMRs and found a potential hint, suggesting that the extension of the keystore database, as defined in the Queue Manager, does NOT need to be specified.

I updated the Queue Manager configuration: -

echo "ALTER QMGR SSLKEYR('/home/mqm/SSL/keystore')" | runmqsc TESTQM

echo "REFRESH SECURITY TYPE(SSL)" | runmqsc TESTQM

and, guess what, it worked :-)

IBM HTTP Server / IBM WebSphere Plugin - Using Transport Layer (TLS) 1.2

The last blog post for today, I promise.

I was seeing this: -

[10/Nov/2015:20:58:15.05163] 0000f090 061fc700 - PLUGIN: ws_common: websphereShouldHandleRequest: Config was successfully reloaded

[10/Nov/2015:20:58:16.28930] 0000f75e 07fff700 - PLUGIN: ws_common: websphereShouldHandleRequest: Config was successfully reloaded
[10/Nov/2015:21:03:07.80560] 0000f090 039f8700 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_SOCKET_CLOSED(gsk rc = 420) PARTNER CERTIFICATE DN=No Information Available, Serial=No Information Available
[10/Nov/2015:21:03:07.81064] 0000f090 039f8700 - ERROR: ws_common: websphereGetStream: Could not open stream
[10/Nov/2015:21:03:07.81073] 0000f090 039f8700 - ERROR: ws_common: websphereExecute: Failed to create the stream
[10/Nov/2015:21:03:07.81075] 0000f090 039f8700 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'Node1_AppClusterMember1' on host 'nemdemo.uk.ibm.com:9443'; will try another one
[10/Nov/2015:21:03:07.81076] 0000f090 039f8700 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request 
[10/Nov/2015:21:03:07.81098] 0000f090 039f8700 - ERROR: ESI: getResponse: failed to get response: rc = 2
[10/Nov/2015:21:03:07.81115] 0000f090 039f8700 - ERROR: ws_common: websphereHandleRequest: Failed to handle request[10/Nov/2015:21:03:52.12777] 0000fa1c 7395a700 - PLUGIN: Plugins loaded.

when attempting to connect to IBM BPM's Process Center URL via IBM HTTP Server / WebSphere Plugin, where I'm using Transport Layer Security (TLS) 1.2 between the Plugin and WAS.

This IBM APAR helped: -

PM74603: THE PLUG-IN CAN NOT CONNECT TO AN APPLICATION SERVER IF IT USES TLSV1.2 AND IS CONFIGURED FOR STRICT SP800-131 COMPLIANCE. 

which mentioned: -

A property was added to allow plugin to enable security compatible with the application server strict server setting.

To enable this property, set StrictSecurity=true on the webserver-><servername>->Plug-in properties->Customer Properties window.

Thus I changed the plugin configuration file: -

vi /opt/ibm/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

Change from: -

<?xml version="1.0" encoding="ISO-8859-1"?><!--HTTP server plugin config file for the webserver PCCell1.Node1.webserver1 generated on 2015.11.10 at 08:45:54 PM GMT-->
<Config ASDisableNagle="false" AcceptAllContent="true" AppServerPortPreference="HostHeader" ChunkedResponse="false" FIPSEnable="false" FailoverToNext="false" HTTPMaxHeaders="300" IISDisableFlushFlag="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" KillWebServerStartUpOnParseErr="false" MarkBusyDown="false" OS400ConvertQueryStringToJobCCSID="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="true" StrictSecurity="false" TrustedProxyEnable="false" VHostMatchingCompat="false">
   <Log LogLevel="Error" Name="/opt/ibm/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>

to: -

<?xml version="1.0" encoding="ISO-8859-1"?><!--HTTP server plugin config file for the webserver PCCell1.Node1.webserver1 generated on 2015.11.10 at 08:45:54 PM GMT-->
<Config ASDisableNagle="false" AcceptAllContent="true" AppServerPortPreference="HostHeader" ChunkedResponse="false" FIPSEnable="false" FailoverToNext="false" HTTPMaxHeaders="300" IISDisableFlushFlag="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" KillWebServerStartUpOnParseErr="false" MarkBusyDown="false" OS400ConvertQueryStringToJobCCSID="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="true" StrictSecurity="true" TrustedProxyEnable="false" VHostMatchingCompat="false">
   <Log LogLevel="Error" Name="/opt/ibm/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>

and restarted IHS.

Once done, it worked like a treat.

I do, of course, need to make the same change within the WAS cell, and then regenerate / propagate the Plugin Configuration.

However, that's tomorrow's job :-)

*UPDATE*

This is the Jython that I used to set the StrictSecurity property to true : -

AdminConfig.create('Property', '(cells/'+cellID+'/nodes/Node1/servers/webserver1|server.xml#PluginProperties_1447187986151)', '[[validationExpression ""] [name "StrictSecurity"] [description ""] [value "true"] [required "false"]]') 

*UPDATE*

Having done this, I simply generated/propagated the Plugin configuration: -

AdminControl.invoke('WebSphere:name=PluginCfgGenerator,process=dmgr,platform=common,node=Dmgr,version=8.5.5.4,type=PluginCfgGenerator,mbeanIdentifier=PluginCfgGenerator,cell='+cellID+',spec=1.0', 'generate', '[/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/config '+cellID+' Node1 webserver1 false]', '[java.lang.String java.lang.String java.lang.String java.lang.String java.lang.Boolean]')

AdminControl.invoke('WebSphere:name=PluginCfgGenerator,process=dmgr,platform=common,node=Dmgr,version=8.5.5.4,type=PluginCfgGenerator,mbeanIdentifier=PluginCfgGenerator,cell='+cellID+',spec=1.0', 'propagate', '[/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/config '+cellID+' Node1 webserver1]', '[java.lang.String java.lang.String java.lang.String java.lang.String]')

and we're good to go.

IBM HTTP Server - Problem with Subject Alternate Names

I'm creating an end-to-end SSL/TLS configuration, using MS Active Directory as my Public Key Infrastructure (PKI) Certificate Authority (CA) Signer.

As part of this, I'm using the Subject Alternate Name (SAN) field in the Certificate Request, in order to allow me to specify BOTH the service name e.g. ibmbpm.uk.ibm.com and the server's host name e.g. nemdemo.uk.ibm.com to be specified.

This was what I'd specified when I created the Certificate Request: -

/opt/ibm/HTTPServer/bin/gskcapicmd -certreq -create -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd -label nemdemo.uk.ibm.com -dn "cn=ibmbpm.uk.ibm.com,dc=uk,dc=ibm,dc=com" -file /home/wasadmin/nemdemo.uk.ibm.com.req -size 2048 -sigalg SHA256WithRSA -san_dnsname nemdemo.uk.ibm.com

This worked perfectly until ....

I see that when accessing the server using the service name rather than the host name.

After some digging, I realised that, whilst I can/should specify the service name as the Distinguished Name (DN), I also need to ensure that the Subject Alternate Name has ALL the values by which this certificate is going to be "known" i.e. BOTH the hostname(s) AND the service name.

Once I correctly specified BOTH: -

/opt/ibm/HTTPServer/bin/gskcapicmd -certreq -create -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd -label nemdemo.uk.ibm.com -dn "cn=ibmbpm.uk.ibm.com,dc=uk,dc=ibm,dc=com" -file /home/wasadmin/nemdemo.uk.ibm.com.req -size 2048 -sigalg SHA256WithRSA -san_dnsname "nemdemo.uk.ibm.com,ibmbpm.uk.ibm.com"

it just simply worked.

This developerWorks article: -

Managing certificates with IBM GSKit - An easy guide to creating, signing, installing, and using certificates with IBM Global Security Kit

also helped: -

You can also request a subject alternative name (SAN) extension by using -san_dnsname or -san_ipaddr options (not supported in version 7). For example:

gsk8capicmd -certreq -create -db server.kdb -stashed -label "My CA signed certificate" -dn "CN=host.mycompany.com,OU=unit,O=company" -san_dnsname "host1.mycompany.com,host2.mycompany.com" -san_ipaddr "10.10.10.1,10.10.10.2" -file cert_request.arm

What we've got here is a failure to communicate - SSLC0008E: Unable to initialize SSL connection

I kept seeing this: -

[10/11/15 17:45:31:220 GMT] 000000c9 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

[10/11/15 17:55:31:242 GMT] 000000c9 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?        

[10/11/15 18:05:31:250 GMT] 000000c8 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

[10/11/15 18:15:31:255 GMT] 000000c8 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

every ten minutes or so, in the SystemOut.log file of my Deployment Manager.

This after I've moved everything in the WAS cell to use an Active Directory CA-signed SSL certificate.

I saw this Technote: -

PM61843: JAVAX.NET.SSL.SSLEXCEPTION: UNRECOGNIZED SSL MESSAGE, PLAINTEXT EXCEPTIONS ARE THROWN WHILE RESTARTING THE NODEAGENTS.

which, whilst referencing an iFix that I already have, made me think about the Node Agent.

When I looked at the Node in the DM's Integrated Solutions Console, I saw: -

I had to manually kill the Node Agent, and forcibly resynchronise it with the cell: -

/opt/ibm/WebSphere/AppServer/profiles/AppSrv01/bin/syncNode.sh `hostname`

After I did this, and restarted the Node Agent, the message continued to pop up ...

I've restarted BOTH the Deployment Manager and the Node Agent, and will keep an eye ( or two ) on it ....

What do I need to do in BPM when I want to move my databases to a different database server?

This from developerWorks Answers: -

What do I need to do in BPM when I want to move my databases to a different database server?

I plan to move my database to a new hardware. What steps do I need to perform in my BPM environment to be able to connect to the moved databases?

Want to know how ? Then read the post ....

LDAP or LDAPS - THAT is THE question

So I did this: -

AdminTask.addIdMgrLDAPServer('[-id ad2008.uk.ibm.com -host ad2008.uk.ibm.com -bindDN CN=ldapbind,CN=Users,DC=uk,DC=ibm,DC=com -bindPassword P455w0rd -referal ignore -sslEnabled false -ldapServerType AD -sslConfiguration -certificateMapMode exactdn -certificateFilter -authentication simple -port 636]')

and got this: -

WASX7015E: Exception running command: "AdminTask.addIdMgrLDAPServer('[-id ad2008.uk.ibm.com -host ad2008.uk.ibm.com -bindDN CN=ldapbind,CN=Users,DC=uk,DC=ibm,DC=com -bindPassword ***** -referal ignore -sslEnabled false -ldapServerType AD -sslConfiguration -certificateMapMode exactdn -certificateFilter -authentication simple -port 636]') "; exception information:
com.ibm.websphere.wim.exception.WIMConfigurationException: com.ibm.websphere.wim.exception.WIMConfigurationException: CWWIM5020E  Could not connect to the ldap://ad2008.uk.ibm.com:636 repository using properties: [port=636],[bindDN=CN=ldapbind,CN=Users,DC=uk,DC=ibm,DC=com],[certificateMapMode=exactdn],[sslConfiguration=],[securityDomainName=admin],[sslEnabled=false],[connectTimeout=20],[connectionPool=false],[id=ad2008.uk.ibm.com],[ldapServerType=AD],[host=ad2008.uk.ibm.com],[referal=ignore],[derefAliases=always],[certificateFilter=],[authentication=simple],[bindPassword=****]. Exception occurred: javax.naming.ServiceUnavailableException.

Can you see what I did there ?

Yep, I tried to bind to the LDAP SSL port 636 but forgot to tell WAS that the port is using SSL ;-)

Once I changed my command: -

AdminTask.addIdMgrLDAPServer('[-id ad2008.uk.ibm.com -host ad2008.uk.ibm.com -bindDN CN=ldapbind,CN=Users,DC=uk,DC=ibm,DC=com -bindPassword P455w0rd -referal ignore -sslEnabled true -ldapServerType AD -sslConfiguration -certificateMapMode exactdn -certificateFilter -authentication simple -port 636]')

I got this: -

'CWWIM5027W The configuration is not complete. Saving an incomplete configuration can cause startup problems.'

which is as I'd expect.

Can you say "Doh!" ? Can you ?

MS Active Directory 2008 - Why no TLS 1.2 ?

As part of my ongoing mission to discover everything about everything, I was trying to retrieve a Signer Certificate from Active Directory 2008 R2 via LDAP, into my WAS 8.5.5 cell, using this command: -

cellID = AdminControl.getCell()

AdminTask.retrieveSignerFromPort('[-keyStoreName CellDefaultTrustStore -keyStoreScope (cell):'+cellID+' -host ad2008.uk.ibm.com -port 636 -certificateAlias ad2008.uk.ibm.com -sslConfigName CellDefaultSSLSettings -sslConfigScopeName (cell):'+cellID+' ]')

However, the command failed with: -

javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLHandshakeException: Server chose TLSv1, but that protocol version is not enabled or not supported by the client.

This occurs because I have WAS set to enforce TLS 1.2, whereas AD was only offering TLS v1.0, by default.

Thankfully this post came to my rescue: -

How to enable TLS 1.2 on Windows Server 2008 R2

It describes how one can edit the Registry (!) to enable TLS 1.2 and set it to by enabled as the default protocol.

From this, I created a registry file - enableTLS12.reg : -

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

which I can use whenever needed in the future.

Now the Signer Certificate imports without fail: -

'Signer Certificate Successfully added to keyStore.'

which is nice.

Microsoft Windows Server 2008 R2 - Certification Authority and the Missing Template

I'm trying to automate the setup of a Public Key Infrastructure (PKI) using MS Windows Server 2008, in order to understand, document and automate the process of using AD to sign certificates for WebSphere Application Server (WAS), IBM HTTP Server, DB2 etc.

As part of this, I've generated a Certificate Request on my WAS VM: -

AdminTask.createCertificateRequest('[-keyStoreName CellDefaultKeyStore -keyStoreScope (cell):'+cellID+' -certificateAlias nemdemo.uk.ibm.com -certificateSize 2048 -certificateCommonName nemdemo.uk.ibm.com -certificateOrganization -certificateOrganizationalUnit -certificateLocality -certificateState -certificateZip -certificateCountry -certificateRequestFilePath /home/wasadmin/nemdemo.req -signatureAlgorithm SHA256withRSA ]')

and have sent the generated file - nemdemo.req - to my AD server ( thanks pscp.exe ).

From there, I then attempted to generate a certificate: -

certreq nemdemo.req

which, alas, failed with: -

CertError: lib\policyserverlist.cpp(835): Lookup: error 0x80094004 (-2146877436)

CertError: lib\policyserverlist.cpp(858): _GetStringProperty: error 0x80094004 (
-2146877436)
CertError: lib\policyserverlist.cpp(835): Lookup: error 0x80094004 (-2146877436)

Active Directory Enrollment Policy
  {30823ACA-B85B-4870-9DEF-1BD6F0377089}
  ldap:
CertError: certlib\comlib.cpp(1814): get_Property(CAPropWebServers): error 0x800
94004 (-2146877436)
CertError: certlib\comlib.cpp(2393): myGetCAStringArrayProperty: error 0x8009400
4 (-2146877436)
RequestId: 39
RequestId: "39"
Certificate not issued (Denied) Denied by Policy Module  0x80094801, The request
 does not contain a certificate template extension or the CertificateTemplate re
quest attribute.
 The request contains no certificate template information. 0x80094801 (-21468753
91)
CertError: certreq\certreq.cpp(4247): Denied(LastStatus): error 0x80094801 (-214
6875391)
CertError: certreq\certreq.cpp(4629): CallServerAndStoreCert: error 0x80094801 (
-2146875391)
CertError: certreq\certreq.cpp(14299): verbSubmitRequest: error 0x80094801 (-214
6875391)
Certificate Request Processor: The request contains no certificate template info
rmation. 0x80094801 (-2146875391)
Denied by Policy Module  0x80094801, The request does not contain a certificate
template extension or the CertificateTemplate request attribute.

CertError: certlib\main.cpp(200): ArgvMain: error 0x80094801 (-2146875391)

A quick Google search brought me here: -

The request contains no certificate template information 0x80094801 (-2146875391). Denied by Policy Module 0x80094801  

which said, in part, this: -

I tried that, verbatim: -

certreq -submit -attrib ,,CertificateTemplate:Webserver nemdemo.req

but this again failed: -

CertError: lib\policyserverlist.cpp(835): Lookup: error 0x80094004 (-2146877436)

CertError: lib\policyserverlist.cpp(858): _GetStringProperty: error 0x80094004 (
-2146877436)
CertError: lib\policyserverlist.cpp(835): Lookup: error 0x80094004 (-2146877436)

Active Directory Enrollment Policy
  {30823ACA-B85B-4870-9DEF-1BD6F0377089}
  ldap:
CertError: certlib\comlib.cpp(1814): get_Property(CAPropWebServers): error 0x800
94004 (-2146877436)
CertError: certlib\comlib.cpp(2393): myGetCAStringArrayProperty: error 0x8009400
4 (-2146877436)
RequestId: 40
RequestId: "40"
Certificate not issued (Denied) Denied by Policy Module  0x80094801, The request
 does not contain a certificate template extension or the CertificateTemplate re
quest attribute.
 The request contains no certificate template information. 0x80094801 (-21468753
91)
CertError: certreq\certreq.cpp(4247): Denied(LastStatus): error 0x80094801 (-214
6875391)
CertError: certreq\certreq.cpp(4629): CallServerAndStoreCert: error 0x80094801 (
-2146875391)
CertError: certreq\certreq.cpp(14299): verbSubmitRequest: error 0x80094801 (-214
6875391)
Certificate Request Processor: The request contains no certificate template info
rmation. 0x80094801 (-2146875391)
Denied by Policy Module  0x80094801, The request does not contain a certificate
template extension or the CertificateTemplate request attribute.

CertError: certlib\main.cpp(200): ArgvMain: error 0x80094801 (-2146875391)

Then I read this: -

0x80094801 – the request contains no certificate template information

which said: -

certreq -submit -attrib "CertificateTemplate:WebServer" <Cert Request.req>

At which point, I slapped myself on the head ....

I had read the earlier post, which contained: -

,,

and read that as double-comma, where it was, in fact, double-quotes :-)

Once I tried the CORRECT syntax: -

certreq -submit -attrib "CertificateTemplate:Webserver" nemdemo.req

I got this: -

CertError: lib\policyserverlist.cpp(835): Lookup: error 0x80094004 (-2146877436)

CertError: lib\policyserverlist.cpp(858): _GetStringProperty: error 0x80094004 (
-2146877436)
CertError: lib\policyserverlist.cpp(835): Lookup: error 0x80094004 (-2146877436)

Active Directory Enrollment Policy
  {30823ACA-B85B-4870-9DEF-1BD6F0377089}
  ldap:
CertError: certlib\comlib.cpp(1814): get_Property(CAPropWebServers): error 0x800
94004 (-2146877436)
CertError: certlib\comlib.cpp(2393): myGetCAStringArrayProperty: error 0x8009400
4 (-2146877436)
RequestId: 41
RequestId: "41"
CertReq: ICertRequest::Submit(Issued) --> Issued
Certificate retrieved(Issued) Issued

which is MUCH better.

I then went one step further: -

certreq -submit -attrib "CertificateTemplate:Webserver" nemdemo.req nemdemo.cer

which means that I get the certificate written back to the current directory.

Now if I can eliminate this popup: -

I'd be happy as Larry.

Who is Larry ?