WebSphere Application Server - manageprofiles.sh and the missing Java security policies

We saw a problem the other day, whilst creating a new IBM BPM Deployment Environment via the BPMConfig.sh script.

When we dug into the problem further, it was apparent that the underlying manageprofiles.sh script was failing.

Having resolved the problem ( more later ), I replicated it on a clean VM this morning.

So, to start with, I attempt to create a new WAS profile: -

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -create -templatePath /opt/ibm/WebSphere/AppServer/profileTemplates/managed -profileName AppSrv02 -nodeName Node1

This fails with: -

INSTCONFFAILED: The profile could not be created.  For more information, consult the /opt/ibm/WebSphere/AppServer/logs/manageprofiles/AppSrv02_create.log file.

When one digs into the logs: -

view /opt/ibm/WebSphere/AppServer/logs/manageprofiles/AppSrv02_create.log 

there are a fair number of failures: -

    <message>Task stopped for: fail - FAILURE</message>
    <message>Target stopped for: failIfError - FAILURE</message>

and exceptions: -

    <message>Exception was thrown, type of exception is: class org.apache.tools.ant.BuildException</message>

but there is also this: -

    <date>2017-04-13T07:41:42</date>
    <millis>1492065702583</millis>
    <sequence>2363</sequence>
    <logger>com.ibm.ws.install.configmanager.actionengine.ant.utils.ANTLogToCmtLogAdapter</logger>
    <level>WARNING</level>
    <class>com.ibm.ws.install.configmanager.logging.LogUtils</class>
    <method>logException</method>
    <thread>0</thread>
    <message>/opt/ibm/WebSphere/AppServer/profileTemplates/managed/actions/scripts/importExternalLogs.xml:30: Failure occured while attempting to Setting the security settings in the default security template.
        at org.apache.tools.ant.taskdefs.Exit.execute(Exit.java:139)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
        at org.apache.tools.ant.Task.perform(Task.java:364)
        at org.apache.tools.ant.Target.execute(Target.java:341)
        at org.apache.tools.ant.Target.performTasks(Target.java:369)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216)

which was the thing that led me to root cause.

Applying the First Rule of Problem Determination - WHAT HAS CHANGED - I realised that the last-but-one thing that we had done was to switch the Java SDK Policy Files from restricted to unrestricted, in order to support AES key lengths greater than 256 bits.

This is documented in way more detail here: -

By default, the IBM® SDK, on all platforms, provides strong but limited jurisdiction policy files. To use unlimited jurisdiction policy files by default, place the US_export_policy.jar and the local_policy.jar in the jre/lib/security/ directory of the SDK.

IBM SDK Policy files

When I traced back, I realised that, as part of this change, we had backed out the old policy files: -

cd /opt/ibm/WebSphere/AppServer/java_1.7.1_64/jre/lib/security/
mv US_export_policy.jar US_export_policy.RAJ
mv local_policy.jar local_policy.RAJ

*BUT* not unpacked the replacement unrestricted policy files :-(

So this is what we had: -

ls -al

total 144
drwxr-xr-x  2 wasadmin wasadmins   4096 Apr 13 07:40 .
drwxr-xr-x 16 wasadmin wasadmins   4096 Mar 29 15:36 ..
-rwxr-xr-x  1 wasadmin wasadmins   4054 Jul 20  2016 blacklist
-rwxr-xr-x  1 wasadmin wasadmins 103822 Mar 24  2014 cacerts
-rwxr-xr-x  1 wasadmin wasadmins   2532 Mar 24  2014 java.policy
-rwxr-xr-x  1 wasadmin wasadmins  10560 Mar 24  2014 java.security
-rwxr-xr-x  1 wasadmin wasadmins     98 Jul 20  2016 javaws.policy
-rwxr-xr-x  1 wasadmin wasadmins   2640 Jul 20  2016 local_policy.RAJ
-rwxr-xr-x  1 wasadmin wasadmins      0 Jul 20  2016 trusted.libraries
-rwxr-xr-x  1 wasadmin wasadmins   2175 Jul 20  2016 US_export_policy.RAJ

In other words, the Java SDK ( JRE ) couldn't find the actual local_policy.jar and US_export_policy.jar files :-(

Once we put the the unrestricted policy files in place: -

unzip ~/unrestrictedpolicyfiles.zip 
Archive:  /home/wasadmin/unrestrictedpolicyfiles.zip
  inflating: US_export_policy.jar    
  inflating: local_policy.jar        

ls -al

total 152
drwxr-xr-x  2 wasadmin wasadmins   4096 Apr 13 07:56 .
drwxr-xr-x 16 wasadmin wasadmins   4096 Mar 29 15:36 ..
-rwxr-xr-x  1 wasadmin wasadmins   4054 Jul 20  2016 blacklist
-rwxr-xr-x  1 wasadmin wasadmins 103822 Mar 24  2014 cacerts
-rwxr-xr-x  1 wasadmin wasadmins   2532 Mar 24  2014 java.policy
-rwxr-xr-x  1 wasadmin wasadmins  10560 Mar 24  2014 java.security
-rwxr-xr-x  1 wasadmin wasadmins     98 Jul 20  2016 javaws.policy
-r--r--r--  1 wasadmin wasadmins   2253 Oct 12  2012 local_policy.jar
-rwxr-xr-x  1 wasadmin wasadmins   2640 Jul 20  2016 local_policy.RAJ
-rwxr-xr-x  1 wasadmin wasadmins      0 Jul 20  2016 trusted.libraries
-r--r--r--  1 wasadmin wasadmins   2240 Oct 12  2012 US_export_policy.jar
-rwxr-xr-x  1 wasadmin wasadmins   2175 Jul 20  2016 US_export_policy.RAJ

the manageprofiles.sh script just bloomin' worked: -

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -create -templatePath /opt/ibm/WebSphere/AppServer/profileTemplates/managed -profileName AppSrv02 -nodeName Node1

INSTCONFSUCCESS: Success: Profile AppSrv02 now exists. Please consult /opt/ibm/WebSphere/AppServer/profiles/AppSrv02/logs/AboutThisProfile.txt for more information about this profile.

although it was necessary to clean up the profile path from the previous failed run: -

rm -Rf /opt/ibm/WebSphere/AppServer/profiles/AppSrv02/

to avoid this: -

The following validation errors were present with the command line arguments: 
profilePath: The profile path is not valid.

So, in conclusion, REMEMBER WHAT YOU LAST DID ( aka What Has Changed )