Friday, 12 October 2018

IBM BPM - Tinkering with Groups via REST

I’m digging into an issue with the IBM Master Data Management (MDM) Data Stewardship Center (DSC), where a button ( Add ) doesn’t appear for my user, DSUser1: -




even though the user is a member of the correct DataStewardGroup: -





As part of the investigation, I wanted to look at the groups and teams available to IBM BPM.

There’s a rather useful REST API for this, accessible via the BPM REST API Tester: -




also accessible via cURL : -

curl -k -u wasadmin:passw0rd -o /tmp/groups.txt --request GET --url https://lt.uk.ibm.com:9446/rest/bpm/wle/v1/groups?includeDeleted=false&parts=all

Knowing the group name ( DataStewardGroup ), I then used the REST API to get at that: -

curl -k -u wasadmin:passw0rd --request GET --url https://lt.uk.ibm.com:9446/rest/bpm/wle/v1/group/DataStewardGroup?includeDeleted=false&parts=all

which returns a single member - DSUser1 : -



I then dug into the BPM teams, as opposed to the groups: -




and the snapshot ID: -




Armed with the Team ID ( 24.fccd301e-1a55-4873-b69f-05c43b251545 ), Snapshot ID ( 2064.f96be189-3d91-4b8e-b485-21af8e0af5ce ) and Branch ID ( 2063.d1ecb65f-8778-4ddc-8c4f-b94ef28781c9 ), I was then able to dig into the details of the DataStewardTeam: -

https://lt.uk.ibm.com:9446/rest/bpm/wle/v1/team/24.fccd301e-1a55-4873-b69f-05c43b251545?snapshotId=2064.f96be189-3d91-4b8e-b485-21af8e0af5ce&branchId=2063.d1ecb65f-8778-4ddc-8c4f-b94ef28781c9




all of which seems to confirm that DSUser1 is a member of the DataStewardGroup which is mapped to the DataStewardTeam

So why doesn’t the Add button appear ?

Ah, well, keep on digging ….

Some follow-up reading: -



Wednesday, 10 October 2018

IBM Master Data Management and IBM BPM - Process Server, rather than Process Center

Having spent the past few months installing / configuring the Master Data Management (MDM) Data Stewardship Center (DSC) with IBM Business Process Manager (BPM), I realised that I've only ever worked with Process Center.

Therefore, the default installation script: -

./InstallGovernanceApplication.sh install_configure_governance_apps

only works for a Process Center, in part because the script imports Process Application snapshots as .TWX files, via the underlying sub-script: -

./InstallGovernanceApplication.sh run_import_process_application

which doesn't work for a Process Server, where one has to install off-line packages ( .ZIP files ).

This is an issue where one has an offline Process Server, which isn't connected to Process Center.

As ever, there is an alternate path ….

In essence, one runs a process (!) on the Process Center environment ( where DSC has previously been installed/configured ), namely: -

./InstallGovernanceApplication.sh get_installable_packages_offline_server

which "dumps" the exported packages ( .ZIP ) here: -

ls -al /opt/ibm/MDMDSG/mdmg/offlinePackages/

total 327388
drwxr-xr-x. 2 wasadmin wasadmins     4096 Oct 10 15:53 .
drwxr-xr-x. 8 wasadmin wasadmins     4096 Sep 18 15:32 ..
-rw-r--r--  1 wasadmin wasadmins 42576778 Oct 10 14:52 MDMDASH.zip
-rw-r--r--  1 wasadmin wasadmins 44511046 Oct 10 14:52 MDMFD.zip
-rw-r--r--  1 wasadmin wasadmins 50974139 Oct 10 14:52 MDMHDQ.zip
-rw-r--r--  1 wasadmin wasadmins 50445365 Oct 10 14:51 MDMPM.zip
-rw-r--r--  1 wasadmin wasadmins 51503903 Oct 10 14:51 MDMSDP.zip
-rw-r--r--  1 wasadmin wasadmins 45311643 Oct 10 14:51 MDMSDV.zip
-rw-r--r--  1 wasadmin wasadmins 49904581 Oct 10 14:52 VST.zip

( having previously created an offline Process Server instance within Process Center - the name is relatively irrelevant - I called mine Boris - and then added the name to MDMBPMIntegration.properties )

offlineServerName=Boris

Having moved these to the Process Server box ( and extracted them to /opt/ibm/MDMDSG/mdmg/offlinePackages/ ), I updated the properties file: -

vi /opt/ibm/MDMDSG/mdmg/install/properties/MDMBPMIntegration.properties

setting the PS_ parameters.

Once done, I then ran a subtly different task from before: -

./InstallGovernanceApplication.sh install_configure_governance_apps_offline_server

and waited.

Now I have a bunch of new Snapshots: -


and the DSC: -


Nice!

Tuesday, 9 October 2018

IBM Operational Decision Manager 8.10 - Arriving in December

This: -


and this: -


Due in early December, this is going to be supported on a wider range of platforms, including IBM Cloud Private ( Docker/Kubernetes orchestration platform ) and WebSphere Liberty Profile.

It'll also include an improved version of Decision Composer, 







etc.

Restoring WebSphere Application Server Profiles - NullPointerExceptions

So, for some reason, I was having a crisis of confidence  / memory.

I was trying/failing to restore a WebSphere Application Server (WAS) profile, using the following command: -

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -restoreProfile -backupFile /home/wasadmin/wasbackups/BPM86_Dmgr01.zip 

which failed with: -

...
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:96)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
at org.eclipse.core.launcher.Main.run(Main.java:981)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:407)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:170)
Caused by: java.lang.NullPointerException: entry
at java.util.zip.ZipFile.getInputStream(ZipFile.java:404)
at com.ibm.ws.profile.utils.FileUtils.getInputStreamFromZipFile(FileUtils.java:364)
at com.ibm.wsspi.profile.WSProfile.restore(WSProfile.java:2097)
at com.ibm.wsspi.profile.WSProfile.restore(WSProfile.java:2069)
at com.ibm.ws.profile.cli.WSProfileCLIRestoreProfileInvoker.executeWSProfileAccordingToMode(WSProfileCLIRestoreProfileInvoker.java:106)
at com.ibm.ws.profile.cli.WSProfileCLIModeInvoker.invokeWSProfile(WSProfileCLIModeInvoker.java:462)
at com.ibm.ws.profile.WSProfileJob.run(WSProfileJob.java:94)
at com.ibm.ws.profile.utils.WSProfileUtils.executeAllJobs(WSProfileUtils.java:793)
at com.ibm.wsspi.profile.WSProfileCLI.invokeWSProfile(WSProfileCLI.java:228)
at com.ibm.wsspi.profile.WSProfileCLI.main(WSProfileCLI.java:62)
... 26 more
...

view /opt/ibm/WebSphere/AppServer/logs/manageprofiles/restoreProfile.log 

...
  <record>
    <date>2018-10-09T12:31:06</date>
    <millis>1539084666088</millis>
    <sequence>24</sequence>
    <logger>com.ibm.ws.profile.cli.WSProfileCLIModeInvoker</logger>
    <level>WARNING</level>
    <class>com.ibm.ws.profile.cli.WSProfileCLIModeInvoker</class>
    <method>getArgumentValue</method>
    <thread>1</thread>
    <message>Could not resolve templatePath from command line</message>
  </record>


It took me a while of faffing around ….. including creating new profiles ….

And then I realised where I was going wrong ….

Given that I had backed up the profiles: -

mkdir wasbackups
cd wasbackups/
/opt/ibm/WebSphere/AppServer/profiles/AppSrv01/bin/backupConfig.sh BPM86_AppSrv01.zip
/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/bin/backupConfig.sh BPM86_Dmgr01.zip

perhaps it'd have made more sense to use …. restoreConfig.sh instead.

Guess what …..

/opt/ibm/WebSphere/AppServer/bin/restoreConfig.sh /home/wasadmin/wasbackups/BPM86_Dmgr01.zip 

ADMU0116I: Tool information is being logged in file
           /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/logs/restoreConfig.log
ADMU0128I: Starting tool with the Dmgr01 profile
ADMU0505I: Servers found in configuration:
ADMU0506I: Server name: dmgr
ADMU2010I: Stopping all server processes for node Dmgr
ADMU0512I: Server dmgr cannot be reached. It appears to be stopped.
ADMU5502I: The directory /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/config
           already exists; renaming to
           /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/config.old
ADMU5504I: Restore location successfully renamed
ADMU5505I: Restoring file /home/wasadmin/wasbackups/BPM86_Dmgr01.zip to
           location /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/config
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ADMU5506I: 3,179 files successfully restored
ADMU6001I: Begin App Preparation -
ADMU6009I: Processing complete.
ADMU6002I: Begin Asset Preparation -
ADMU6009I: Processing complete.

Monday, 8 October 2018

Upgrading from IBM 8.6 to IBM BAW 18.0.0.1 - Disk is crucial

I saw this: -

...
[08/10/18 11:54:56:571 BST] 00000001 BPMConfig     E com.ibm.bpm.config.BPMConfig main ADMA5002E: Application binaries cannot be saved in /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/wstemp/anonymous1538995571224/workspace/cells/PCCell1/applications/navigatorEAR_AppCluster.ear/navigatorEAR_AppCluster.ear: org.eclipse.jst.j2ee.commonarchivecore.internal.exception.SaveFailureException: navigator.war
                                 com.ibm.bpm.config.util.ConfigException: ADMA5002E: Application binaries cannot be saved in /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/wstemp/anonymous1538995571224/workspace/cells/PCCell1/applications/navigatorEAR_AppCluster.ear/navigatorEAR_AppCluster.ear: org.eclipse.jst.j2ee.commonarchivecore.internal.exception.SaveFailureException: navigator.war
        at com.ibm.bpm.config.util.jmx.AppManagementNotificationListener.appEventReceived(AppManagementNotificationListener.java:117)
        at com.ibm.ws.management.application.SchedulerImpl.propagateTaskEvent(SchedulerImpl.java:245)
        at com.ibm.ws.management.application.task.BackupAppTask.install(BackupAppTask.java:195)
        at com.ibm.ws.management.application.task.BackupAppTask.performTask(BackupAppTask.java:92)
        at com.ibm.ws.management.application.SchedulerImpl.run(SchedulerImpl.java:315)
        at java.lang.Thread.run(Thread.java:811)


whilst upgrading from IBM Business Process Manager 8.6 CF2013.03 to IBM Business Automation Workflow 18.0.01, as per this: -


Something made me think that it MIGHT be disk-space related, given the exception.

Checking free disk space: -

df -kmh

Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   48G   44G  3.8G  93% /
devtmpfs               3.8G     0  3.8G   0% /dev
tmpfs                  3.9G   60K  3.9G   1% /dev/shm
tmpfs                  3.9G   12M  3.8G   1% /run
tmpfs                  3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1              497M  210M  288M  43% /boot
tmpfs                  781M     0  781M   0% /run/user/1004
vmhgfs-fuse            477G  405G   73G  85% /mnt
tmpfs                  781M     0  781M   0% /run/user/0


kinda reinforced that suspicion.

So I checked the logs further: -

fgrep -R device /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/logs

and found: -=

...
/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/logs/DBUpgrade_Mon-Oct-08-11.40.36-BST-2018_0.log:INSERT INTO SYNCCONFIGURATION ( ID, ATTRIBUTES ) VALUES ('com.ibm.ecm.sync.WebAppListener.deviceCleanupInitialDelay', 'name=Initial deplay for the database clean up task;descr=Initial deplay for the database clean up task (in days);attributes=1')
/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/logs/DBUpgrade_Mon-Oct-08-11.40.36-BST-2018_0.log:INSERT INTO SYNCCONFIGURATION ( ID, ATTRIBUTES ) VALUES ('com.ibm.ecm.sync.WebAppListener.deviceCleanupPeriod', 'name=Device clean up cycle time;descr=Device clean up cycle time (in days);attributes=7')
/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/logs/DBUpgrade_Mon-Oct-08-11.40.36-BST-2018_0.log:INSERT INTO SYNCCONFIGURATION ( ID, ATTRIBUTES ) VALUES ('com.ibm.ecm.sync.tools.cleanup.CleanupTask.deviceExpiration','name=Device expiration;descr=device expiration (in days);attributes=90')
/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc_20181008-114610/ffdc.1987123897687396422.txt:Caused by: java.io.IOException: No space left on device
/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc_20181008-114610/ffdc.1987123897687396422.txt:   java.lang.Throwable::detailMessage:No space left on device

...

which kinda confirmed things.

Off to add more disk ……

Thursday, 4 October 2018

BPMConfig - Validation of a Deployment Environment

So this is a new one to me ….

In the context of changing WAS / BPM passwords, and then validating the changes, the IBM Knowledge Center has this to say: -



Having documented the password change process for my client, I thought I'd give the validation a try 

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -validate -profile Dmgr02 -de De1 -outputDir /tmp/val

The output was: -

ls -R -al /tmp/val/

/tmp/val/:
total 4
drwxr-xr-x   3 wasadmin wasadmins   17 Oct  4 09:04 .
drwxrwxrwt. 19 root     root      4096 Oct  4 09:34 ..
drwxr-xr-x   3 wasadmin wasadmins   54 Oct  4 09:04 html

/tmp/val/html:
total 36
drwxr-xr-x 3 wasadmin wasadmins    54 Oct  4 09:04 .
drwxr-xr-x 3 wasadmin wasadmins    17 Oct  4 09:04 ..
-rw-r--r-- 1 wasadmin wasadmins 34573 Oct  4 09:04 configValidationReport_De1.html
drwxr-xr-x 3 wasadmin wasadmins    51 Oct  4 09:04 css

/tmp/val/html/css:
total 12
drwxr-xr-x 3 wasadmin wasadmins   51 Oct  4 09:04 .
drwxr-xr-x 3 wasadmin wasadmins   54 Oct  4 09:04 ..
drwxr-xr-x 2 wasadmin wasadmins 4096 Oct  4 09:04 images
-rw-r--r-- 1 wasadmin wasadmins 1966 Oct  4 09:04 style.css
-rw-r--r-- 1 wasadmin wasadmins  849 Oct  4 09:04 table.css

/tmp/val/html/css/images:
total 40
drwxr-xr-x 2 wasadmin wasadmins 4096 Oct  4 09:04 .
drwxr-xr-x 3 wasadmin wasadmins   51 Oct  4 09:04 ..
-rw-r--r-- 1 wasadmin wasadmins  155 Oct  4 09:04 error.gif
-rw-r--r-- 1 wasadmin wasadmins  157 Oct  4 09:04 ibm-logo-white.gif
-rw-r--r-- 1 wasadmin wasadmins  142 Oct  4 09:04 part_start.gif
-rw-r--r-- 1 wasadmin wasadmins  146 Oct  4 09:04 part_stop.gif
-rw-r--r-- 1 wasadmin wasadmins  145 Oct  4 09:04 running.gif
-rw-r--r-- 1 wasadmin wasadmins  146 Oct  4 09:04 stop.gif
-rw-r--r-- 1 wasadmin wasadmins  297 Oct  4 09:04 success.gif
-rw-r--r-- 1 wasadmin wasadmins  149 Oct  4 09:04 unknown.gif
-rw-r--r-- 1 wasadmin wasadmins  158 Oct  4 09:04 warning.gif

and the HTML page looked like this: -


Nice !

Monday, 1 October 2018

WebSphere Application Server and the Case of the Bad File Store

We saw an interesting situation today, with a client's IBM WebSphere Application Server (WAS) 8.5.5.13 environment.

Whereas most clients, in my recent experience, tend to use a relational database such as DB2, Oracle or SQL Server, to host the Service Integration Bus (SIB) Messaging Engine (ME) datastore, this particular client was using the local Linux box's file-system.

When the cluster hosting the SIB came up, a message similar to this: -

CWSIS1561E:The messaging engine's unique id does not match that found in the filestore. ME_UUID=A85B64156006973D, ME_UUID(FS)=AAF19AC8C805638A

was posted in SystemOut.log.

Having checked the SIB/ME configuration, I could see that we had three file-system-based artefacts: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msglog/Log
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msgstore/PermanentStore
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/tempmsgstore/TemporaryStore

which translates as: -

Log File
Permanent Store File
Temporary Store File

as per this: -



Thankfully, this developerWorks Answers post provided the insight: -


CWSIS1561E errors are usually caused by deleting the messaging engine, without deleting its filestore, then recreating the messaging engine with the same name. To fix the issue, you will have to delete the existing filestore files for the messaging engine and restart the JVM. A new filestore will be created during the messaging engine startup.

Note1: You must treat the log file, the temporary store file, and the permanent store file as one unit; that is, delete operation must be performed on all three files.

Note2:By performing the suggested action, all the messages in the filestore will be lost.

To be on the safe side, we backed up the existing files ( all of which were there, with the correct ownership and permissions ), and then deleted the subdirectories: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msglog/
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msgstore/
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/tempmsgstore/

and bounced WAS.

Job done :-)

This also provided some useful context: -