IBM BPM - Failing to uninstall

I had an interesting, and only half-resolved, issue with an Service Component Archive (SCA) module that I'd created using IBM Integration Designer 8.5.6 and deployed to the in-built IBM BPM Advanced 8.5.6 runtime ( known as the Unit Test Environment or Integrated Test Environment ).

I got into a position where I was unable to publish, start or uninstall an application ( merely a basic SCA module containing an Interface, a MQ Binding and a BPEL flow ).

This is what I saw in the WAS SystemOut.log file: -

...

[28/06/15 05:30:59:528 BST] 00000111 AdminHelper   A   ADMN1010I: An attempt is made to stop the BPMDemoApp application. (User ID = defaultWIMFileBasedRealm/wasadmin)
[28/06/15 05:30:59:532 BST] 00000111 CompositionUn A   WSVR0192I: Stopping composition unit WebSphere:cuname=BPMDemoApp in BLA WebSphere:blaname=BPMDemoApp.
[28/06/15 05:30:59:534 BST] 00000111 ApplicationMg A   WSVR0217I: Stopping application: BPMDemoApp
[28/06/15 05:30:59:986 BST] 00000111 Core          I   CWSCA3011I: The SCA module "BPMDemoApp" is stopping.
[28/06/15 05:30:59:990 BST] 00000111 ActivationSpe I   J2CA0524I: The Message Endpoint for ActivationSpec sca/BPMDemo/ActivationSpec (com.ibm.ws.sib.ra.inbound.impl.SibRaActivationSpecImpl) and MDB Application BPMDemoApp#BPMDemo#BPMDemo is deactivated.
[28/06/15 05:30:59:992 BST] 00000111 Core          I   CWSCA3012I: The SCA module has been stopped.
[28/06/15 05:31:00:007 BST] 00000111 ApplicationMg A   WSVR0220I: Application stopped: BPMDemoApp
[28/06/15 05:31:06:432 BST] 00000111 CompositionUn A   WSVR0193I: Composition unit WebSphere:cuname=BPMDemoApp in BLA WebSphere:blaname=BPMDemoApp stopped.
[28/06/15 05:31:06:838 BST] 00000113 ServletWrappe I com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0242I: [isclite] [/ibm/console] [/secure/javascriptToSession.jsp]: Initialization successful.
[28/06/15 05:31:12:417 BST] 00000113 ServletWrappe I com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0242I: [isclite] [/ibm/console] [/com.ibm.ws.console.appmanagement/RemoveSplat.jsp]: Initialization successful.
[28/06/15 05:31:15:971 BST] 00000125 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/logs/ffdc/server1_267904e7_15.06.28_05.31.15.9676531005148371738678.txt com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions 290
[28/06/15 05:31:15:983 BST] 00000125 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/logs/ffdc/server1_267904e7_15.06.28_05.31.15.971705321591203166738.txt com.ibm.ws.management.application.SchedulerImpl.run 328
[28/06/15 05:31:15:985 BST] 00000125 UninstallSche I   ADMA5107E: The application BPMDemoApp cannot be uninstalled.
...

with this in the FFDC logs: -

...

[28/06/15 05:31:15:968 BST]     FFDC Exception:java.lang.ClassCastException SourceId:com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions ProbeId:290 Reporter:java.lang.Class@2bc12930
java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler
at com.ibm.task.admin.HumanTaskTaskProvider.provideServerUninstallExtensions(HumanTaskTaskProvider.java:187)
at com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions(ExtensionHelper.java:286)
at com.ibm.ws.management.application.UninstallSchedulerImpl.setupTasks(UninstallSchedulerImpl.java:156)
at com.ibm.ws.management.application.SchedulerImpl.run(SchedulerImpl.java:303)
at java.lang.Thread.run(Thread.java:801)
...

[28/06/15 05:31:15:971 BST]     FFDC Exception:com.ibm.websphere.management.application.client.AppDeploymentException SourceId:com.ibm.ws.management.application.SchedulerImpl.run ProbeId:328 Reporter:java.lang.Class@2bc12930
com.ibm.websphere.management.application.client.AppDeploymentException: java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler [Root exception is java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler]
        at com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions(ExtensionHelper.java:294)
        at com.ibm.ws.management.application.UninstallSchedulerImpl.setupTasks(UninstallSchedulerImpl.java:156)
        at com.ibm.ws.management.application.SchedulerImpl.run(SchedulerImpl.java:303)
        at java.lang.Thread.run(Thread.java:801)
Caused by: java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler
        at com.ibm.task.admin.HumanTaskTaskProvider.provideServerUninstallExtensions(HumanTaskTaskProvider.java:187)
        at com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions(ExtensionHelper.java:286)
        ... 3 more

...

I then tried to uninstall the application using an old-but-good Jacl script: -

/opt/IBM/WebSphere/AppServer/profiles/bpmprofile/bin/wsadmin.sh -lang jacl -user wasadmin -password passw0rd -f /opt/IBM/WebSphere/AppServer/ProcessChoreographer/admin/bpcTemplates.jacl -uninstall BPMDemoApp -force

WASX7209I: Connected to process "server1" on node Node1 using SOAP connector;  The type of process is: UnManagedProcess
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[-uninstall, BPMDemoApp, -force]"

Script 'bpcTemplates.jacl' has been deprecated in BPM 8.5.5 and replaced by /opt/IBM/WebSphere/AppServer/ProcessChoreographer/admin/listTemplates.py and manageTemplates.py


Application 'BPMDemoApp' is deployed on server 'server1' of node 'Node1'.
   Process templates:
     No process templates found in 'BPMDemoApp'.
   Task templates:
     No task templates found in 'BPMDemoApp'.
ADMA5011I: The cleanup of the temp directory for application BPMDemoApp is complete.
ADMA5107E: The application BPMDemoApp cannot be uninstalled.
Error stopping and force uninstalling BPMDemoApp: com.ibm.ws.scripting.ScriptingException: WASX7133E: Application uninstall of BPMDemoApp failed: see previous messages for details..

which struck out :-(

<CAVEAT> Don't try this at home, kids</CAVEAT>

Finally, I took the "nuclear" option of physically removing the EAR file: -

cd /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/config/cells/PSCell1/applications
rm -Rf BPMDemoApp.ear/

cd /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/installedApps/PSCell1
rm -Rf BPMDemoApp.ear/

and validated that, at least from a file-system perspective, it had gone: -

find /opt/IBM/WebSphere/AppServer/ -name BPMDemo

which returned NOWT.

<CAVEAT> Don't try this at home, kids</CAVEAT>

The application is still "there" as far as WAS is concerned, as it appears in the Applications perspective in the ISC, but it's not stopping my server from starting, so I can ignore it - for now.

And this is only my personal test VM.