IBM BPM Advanced - Problems with the Internal Document Store - Be careful what you change, and when

Ooops, I broke it, it was me :-)

Whilst migrating a BPM Advanced 8.5.5 environment from a file-based user registry ( fileRegistry.xml ) to Microsoft Active Directory ( via LDAP ), I inadvertently broke the IBM BPM Document Store ( based upon FileNet ).

I started seeing exceptions such as: -

[24/09/15 16:16:41:190 BST] 0000006d EmbeddedECMIn E   CWTDS1100E: An error occurred while validating or creating the default configuration for the IBM BPM document store.
                                 com.ibm.bpm.embeddedecm.exception.UserMissesWritePermissionException: CWTDS0022E: The configuration was changed in a way that the technical user 'DepEnvAdmin' of the IBM BPM document store fails to change the object 'Domain'.
Explanation: The technical user defined in the BPM role type 'EmbeddedECMTechnicalUser' is not permitted to perform changes on an object.
Action: Revert the recent configuration changes. Ensure that the user defined by the BPM role type 'EmbeddedECMTechnicalUser' has access to the object. Verify this using the admin task 'getDocumentStoreStatus'.

in the AppCluster SystemOut.log.

When I tried to work with the Document Store Authorisation service: -

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -add uid=DepEnvAdmin,o=defaultWIMFileBasedRealm]')

I saw: -

WASX7015E: Exception running command: "AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -add uid=DepEnvAdmin,o=defaultWIMFileBasedRealm]')"; exception information:
com.ibm.bpm.embeddedecm.exception.UnexpectedFailureException: com.ibm.bpm.embeddedecm.exception.UnexpectedFailureException: CWTDS0000E: An unexpected failure occurred. Details: 'FNRCE0057: E_READ_ONLY: The method failed because an object or property is read-only.'
Explanation: An exception was thrown.
Action: Check the server log files.

The root cause ?

I'd switched from the file-based user - deadline - to the LDAP user - DepEnvAdmin ( that's the subject of another blog post ) *BUT* I neglected to "tell" the Document Store before I deleted the deadmin user from the file-based registry: -

AdminTask.deleteUser('[-uniqueName uid=deadmin,o=defaultWIMFileBasedRealm]')

I dug around, and found this IBM Technote: -

IBM Business Process Manager (BPM) cannot communicate with the IBM BPM document store

and this: -

Administering the technical user for the IBM BPM document store

Thankfully, I *DID* have a backup of the file-based registry: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/PCCell1/fileRegistry.xml

<?xml version="1.0" encoding="UTF-8"?>
<xml.type:datagraph xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wim="http://www.ibm.com/websphere/wim"
    xmlns:xml.type="commonj.sdo">
  <changeSummary xmlns=""/>
  <wim:Root>
    <wim:entities xsi:type="wim:PersonAccount">
      <wim:identifier externalId="be7f5abc-c40e-41ff-81e6-f7d1cfbaf214" externalName="uid=wasadmin,o=defaultWIMFileBasedRealm"
          uniqueId="be7f5abc-c40e-41ff-81e6-f7d1cfbaf214" uniqueName="uid=wasadmin,o=defaultWIMFileBasedRealm"/>
      <wim:parent>
        <wim:identifier uniqueName="o=defaultWIMFileBasedRealm"/>
      </wim:parent>
      <wim:createTimestamp>2015-09-22T10:27:30.890Z</wim:createTimestamp>
      <wim:password>U0hBLTE6dXhtbjllY2o3YjQzOm9lY0tsbXh2ZXVGeWZXY00xTUl5NlNNR0x5UT0K</wim:password>
      <wim:uid>wasadmin</wim:uid>
      <wim:cn>wasadmin</wim:cn>
      <wim:sn>wasadmin</wim:sn>
    </wim:entities>
    <wim:entities xsi:type="wim:PersonAccount">
      <wim:identifier externalId="ffe99ec4-a1c6-4d52-93f9-02d7d80e55b6" externalName="uid=deAdmin,o=defaultWIMFileBasedRealm"
          uniqueId="ffe99ec4-a1c6-4d52-93f9-02d7d80e55b6" uniqueName="uid=deAdmin,o=defaultWIMFileBasedRealm"/>
      <wim:parent>
        <wim:identifier uniqueName="o=defaultWIMFileBasedRealm"/>
      </wim:parent>
      <wim:createTimestamp>2015-09-22T11:41:43.017+01:00</wim:createTimestamp>
      <wim:password>U0hBLTE6aGxsdjlsZWtlMW1qOlhVQ2tNUm9rUmR6bnM5WWRVTTEvUkNNVkFFND0K</wim:password>
      <wim:uid>deAdmin</wim:uid>
      <wim:cn>deAdmin</wim:cn>
      <wim:sn>deAdmin</wim:sn>
    </wim:entities>
  </wim:Root>
</xml.type:datagraph>

I restored this from my backup, and restarted the AppCluster AND the Deployment Manager.

This time around the cluster started without exception, and I was able to access the Document Store using WSadmin: -

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -list]')

"Authorization on the domain for the IBM BPM document store\nAuthorization on the object store for the IBM BPM document store\nCWTDS2035I: Access is granted to the IBM BPM document store object store 'uid=deAdmin,o=defaultWIMFileBasedRealm' with access mask '838,205,440'."

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -add CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com]')

"CWTDS1005I: Domain access for the IBM BPM document store granted for principal 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com'.\nCWTDS1006I: Object store access for the IBM BPM document store granted for principal 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com'.\nCWTDS2027I: The access to the IBM BPM document store was successfully modified."

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -list]')

"Authorization on the domain for the IBM BPM document store\nCWTDS2034I: Access is granted to the IBM BPM document store domain 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com' with access mask '459,267'.\nCWTDS2034I: Access is granted to the IBM BPM document store domain 'uid=deAdmin,o=defaultWIMFileBasedRealm' with access mask '459,267'.\nAuthorization on the object store for the IBM BPM document store\nCWTDS2035I: Access is granted to the IBM BPM document store object store 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com' with access mask '838,205,440'.\nCWTDS2035I: Access is granted to the IBM BPM document store object store 'uid=deAdmin,o=defaultWIMFileBasedRealm' with access mask '838,205,440'."

I don't yet have a precise sequence of events BUT it's important to get this Document Store (ECM) stuff sorted BEFORE making any other substantive changes to WAS/BPM security i.e. updating J2C Authentication Aliases, deleting users etc.

Definitely read this: -

Administering the technical user for the IBM BPM document store

first :-)