CWWIM4529E and SECJ0369E seen when authenticating a user in WebSphere Application Server

If you see exceptions such as: -

[08/04/13 13:11:01:956 BST] 0000001d LTPAServerObj E   SECJ0369E: Authentication failed when using LTPA. The exception is com.ibm.websphere.wim.exception.PasswordCheckFailedException: CWWIM4529E  The password verification for the 'wasadmin' principal name failed. Root cause: 'javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid Credentials]; Resolved object: 'com.sun.jndi.ldap.LdapCtx@a9d2c4f9''..

[08/04/13 12:35:42:483 BST] 0000001c LTPAServerObj E   SECJ0369E: Authentication failed when using LTPA. The exception is com.ibm.websphere.wim.exception.WIMException.
[08/04/13 12:35:42:485 BST] 0000001c FormLoginExte E   SECJ0118E: Authentication error during authentication for user wasadmin

[08/04/13 12:34:41:460 BST] 0000001c UserRegistryI E   SECJ0363E: Cannot create credential for the user wasadmin because of the following exception com.ibm.websphere.wim.exception.WIMException
[08/04/13 12:34:41:469 BST] 0000001c LTPAServerObj E   SECJ0373E: Cannot create credential for the user wasadmin due to failed validation of the LTPA token. The exception is com.ibm.websphere.wim.exception.WIMException

and, most tellingly: -

[08/04/13 12:46:28:546 BST] 00000024 exception     E com.ibm.ws.wim.adapter.file.was.FileAdapter create
                                 com.ibm.websphere.wim.exception.EntityAlreadyExistsException: CWWIM4501E  An entity with same unique name, 'uid=wasadmin,o=defaultWIMFileBasedRealm', or same RDN value already exists.

and ( in SystemErr.log ): -

[08/04/13 12:46:28:557 BST] 00000024 SystemErr     R com.ibm.websphere.wim.exception.EntityAlreadyExistsException: CWWIM4501E  An entity with same unique name, 'uid=wasadmin,o=defaultWIMFileBasedRealm', or same RDN value already exists.

there's a very good chance that you do have TWO wasadmin accounts :-)

In our case, we were using wasadmin from the File-Based Registry, which was working perfectly UNTIL we added LDAP.

After much faffing about, we realised that we did indeed have a duplicate ID problem and, when we checked the LDAP, lo and behold, there was another account called wasadmin.

Strangely, this had worked a few weeks back, so I can only assume that (a) the LDAP-based wasadmin was new or (b) it had previously had the same password as the local account, but had been subsequently changed.

Bottom line, when adding LDAP, check that you're not going to hit this problem …. BEFORE you hit this problem.

If in doubt, make sure that the account you use when creating the profile ( which automagically creates the File-Based Registry ) is going to be unique, unique, unique.

:-)

IBM Rational Software Architect Version 8.5 for Mac OSX - A Technology Preview

IBM® Rational® Software Architect is an advanced and comprehensive application design, modeling and development tool for end-to-end software delivery. The latest version is updated with the latest in design and modeling technologies, comprehensive support for emerging technologies around BPMN2, SOA and Java™ Enterprise Edition 5, and delivers the best of breed tooling that integrates with IBM's application lifecycle management solutions.

Evaluate: IBM Rational Software Architect

Problems with IBM Business Monitor Messaging Engine ( SI Bus ) following a teardown

*CAVEAT*

This post relates to my OWN individual experiences on my OWN personal VMware environment. This is NOT NOT NOT a recipe for everyone; your mileage may vary. If in doubt, PLEASE raise a PMR with IBM Support

*CAVEAT*

Having performed a fresh installation of IBM Business Monitor 8.0.1.1 against Oracle 11g R2 after a "teardown" - where I cleaned up the database objects created the first time around - I noticed that the Messaging Engine cluster ( that hosts the Service Integration Bus ) kept restarting.

When I checked SystemOut.log for the offending cluster member, I found: -

...

[28/03/13 09:54:01:606 GMT] 0000001b SibMessage    I   [CEI.BAMCELL.BUS:BAMSR01.Messaging.000-CEI.BAMCELL.BUS] CWSIS1538I: The messaging engine, ME_UUID=3D59E737F07528C9, INC_UUID=62A8E276B06B1903, is attempting to obtain an exclusive lock on the data store.

[28/03/13 09:54:01:766 GMT] 0000001c SibMessage    I   [CEI.BAMCELL.BUS:BAMSR01.Messaging.000-CEI.BAMCELL.BUS] CWSIS1545I: A single previous owner was found in the messaging engine's data store, ME_UUID=09BF782E0B664719, INC_UUID=78437FD9A71F6596
[28/03/13 09:54:01:768 GMT] 0000001d SibMessage    I   [MONITOR.BAMCELL.Bus:BAMSR01.Messaging.000-MONITOR.BAMCELL.Bus] CWSIS1545I: A single previous owner was found in the messaging engine's data store, ME_UUID=E2ABE650D061BE5C, INC_UUID=ADD9DFC1AA982A5A
[28/03/13 09:54:01:771 GMT] 0000001c SibMessage    E   [CEI.BAMCELL.BUS:BAMSR01.Messaging.000-CEI.BAMCELL.BUS] CWSIS1535E: The messaging engine's unique id does not match that found in the data store. ME_UUID=3D59E737F07528C9, ME_UUID(DB)=09BF782E0B664719
[28/03/13 09:54:01:784 GMT] 0000001b SibMessage    I   [CEI.BAMCELL.BUS:BAMSR01.Messaging.000-CEI.BAMCELL.BUS] CWSIS1593I: The messaging engine, ME_UUID=3D59E737F07528C9, INC_UUID=62A8E276B06B1903, has failed to gain an initial lock on the data store.
[28/03/13 09:54:01:788 GMT] 0000001a SibMessage    I   [MONITOR.BAMCELL.Bus:BAMSR01.Messaging.000-MONITOR.BAMCELL.Bus] CWSIS1537I: The messaging engine, ME_UUID=E2ABE650D061BE5C, INC_UUID=5634F9A5B06B1901, has acquired an exclusive lock on the data store.
…

and: -

...

[28/03/13 09:55:53:555 GMT] 0000000f SibMessage    E   [CEI.BAMCELL.BUS:BAMSR01.Messaging.000-CEI.BAMCELL.BUS] CWSID0046E: Messaging engine BAMSR01.Messaging.000-CEI.BAMCELL.BUS detected an error and cannot continue to run in this server.
[28/03/13 09:55:53:555 GMT] 0000000f HAGroupImpl   I   HMGR0130I: The local member of group IBM_hc=BAMSR01.Messaging,WSAF_SIB_BUS=CEI.BAMCELL.BUS,WSAF_SIB_MESSAGING_ENGINE=BAMSR01.Messaging.000-CEI.BAMCELL.BUS,type=WSAF_SIB has indicated that is it not alive. The JVM will be terminated.
[28/03/13 09:55:53:566 GMT] 0000000f SystemOut     O Panic:component requested panic from isAlive
[28/03/13 09:55:53:567 GMT] 0000000f SystemOut     O java.lang.RuntimeException: emergencyShutdown called:
[28/03/13 09:55:53:567 GMT] 0000000f SystemOut     O    at com.ibm.ws.runtime.component.ServerImpl.emergencyShutdown(ServerImpl.java:632)
[28/03/13 09:55:53:567 GMT] 0000000f SystemOut     O    at com.ibm.ws.hamanager.runtime.RuntimeProviderImpl.panicJVM(RuntimeProviderImpl.java:92)
[28/03/13 09:55:53:569 GMT] 0000000f SystemOut     O    at com.ibm.ws.hamanager.coordinator.impl.JVMControllerImpl.panicJVM(JVMControllerImpl.java:56)
[28/03/13 09:55:53:569 GMT] 0000000f SystemOut     O    at com.ibm.ws.hamanager.impl.HAGroupImpl.doIsAlive(HAGroupImpl.java:882)
[28/03/13 09:55:53:569 GMT] 0000000f SystemOut     O    at com.ibm.ws.hamanager.impl.HAGroupImpl$HAGroupUserCallback.doCallback(HAGroupImpl.java:1388)
[28/03/13 09:55:53:569 GMT] 0000000f SystemOut     O    at com.ibm.ws.hamanager.impl.Worker.run(Worker.java:64)
[28/03/13 09:55:53:569 GMT] 0000000f SystemOut     O    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)
...

The first set of exceptions ( CWSIS1545I and CWSIS1535E ) led me to the solution, aided by this IBM Technote: -

CWSIS1535E The messaging engine's unique id (ME_UUID) does not match that found in the data store.

Resolution

I realised that, when I'd cleaned down the database objects from the previous installation of BAM, I'd neglected to remove the schemas for the Messaging Engine.

In Oracle, I used SQL*Plus: -

$ sqlplus / as SYSDBA

and ran: -

SQL> select username from dba_users;

USERNAME
------------------------------
COGNOS
IBMBUSSP
MONITOR
MONME00
MONCM00
SCOTT
…

This showed the two schema user objects - MONME00 and MONCM00 - which I then removed: -

SQL> drop user MONCM00 cascade;

User dropped.

SQL> drop user MONME00 cascade;

User dropped.

and then restarted the ME cluster member.

This automatically recreated the objects ( this is almost certainly NOT the default behaviour - most DBAs would prefer to have more control over the creation of database objects such as schemas and users ) and the ME came up without exception.

Job done :-)

*CAVEAT*

This post relates to my OWN individual experiences on my OWN personal VMware environment. This is NOT NOT NOT a recipe for everyone; your mileage may vary. If in doubt, PLEASE raise a PMR with IBM Support

*CAVEAT*

Good Decision! Five Useful Technical Patterns for Operational Decision Management

Technical patterns are useful as best practice guides for both selecting and using technology. The technical patterns for IBM Operational Decision Manager (ODM) technology can be divided into two sets, those that align with its business rules capabilities and those that align with the business events capabilities.

1. Complex or Volatile Routing

2. Input Data Validation

3. Data Enrichment and Calculation

4. Data Augmentation of Decision Service Requests

5. Application Modernization

Want to know more ? Then please visit Cheryl Wilson's blog here: -

Five Useful Technical Patterns for Operational Decision Management

Updating IBM Business Process Manager 8

I'm writing this down as I'm going to need it at some point in the future :-)

So here's a Bash script to list the fixes available to me: -

list_fixes.sh

#!/bin/sh 
for z in /store/BPM801/Fixes/*.zip 
do 
        echo $z 
Done

and one to list more detail about each fix: -

list_fix_detail.sh

#!/bin/sh 
for z in /store/BPM801/Fixes/*.zip 
        do 
        unzip -c $z repository.xml | grep 'information name=' | sed 's/^.\{25\}//' | sed "s/'.*$//g" 
done

and one to expand a list of fixes to a target directory structure: -

unpack_fixes.sh

#!/bin/sh 
for z in /store/BPM801/Fixes/*.zip 
do 
        unzip $z -d `echo $z |  sed 's/^.\{20\}//' | sed 's/.zip//g'` 
done

This is how we'd install a single fix: -

$ /opt/IBM/InstallationManager/eclipse/tools/imcl install 8.0.1.0-WS-BPMPC-IFPD44937 -repositories /tmp/BPM801Fixes/8.0.1.0-WS-BPM-IFJR44937 -installationDirectory /opt/IBM/WebSphere/AppServer/

Installed 8.0.1.0-WS-BPMPC-IFPD44937_8.0.1000.20121203_1006 to the /opt/IBM/WebSphere/AppServer directory.

and here's a script to install a whole bunch of fixes: -

install_fixes.sh

#!/bin/sh 
for z in /store/BPM801/Fixes/*.zip 
        do 
        /opt/IBM/InstallationManager/eclipse/tools/imcl install `unzip -c $z repository.xml | grep 'information name=' | sed 's/^.\{25\}//' | sed "s/'.*$//g"` -repositories /tmp/BPM801Fixes/`echo $z |  sed 's/^.\{20\}//' | sed 's/.zip//g'` -installationDirectory /opt/IBM/WebSphere/AppServer -log ~/$z.txt

done

and here's how we validated what's installed: -

$ /opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

com.ibm.cic.agent_1.6.0.20120831_1216 
com.ibm.websphere.IHS.v80_8.0.5.20121022_1902 
com.ibm.bpm.ADV.V80_8.0.1000.20121102_2136 
com.ibm.websphere.ND.v80_8.0.5.20121022_1902 
8.0.1.0-WS-BPM-IFJR44669_8.0.1000.20121113_1004 
8.0.1.0-WS-BPM-IFJR44742_8.0.1000.20121119_1510 
8.0.1.0-WS-BPM-IFJR44786_8.0.1000.20121214_1109 
8.0.1.0-WS-BPM-IFJR44894_8.0.1000.20121129_1038 
8.0.1.0-WS-BPM-IFJR44978_8.0.1000.20121212_1322 
8.0.1.0-WS-BPM-IFJR45014_8.0.1000.20121207_1055 
8.0.1.0-WS-BPM-IFJR45047_8.0.1000.20121211_1048 
8.0.1.0-WS-BPM-IFJR45071_8.0.1000.20121218_1447 
8.0.1.0-WS-BPM-IFJR45089_8.0.1000.20130124_1057 
8.0.1.0-WS-BPM-IFJR45113_8.0.1000.20121217_2010 
8.0.1.0-WS-BPM-IFJR45135_8.0.1000.20130121_1715 
8.0.1.0-WS-BPM-IFJR45227_8.0.1000.20130109_1543 
8.0.1.0-WS-BPM-IFJR45386_8.0.1000.20130119_0019 
8.0.1.0-WS-BPM-IFJR45389_8.0.1000.20130118_0935 
8.0.1.0-WS-BPMADV-IFJR45540_8.0.1000.20130131_1111 
8.0.1.0-WS-BPMPC-IFJR45179_8.0.1000.20130131_1354 
8.0.1.0-WS-BPMPC-IFJR45385_8.0.1000.20130121_1156 
8.0.1.0-WS-BPMPC-IFPD44937_8.0.1000.20121203_1006 
8.0.1.0-WS-BPMPC-IFPD45014_8.0.1000.20121207_1108 
8.0.1.0-WS-BPMPC-IFPD45227_8.0.1000.20130109_1602 
8.0.1.0-WS-BSPACE-IFJR44853_8.0.1000.20121127_1628 
8.0.1.0-WS-BSPACE-IFJR45108_8.0.1000.20121212_1039 
com.ibm.websphere.PLG.v80_8.0.5.20121022_1902

More about Oracle DB ...

Following on from my earlier post about Oracle DB: -

Oracle Database - My First Few Baby Steps .... 

one of my colleagues picked up on my point about starting the SQLPlus application with authentication: -

$ sqlplus sys/passw0rd@orcl as SYSDBA

by saying: -

"Whenever you use "as sysdba" as a mechanism to log on, Oracle will assume you are logging in using OS level authentication.  As such you don't need to provide a username and password.  This method of logon is only available to a few ....

$export ORACLE_SID=orcl 
$sqlplus / as sysdba

For the above to work, the OS user you are logged on with as you run the above, must be a member of the DBA OS user group.  So how come the way you've logged on works?  Simple, Oracle ignores any credentials provided when you use 'as sysdba'.  So try it, change the username and password to anything and you should still get on ( It's important that someone, usually the sys user, can be authenticated in this manner, i.e. externally to the database, as when the db's shut down, someone needs to be capable of starting the thing up.  As all credentials are only available for querying once the db is up, this would provide a catch 22 situation for db startup.  Not sure what DB2 and SQL Server etc. employ? ).  As for all other 'normal' users, they can't log on to the db until the dba has started the db, by which time the Oracle data dictionary is open and can now be queried for authentication purposes.

The 'create user' command is fine, but it will create a user called 'monitor' which won't be able to effectively log on interactively, but if as I suspect it's a system account, you wouldn't want anyone to log on as that user? If you do want the monitor user to actually be capable of logging on you would grant the user that privilege with 'grant create session to monitor'.  You would normally allocate a default tablespace for the newly created user, otherwise it will use whichever tablespace is defined as the catch all default tablespace which isn't a great idea going forwards as then every user gets thrown in to this catch all tablespace and makes management more difficult.  A newly created standard user would often be created along the lines of ...

CREATE USER monitor
    IDENTIFIED BY passw0rd 
    DEFAULT TABLESPACE example 
    QUOTA 10M ON example 
    TEMPORARY TABLESPACE temp
    PASSWORD EXPIRE;

Clearly one wouldn't expire the password on a system account.  Often a newly created user is simply created using an existing profile which is fit for purpose and the profile would give most of the above and more in one slice."

PS With regard to his comment about DB2, the answer is pretty simple - DB2 "delegates" authentication to the OS, so one can only start the database instance ( or the DAS ) once one has authenticated to the underlying OS e.g. su - db2inst1 -c db2start or su - dasusr1 -c "db2admin start" - in both cases, one will likely need to pass the password for the Unix account.

IBM Business Monitor 8.0.1.1 and the Dispatcher

As per my previous posts, I've just started to dabble with Oracle 11g R2 as my database for IBM Business Monitor, instead of using DB2 which is my default position.

The installation had been relatively smooth, but I then had an issue with the Cognos Dispatcher which, I was sure, was Oracle-related.

Think again ….

I was seeing: -

127.0.0.1:9097 31198 2013-03-26 12:00:54.796 +0 pogoStartup na na 0 Thread-54 DISP 6235 1 Audit.Other.dispatcher.DISP.pogo pogo com.cognos.pogo.contentmanager.coordinator.ActiveCMControl Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>DPR-DPR-1004 Expecting a BI Bus XML response but got:   </messageString></message></messages> DPR-CMI-4007 Unable to perform an active Content Manager election on the local IP node. For more information, see the dispatcher and Content Manager detailed logs. Ensure that the local Content Manager service is started.DPR-DPR-1004 Expecting a BI Bus XML response but got: com.cognos.pogo.bibus.CommandExecutionException: DPR-DPR-1004 Expecting a BI Bus XML response but got: at com.cognos.pogo.bibus.BIBusCommand.handleDefaultException(BIBusCommand.java:294) at 

in the cogserver.log after building a new BAM 8011 environment against Oracle 11g R2.

Whilst poking about in cogstartup.xml, I realised that the Dispatcher was sitting on port 9097 rather than 9081: -

$ cat ../configuration/cogstartup.xml | grep bam8011.uk.ibm.com

    <crn:value xsi:type="xsd:anyURI">http://bam8011.uk.ibm.com:9097/p2pd/servlet/dispatch</crn:value>

      <crn:item xsi:type="xsd:anyURI" order="0">http://bam8011.uk.ibm.com:9097/p2pd/servlet/dispatch/ext</crn:item>

    <crn:value xsi:type="cfg:localURI">http://bam8011.uk.ibm.com:9097/p2pd/servlet/dispatch</crn:value>

    <crn:value xsi:type="cfg:localURI">http://bam8011.uk.ibm.com:9097/p2pd/servlet/dispatch</crn:value>

    <crn:value xsi:type="xsd:anyURI">http://bam8011.uk.ibm.com:9097/p2pd/servlet/dispatch</crn:value>

      <crn:item xsi:type="xsd:anyURI">http://bam8011.uk.ibm.com:9097/p2pd/servlet</crn:item>

          <crn:item xsi:type="xsd:string">bam8011.uk.ibm.com:9097</crn:item>

Given that Monitor/Cognos can't currently access the Dispatcher on an HTTPS port, I'd previously handled this by adding a Virtual Host alias to default_host for port 9081.

Therefore, you can guess the problem ....

Yep, whilst I could access port 9097: -

$ telnet bam8011.uk.ibm.com 9097

I was then getting: -

SRVE0255E: A WebGroup/Virtual Host to handle /p2pd has not been defined.

SRVE0255E: A WebGroup/Virtual Host to handle bam8011.uk.ibm.com:9043 has not been defined.

when I tried to access the Dispatcher via a browser.

Once I added a new alias for 9097, and restarted the Support cluster, I got further forward, but was then seeing: -

Failure SecureErrorId: 2013-03-26-14:23:03.321-#1  Original Error: DPR-ERR-2088 The requested Server Group '' does not exist.  Handler trace back: [the_dispatcher] com.cognos.pogo.handlers.performance.PerformanceIndicationHandler [the_dispatcher] com.cognos.pogo.handlers.logic.ChainHandler [service_lookup] com.cognos.pogo.handlers.engine.ServiceLookupHandler [load_balancer] com.cognos.pogo.handlers.logic.ChainHandler [lb_forwarder] com.cognos.p2plb.clerver.LoadBalanceHandler

Thinking semi-laterally, I re-generated and re-propagated the Plugin, and got even further forward: -

 

Once I restarted the IHS server, I was in like Flynn

 

Note that this allows me to access the Dispatched on an HTTPS port ( 9443 in my case ), *BUT* the non-HTTPS port ( now 9097 rather than 9081 ) is required because of a current issue with Monitor/Cognos.

*UPDATE 27/03/2013* I've just completed a new, clean, installation of Monitor 8.0.1.1, and, quelle surprise, my Dispatcher is back on port 9081, rather than 9097. I'm not too sure what happened; I can only assume that something was hogging 9081 when I did the previous install, meaning that port 9097 was allocated instead. I can't explain it but ....