Strange behaviour with CUSTOMIZATION and COMMUNITY databases in WebSphere Portal 7

I saw some strange behaviour with my WebSphere Portal 7.0.0.2 environment, after a migration from Derby to DB2 UDB.

This occurred AFTER I'd restarted WebSphere Portal, immediately following a successful database migration.

As I watched the SystemOut.log during the startup, I saw errors relating to the Customization and Community databases, including: -

...
3/17/12 8:21:26:235 GMT] 00000012 DataStoreCont E com.ibm.wps.datastore.impl.DataStoreContext handleException EJPDB0002E: Error occurred during database access.
                                 com.ibm.wps.datastore.domains.FFDCDomainUnavailableException: EJPDB0101E: Database domain [Domain: cust] is currently unavailable.
...
Caused by: com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][2057][11264][4.8.87] The application server rejected establishment of the connection.
An attempt was made to access a database, cust, which was either not found or does not support transactions. ERRORCODE=-4499, SQLSTATE=08004DSRA0010E: SQL State = 08004, Error Code = -4,499
...
Caused by: java.sql.SQLNonTransientException: [jcc][t4][2057][11264][4.8.87] The application server rejected establishment of the connection.
An attempt was made to access a database, cust, which was either not found or does not support transactions. ERRORCODE=-4499, SQLSTATE=08004DSRA0010E: SQL State = 08004, Error Code = -4,499
...
3/17/12 8:21:29:769 GMT] 00000012 DataStoreCont E com.ibm.wps.datastore.impl.DataStoreContext handleException EJPDB0002E: Error occurred during database access.
                                 com.ibm.wps.datastore.domains.FFDCDomainUnavailableException: EJPDB0101E: Database domain [Domain: comm] is currently unavailable.
 
Caused by: com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][2057][11264][4.8.87] The application server rejected establishment of the connection.
An attempt was made to access a database, comm, which was either not found or does not support transactions. ERRORCODE=-4499, SQLSTATE=08004DSRA0010E: SQL State = 08004, Error Code = -4,499
...
Caused by: java.sql.SQLNonTransientException: [jcc][t4][2057][11264][4.8.87] The application server rejected establishment of the connection.
An attempt was made to access a database, comm, which was either not found or does not support transactions. ERRORCODE=-4499, SQLSTATE=08004DSRA0010E: SQL State = 08004, Error Code = -4,499
...

The first thing I did was to log into the WAS Integrated Solutions Console ( https://portal.uk.ibm.com:10032/ibm/console ) and test the JDBC data source for Customization - wpdbDS_customization - which immediately failed with: -

The test connection operation failed for data source wpdbDS_community on server WebSphere_Portal at node portal with the following exception: java.sql.SQLNonTransientException: [jcc][t4][2030][11211][4.8.87] A communication error occurred during operations on the connection's underlying socket, socket input stream, or socket output stream. Error location: Reply.fill(). Message: Connection reset. ERRORCODE=-4499, SQLSTATE=08001DSRA0010E: SQL State = 08001, Error Code = -4,499.

with the corresponding error: -

...
[3/17/12 16:10:08:873 GMT] 0000002f DSConfigurati W   DSRA8201W: DataSource Configuration: DSRA8040I: Failed to connect to the DataSource.  Encountered java.sql.SQLNonTransientException: [jcc][t4][2030][11211][4.8.87] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream.  Error location: Reply.fill().  Message: Connection reset. ERRORCODE=-4499, SQLSTATE=08001DSRA0010E: SQL State = 08001, Error Code = -4,499.
java.sql.SQLNonTransientException: [jcc][t4][2030][11211][4.8.87] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream.  Error location: Reply.fill().  Message: Connection reset. ERRORCODE=-4499, SQLSTATE=08001DSRA0010E: SQL State = 08001, Error Code = -4,499
...

in SystemOut.log.

I repeated this test for the other WebSphere Portal-related data sources: -

wpdbDS_community
wpdbDS_feedback
wpdbDS_jcr
wpdbDS_likeminds
wpdbDS_release

and saw the same "The test connection operation failed..." for Community; the other four worked just fine.

Hmmm, I thought.

I then compared and contrasted the two offending data sources with the four that worked, and then checked the DB2 directory ( as user db2inst1 on the DB2 server ): -

$ db2 list db directory

 Database alias                       = FDBKDB
 Database alias                       = COMMUN
 Database alias                       = JCRDB
 Database alias                       = CUSTOM
 Database alias                       = LMDB
 Database alias                       = RELEASE

It didn't take me long to realise that, for some strange reason, the Database Transfer wizard had apparently mis-entered the database names for Customization and Community.

instead of being stored as CUSTOM and COMMUN respectively, they were actually stored as CUST and COMM, hence the problems.

Once I manually updated the data sources, the Test Connection button worked a treat.

Now to restart WebSphere Portal ....