Fun with DB2 UDB following loss of / file system

As per my earlier post: -

Fun with Linux disks

I saw this when testing JDBC connections: -

 The test connection operation failed for data source Monitor_Admin_Database on server dmgr at node BAMDMNODENode with the following exception: java.sql.SQLNonTransientException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082. View JVM logs for further details.

having seen exceptions such as: -

com.ibm.wbimonitor.lifecycle.LifecycleStopRequestScanTask run() CWMLC0012E: Unexpected exception [com.ibm.wbimonitor.persistence.spi.MonitorPersistenceException: com.ibm.websphere.ce.cm.StaleConnectionException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082].

[3/4/13 13:24:10:878 GMT] 00000014 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/logs/ffdc/dmgr_d363f0f1_13.03.04_13.24.10.8779174053313322880198.txt com.ibm.wbimonitor.lifecycle.LifecycleUpdateScanTask 106

[3/4/13 13:24:10:879 GMT] 00000014 LifecycleUpda E com.ibm.wbimonitor.lifecycle.LifecycleUpdateScanTask run() CWMLC0012E: Unexpected exception [com.ibm.wbimonitor.persistence.spi.MonitorPersistenceException: com.ibm.websphere.ce.cm.StaleConnectionException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082].

I validated the password ( su - db2inst1 ), and also updated the J2C alias for the db2inst1 user, but to no avail.

There were a few Google hits relating to the "ROOT CAPABILITY REQUIRED" part of the error message, including that made reference to the db2ckpwd process ( which should be running as root ).

This led me to the realization that the DAS ( dasusr1 ) and instance ( db2inst1 ) were "orphaned", as validated by these two commands: -

$ /opt/ibm/db2/V9.7/instance/db2ilist

and

$ /opt/ibm/db2/V9.7/instance/daslist

both of which returned no data.

In the end, I had to remove and recreate the db2inst1, db2fenc1 and dasusr1 users, and recreate the instances: -

$ userdel -r dasusr1 
$ userdel -r db2fenc1 
$ userdel -r db2inst1

$ useradd -u 501 -g db2iadm1 -G db2iadm1,dasadm1 db2inst1 
$ useradd -u 502 -g db2fadm1 db2fenc1 
$ useradd -u 500 -g dasadm1 dasusr1

$ passwd db2inst1 
$ passwd db2fenc1

$ /opt/ibm/db2/V9.7/instance/dascrt -u dasusr1 
$ /opt/ibm/db2/V9.7/instance/db2icrt -a SERVER -u db2fenc1 db2inst1

and set up autostart etc.

$ su - db2inst1

$ db2iauto -on db2inst1

$ db2set DB2AUTOSTART=YES 
$ db2set DB2COMM=tcpip 

$ db2 update dbm config using SVCENAME db2c_db2inst1 

DB20000I  The UPDATE DATABASE MANAGER CONFIGURATION command completed successfully.

$ db2 get dbm config | grep SVCE

 TCP/IP Service name                          (SVCENAME) = db2c_db2inst1 
 SSL service name                         (SSL_SVCENAME) =

$ db2stop

03/04/2013 13:53:25     0   0   SQL1064N  DB2STOP processing was successful. 
SQL1064N  DB2STOP processing was successful.

$ db2start

03/04/2013 13:53:29     0   0   SQL1063N  DB2START processing was successful. 
SQL1063N  DB2START processing was successful.

and we are back in the game.