Friday, 21 November 2014

DB2 Buffer Pools - Automagically tuning

So I picked this nugget of wisdom up this week.

One of my most excellent DB2 SME colleagues pointed out the wonderful db2top utility, specifically in the context of monitoring Buffer Pools.

You see, ever since I learned to install IBM Operational Decision Manager (ODM), I've been creating a Buffer Pool, bp32k, as required by the documentation and, more importantly, the product: -

db2 create bufferpool BP32K size 8000 pagesize 32 K

The DB2 SME, let's call him ... John, pointed out that a Buffer Pool of 8,000 pages x 32K may not always be large enough for one's requirements.

He showed me how db2top can be used to see this, as per the following example ( NOT from ODM, hence the different BP names ): -


In this example, we have 5 Buffer Pools, one of which IBMDEFAULTBP, is getting the most hits.

John pointed out that a Buffer Pool can be reconfigured to support automatic tuning, allowing it to grow IF the need arises.

Failure to grow means that the Buffer Pool is going to start paging to disk, and we do NOT want paging to occur.

So we can check what Buffer Pools we have in a database: -

db2 connect to bpmdb1

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.1.3
 SQL authorization ID   = DB2INST1
 Local database alias   = BPMDB1

db2 "SELECT BPNAME, NPAGES, PAGESIZE FROM SYSCAT.BUFFERPOOLS"

BPNAME                                                                                                                           NPAGES      PAGESIZE   
-------------------------------------------------------------------------------------------------------------------------------- ----------- -----------
IBMDEFAULTBP                                                                                                                              -2       32768

  1 record(s) selected.


and we can then check whether automatic configuration is enabled: -

db2 "select BP_NAME,AUTOMATIC FROM TABLE(MON_GET_BUFFERPOOL('',-2))"

BP_NAME                                                                                                                          AUTOMATIC
-------------------------------------------------------------------------------------------------------------------------------- ---------
IBMDEFAULTBP                                                                                                                             1
IBMSYSTEMBP4K                                                                                                                            0
IBMSYSTEMBP8K                                                                                                                            0
IBMSYSTEMBP16K                                                                                                                           0
IBMSYSTEMBP32K                                                                                                                           0

  5 record(s) selected.


*IF* the Buffer Pool hadn't been set to auto-tune, we could've then changed it as follows: -

db2 "alter bufferpool IBMDEFAULTBP immediate size 1000 automatic"

Bottom line, db2top is your friend, and one should always follow the same process: -

Instrument > Test > Monitor > Tune > Test > Monitor > Tune ............


IBM Business Process Manager - Missing the Bus

I've just built a single cell, two node three cluster IBM BPM Advanced 8.5.5 environment, against a remote DB2 ESE 10.1.0.3 server.

So I was a little startled when, after starting the Deployment Environment, the Service Integration Bus (SIbus) failed to properly start.

This is what I saw in one of my Cluster Member logs: -

[21/11/14 13:17:03:719 GMT] 00000073 SibMessage    I   [BPM.ProcessServer.Bus:MECluster.000-BPM.ProcessServer.Bus] CWSIS1593I: The messaging engine, ME_UUID=E997A9EFA09498FC, INC_UUID=6DC2A53AD19710D7, has failed to gain an initial lock on the data store.
[21/11/14 13:17:03:719 GMT] 00000073 SibMessage    I   [BPM.ProcessServer.Bus:MECluster.000-BPM.ProcessServer.Bus] CWSIS1538I: The messaging engine, ME_UUID=E997A9EFA09498FC, INC_UUID=6DC2A53AD19710D7, is attempting to obtain an exclusive lock on the data store.


This was a clean build, so the Messaging Engine database should have been OK.

The tables were definitely there: -

SIB000                          DB2USER1        T     2014-11-21-13.43.55.547439
SIB001                          DB2USER1        T     2014-11-21-13.43.55.682333
SIB002                          DB2USER1        T     2014-11-21-13.43.55.819494
SIBCLASSMAP                     DB2USER1        T     2014-11-21-13.43.55.334938
SIBKEYS                         DB2USER1        T     2014-11-21-13.43.55.947883
SIBLISTING                      DB2USER1        T     2014-11-21-13.43.55.420531
SIBOWNER                        DB2USER1        T     2014-11-21-13.43.55.151963
SIBOWNERO                       DB2USER1        T     2014-11-21-13.43.55.081007
SIBXACTS                        DB2USER1        T     2014-11-21-13.43.56.039355

and yet .... they were ALL empty :-(

As this is MY own environment, I called the ball and dropped the SIB tables: -

db2 drop table db2user1.sib000
db2 drop table db2user1.sib001
db2 drop table db2user1.sib002
db2 drop table db2user1.sibclassmap
db2 drop table db2user1.sibkeys
db2 drop table db2user1.siblisting
db2 drop table db2user1.sibowner
db2 drop table db2user1.sibownero
db2 drop table db2user1.sibxacts

and restarted the MECluster

This time around, the tables were nicely populated e.g.

db2 "select id from db2user1.sib000"

...
                 252
                 253
                 254
                 255
                 256
                 257
                 258
                 259
                 260
                 261
                 262
                 263
                 264
                 265
                 266
                 272

  269 record(s) selected.

...

and the SIbus comes up nicely: -

with JVM1 reports: -

[21/11/14 13:43:58:431 GMT] 0000006a SibMessage I [BPM.ProcessServer.Bus:MECluster.000-BPM.ProcessServer.Bus] CWSID0016I: Messaging engine MECluster.000-BPM.ProcessServer.Bus is in state Started.

and JVM2 reports: -   

[21/11/14 13:47:23:859 GMT] 00000065 SibMessage I [BPM.ProcessServer.Bus:MECluster.000-BPM.ProcessServer.Bus] CWSID0016I: Messaging engine MECluster.000-BPM.ProcessServer.Bus is in state Joined. 

In other words, the Bus Member on node 1 is active, with the Bus Member on node 2 standing by to take over.

When I stopped the MEClusterMember1 on node 1, I see this from node 2: -

[21/11/14 13:51:53:684 GMT] 00000097 SibMessage I [BPM.ProcessServer.Bus:MECluster.000-BPM.ProcessServer.Bus] CWSID0016I: Messaging engine MECluster.000-BPM.ProcessServer.Bus is in state Started. 

which again is as expected.

And, as  a final acid test, when I restart MEClusterMember1, I see this: -

[21/11/14 13:55:33:043 GMT] 00000062 SibMessage I [BPM.ProcessServer.Bus:MECluster.000-BPM.ProcessServer.Bus] CWSID0016I: Messaging engine MECluster.000-BPM.ProcessServer.Bus is in state Joined.

and stop MEClusterMember2, I see this: -

[21/11/14 13:57:33:123 GMT] 0000008f SibMessage I [BPM.ProcessServer.Bus:MECluster.000-BPM.ProcessServer.Bus] CWSID0016I: Messaging engine MECluster.000-BPM.ProcessServer.Bus is in state Started.

both messaging coming from node 1.

This shows that, once I dropped and recreated the SIB tables, the bus comes up nicely, and failover works both ways - node 1 to node 2 and node 2 to node 1.

This ties up with the IBM BPM pattern, known as 1-of-n, where only one ME / Bus Member can be active at any one time, regardless of the number of nodes in the cell / members in the cluster.

Which is nice.

So what went wrong ? I do not know, but I know how to resolve it AND, more importantly, watch for problems.

Some background reading: -



Book Review - Anti-Hacker Tool Kit, Fourth Edition, by Mike Shema

Again, following on from earlier posts: -




here's my latest book review, on which I am working in conjunction with the British Computer Society.


From their site: -

Welcome to the fourth edition of the Anti-Hacker Tool Kit. This is a book about the tools that hackers use to attack and defend systems. Knowing how to conduct advanced configuration for an operating system is a step toward being a hacker. Knowing how to infiltrate a system is a step along the same path. Knowing how to monitor an attacker's activity and defend a system are more points on the path to hacking. In other words, hacking is more about knowledge and creativity than it is about having a collection of tools.

and here is my review: -

As someone with an active interest in IT security, and as someone who practices in the area, albeit from an IT infrastructure perspective, I am always looking for new insights into the tools, techniques and tricks of the trade.

This book absolutely lives up to it's title, as it is an A-Z cookbook, taking one through the details of building a full penetration testing environment, using freely available and, in many cases, open source software.

If I have one niggle, it's that the book does not immediately set out the context of IT security in general or in specific, choosing to jump right on into the detail. As an example, the book starts with a chapter on Source Code Management, specifically Git, which is an interesting choice.

I would have preferred to see more detail on the typical vulnerabilities of common IT systems, perhaps with examples of "popular" attack vectors and well-known security breaches, also emphasising that security is as much about the people as the technology.

If one is looking for a primer on IT, including application development, virtualization, Unix/Linux etc., this book is definitely worth adding to one's library.

In terms of the specific intention, the book gets back into gear in Chapter 2 onwards, focusing on vulnerability scanning, auditing and monitoring, continuing to develop on the tooling theme from the earlier chapters.

Whilst this is definitely a book that one can and should read from (virtual) cover to cover, it's also useful to dip into for specific pieces of advice and guidance.

For anyone interested in IT security, and we should ALL be very interested in IT security, this book is one that I would strongly recommend adding to the library of must-read books.

Equally, I would also encourage this book to be part of any IT curriculum, as it is a relatively concise ( ~450 page ) tutorial for any budding practitioner of the art.

In conclusion, I recommend this book to anyone keen to know more about information security, software engineering and the fundamental building blocks of modern computer systems.

The book lives up to it's title as a tool kit, something one can dip into to find precisely the right tool for the job.

Thursday, 20 November 2014

DB2 - Still 8 characters after all these years ...

So I'm more than familiar with systems that are "limited" to 8 characters, but I didn't expect DB2 to have the same limitation, leastways not in 2014 :-)

This came up using DB2 Enterprise Server Edition 10.1.0.3 on AIX, but I've since recreated it using DB2 on Red Hat Enterprise Linux 6.6 as well.

The problem occurs when I tried to create a DB2 instance against an Unix user ID that's 9 characters long.

This is because I'm hosting 10 instances on one OS - db2inst1 through db2inst10.

Therefore, all are the same length APART from db2inst10 which is: -

db2inst10
^^^^^^^^^
|||||||||
123456789

9 characters in length.

When I try and create the 10th instance: -

/opt/ibm/db2/V10.1/instance/db2icrt -a SERVER -u db2fenc10 db2inst10

I get: -

DBI1446I  The db2icrt command is running, please wait.

DB2 installation is being initialized.
 
DBI1131E  The user ID db2inst1 is invalid.

Explanation: 

An attempt to access the given user ID failed. One of the following
situations has occurred: 
*  This user ID does not exist on the system.
*  The home directory of this user is not set up properly.
*  One of the user attributes needed by DB2 is unset.
*  The UID of this user is 0

User response: 

Make sure a valid user ID with valid home directory, shell, primary
group and secondary group has been used. Create a new user if necessary.

A major error occurred during the execution that caused this program to
terminate prematurely. If the problem persists, contact your technical service
representative.

For more information see the DB2 installation log at "/tmp/db2icrt.log.2716".
DBI1264E  Errors were encountered in running db2icrt. Please
      refer to the installation log file /tmp/db2icrt.log.2716 for more
      information.

Explanation: 

All processed and failed operations have been saved into this log file.

User response: 

Do not modify this file in any way. This file is for IBM Technical
Support reference.

Note that the exception has truncated the user ID to 8 characters :-(

Just for the record, here's my two 9-character user IDs; fenced user and instance owner.

cat /etc/passwd

...
db2fenc10:x:503:504::/home/db2fenc10:/bin/bash
db2inst10:x:504:505::/home/db2inst10:/bin/bash
...

In the case of the AIX environment, I already had db2inst1 created: -

/opt/ibm/db2/V10.1/instance/db2ilist 

db2inst1

so I instead see this: -

DBI1446I  The db2icrt command is running, please wait.

DB2 installation is being initialized.

 The instance "db2inst1" already exists. Specify a new instance name.

A major error occurred during the execution that caused this program to
terminate prematurely. If the problem persists, contact your technical service
representative.

For more information see the DB2 installation log at "/tmp/db2icrt.log.12179".
DBI1264E  Errors were encountered in running db2icrt. Please
      refer to the installation log file /tmp/db2icrt.log.12179 for more
      information.

Explanation: 

All processed and failed operations have been saved into this log file.

User response: 

Do not modify this file in any way. This file is for IBM Technical
Support reference.


Bottom line, I appear to be stalled if I want to use a 9-character instance name, so I'm now reduced to using hexadecimal ( Base 16 ), giving me db2insta instead of db2inst10.

Which is nice :-)

Friday, 14 November 2014

Hmmmm, HTTP404 and SRVE0190E seen with IBM HTTP Server and WebSphere Application Server

Hmm, so I am seeing this: -

Error 404: java.io.FileNotFoundException: SRVE0190E: File not found: /index.html

when I attempt to access a HTML page from IBM HTTP Server via HTTPS: -


even though I can get the page via HTTP: -


This is part of an IBM BPM Advanced 8.5.5 infrastructure, and the most recent change was to add IHS into the mix, federate it into the WAS cell, and add a Virtual Host entry for port 8443: -

cellID=AdminControl.getCell() 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8443"]]') 
AdminConfig.save() 
AdminNodeManagement.syncActiveNodes() 
quit 


I tried enabling further debugging in IHS by changing: -

LogLevel warn

to: -

LogLevel debug

in: -

/opt/IBM/HTTPServer/conf/httpd.conf

but the only thing that popped up in access_log was: -

192.168.1.70 - - [14/Nov/2014:15:08:26 +0000] "GET / HTTP/1.1" 404 42

and nothing useful in error_log.

I wondered whether the problem was related to the WAS Plugin, which was configured at the end of httpd.conf : -

LoadModule was_ap22_module "/opt/IBM/WebSphere/Plugins/bin/64bits/mod_was_ap22_http.so"
WebSpherePluginConfig /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

so I temporarily commented out both lines, and restarted IHS.

At that point, getting index.html via HTTP *and* HTTPS worked a treat.

So it's definitely somehow related to the Plugin and/or WAS.

I enabled debug in the plugin, by changing: -

   <Log LogLevel="Error" Name="/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>

to: -

   <Log LogLevel="Debug" Name="/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>

in: -

/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml 

and can see: -

[14/Nov/2014:15:01:23.89900] 000045c2 057fb700 - DETAIL:    GET /index.html HTTP/1.1
[14/Nov/2014:15:01:23.89901] 000045c2 057fb700 - DETAIL:    Host: bpm855.uk.ibm.com:8443
[14/Nov/2014:15:01:23.89901] 000045c2 057fb700 - DETAIL:    Cache-Control: max-age=0
[14/Nov/2014:15:01:23.89902] 000045c2 057fb700 - DETAIL:    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
[14/Nov/2014:15:01:23.89902] 000045c2 057fb700 - DETAIL:    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36
[14/Nov/2014:15:01:23.89903] 000045c2 057fb700 - DETAIL:    DNT: 1
[14/Nov/2014:15:01:23.89903] 000045c2 057fb700 - DETAIL:    Accept-Encoding: gzip,deflate,sdch
[14/Nov/2014:15:01:23.89904] 000045c2 057fb700 - DETAIL:    Accept-Language: en-US,en;q=0.8
[14/Nov/2014:15:01:23.89905] 000045c2 057fb700 - DETAIL:    Cookie: JSESSIONID=00002GxGEWjz6XsUDTV69LDQYmi:196509vjn; CoreID6=40736703432114159613577&ci=50200000|ESTKCS; UnicaNIODID=JOlYGvtFslm-Y5kRjJ7; CoreM_State=72~-1~-1~-1~-1~3~3~5~3~3~7~7~|~~|~~|~~|~||||||~|~~|~~|~~|~~|~~|~~|~~|~; CoreM_State_Content=6~|~38D1983DAF0161AD~|~0; 50200000|ESTKCS_clogin=v=1&l=1415961357&e=1415963160646; LtpaToken2=ELcJ3s000hNQAmekT9IFK4NhSC419BFnN4WsQxjSBdXOIt78jL/UVc/tSQd4p+wBCpygJwkLG51qhZQcYOoZN0PNsYOwLyow8ERnvf4jeWyqw45ZetsBSErQjemaZHoHXMyO8c85FdH4edxT5M5AFdojKPpKKAabTKe94Jm2KAlsGb6Mw5BkQMkVIIocUMqTFqpoyBxmxGbADGAjJ86eiI+1MtYQXVFYggAIevtQllckeB77xEkionOnm1q0POYq+5CLdjVpGxXE0sizd2vua5iySBjWoIuIR6pfhfBw8DE8htetEXvHzJlDBErGXcUdo43zMeQQINzSxAYzWsEvlxjxCqYEuEpYSzplj7Kf8RmM4QSjtYWc+v9ZmHYgGBA9; sessionCode=514564614; TJE=; TE3=
[14/Nov/2014:15:01:23.89905] 000045c2 057fb700 - DETAIL:    $WSCS: TLS_RSA_WITH_AES_128_GCM_SHA256
[14/Nov/2014:15:01:23.89906] 000045c2 057fb700 - DETAIL:    $WSIS: true
[14/Nov/2014:15:01:23.89907] 000045c2 057fb700 - DETAIL:    $WSSC: https
[14/Nov/2014:15:01:23.89907] 000045c2 057fb700 - DETAIL:    $WSPR: HTTP/1.1
[14/Nov/2014:15:01:23.89908] 000045c2 057fb700 - DETAIL:    $WSRA: 192.168.1.70
[14/Nov/2014:15:01:23.89908] 000045c2 057fb700 - DETAIL:    $WSRH: 192.168.1.70
[14/Nov/2014:15:01:23.89909] 000045c2 057fb700 - DETAIL:    $WSSN: bpm855.uk.ibm.com
[14/Nov/2014:15:01:23.89909] 000045c2 057fb700 - DETAIL:    $WSSP: 8443
[14/Nov/2014:15:01:23.89910] 000045c2 057fb700 - DETAIL:    $WSSI: wkUAAOMXLq4ktsQ6fdbYIboGeENYWFhYQxlmVAAAAAI=
[14/Nov/2014:15:01:23.89910] 000045c2 057fb700 - DETAIL:    Surrogate-Capability: WS-ESI="ESI/1.0+"
[14/Nov/2014:15:01:23.89911] 000045c2 057fb700 - DETAIL:    _WS_HAPRT_WLMVERSION: -1
[14/Nov/2014:15:01:23.89916] 000045c2 057fb700 - DEBUG: ws_common: websphereExecute: Wrote the request; reading the response (timeout 900)
[14/Nov/2014:15:01:23.89917] 000045c2 057fb700 - DETAIL: lib_htresponse: htresponseRead: Reading the response: f002ab30
[14/Nov/2014:15:01:23.90631] 000045c2 057fb700 - DETAIL:    HTTP/1.1 404 Not Found
[14/Nov/2014:15:01:23.90637] 000045c2 057fb700 - DETAIL:    X-Powered-By: Servlet/3.0
[14/Nov/2014:15:01:23.90638] 000045c2 057fb700 - DETAIL:    Content-Type: text/html;charset=ISO-8859-1
[14/Nov/2014:15:01:23.90639] 000045c2 057fb700 - DETAIL:    $WSEP:
[14/Nov/2014:15:01:23.90639] 000045c2 057fb700 - DETAIL:    Content-Language: en-GB


in: -

/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log

So this suggests that the request is being forwarded, by IHS, via the WAS Plugin, to WAS.

So I drilled into the plugin-cfg.xml file a bit more.

Logic dictated that the problem was somehow related to the context root of the URL, in that the page I'm requesting - /index.html - doesn't have the usual context root of, say, /ProcessCenter or /ProcessPortal.

Lo and behold, in plugin-cfg.xml, I can see: -

   <ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="false" LoadBalance="Round Robin" Name="dmgr_Dmgr_Cluster" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
      <Server ConnectTimeout="0" ExtendedHandshake="false" MaxConnections="-1" Name="Dmgr_dmgr" ServerIOTimeout="900" WaitForContinue="false"/>
   </ServerCluster>
   <UriGroup Name="default_host_AppCluster_URIs">
...
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/*"/>
...
   </UriGroup>
   <Route ServerCluster="AppCluster" UriGroup="default_host_AppCluster_URIs" VirtualHostGroup="default_host"/>

Now where did that little sucker come from ?

I checked the source, which is held at the WAS cell level: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/PCCell1/nodes/Node1/servers/webserver1

which has the same thing.

I'm going to retrace my steps but .... at least I have a successful resolution / circumvention, which is nice.

New - IBM Business Process Management Developer Center

IBM Business Process Management Developer Center

For Developers of Reusable Smarter Process Applications and Toolkits

Think big! Scale your business rules solutions up to the world of big data

This is from my ISSW colleague, Nigel Crowther, and is definitely worth a read: -


<snip>
Traditional business rule applications process records of a few megabytes of data at a time. Records are usually processed as client server requests or in a batch, one record at a time. As solutions move to the cloud, and applications apply rules to terabytes of data, these traditional approaches cannot keep up. To scale business rules solutions up to the world of big data, consider using the Business Rules and the IBM® Analytics for Hadoop services in IBM Bluemix™.

This tutorial describes a generic application called RulesAdaptor that uses services in IBM Bluemix to integrate the business rules of IBM Operational Decision Manager and the big data capabilities of Apache™ Hadoop®. This application opens up the possibility for data scientists to analyze big data with business rules.
</snip>