Oracle Database 11gR2 - Get the dependencies right ...

Having just installed Oracle 11.2 ( 11gR2 ) onto a Red Hat Enterprise Linux 7.4 VM, I was kinda getting fed up with the sqlplus command not working …..

sqlplus 

sqlplus: error while loading shared libraries: libclntsh.so.11.1: cannot open shared object file: No such file or directory

ls -al `locate libclntsh.so`

-rwxrwx---. 1 oracle oracle 48797739 Dec 12 08:49 /home/oracle/app/oracle/product/11.2.0/dbhome_1/inventory/backup/2017-12-12_08-48-58AM/Scripts/ext/lib/libclntsh.so.11.1
-rwxrwx---. 1 oracle oracle 48797739 Dec 12 08:49 /home/oracle/app/oracle/product/11.2.0/dbhome_1/inventory/Scripts/ext/lib/libclntsh.so.11.1
-rwxr-xr-x. 1 oracle oracle        0 Dec 12 08:50 /home/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so

echo $LD_LIBRARY_PATH

/home/oracle/app/oracle/product/11.2.0/dbhome_1/lib:

ls -al /home/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so*

-rwxr-xr-x. 1 oracle oracle 0 Dec 12 08:50 /home/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so

and yet ….

As is ever the case, Google came to my rescue; more importantly, Cody Burleson: -

sqlplus: error while loading shared libraries tips

which told me about the relink command: -

~/app/oracle/product/11.2.0/dbhome_1/bin/relink

The log for this: -

/home/oracle/app/oracle/product/11.2.0/dbhome_1/install/relink.log

showed me the error of my ways: -

…

INFO: gcc -o tnslsnr -m64

…

INFO: /bin/sh: gcc: command not found

INFO: make: *** [tnslsnr] Error 127
…

Ah, dependency time :-)

So I installed gcc using Yum, and reinstalled Oracle: -

rm -Rf /home/oracle/app/
/mnt/database/runInstaller -silent -responseFile ~/db_install.rsp 

and now it works: -

sqlplus

SQL*Plus: Release 11.2.0.1.0 Production on Tue Dec 12 09:46:19 2017

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Enter user-name: 

Next time (!), check the dependencies, doofus !

DB2 on Windows - where does the db2nodes.cfg file live ?

I was briefly tinkering with the need to update the db2nodes.cfg file to help DB2 Express 11.1 cope with the hostname change of a Windows Server 2012 R2 boxen.

I'd created a new VM, via OpenStack, using a snapshot from an existing VM - which I'd NOT yet deleted.

Therefore, OpenStack, being the nice IaaS solution that it is, kindly helped me out by creating the new instance with a new hostname.

This borked DB2, as you'd expect ….

I've seen similar issues with DB2 on Linux: -

SQL10003C There are not enough system resources to process the request. The request cannot be processed.

and it's typically just a case of updating the db2nodes.cfg file, which lives under the DB2 instance's home directory: -

/home/db2inst1/sqllib/db2nodes.cfg

On Windows …. yeah, good luck with that.

Of course, I found it ….

It's HERE

C:\ProgramData\IBM\DB2\DB2COPY1\BPMINST\db2nodes.cfg

Easy when you know how.

For the record, DB2 was installed, via IBM Installation Manager, as part of an IBM BPM 8.6 installation.

But that's not important right now ….

For more info about changing DB2 hostnames ( ideally you don't want to do this but … ), please go here: -

Changing hostname of the DB2 server

which says, in part: -

…

Locate db2nodes.cfg in one of the following directories:

• UNIX/Linux:
<db2 instance home directory>/sqllib/db2nodes.cfg

• All Windows flavors running DB2 v8 and v9.1: 
Program Files\IBM\SQLLIB\DB2\db2nodes.cfg 

• Windows XP and 2003 running DB2 v9.5: 
Documents and Settings\All Users\Application Data\IBM\DB2\<DB2COPY>\DB2\db2nodes.cfg

NOTE: Application Data is a hidden folder

• Windows Vista and later operating systems: ProgramData\IBM\DB2\<DB2COPY>\DB2\db2nodes.cfg

…

Ah, Windows Vista, how quaint !

Windows Server 2012 - Ooops, I locked my account

One of my colleagues had a bad day with a shared Windows Server 2012 R2 environment, in that he locked out the main account that we use to access the box, via Remote Desktop Protocol (RDP).

This is NOT an Active Directory environment, so there's no domain controller in the mix; these are merely local accounts.

Thankfully, I had another account that WASN'T locked out, so I RDP'd into the box using that account and used the Advanced User Accounts Control Panel ( aka netplwiz ), which did the trick: -

and: -

The "Account is locked out" checkbox WAS checked, it's checked no longer :-)

Thanks to the internet for the assist: -

LocalSystem user account locked out

Nagios and NRPE - There's more ...

Following on from my earlier post: -

Nagios Remote Plugin Executor (NRPE) and xinetd on Red Hat Enterprise Linux

things are starting to become more clear.

As far as I can establish, on the NRPE client/agent side, we have several components at work here.

So we have the NRPE agent itself, as started using xinetd : -

/etc/xinetd.d/nrpe

 
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream    
port = 5666    
        wait            = no
        user            = nagios
group = nagios
        server          = /usr/local/nagios/bin/nrpe
        server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable         = no
# only_from       = 192.168.153.130
}

and then we have the command(s) that we want to invoke from the Nagios server itself: -

ls -1 /usr/local/nagios/libexec

check_disk
check_load
check_nrpe
check_procs
check_swap
check_users
diskfree.py
hello.pl

and then we have the configuration file that acts as the lookup/aliasing agent: -

cat /usr/local/nagios/etc/nrpe.cfg

…

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -r -w .15,.10,.05 -c .30,.25,.20
command[check_sda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/sda1
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
command[sayHello]=/usr/local/nagios/libexec/hello.pl
command[diskFree]=/usr/local/nagios/libexec/diskfree.py
…

This means that, from the Nagios server, I can invoke any of the above commands: -

/usr/local/nagios/libexec/check_nrpe -H mfp.uk.ibm.com -c check_users

USERS OK - 2 users currently logged in |users=2;5;10;0

/usr/local/nagios/libexec/check_nrpe -H mfp.uk.ibm.com -c check_load

CRITICAL - load average: 0.39, 0.24, 0.17|load1=0.390;0.150;0.300;0; load5=0.240;0.100;0.250;0; load15=0.170;0.050;0.200;0; 

/usr/local/nagios/libexec/check_nrpe -H mfp.uk.ibm.com -c check_sda1

DISK OK - free space: /boot 274 MB (55.29% inode=100%);| /boot=222MB;396;446;0;496

/usr/local/nagios/libexec/check_nrpe -H mfp.uk.ibm.com -c check_zombie_procs

PROCS OK: 0 processes with STATE = Z | procs=0;5;10;0;

/usr/local/nagios/libexec/check_nrpe -H mfp.uk.ibm.com -c check_total_procs

PROCS WARNING: 185 processes | procs=185;150;200;0;

/usr/local/nagios/libexec/check_nrpe -H mfp.uk.ibm.com -c sayHello

Hello, World!

/usr/local/nagios/libexec/check_nrpe -H mfp.uk.ibm.com -c diskFree

FREE SPACE OK: '/' is 26.00% full

From a Nagios console perspective, I can define all/any of these

sudo vi /usr/local/nagios/etc/servers/mfp.cfg 

…

define command {
        command_name                    check_nrpe_with_args
        command_line                    $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}
…

define service {
        use                             generic-service
        host_name                       mfp.uk.ibm.com
        service_description             Check_Load
        check_command                   check_nrpe_with_args!check_load
        notifications_enabled           1
}
define service {
        use                             generic-service
        host_name                       mfp.uk.ibm.com
        service_description             Hello_World
        check_command                   check_nrpe!sayHello
        notifications_enabled           0
}
define service {
        use                             generic-service
        host_name                       mfp.uk.ibm.com
        service_description             Disk_Free
        check_command                   check_nrpe!diskFree
        notifications_enabled           0
}
…

sudo systemctl restart nagios.service

and then check the Nagios dashboard: -

Finally, purely for the record, whilst the shipped plugins are mainly C/C++ binaries e.g.

file /usr/lib64/nagios/plugins/check_disk 

/usr/lib64/nagios/plugins/check_disk: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0f75d7633e74455320d40f0b75071f8eabe148de, stripped

but I also have a Perl script: -

file hello.pl 

hello.pl: Perl script, ASCII text executable

and a Python script: -

file diskfree.py 

diskfree.py: Python script, ASCII text executable

( kudos to Linux Magazine for this latter script )

Nagios Remote Plugin Executor (NRPE) and xinetd on Red Hat Enterprise Linux

As per a few previous posts, I'm tinkering ( there's that word again, I must think of another simile ) with Nagios.

This time I'm looking to have my Nagios monitoring server ( which runs on Ubuntu ) execute an agent on a remote boxen running Red Hat Enterprise Linux (RHEL).

Having installed the pre-requisites ( and I do need to document those steps more clearly ), I setup a NRPE script to be started using xinetd ( A Powerful Replacement For Inetd )

cat /etc/xinetd.d/nrpe 

# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream    
port = 5666    
        wait            = no
        user            = nagios
group = nagios
        server          = /usr/local/nagios/bin/nrpe
        server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable         = no
# only_from       = 192.168.153.130
}

Having started the service: -

systemctl restart xinetd.service

I was expecting to see NRPE listening on port 5666: -

netstat -aon | grep LISTEN | grep 5666

but saw nothing alas.

I checked the logs: -

tail -f /var/log/messages

Dec  6 14:00:01 mfp systemd: Starting Session 222 of user root.
Dec  6 14:00:14 mfp xinetd[5771]: unexpected signal: 18 (Continued) in signal pipe
Dec  6 14:00:14 mfp systemd: Stopping Xinetd A Powerful Replacement For Inetd...
Dec  6 14:00:14 mfp xinetd[5771]: Exiting...
Dec  6 14:00:14 mfp systemd: Starting Xinetd A Powerful Replacement For Inetd...
Dec  6 14:00:14 mfp systemd: PID file /var/run/xinetd.pid not readable (yet?) after start.
Dec  6 14:00:14 mfp xinetd[6205]: service/protocol combination not in /etc/services: nrpe/tcp
Dec  6 14:00:14 mfp xinetd[6205]: xinetd Version 2.3.15 started with libwrap loadavg labeled-networking options compiled in.
Dec  6 14:00:14 mfp xinetd[6205]: Started working: 0 available services
Dec  6 14:00:14 mfp systemd: Started Xinetd A Powerful Replacement For Inetd.

which makes sense.

Following this: -

xinetd[11497]: service/protocol combination not in /etc/services: nrpe/tcp

I updated the services table: -

vi /etc/services

and changed it from: -

...
cfengine        5308/tcp                        # CFengine
cfengine        5308/udp                        # CFengine
cvsup           5999/tcp        CVSup           # CVSup file transfer/John Polstra/FreeBSD
cvsup           5999/udp        CVSup           # CVSup file transfer/John Polstra/FreeBSD
...

to: -

...
cfengine        5308/tcp                        # CFengine
cfengine        5308/udp                        # CFengine

nrpe            5666/tcp                        # NRPE

cvsup           5999/tcp        CVSup           # CVSup file transfer/John Polstra/FreeBSD
cvsup           5999/udp        CVSup           # CVSup file transfer/John Polstra/FreeBSD
...

Once I again restarted xinetd : -

systemctl restart xinetd.service

the nrpe service started listening on port 5666: -

netstat -aon | grep LISTEN | grep 5666

tcp6       0      0 :::5666                 :::*                    LISTEN      off (0.00/0/0)

with no issues in the logs: -

tail -f /var/log/messages

Dec  6 14:09:54 mfp systemd: Starting Xinetd A Powerful Replacement For Inetd...
Dec  6 14:09:54 mfp systemd: PID file /var/run/xinetd.pid not readable (yet?) after start.
Dec  6 14:09:54 mfp xinetd[6803]: xinetd Version 2.3.15 started with libwrap loadavg labeled-networking options compiled in.
Dec  6 14:09:54 mfp xinetd[6803]: Started working: 1 available service
Dec  6 14:09:54 mfp systemd: Started Xinetd A Powerful Replacement For Inetd.

Thanks, Google, you again served me well ...

WebSphere Liberty Profile - Monitoring via JMX over REST using Jython

For this, I'm using two excellent IBM developerWorks articles as inspiration: -

Retrieve performance metrics from the WebSphere Liberty profile, Part 1 - Setup and configuration

Retrieve performance metrics from the WebSphere Liberty profile, Part 2 - Data collection by using Jython

Without reposting the entire pair of articles ( which would be a daft idea ), here's a short-cut of what I ended up doing.

For the record, I am using Liberty 17.0.0.3: -

/opt/ibm/WebSphere/Liberty/bin/server version

WebSphere Application Server 17.0.0.3 (1.0.18.cl170320170927-1854) on IBM J9 VM, version pxa6480sr3fp12-20160919_01 (SR3 FP12) (en_GB)

Java 8: -

java -fullversion

java full version JRE 1.8.0 IBM Linux build pxa6480sr3fp12-20160919_01(SR3 FP12)

and Jython 2.7.0: -

http://www.jython.org/downloads.html

So I started by setting up the Jython runtime environment ( this is on the same Red Hat Enterprise Linux box that's hosting Liberty etc. ): -

Create the directory structure

mkdir /opt/ibm/WebSphere/Liberty/jython
cd /opt/ibm/WebSphere/Liberty/jython

Pull Jython 2.7.0

wget http://search.maven.org/remotecontent?filepath=org/python/jython-standalone/2.7.0/jython-standalone-2.7.0.jar
mv remotecontent\?filepath\=org%2Fpython%2Fjython-standalone%2F2.7.0%2Fjython-standalone-2.7.0.jar jython-standalone-2.7.0.jar

Copy the requisite REST Connector classes - both Jython and Java

cp /opt/ibm/WebSphere/Liberty/clients/jython/restConnector.py .
cp /opt/ibm/WebSphere/Liberty/clients/restConnector.jar .

and then started the Jython environment: -

java -cp jython-standalone-2.7.0.jar:restConnector.jar org.python.util.jython

From there, I established connectivity to my Liberty runtime: -

from restConnector import JMXRESTConnector

JMXRESTConnector.trustStore = '/opt/ibm/WebSphere/Liberty/usr/servers/defaultServer/resources/security/key.jks'
JMXRESTConnector.trustStorePassword = 'passw0rd'
connector = JMXRESTConnector()
connection = connector.connect( 'mfp.uk.ibm.com', 9443, 'appcenteradmin', 'admin')
mconnection=connector.getMBeanServerConnection()

Note that, contrary to the article, this code did not work: -

mconnection = connection.getMBeanServerConnection()

It instead returned: -

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'getMBeanServerConnection'

However, this code DID work: -

mconnection=connector.getMBeanServerConnection()

and I was able to validate connectivity etc. : -

mconnection

com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection@fcbd91c2

dir(mconnection)

['MBeanCount', 'PollingMode', 'ServerPollingThread', '__class__', '__copy__', '__deepcopy__', '__delattr__', '__doc__', '__ensure_finalizer__', '__eq__', '__format__', '__getattribute__', '__hash__', '__init__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__subclasshook__', '__unicode__', 'addNotificationListener', 'class', 'createMBean', 'defaultDomain', 'domains', 'equals', 'getAttribute', 'getAttributes', 'getClass', 'getDefaultDomain', 'getDomains', 'getMBeanCount', 'getMBeanInfo', 'getObjectInstance', 'hashCode', 'invoke', 'isInstanceOf', 'isRegistered', 'notify', 'notifyAll', 'queryMBeans', 'queryNames', 'removeNotificationListener', 'setAttribute', 'setAttributes', 'toString', 'unregisterMBean', 'wait']

mconnection.getClass()

<type 'com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection'>

The article then takes one through creating a pair of Python scripts: -

wlp_collect_conf.py

wlp_collect.py

to create an importable library of functions, including: -

connection = connect()

9443
/opt/ibm/WebSphere/Liberty/usr/servers/defaultServer/resources/security/key.jks
Connecting to the server...
Successfully connected to the server "mfp.uk.ibm.com:9443"

and: -

collect( connection, True,  "WebSphere:type=JvmStats,*")

 MBean details for WebSphere:type=JvmStats

   7 attributes
      UsedMemory [long] = 83109880
      FreeMemory [long] = 41080840
      Heap [long] = 124321792
      UpTime [long] = 22864299
      ProcessCPU [double] = 2.04226135228
      GcCount [long] = 1187
      GcTime [long] = 2457

   0 operations
array(java.lang.Object, [WebSphere:type=JvmStats])

So, in principle, I could use these via Nagios ……

That's the next step ….

WebSphere Liberty Profile - Snooping About

I've written about the SuperSnoop Servlet before: -

WebSphere Application Server - Binary Scanner

WebSphere Liberty Profile on macOS - SuperSnooping

WebSphere Application Server 9 - Snooping About

WebSphere Application Server - Liberty Profile - Oh, what fun

and it's one tool that I use, and reuse, whenever I'm testing a build of WebSphere Application Server etc.

However, it niggled me that, whilst it DOES run on WebSphere Liberty Profile, it doesn't actually work.

Having deployed it: -

cp SuperSnoopWeb.war /opt/ibm/WebSphere/Liberty/usr/servers/defaultServer/dropins/

( I *SO* love the Dropin support on WLP )

and watched the logs: -

[AUDIT   ] CWWKT0016I: Web application available (default_host): http://mfp.uk.ibm.com:9080/SuperSnoopWeb/
[AUDIT   ] CWWKZ0001I: Application SuperSnoopWeb started in 0.085 seconds.

I see this: -

when I hit it, and this in the logs: -

SuperSnoop running
[ERROR   ] SRVE0777E: Exception thrown by application class 'SuperSnoop.getAppServerName:46'
java.lang.NoClassDefFoundError: com/ibm/websphere/management/AdminServiceFactory
        at SuperSnoop.getAppServerName(SuperSnoop.java:46)
        at SuperSnoop.doGet(SuperSnoop.java:160)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1255)
        at [internal classes]

This developerWorks Answer thread covered the specific issue - that of the AdminServiceFactory class not being available to me: -

ClassNotFound: AdminService in Liberty Profile

…

The full profile AdminService API is not provided in the Liberty profile as it relates to an administration model that is not provided, nor compatible with the Liberty profile.

…

but also said this: -

…

Have you tried using the liberty migration tool (tech preview) to scan your application? If so this may be a package we are missing from the list. If you haven't tried it, there are more details on the tool here; you may find it useful: https://www.ibmdw.net/wasdev/2014/04/23/websphere-application-server-migration-toolkit-liberty-tech-preview/

…

Taking this good advice, I downloaded the Migration Toolkit for Application Binaries: -

https://developer.ibm.com/wasdev/downloads/#asset/tools-Migration_Toolkit_for_Application_Binaries

and ran it: -

java -jar binaryAppScanner.jar SuperSnoopWeb.war

Processing the SuperSnoopWeb.war application.
Scanning files.........
The report was saved to the following file: /Users/davidhay/Downloads/wamt/SuperSnoopWeb.war_MigrationReport.html

The resulting HTML document said, in part: -

*BUT*

which is good intel.

So I'll continue to deploy it, in part to test my deployment practices, but won't expect it to do owt :-(

However, one of my blog correspondents, Jeremy Hughes from IBM, did rightly point out that there's a more recent Github project called Ferret: -

A sample servlet application which responds to HTTP requests with information about the request and server

https://github.com/wasdev/sample.ferret

which doesn't have the same drawback re unavailable APIs: -

etc.

and also runs happily on Liberty: -

https://mfp.uk.ibm.com:9443/ferret/