Thursday, 25 June 2015

WAS and Kerberos and SPNEGO - it's been a while - but it's all right now, in fact it it's a gas

It's been a while since first I worked with WAS, Kerberos and SPNEGO - 2011 was SUCH a long time ago.

We're configuring desktop SSO between WAS ND 8.5.5.0 ( specifically for BPM Advanced 8.5.5.0 but that's not important right now ) and MS Active Directory 2008.

I'd previously documented the required steps to create the Service Principal Name (SPN) and Kerberos key tab in AD, and I was confident that there were no issues there.

However, when we started up the WAS JVM ( we were specifically looking at the Deployment Manager ), we saw the following in SystemOut.log: -

[25/06/15 14:05:19:001 BST] 0000015a ServerCredent E com.ibm.ws.security.spnego.ServerCredential initialize CWSPN0014E: An exception occurred during Kerberos initialization. Failure: org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS.

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E com.ibm.ws.security.spnego.ServerCredentialsFactory initializeServer CWSPN0015E: Unable to create a GSSCredential for: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E com.ibm.ws.security.spnego.ServerCredentialsFactory initializeServerCreds CWSPN0017E: Unable to create GSSCredentials for any of the hosts specified in the configuration properties.

[25/06/15 14:05:19:048 BST] 0000015a TrustAssociat E com.ibm.ws.security.spnego.TrustAssociationInterceptorImpl initialize CWSPN0009E: SPNEGO Trust Association Interceptor configuration is not valid. Failure condition: initializeServerCreds: no credentials successfully created.. If you are not using the SPNEGO TAI, you can ignore this message.


and, in the FFDC logs: -

org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS
 at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:35)

[25/06/15 14:05:19:005 BST]     FFDC Exception:org.ietf.jgss.GSSException SourceId:com.ibm.ws.security.spnego.ServerCredentialsFactory.initializeServer ProbeId:68 org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:023 BST]     FFDC Exception:com.ibm.ws.security.spnego.TAIConfigurationException SourceId:com.ibm.ws.security.spnego.TrustAssociationInterceptorImpl.initialize ProbeId:332 Reporter:com.ibm.ws.security.spnego.TrustAssociationInterceptorImpl@7238f7dd

com.ibm.ws.security.spnego.TAIConfigurationException: initializeServerCreds: no credentials successfully created.


We started by checking the WAS configuration and then moved on to the Kerberos key tab: -

/opt/ibm/WebSphere/AppServer/java/jre/bin/java com.ibm.security.krb5.internal.tools.Klist -e -k /home/wasadmin/keytabs/bpm.keytab

which returns: -

Key table: /home/wasadmin/keytabs/bpm.keytab
Number of entries: 1

1] principal: HTTP/bpm101.uk.ibm.com@2008ADMINLDAPS
        KVNO: 4

       Encryption type: RC4 with HMAC

We also checked the WAS Kerberos configuration file, krb5.conf.

[libdefaults]
        default_realm = uk.ibm.com
        default_keytab_name = FILE:/home/wasadmin/keytabs/bpm.keytab
        default_tkt_enctypes = des3-cbc-sha1
        default_tgs_enctypes = des3-cbc-sha1
        forwardable  = true
        renewable  = true
        noaddresses = true
        clockskew  = 300

[realms]
        uk.ibm.com = {
                kdc = 2008ADMINLDAPS.uk.ibm.com:88
                default_domain = uk.ibm.com
        }

[domain_realm]
        .uk.ibm.com = uk.ibm.com

We then started to look at the hostname of the server, as compared with what DNS shows.

Looking back at an older blog post ( from June 2014 ): -


I compiled and tested a Java class: -

import java.net.InetAddress;
import java.net.UnknownHostException;

public class hostStuff
{
public static void main(String[] args)
{

try
{
InetAddress address = InetAddress.getLocalHost();
System.out.println("My IP address ( via InetAddress.getLocalHost() ) is " + address.toString());
System.out.println("My hostname ( via InetAddress.getHostName() ) is " + address.getHostName());
System.out.println("My hostname ( via InetAddress.getCanonicalHostname() ) is  " + address.getCanonicalHostName());
}
catch (UnknownHostException e)
{
       System.out.println("I'm sorry. I don't know my own name.");
}
}
}

which showed that, whilst we were seeing the correct hostname, we were ONLY seeing the short name, rather than the fully-qualified service name.

In other words, we were seeing bpm101 rather than bpm101.uk.ibm.com.

When we checked further, that's what /etc/hosts was configured to use.

We asked the AIX sysadmin to change this - he commented out the entry in the hosts file - and then restarted WAS.

Immediately, SPNEGO burst into life.

Bottom line, it doesn't matter how good the SPNEGO and WAS configuration is, it is CRUCIAL to ensure that the hostname that WAS can see, via the various Java methods, matches what's been configured for the Service Principal Name (SPN).

Life is, as they say, a learning curve :-)



6 comments:

devildoommaker said...

Hi

Could you please share me the required steps to create the Service Principal Name (SPN) and Kerberos key tab in AD? ssadoglu at ereteam dot com

Thanks

Dave Hay said...

Hi, thanks for your comments. I've created a new post here: -

http://portal2portal.blogspot.co.uk/2015/06/ibm-business-process-manager-and.html

with the full details, including the setspn and ktpass commands.

Cheers, Dave

Shaikh Zeeshan said...

Hi Dave,

It was nice to read your blogs. Would you please let me know how to use virtual alias in SSO configurations. we generated the keytab files using virtual alias name in SPN mapping but kerberos is throwing initialization exception and still looking for app server host name instead of virtual alias.

Dave Hay said...

Hi Shaikh

Thanks for your comment. It's important to ensure that the Service Principal Names match genuine DNS domain/service names, both for forward AND reverse lookup.

As an example, if your Service Name is ibmbpm.uk.ibm.com, you need to ensure that DNS can resolve that to a valid IP address AND that the same IP address can be resolved back to the same hostname.

We typically see issues where DNS is incorrectly configured for these service names, often where DNS in one data centre is different to DNS in another data centre ... or where the end-users are.

You can validate this from the actual boxes hosting WAS, using OS commands such as ping, host, nslookup etc.

Final thought - I've presented on the SPNEGO and WAS story in the past, most recently here: -

http://www.websphereusergroup.co.uk/wug/downloads/33/

Cheers, Dave

seanw said...

Nice breakdown, thanks! I wonder if you could have created an additional SPN so that you had both the short and longer names in the directory. Since you were using hostnames instead of DNS (bizarre in the world of AD at least), a matching entry there might also have worked.

In fact, NetBIOS computer names often did this because of their name-length restrictions (e.g., try naming your Windows machine SomethingReallySuperLong and you'll still see it in place).

Dave Hay said...

Hi Sean

Thanks for the kind comments.

Yes, you're right, we could've fixed the problem by creating a new SPN.

However, what the problem illustrated is that AIX was using a short hostname, from /etc/hosts, rather than DNS.

This is a common practice, especially when AIX LPARs are created from a "golden" master, before networking is added ( before Unix has an IP address, can hit DNS etc. ).

Again, thanks for the feedback

Dave