Thursday, 25 June 2015

WAS and Kerberos and SPNEGO - it's been a while - but it's all right now, in fact it it's a gas

It's been a while since first I worked with WAS, Kerberos and SPNEGO - 2011 was SUCH a long time ago.

We're configuring desktop SSO between WAS ND ( specifically for BPM Advanced but that's not important right now ) and MS Active Directory 2008.

I'd previously documented the required steps to create the Service Principal Name (SPN) and Kerberos key tab in AD, and I was confident that there were no issues there.

However, when we started up the WAS JVM ( we were specifically looking at the Deployment Manager ), we saw the following in SystemOut.log: -

[25/06/15 14:05:19:001 BST] 0000015a ServerCredent E initialize CWSPN0014E: An exception occurred during Kerberos initialization. Failure: org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS.

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E initializeServer CWSPN0015E: Unable to create a GSSCredential for: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E initializeServerCreds CWSPN0017E: Unable to create GSSCredentials for any of the hosts specified in the configuration properties.

[25/06/15 14:05:19:048 BST] 0000015a TrustAssociat E initialize CWSPN0009E: SPNEGO Trust Association Interceptor configuration is not valid. Failure condition: initializeServerCreds: no credentials successfully created.. If you are not using the SPNEGO TAI, you can ignore this message.

and, in the FFDC logs: -

org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:005 BST]     FFDC Exception:org.ietf.jgss.GSSException ProbeId:68 org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:023 BST]     FFDC ProbeId:332 initializeServerCreds: no credentials successfully created.

We started by checking the WAS configuration and then moved on to the Kerberos key tab: -

/opt/ibm/WebSphere/AppServer/java/jre/bin/java -e -k /home/wasadmin/keytabs/bpm.keytab

which returns: -

Key table: /home/wasadmin/keytabs/bpm.keytab
Number of entries: 1

1] principal: HTTP/
        KVNO: 4

       Encryption type: RC4 with HMAC

We also checked the WAS Kerberos configuration file, krb5.conf.

        default_realm =
        default_keytab_name = FILE:/home/wasadmin/keytabs/bpm.keytab
        default_tkt_enctypes = des3-cbc-sha1
        default_tgs_enctypes = des3-cbc-sha1
        forwardable  = true
        renewable  = true
        noaddresses = true
        clockskew  = 300

[realms] = {
                kdc =
                default_domain =

[domain_realm] =

We then started to look at the hostname of the server, as compared with what DNS shows.

Looking back at an older blog post ( from June 2014 ): -

I compiled and tested a Java class: -


public class hostStuff
public static void main(String[] args)

InetAddress address = InetAddress.getLocalHost();
System.out.println("My IP address ( via InetAddress.getLocalHost() ) is " + address.toString());
System.out.println("My hostname ( via InetAddress.getHostName() ) is " + address.getHostName());
System.out.println("My hostname ( via InetAddress.getCanonicalHostname() ) is  " + address.getCanonicalHostName());
catch (UnknownHostException e)
       System.out.println("I'm sorry. I don't know my own name.");

which showed that, whilst we were seeing the correct hostname, we were ONLY seeing the short name, rather than the fully-qualified service name.

In other words, we were seeing bpm101 rather than

When we checked further, that's what /etc/hosts was configured to use.

We asked the AIX sysadmin to change this - he commented out the entry in the hosts file - and then restarted WAS.

Immediately, SPNEGO burst into life.

Bottom line, it doesn't matter how good the SPNEGO and WAS configuration is, it is CRUCIAL to ensure that the hostname that WAS can see, via the various Java methods, matches what's been configured for the Service Principal Name (SPN).

Life is, as they say, a learning curve :-)


devildoommaker said...


Could you please share me the required steps to create the Service Principal Name (SPN) and Kerberos key tab in AD? ssadoglu at ereteam dot com


Dave Hay said...

Hi, thanks for your comments. I've created a new post here: -

with the full details, including the setspn and ktpass commands.

Cheers, Dave

Shaikh Zeeshan said...

Hi Dave,

It was nice to read your blogs. Would you please let me know how to use virtual alias in SSO configurations. we generated the keytab files using virtual alias name in SPN mapping but kerberos is throwing initialization exception and still looking for app server host name instead of virtual alias.

Dave Hay said...

Hi Shaikh

Thanks for your comment. It's important to ensure that the Service Principal Names match genuine DNS domain/service names, both for forward AND reverse lookup.

As an example, if your Service Name is, you need to ensure that DNS can resolve that to a valid IP address AND that the same IP address can be resolved back to the same hostname.

We typically see issues where DNS is incorrectly configured for these service names, often where DNS in one data centre is different to DNS in another data centre ... or where the end-users are.

You can validate this from the actual boxes hosting WAS, using OS commands such as ping, host, nslookup etc.

Final thought - I've presented on the SPNEGO and WAS story in the past, most recently here: -

Cheers, Dave

seanw said...

Nice breakdown, thanks! I wonder if you could have created an additional SPN so that you had both the short and longer names in the directory. Since you were using hostnames instead of DNS (bizarre in the world of AD at least), a matching entry there might also have worked.

In fact, NetBIOS computer names often did this because of their name-length restrictions (e.g., try naming your Windows machine SomethingReallySuperLong and you'll still see it in place).

Dave Hay said...

Hi Sean

Thanks for the kind comments.

Yes, you're right, we could've fixed the problem by creating a new SPN.

However, what the problem illustrated is that AIX was using a short hostname, from /etc/hosts, rather than DNS.

This is a common practice, especially when AIX LPARs are created from a "golden" master, before networking is added ( before Unix has an IP address, can hit DNS etc. ).

Again, thanks for the feedback