IBM API Connect and the SshClientException

This post represents a frustrating, but extremely enjoyable and interesting, voyage of discovery, digging into problems with SSH on Ubuntu Linux …..

I saw this whilst attempting to create a new Developer Portal for an IBM API Connect 5.0.7.2 implementation: -

For the record, here's the text of the exception: -

Error

Error while performing action add during communication to the Advanced Portal. Please report this error to your server administrator. Error details: com.ibm.apimgmt.api.util.SshClient$SshClientException: An exception occurred during SSH call: com.jcraft.jsch.JSchException: Algorithm negotiation fail.
Error ID: 59ef3f69e4b07d97f084e2ee

I checked the authorized_keys file on the Portal box: -

cat /home/admin/.ssh/authorized_keys 

command="/home/admin/bin/site_action" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyA1P0bv68VRylLHGwNF+aRYR5FCYAtTJQYRPfbAaE286gPseddNEME0vCxQkAGwqUJX7hZRKNdplw9/o67BHbEedkF6d4O8JPON2ZkFPQTv+cxAhFkDLt86ClRlstvxroqbHYwsRKOl/cOVW/88LEQ90UuQVunUQYanI4A6AJZZ8OvxN+/pgq/bHZULfKF5148IKwY9/90uuavwp6t4Jjm62d2UOHplRv6LiT+qPY2Iykncmqr85X0riUExqwkyyOoVextC450Ui10bMFeQYO4KS0cTHTKd0LuiLUopy4hYmDbyJXNa9t6H6mQVe+P+MjAmJNKx8j4xZqZvojiwUf apim_advanced_portal_ssh_key

which matched that specified within the API Manager Cloud Manager UI: -

I even tried upgrading from the older version of the Developer Portal ( 5.0.7.2 based upon Debian 7 ) to the latest fix pack ( 5.0.8.0 based upon Ubuntu 16.0.4.3 LTS ), but to no avail.

As this is a test environment, running on my own Beast box, I set the Portal to trust ALL certificates: -

set_apim_cert -i

WARNING: This should only be used for development and testing purposes as it is not secure and leaves the Developer Portal exposed to a man-in-the-middle attack.

and checked the status: -

status

Operating System: Ubuntu 16.04.3 LTS
System version: 7.x-5.0.8.0-20170908-0855
Distribution version: 7.x-5.0.8.0-20170907-2206

Free disk space: 22G
 RAM Free/Total: 1941 MB / 3951 MB (49% free)
   Set Hostname: OK
     DNS Server: Reachable (8.8.8.8)
   APIC SSH Key: OK

Configuration:
  APIC Hostname: management.uk.ibm.com
  APIC IP: 192.168.1.150
  Devportal Hostname: portal.uk.ibm.com
  Devportal IP: 192.168.1.151
  APIC Certificate Status (Insecure): WARNING - Only suitable for development and PoC purposes.

Node is standalone

Site web check: All sites OK

Site services:
         Webhooks: All sites Up
  Background sync: All sites Up

Services:
  Queue                      is Up
  Database   [Mysql]         is Up (Standalone)
  Web Server [Nginx]         is Up
  PHP Pool   [Php7.0-fpm]    is Up
  Inetd      [Openbsd-inetd] is Up
  REST       [Restservice]   is Up

SUCCESS: All services are Up.

It took me a while, but I worked out how to debug …

Having switched to the Ubuntu version of the Developer Portal, I was able to turn on debugging in the SSH Daemon ( SSHD ), by editing the sshd_config file: -

sudo vi /etc/ssh/sshd_config 

and changing the logging level from: -

# Logging
SyslogFacility AUTH
LogLevel INFO

to: -

# Logging
SyslogFacility AUTH
#LogLevel INFO
LogLevel DEBUG3

I then restarted the SSHD service: -

sudo /etc/init.d/ssh restart

[ ok ] Restarting ssh (via systemctl): ssh.service.

and watched the logs whilst I reproduced the problem: -

tail -f /var/log/auth.log 

which gave me: -

...
Oct 25 08:55:12 portal sshd[24777]: debug1: kex: algorithm: (no match) [preauth]
Oct 25 08:55:12 portal sshd[24777]: fatal: Unable to negotiate with 192.168.1.150 port 52443: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1 [preauth]
Oct 25 08:55:12 portal sshd[24777]: debug1: do_cleanup [preauth]
...

I then checked the sshd_config file again: -

sudo vi /etc/ssh/sshd_config 

and looked at the KexAlgorithms line: -

KexAlgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

which tied up nicely …. which was confusing :-(

And then …. 

IT JUST STARTED WORKING !!!

To prove it, I deleted the Developer Portal VM, and built it again from the original .OVA file ….

AND IT WORKED !!

First time, out of the box …

So I'm at a complete and utter loss to know what broke …

However, I learned shedloads in diagnosing the problem, so that's all good then :-)

For the record, the Ubuntu version of the Developer Portal ( 5.0.8.0 ) uses SSH-2.0-OpenSSH_7.2p2 - in case it becomes relevant down the line ….

I'm now going to tinker some more, before ditching the APIC Management Server, and rebuilding that with the new 5.0.8.0 OVA file …

In the context of 5.0.8.0, this is what I've downloaded …

APIConnect_Management_5.0.8.0_20170905-1133_a7fe4cd1d442_c04798a.ova (2.99 GB)

5.0.8.0-APIConnect-Portal-Ubuntu16-20170908-0855.ova (879.64 MB)

from IBM Fix Central, as per this: -

IBM API Connect V5.0.8.0 is available

…

Please note: The Linux distribution for the Developer Portal OVA has moved from a Debian V7 base to an Ubuntu V16.04 base. Support for the Debian V7 OVA is being withdrawn in May 2018. You are strongly encouraged to migrate your Developer Portal to the Ubuntu V16.04 base now, as support for Debian V7 upgrades will be removed by May 2018.
…

What's new for this release