wjd.nu

2025-05-29 - supermicro / ikvm / expired certificate

Supermicro computers generally work very well. At our company we've been using them for ages. One thing that does keep giving us the occasional trouble however is the BMC. Today, we'll be looking at expired certificates of the iKVM interface.

Introduction

First, some definitions, which we generally use interchangeably:

The Baseboard Management Controller (BMC) is the chipset core of the Intelligent Platform Management Interface (IPMI) implementation, which we mostly use to remotely attach a Keyboard, Video and Mouse (KVM, over IP).

iKVM — for IP-KVM or IPMI-KVM? — is the Java implementation that comes with the Supermicro BMC Firmware. The Supermicro BMC Interface (or IPMI) provides access to the KVM via either a HTML interface or via its iKVM Java application.

The use case for IPMI KVM access is when we've lost, or never had, regular ssh or vpn access to a remote server. Often you'll want more than just a keyboard; you also want to attach a virtual CD-ROM or USB-drive to get access to installation/recovery tools.

The HTML interface does not provide the capability to mount a virtual drive. The Java application does. So, we prefer working with that one. We'll be calling that iKVM, and BMC will refer to the hardware, firmware and the web interface of the Supermicro IPMI.

(Note that the Java application is dated and buggy, but with the right tools (vcutil ipmikvm and xpaste) it's usable.)

As mentioned, the Supermicro BMC and iKVM have been a recurring item in my notes to self section, all the way back to 2016:

• 2016-10-25 - A better way of packaging iKVM
• 2020-02-12 - Using the ipmikvm wrapper to launch iKVM
• 2021-06-17 - iKVM TLS issues caused by a wrong clock
• 2022-09-09 - Getting video to work on a Supermicro X9DRW BMC

That issue with the inaccurate BMC clock was a particularly insidious one: the clock was set in the past, causing it to reject TLS connections from iKVM because it considered the TLS certificate not yet valid. But it was hard to debug because the server side did the rejection and only provided an uninformative status code.

Today, there's another “fun” one, also related to certificates and time.

Retry =10 (Connection failed)

There was a connectivity problem on a machine and we needed get KVM access.

No biggie! Just run vcutil ipmikvm.

$ ipmikvm 10.10.10.10
attempting login on '10.10.10.10' with user ADMIN
...
+ exec java -Djava.library.path=.local/lib/ipmikvm/iKVM__V1.69.42.0x0 -cp ...

connect failed sd:16
Retry =1
Retry =2
Retry =3
Retry =4
Retry =5
Retry =6
Retry =7
Retry =8
Retry =9
Retry =10

Okay, that's not good. Generally we get about one to three retries and then it works.

We could log into the web interface after clicking past the self-signed and expired certificate warning:

In the web interface we could use the HTML interface to the KVM and the power buttons. But using iKVM would consistently fail.

Expired certificate

A few moments of debugging later, and the expired certificate was assigned blame:

$ easycert -T 10.10.10.10 443
Certificate chain
 0 s: {1B:29:EF:3D:CA:7B:26:2D:A0:AE:AA:71:39:E9:5C:91:0C:C1:28:A8} [9adfe6f5] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI
   i: {1B:29:EF:3D:CA:7B:26:2D:A0:AE:AA:71:39:E9:5C:91:0C:C1:28:A8} [9adfe6f5] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI
---
Expires in -578 days

vcutil easycert is a quick and easy wrapper around the openssl CLI to check certificate chains and expiry. As you can see, it shows that the certificate is (a) self-signed and (b) expired 578 days ago.

On the https port 443, the certificate had expired almost two years ago. Doing the same check on the main iKVM port 5900:

$ easycert -T 10.10.10.10 5900
Certificate chain
 0 s: {BF:00:C8:10:AC:B2:58:E9:E7:4A:D9:7E:06:A2:DB:79:F6:3B:8F:38} [9adfe6f5] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI
   i: {BF:00:C8:10:AC:B2:58:E9:E7:4A:D9:7E:06:A2:DB:79:F6:3B:8F:38} [9adfe6f5] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI
---
Expires in -578 days

Exactly the same. Only the X509v3 Subject Key Identifier was different. (Insert foreshadowing here.)

I tried to update the certificate in the web interface. The openssl command to create fresh certificate with the same subject and encryption algorithm:

$ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout ipmi-priv.pem -out ipmi-pub.pem \
    -subj '/C=US/ST=California/L=San Jose/O=Super Micro Computer/OU=Software/CN=IPMI'
...

In the BMC web interface, the Date and Time were set correctly this time. And in the Configuration > Network > SSL Certificates there's a form to upload the freshly generated certificates.

After uploading them, the web certificate was not updated yet. This required a BMC reset located through Maintenance > BMC Reset > Unit Reset. The reset took more than a minute, but afterwards, the certificate on port 443 was updated to the freshly generated one. Good.

$ easycert -T 10.10.10.10 443
Certificate chain
 0 s: {B6:52:AB:9A:A1:C4:67:9C:8F:40:FF:C3:A8:75:78:8D:85:7F:6B:22} [9adfe6f5] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI
   i: {B6:52:AB:9A:A1:C4:67:9C:8F:40:FF:C3:A8:75:78:8D:85:7F:6B:22} [9adfe6f5] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI
---
Expires in 3649 days

But unfortunately, the certificate on port 5900 wasn't. It still showed the same expired certificate.

I tried — in vain — to run ipmikvm again, but as expected, it failed with the same Retry =10 and Connection failed messages.

Updating firmware

Next, I did some frantic Googling and ChatGPTing to find another button/option to update the certificates, but the general concensus was that on this board, the TLS certificates were baked into the BMC firmware and could not be independently updated.

Supermicro lists on its downloads page that you shouldn't update the BMC firmware unless you really really have to:

But, I do want iKVM to work again. Let's find the right firmware. Start out by figuring out what hardware we're dealing with:

# for x in system-manufacturer system-product-name \
           baseboard-manufacturer baseboard-product-name
  do printf '%-24s: %s\n' "$x" "$(dmidecode -s $x)"; done

system-manufacturer     : S3S
system-product-name     : AS -1114S-WN10RT
baseboard-manufacturer  : Supermicro
baseboard-product-name  : H12SSW-NTR

Google turns up this AS-1114S-WN10RT which seems plausible, especially since it speaks of the H12SSW-NTR motherboard. It has two downloads available:

BIOS / BMC / Bundle Firmware for AS-1114S-WN10RT and BIOS / BMC / Bundle Firmware for AS-1114S-WN10RT.

Oh wait, that's the same download. I want to update the BIOS even less than I want to update the BMC. And it also mentions that updating needs to be done via the EFI shell, which requires the system to be rebooted, which I also don't want.

Is there another way?

Faking the time

So, as far as I understand, the trouble is that a component in the iKVM application validates the certificate expiry time. This could be inside the bundled Java application, or in an auxiliary application like stunnel.

In any case, unlike the earlier issues where iKVM was rejected due to the BMC clock being set incorrectly, this problem looks to be client side. And on the client, we have control. Like control over the “current time”.

Mr. Chat, whip me up a quick LD_PRELOAD shared libary that fakes the time being 2 years in the past.

In mere seconds, I received an almost functional faketime.c. Fixed, and annotated with docs:

/**
 * LD_PRELOAD library to "fake" the current time. Takes a
 * FAKETIME_OFFSET=<seconds> as environment variable.
 *
 * Compile:
 *   gcc -shared -fPIC -o faketime.so faketime.c -ldl -Wall
 * Run:
 *   FAKETIME_OFFSET=$((2*365*24*60*60)) LD_PRELOAD=$PWD/faketime.so date
 */
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdlib.h>
#include <sys/time.h>
#include <time.h>

static time_t offset_seconds = 0;

time_t time(time_t *tloc) {
    static time_t (*real_time)(time_t *) = NULL;
    time_t real, fake;
    if (!real_time)
        real_time = dlsym(RTLD_NEXT, "time");
    real = real_time(NULL);
    fake = real - offset_seconds;
    if (tloc)
        *tloc = fake;
    return fake;
}

int gettimeofday(struct timeval *tv, void *tz) {
    static int (*real_gettimeofday)(struct timeval *, struct timezone *) = NULL;
    int res;
    if (!real_gettimeofday)
        real_gettimeofday = dlsym(RTLD_NEXT, "gettimeofday");
    res = real_gettimeofday(tv, tz);
    if (res == 0)
        tv->tv_sec -= offset_seconds;
    return res;
}

int clock_gettime(clockid_t clk_id, struct timespec *tp) {
    static int (*real_clock_gettime)(clockid_t, struct timespec *) = NULL;
    int res;
    if (!real_clock_gettime)
        real_clock_gettime = dlsym(RTLD_NEXT, "clock_gettime");
    res = real_clock_gettime(clk_id, tp);
    if (res == 0 && (clk_id == CLOCK_REALTIME || clk_id == CLOCK_REALTIME_COARSE))
        tp->tv_sec -= offset_seconds;
    return res;
}

__attribute__((constructor))
void init() {
    const char *env = getenv("FAKETIME_OFFSET");
    if (env) {
        offset_seconds = strtoll(env, NULL, 10);
    } else {
        /* Default to 2 years (730 days) if not set */
        offset_seconds = 730 * 24 * 60 * 60;
    }
}

A quick polish, compilation with gcc -shared -fPIC -o faketime.so faketime.c -ldl and test:

$ LD_PRELOAD=$PWD/faketime.so date
Tue May 30 11:29:26 UTC 2023

$ FAKETIME_OFFSET=$((-2*24*60*60)) LD_PRELOAD=$PWD/faketime.so date
Sat May 31 11:29:53 UTC 2025

Yup, it works, even with negative offsets for future dates.

But does it also work with iKVM? Drum rolls...

$ FAKETIME_OFFSET=$((2*365*86400)) LD_PRELOAD=$PWD/faketime.so ipmikvm 10.10.10.10
attempting login on '10.10.10.10' with user ADMIN
...
connect failed sd:16
Retry =1
Gtk-Message: 13:31:13.519: Failed to load module "canberra-gtk-module"
a singal 17 is raised
GetFileDevStr:4051 media_type = 40

Success! A working iKVM viewer. And another utility library for under my belt.

BMC upgrade anyway

When writing this post, I did fetch the BIOS firmware again, to double check some things, and I noticed that inside H12SSW-NTR_3.3_AS01.05.21_SAA1.2.0-p2.zip, there is both a BIOS zip and a and a BMC BMC_H12AST2500-ROT-2201MS_20250430_01.05.21_STDsp.zip file.

Through Maintenance > Firmware Management > Update and deselecting “Preserve SSL Certificate” the BMC firmware could be updated. (I guess that the SSL certificate update question only applies to the web certificate though.)

The update appeared to stall at Upload firmware : 75% for a while, but it completed after a few minutes. The web interface was ungrayed, but a FW Update Mode: Current system is in FW update mode, any configuration changes are not recommended notice stayed visible and the current and new BMC firmware versions were listed, along with an Update button (which you need to press, or nothing happens, pro-tip).

Updating from 01.00.25 to 01.05.21... took about 10 more minutes. After System is rebooting, Please wait! it might be done. You'll have to check yourself.

And after all that waiting, I was finally greeted with new certificates. Both for the web interface and the iKVM interface:

$ easycert -T 10.10.10.10 443
Certificate chain
 0 s: {9C:70:1A:CF:57:9E:B9:A1:CE:EB:B8:99:47:3C:A8:4D:88:1B:BD:16} [a8c7e6cd] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
   i: {9C:70:1A:CF:57:9E:B9:A1:CE:EB:B8:99:47:3C:A8:4D:88:1B:BD:16} [a8c7e6cd] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
---
Expires in 3622 days

$ easycert -T 10.10.10.10 5900
Certificate chain
 0 s: {68:6D:88:26:E2:B4:34:3F:CC:06:FC:E7:02:99:2B:6C:FC:57:61:7D} [a8c7e6cd] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
   i: {68:6D:88:26:E2:B4:34:3F:CC:06:FC:E7:02:99:2B:6C:FC:57:61:7D} [a8c7e6cd] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
---
Expires in 3622 days

Ok. No faketime.so needed anymore.

Hardware crash

Unfortunately, the BMC firmware upgrade came with a “present”:

mlx5_core 0000:81:00.0: temp_warn:165:(pid 0): High temperature on sensors with bit set 0 1
mlx5_core 0000:81:00.1: temp_warn:165:(pid 0): High temperature on sensors with bit set 0 1
...
mlx5_core 0000:81:00.0: mlx5_crdump_collect:50:(pid 1100352): crdump: failed to lock vsc gw err -16
mlx5_core 0000:81:00.0: mlx5_health_try_recover:327:(pid 1100352): handling bad device here
mlx5_core 0000:81:00.0: mlx5_error_sw_reset:233:(pid 1100352): start
mlx5_core 0000:81:00.1: mlx5_crdump_collect:50:(pid 1031132): crdump: failed to lock vsc gw err -16
mlx5_core 0000:81:00.1: mlx5_health_try_recover:327:(pid 1031132): handling bad device here
mlx5_core 0000:81:00.1: mlx5_error_sw_reset:233:(pid 1031132): start
mlx5_core 0000:81:00.0: NIC IFC still 7 after 1000ms.
mlx5_core 0000:81:00.0: mlx5_error_sw_reset:266:(pid 1100352): end
mlx5_core 0000:81:00.1: NIC IFC still 7 after 1000ms.
mlx5_core 0000:81:00.1: mlx5_error_sw_reset:266:(pid 1031132): end
mlx5_core 0000:81:00.0: mlx5_wait_for_pages:735:(pid 1100352): Skipping wait for vf pages stage
mlx5_core 0000:81:00.1: mlx5_wait_for_pages:735:(pid 1031132): Skipping wait for vf pages stage
mlx5_core 0000:81:00.0: mlx5_health_try_recover:330:(pid 1100352): health recovery flow aborted, PCI reads still not working
mlx5_core 0000:81:00.1: mlx5_health_try_recover:330:(pid 1031132): health recovery flow aborted, PCI reads still not working

Not sure if the upgrade was the culprit, but mid-upgrade the Mellanox network devices both crashed. Unloading and reloading the kernel modules did not revive them, so a reboot was needed.

Luckily I didn't have to fight the expired certificate this time.