EKCD: Preload error strings, PRNG seed; use OAEP padding

Preload OpenSSL ERR string data so that the formatted error messages are
vaguely meaningful. Add OpenSSL error information to the RSA_public_encrypt()
operation failure case in one-time key generation.

For obsolescent OpenSSL versions (*cough* FIPS *cough*), pre-seed the PRNG
before entering Cap mode, as old versions of OpenSSL are unaware of kernel
RNG interfaces aside from /dev/random (such as the long-supported kern.arnd, or
the slightly more recent getentropy(3) or getrandom(2)). (RSA_public_encrypt()
wants a seeded PRNG to randomize the "PS" portion of PKCS 1.5 padding or the
"MGF" pseudo-random function in OAEP padding.)

Switch dumpon to encrypt the one-time key with OAEP padding (recommended since
1998; RFC2437) rather than the obsolescent PKCS 1.5 padding (1993; RFC2313).

Switch decryptcore to attempt OAEP decryption first, and try PKCS 1.5
decryption on failure. This is intended only for transition convenience, and
we should obsolete support for non-OAEP padding in a release or two.

Reviewed by:	markj
MFC After:	2 weeks
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D24534

2 files changed, 24 insertions, 3 deletions

diff --git a/sbin/decryptcore/decryptcore.c b/sbin/decryptcore/decryptcore.c
index 7fbe237487c5..80050c9a0bfb 100644
--- a/sbin/decryptcore/decryptcore.c
+++ b/sbin/decryptcore/decryptcore.c
@@ -219,6 +219,10 @@ decrypt(int ofd, const char *privkeyfile, const char *keyfile,
 
 	if (RSA_private_decrypt(kdk->kdk_encryptedkeysize,
 	    kdk->kdk_encryptedkey, key, privkey,
+	    RSA_PKCS1_OAEP_PADDING) != sizeof(key) &&
+	    /* Fallback to deprecated, formerly-used PKCS 1.5 padding. */
+	    RSA_private_decrypt(kdk->kdk_encryptedkeysize,
+	    kdk->kdk_encryptedkey, key, privkey,
 	    RSA_PKCS1_PADDING) != sizeof(key)) {
 		pjdlog_error("Unable to decrypt key: %s",
 		    ERR_error_string(ERR_get_error(), NULL));
diff --git a/sbin/dumpon/dumpon.c b/sbin/dumpon/dumpon.c
index d705193d2418..2b710a611af3 100644
--- a/sbin/dumpon/dumpon.c
+++ b/sbin/dumpon/dumpon.c
@@ -77,6 +77,7 @@ __FBSDID("$FreeBSD$");
 #ifdef HAVE_CRYPTO
 #include <openssl/err.h>
 #include <openssl/pem.h>
+#include <openssl/rand.h>
 #include <openssl/rsa.h>
 #endif
 
@@ -224,6 +225,18 @@ genkey(const char *pubkeyfile, struct diocskerneldump_arg *kdap)
 	if (fp == NULL)
 		err(1, "Unable to open %s", pubkeyfile);
 
+	/*
+	 * Obsolescent OpenSSL only knows about /dev/random, and needs to
+	 * pre-seed before entering cap mode.  For whatever reason,
+	 * RSA_pub_encrypt uses the internal PRNG.
+	 */
+#if OPENSSL_VERSION_NUMBER < 0x10100000L
+	{
+		unsigned char c[1];
+		RAND_bytes(c, 1);
+	}
+#endif
+
 	if (caph_enter() < 0)
 		err(1, "Unable to enter capability mode");
 
@@ -286,8 +299,9 @@ genkey(const char *pubkeyfile, struct diocskerneldump_arg *kdap)
 	arc4random_buf(kdap->kda_key, sizeof(kdap->kda_key));
 	if (RSA_public_encrypt(sizeof(kdap->kda_key), kdap->kda_key,
 	    kdap->kda_encryptedkey, pubkey,
-	    RSA_PKCS1_PADDING) != (int)kdap->kda_encryptedkeysize) {
-		errx(1, "Unable to encrypt the one-time key.");
+	    RSA_PKCS1_OAEP_PADDING) != (int)kdap->kda_encryptedkeysize) {
+		errx(1, "Unable to encrypt the one-time key: %s",
+		    ERR_error_string(ERR_get_error(), NULL));
 	}
 	RSA_free(pubkey);
 }
@@ -470,8 +484,11 @@ main(int argc, char *argv[])
 		usage();
 
 #ifdef HAVE_CRYPTO
-	if (cipher != KERNELDUMP_ENC_NONE && pubkeyfile == NULL)
+	if (cipher != KERNELDUMP_ENC_NONE && pubkeyfile == NULL) {
 		errx(EX_USAGE, "-C option requires a public key file.");
+	} else if (pubkeyfile != NULL) {
+		ERR_load_crypto_strings();
+	}
 #else
 	if (pubkeyfile != NULL)
 		errx(EX_UNAVAILABLE,"Unable to use the public key."