Workaround: Fix OpenSSL::PKey.read that cannot parse PKey in the FIPS mode.

This commit is a workaround to avoid the error below that the
`OpenSSL::PKey.read` fails with the OpenSSL 3.0 FIPS mode.

```
$ openssl genrsa -out key.pem 4096

$ ruby -e "require 'openssl'; OpenSSL::PKey.read(File.read('key.pem'))"
-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
  from -e:1:in `<main>'
```

The root cause is on the OpenSSL side. The `OSSL_DECODER_CTX_set_selection`
doesn't apply the selection value properly if there are multiple providers, and
a provider (e.g.  "base" provider) handles the decoder implementation, and
another provider (e.g. "fips" provider) handles the keys.

The workaround is to create `OSSL_DECODER_CTX` variable each time without using
the `OSSL_DECODER_CTX_set_selection`.

1 files changed, 21 insertions, 5 deletions

diff --git a/ext/openssl/ossl_pkey.c b/ext/openssl/ossl_pkey.c
index 47625667..a8e97d0b 100644
--- a/ext/openssl/ossl_pkey.c
+++ b/ext/openssl/ossl_pkey.c
@@ -101,10 +101,9 @@ ossl_pkey_read_generic(BIO *bio, VALUE pass)
         goto out;
     OSSL_BIO_reset(bio);
 
-    /* Then check PEM; multiple OSSL_DECODER_from_bio() calls may be needed */
-    if (OSSL_DECODER_CTX_set_input_type(dctx, "PEM") != 1)
-        goto out;
     /*
+     * Then check PEM; multiple OSSL_DECODER_from_bio() calls may be needed.
+     *
      * First check for private key formats. This is to keep compatibility with
      * ruby/openssl < 3.0 which decoded the following as a private key.
      *
@@ -124,8 +123,19 @@ ossl_pkey_read_generic(BIO *bio, VALUE pass)
      *
      * Note that normally, the input is supposed to contain a single decodable
      * PEM block only, so this special handling should not create a new problem.
+     *
+     * Note that we need to create the OSSL_DECODER_CTX variable each time when
+     * we use the different selection as a workaround.
+     * https://github.com/openssl/openssl/issues/20657
      */
-    OSSL_DECODER_CTX_set_selection(dctx, EVP_PKEY_KEYPAIR);
+    OSSL_DECODER_CTX_free(dctx);
+    dctx = NULL;
+    dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "PEM", NULL, NULL,
+                                         EVP_PKEY_KEYPAIR, NULL, NULL);
+    if (!dctx)
+        goto out;
+    if (OSSL_DECODER_CTX_set_pem_password_cb(dctx, ossl_pem_passwd_cb, ppass) != 1)
+        goto out;
     while (1) {
         if (OSSL_DECODER_from_bio(dctx, bio) == 1)
             goto out;
@@ -139,7 +149,13 @@ ossl_pkey_read_generic(BIO *bio, VALUE pass)
     }
 
     OSSL_BIO_reset(bio);
-    OSSL_DECODER_CTX_set_selection(dctx, 0);
+    OSSL_DECODER_CTX_free(dctx);
+    dctx = NULL;
+    dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "PEM", NULL, NULL, 0, NULL, NULL);
+    if (!dctx)
+        goto out;
+    if (OSSL_DECODER_CTX_set_pem_password_cb(dctx, ossl_pem_passwd_cb, ppass) != 1)
+        goto out;
     while (1) {
         if (OSSL_DECODER_from_bio(dctx, bio) == 1)
             goto out;