Fixes related to separation of DH and DHX types

Fix dh_rfc5114 option in genpkey.

Fixes #14145
Fixes #13956
Fixes #13952
Fixes #13871
Fixes #14054
Fixes #14444

Updated documentation for app to indicate what options are available for
DH and DHX keys.

DH and DHX now have different keymanager gen_set_params() methods.

Added CHANGES entry to indicate the breaking change.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14883)

3 files changed, 222 insertions, 60 deletions

diff --git a/doc/man1/openssl-genpkey.pod.in b/doc/man1/openssl-genpkey.pod.in
index 7f4ebef439..aa08b01f4f 100644
--- a/doc/man1/openssl-genpkey.pod.in
+++ b/doc/man1/openssl-genpkey.pod.in
@@ -63,7 +63,7 @@ name accepted by EVP_get_cipherbyname() is acceptable such as B<des3>.
 
 =item B<-algorithm> I<alg>
 
-Public key algorithm to use such as RSA, DSA or DH. If used this option must
+Public key algorithm to use such as RSA, DSA, DH or DHX. If used this option must
 precede any B<-pkeyopt> options. The options B<-paramfile> and B<-algorithm>
 are mutually exclusive. Engines may add algorithms in addition to the standard
 built-in ones.
@@ -74,11 +74,8 @@ X25519, X448, ED25519 and ED448.
 Valid built-in algorithm names for parameter generation (see the B<-genparam>
 option) are DH, DSA and EC.
 
-Note that the algorithm name X9.42 DH may be used as a synonym for the DH
-algorithm. These are identical and do not indicate the type of parameters that
-will be generated. Use the B<dh_paramgen_type> option to indicate whether PKCS#3
-or X9.42 DH parameters are required. See L</DH Parameter Generation Options>
-below for more details.
+Note that the algorithm name X9.42 DH may be used as a synonym for DHX keys and
+PKCS#3 refers to DH Keys. Some options are not shared between DH and DHX keys.
 
 =item B<-pkeyopt> I<opt>:I<value>
 
@@ -182,6 +179,18 @@ B<named_curve> or B<explicit>. The default value is B<named_curve>.
 
 =back
 
+=head2 DH Key Generation Options
+
+=over 4
+
+=item B<group>:I<name>
+
+The B<paramfile> option is not required if a named group is used here.
+See the L</DH Parameter Generation Options> section below.
+
+=back
+
+
 =head1 PARAMETER GENERATION OPTIONS
 
 The options supported by each algorithm and indeed each implementation of an
@@ -214,7 +223,6 @@ ignored. If not set, then a digest will be used that gives an output matching
 the number of bits in B<q>, i.e. B<sha1> if q length is 160, B<sha224> if it 224
 or B<sha256> if it is 256.
 
-
 =item B<properties>:I<query>
 
 The I<digest> property I<query> string to use when fetching a digest from a provider.
@@ -243,36 +251,128 @@ generate valid primes.
 
 =head2 DH Parameter Generation Options
 
+For most use cases it is recommended to use the B<group> option rather than
+the B<type> options. Note that the B<group> option is not used by default if
+no parameter generation options are specified.
+
 =over 4
 
+=item B<group>:I<name>
+
+=item B<dh_param>:I<name>
+
+Use a named DH group to select constant values for the DH parameters.
+All other options will be ignored if this value is set.
+
+Valid values that are associated with the B<algorithm> of B<"DH"> are:
+"ffdhe2048", "ffdhe3072", "ffdhe4096", "ffdhe6144", "ffdhe8192",
+"modp_1536", "modp_2048", "modp_3072", "modp_4096", "modp_6144", "modp_8192".
+
+Valid values that are associated with the B<algorithm> of B<"DHX"> are the
+RFC5114 names "dh_1024_160", "dh_2048_224", "dh_2048_256".
+
+=item B<dh_rfc5114>:I<num>
+
+If this option is set, then the appropriate RFC5114 parameters are used
+instead of generating new parameters. The value I<num> can be one of
+1, 2 or 3 that are equivalant to using the option B<group> with one of
+"dh_1024_160", "dh_2048_224" or "dh_2048_256".
+All other options will be ignored if this value is set.
+
+=item B<pbits>:I<numbits>
+
 =item B<dh_paramgen_prime_len>:I<numbits>
 
 The number of bits in the prime parameter I<p>. The default is 2048.
 
+=item B<qbits>:I<numbits>
+
 =item B<dh_paramgen_subprime_len>:I<numbits>
 
-The number of bits in the sub prime parameter I<q>. The default is 256 if the
-prime is at least 2048 bits long or 160 otherwise. Only relevant if used in
-conjunction with the B<dh_paramgen_type> option to generate X9.42 DH parameters.
+The number of bits in the sub prime parameter I<q>. The default is 224.
+Only relevant if used in conjunction with the B<dh_paramgen_type> option to
+generate DHX parameters.
+
+=item B<safeprime-generator>:I<value>
 
 =item B<dh_paramgen_generator>:I<value>
 
 The value to use for the generator I<g>. The default is 2.
+The B<algorithm> option must be B<"DH"> for this parameter to be used.
+
+=item B<type>:I<string>
+
+The type name of DH parameters to generate. Valid values are:
+
+=over 4
+
+=item "generator"
+
+Use a safe prime generator with the option B<safeprime_generator>
+The B<algorithm> option must be B<"DH">.
+
+=item "fips186_4"
+
+FIPS186-4 parameter generation.
+The B<algorithm> option must be B<"DHX">.
+
+=item "fips186_2"
+
+FIPS186-4 parameter generation.
+The B<algorithm> option must be B<"DHX">.
+
+=item "group"
+
+Can be used with the option B<pbits> to select one of
+"ffdhe2048", "ffdhe3072", "ffdhe4096", "ffdhe6144" or "ffdhe8192".
+The B<algorithm> option must be B<"DH">.
+
+=item "default"
+
+Selects a default type based on the B<algorithm>. This is used by the
+OpenSSL default provider to set the type for backwards compatability.
+If B<algorithm> is B<"DH"> then B<"generator"> is used.
+If B<algorithm> is B<"DHX"> then B<"fips186_2"> is used.
+
+=back
 
 =item B<dh_paramgen_type>:I<value>
 
-The type of DH parameters to generate. Use 0 for PKCS#3 DH and 1 for X9.42 DH.
-The default is 0.
+The type of DH parameters to generate. Valid values are 0, 1, 2 or 3
+which correspond to setting the option B<type> to
+"generator", "fips186_2", "fips186_4" or "group".
 
-=item B<dh_rfc5114>:I<num>
+=item B<digest>:I<digest>
 
-If this option is set, then the appropriate RFC5114 parameters are used
-instead of generating new parameters. The value I<num> can be one of
-1, 2 or 3 corresponding to RFC5114 DH parameters consisting of
-1024 bit group with 160 bit subgroup, 2048 bit group with 224 bit subgroup
-and 2048 bit group with 256 bit subgroup as mentioned in RFC5114 sections
-2.1, 2.2 and 2.3 respectively. If present this overrides all other DH parameter
-options.
+The digest to use during parameter generation. Must be one of B<sha1>, B<sha224>
+or B<sha256>. If set, then the number of bits in B<qbits> will match the output
+size of the specified digest and the B<qbits> parameter will be
+ignored. If not set, then a digest will be used that gives an output matching
+the number of bits in B<q>, i.e. B<sha1> if q length is 160, B<sha224> if it is
+224 or B<sha256> if it is 256.
+This is only used by "fips186_4" and "fips186_2" key generation.
+
+=item B<properties>:I<query>
+
+The I<digest> property I<query> string to use when fetching a digest from a provider.
+This is only used by "fips186_4" and "fips186_2" key generation.
+
+=item B<gindex>:I<index>
+
+The index to use for canonical generation and verification of the generator g.
+Set this to a positive value ranging from 0..255 to use this mode. Larger values
+will only use the bottom byte.
+This I<index> must then be reused during key validation to verify the value of g.
+If this value is not set then g is not verifiable. The default value is -1.
+This is only used by "fips186_4" and "fips186_2" key generation.
+
+=item B<hexseed>:I<seed>
+
+The seed I<seed> data to use instead of generating a random seed internally.
+This should be used for testing purposes only. This will either produced fixed
+values for the generated parameters OR it will fail if the seed did not
+generate valid primes.
+This is only used by "fips186_4" and "fips186_2" key generation.
 
 =back
 
@@ -313,25 +413,49 @@ Generate DSA key from parameters:
 
  openssl genpkey -paramfile dsap.pem -out dsakey.pem
 
-Generate 2048 bit DH parameters:
+Generate 4096 bit DH Key using safe prime group ffdhe4096:
+
+ openssl genpkey -algorithm DH -out dhkey.pem -pkeyopt group:ffdhe4096
+
+Generate 2048 bit X9.42 DH key with 256 bit subgroup using RFC5114 group3:
+
+ openssl genpkey -algorithm DHX -out dhkey.pem -pkeyopt dh_rfc5114:3
+
+Generate a DH key using a DH parameters file:
+
+ openssl genpkey -paramfile dhp.pem -out dhkey.pem
+
+Output DH parameters for safe prime group ffdhe2048:
+
+ openssl genpkey -genparam -algorithm DH -out dhp.pem -pkeyopt group:ffdhe2048
+
+Output 2048 bit X9.42 DH parameters with 224 bit subgroup using RFC5114 group2:
+
+ openssl genpkey -genparam -algorithm DHX -out dhp.pem -pkeyopt dh_rfc5114:2
+
+Output 2048 bit X9.42 DH parameters with 224 bit subgroup using FIP186-4 keygen:
+
+ openssl genpkey -genparam -algorithm DHX -out dhp.pem -text \
+     -pkeyopt pbits:2048 -pkeyopt qbits:224 -pkeyopt digest:SHA256 \
+     -pkeyopt gindex:1 -pkeyopt dh_paramgen_type:2
+
+Output 1024 bit X9.42 DH parameters with 160 bit subgroup using FIP186-2 keygen:
+
+ openssl genpkey -genparam -algorithm DHX -out dhp.pem -text \
+     -pkeyopt pbits:1024 -pkeyopt qbits:160 -pkeyopt digest:SHA1 \
+     -pkeyopt gindex:1 -pkeyopt dh_paramgen_type:1
+
+Output 2048 bit DH parameters:
 
  openssl genpkey -genparam -algorithm DH -out dhp.pem \
      -pkeyopt dh_paramgen_prime_len:2048
 
-Generate 2048 bit X9.42 DH parameters:
+Output 2048 bit DH parameters using a generator:
 
  openssl genpkey -genparam -algorithm DH -out dhpx.pem \
      -pkeyopt dh_paramgen_prime_len:2048 \
      -pkeyopt dh_paramgen_type:1
 
-Output RFC5114 2048 bit DH parameters with 224 bit subgroup:
-
- openssl genpkey -genparam -algorithm DH -out dhp.pem -pkeyopt dh_rfc5114:2
-
-Generate DH key from parameters:
-
- openssl genpkey -paramfile dhp.pem -out dhkey.pem
-
 Generate EC parameters:
 
  openssl genpkey -genparam -algorithm EC -out ecp.pem \
@@ -367,7 +491,7 @@ The B<-engine> option was deprecated in OpenSSL 3.0.
 
 =head1 COPYRIGHT
 
-Copyright 2006-2020 The OpenSSL Project Authors. All Rights Reserved.
+Copyright 2006-2021 The OpenSSL Project Authors. All Rights Reserved.
 
 Licensed under the Apache License 2.0 (the "License").  You may not use
 this file except in compliance with the License.  You can obtain a copy
diff --git a/doc/man7/EVP_PKEY-DH.pod b/doc/man7/EVP_PKEY-DH.pod
index 72d20f6f1c..60865a7120 100644
--- a/doc/man7/EVP_PKEY-DH.pod
+++ b/doc/man7/EVP_PKEY-DH.pod
@@ -2,7 +2,8 @@
 
 =head1 NAME
 
-EVP_PKEY-DH, EVP_KEYMGMT-DH - EVP_PKEY DH keytype and algorithm support
+EVP_PKEY-DH, EVP_PKEY-DHX, EVP_KEYMGMT-DH
+- EVP_PKEY DH and DHX keytype and algorithm support
 
 =head1 DESCRIPTION
 
@@ -14,25 +15,30 @@ applications that cannot be upgraded to use the approved safe-prime groups.
 
 See L<EVP_PKEY-FFC(7)> for more information about FFC keys.
 
-For B<DH> that is not a named group the FIPS186-4 standard specifies that the
+The B<DH> key type uses PKCS#3 format which saves p and g, but not the 'q' value.
+The B<DHX> key type uses X9.42 format which saves the value of 'q' and this
+must be used for FIPS186-4.
+
+For B<DHX> that is not a named group the FIPS186-4 standard specifies that the
 values used for FFC parameter generation are also required for parameter
 validation. This means that optional FFC domain parameter values for
-I<seed>, I<pcounter> and I<gindex> may need to be stored for validation purposes.
-For B<DH> the I<seed> and I<pcounter> can be stored in ASN1 data
-(but the I<gindex> is not).
+I<seed>, I<pcounter> and I<gindex> or I<hindex> may need to be stored for
+validation purposes.
+For B<DHX> the I<seed> and I<pcounter> can be stored in ASN1 data
+(but the I<gindex> or I<hindex> can not be stored).
 
-=head2 DH parameters
+=head2 DH and DHX domain parameters
 
 In addition to the common FCC parameters that all FFC keytypes should support
-(see L<EVP_PKEY-FFC(7)/FFC parameters>) the B<DH> keytype
-implementation supports the following:
+(see L<EVP_PKEY-FFC(7)/FFC parameters>) the B<DHX> and B<DH> keytype
+implementations support the following:
 
 =over 4
 
 =item "group" (B<OSSL_PKEY_PARAM_GROUP_NAME>) <UTF8 string>
 
-Set or gets a string that associates a B<DH> named safe prime group with known
-values for I<p>, I<q> and I<g>.
+Sets or gets a string that associates a B<DH> or B<DHX> named safe prime group
+with known values for I<p>, I<q> and I<g>.
 
 The following values can be used by the OpenSSL's default and FIPS providers:
 "ffdhe2048", "ffdhe3072", "ffdhe4096", "ffdhe6144", "ffdhe8192",
@@ -41,31 +47,46 @@ The following values can be used by the OpenSSL's default and FIPS providers:
 The following additional values can also be used by OpenSSL's default provider:
 "modp_1536", "dh_1024_160", "dh_2048_224", "dh_2048_256".
 
-DH named groups can be easily validated since the parameters are well known.
+DH/DHX named groups can be easily validated since the parameters are well known.
 For protocols that only transfer I<p> and I<g> the value of I<q> can also be
 retrieved.
 
-=item "safeprime-generator" (B<OSSL_PKEY_PARAM_DH_GENERATOR>) <integer>
+=back
 
-Used for DH generation of safe primes using the old generator code.
-It is recommended to use a named safe prime group instead, if domain parameter
-validation is required. The default value is 2.
+=head2 DH and DHX additional parameters
 
-These are not named safe prime groups so setting this value for the OpenSSL FIPS
-provider will instead choose a named safe prime group based on the size of I<p>.
+=over 4
 
 =item "encoded-pub-key" (B<OSSL_PKEY_PARAM_ENCODED_PUBLIC_KEY>) <octet string>
 
 Used for getting and setting the encoding of the DH public key used in a key
 exchange message for the TLS protocol.
+See EVP_PKEY_set1_encoded_public_key() and EVP_PKEY_get1_encoded_public_key().
+
+=back
+
+=head2 DH additional domain parameters
+
+=over 4
+
+=item "safeprime-generator" (B<OSSL_PKEY_PARAM_DH_GENERATOR>) <integer>
+
+Used for DH generation of safe primes using the old safe prime generator code.
+The default value is 2.
+It is recommended to use a named safe prime group instead, if domain parameter
+validation is required. 
+
+Randomly generated safe primes are not allowed by FIPS, so setting this value
+for the OpenSSL FIPS provider will instead choose a named safe prime group
+based on the size of I<p>.
 
 =back
 
-=head2 DH domain parameter / key generation parameters
+=head2 DH and DHX domain parameter / key generation parameters
 
 In addition to the common FFC key generation parameters that all FFC key types
 should support (see L<EVP_PKEY-FFC(7)/FFC key generation parameters>) the
-B<DH> keytype implementation supports the following:
+B<DH> and B<DHX> keytype implementation supports the following:
 
 =over 4
 
@@ -91,6 +112,7 @@ type.
 =item "generator"
 
 A safe prime generator. See the "safeprime-generator" type above.
+This is only valid for B<DH> keys.
 
 =back
 
diff --git a/doc/man7/EVP_PKEY-FFC.pod b/doc/man7/EVP_PKEY-FFC.pod
index 7e238d7b44..e345580ec1 100644
--- a/doc/man7/EVP_PKEY-FFC.pod
+++ b/doc/man7/EVP_PKEY-FFC.pod
@@ -2,7 +2,7 @@
 
 =head1 NAME
 
-EVP_PKEY-FFC - EVP_PKEY DSA and DH shared FFC parameters.
+EVP_PKEY-FFC - EVP_PKEY DSA and DH/DHX shared FFC parameters.
 
 =head1 DESCRIPTION
 
@@ -11,9 +11,9 @@ cryptography using finite field mathematics. DSA is an example of FFC and
 Diffie-Hellman key establishment algorithms specified in SP800-56A can also be
 implemented as FFC.
 
-The B<DSA> and B<DH> keytypes are implemented in OpenSSL's default and FIPS
-providers.
-The implementations support the basic DSA and DH keys, containing the public
+The B<DSA>, B<DH> and B<DHX> keytypes are implemented in OpenSSL's default and
+FIPS providers.
+The implementations support the basic DSA, DH and DHX keys, containing the public
 and private keys I<pub> and I<priv> as well as the three main domain parameters
 I<p>, I<q> and I<g>.
 
@@ -26,10 +26,14 @@ For B<DH> the I<seed> and I<pcounter> can be stored in ASN1 data
 (but the I<gindex> is not). For B<DSA> however, these fields are not stored in
 the ASN1 data so they need to be stored externally if validation is required.
 
+The B<DH> key type uses PKCS#3 format which saves p and g, but not the 'q' value.
+The B<DHX> key type uses X9.42 format which saves the value of 'q' and this
+must be used for FIPS186-4.
+
 =head2 FFC parameters
 
 In addition to the common parameters that all keytypes should support (see
-L<provider-keymgmt(7)/Common parameters>), the B<DSA> and B<DH> keytype
+L<provider-keymgmt(7)/Common parameters>), the B<DSA>, B<DH> and B<DHX> keytype
 implementations support the following.
 
 =over 4
@@ -42,18 +46,30 @@ The public key value.
 
 The private key value.
 
-=item "p" (B<OSSL_PKEY_PARAM_FFC_P>) <unsigned integer>
+=back
 
-A DSA or Diffie-Hellman prime "p" value.
+=head2 FFC DSA, DH and DHX domain parameters
 
-=item "q" (B<OSSL_PKEY_PARAM_FFC_Q>) <unsigned integer>
+=over 4
 
-A DSA or Diffie-Hellman prime "q" value.
+=item "p" (B<OSSL_PKEY_PARAM_FFC_P>) <unsigned integer>
+
+A DSA or Diffie-Hellman prime "p" value.
 
 =item "g" (B<OSSL_PKEY_PARAM_FFC_G>) <unsigned integer>
 
 A DSA or Diffie-Hellman generator "g" value.
 
+=back
+
+=head2 FFC DSA and DHX domain parameters
+
+=over 4
+
+=item "q" (B<OSSL_PKEY_PARAM_FFC_Q>) <unsigned integer>
+
+A DSA or Diffie-Hellman prime "q" value.
+
 =item "seed" (B<OSSL_PKEY_PARAM_FFC_SEED>) <octet string>
 
 An optional domain parameter I<seed> value used during generation and validation
@@ -88,7 +104,7 @@ An optional informational cofactor parameter that should equal to (p - 1) / q.
 
 =head2 FFC key generation parameters
 
-The following key generation types are available for DSA and DH algorithms:
+The following key generation types are available for DSA and DHX algorithms:
 
 =over 4