Add SSL_CTX early callback

Provide a callback interface that gives the application the ability
to adjust the nascent SSL object at the earliest stage of ClientHello
processing, immediately after extensions have been collected but
before they have been processed.

This is akin to BoringSSL's "select_certificate_cb" (though it is not
API compatible), and as the name indicates, one major use is to examine
the supplied server name indication and select what certificate to
present to the client.  However, it can also be used to make more
sweeping configuration changes to the SSL object according to the
selected server identity and configuration.  That may include adjusting
the permitted TLS versions, swapping out the SSL_CTX object (as is
traditionally done in a tlsext_servername_callback), changing the
server's cipher list, and more.

We also wish to allow an early callback to indicate that it needs to perform
additional work asynchronously and resume processing later.  To that effect,
refactor the second half of tls_process_client_hello() into a subroutine to be
called at the post-processing stage (including the early callback itself), to
allow the callback to result in remaining in the same work stage for a later
call to succeed.  This requires allocating for and storing the CLIENTHELLO_MSG
in the SSL object to be preserved across such calls, but the storage is
reclaimed after ClientHello processing finishes.

Information about the CliehtHello is available to the callback by means of
accessor functions that can only be used from the early callback.  This allows
extensions to make use of the existing internal parsing machinery without
exposing structure internals (e.g., of PACKET), so that applications do not
have to write fragile parsing code.

Applications are encouraged to utilize an early callback and not use
a servername_callback, in order to avoid unexpected behavior that
occurs due to the relative order of processing between things like
session resumption and the historical servername callback.

Also tidy up nearby style by removing unnecessary braces around one-line
conditional bodies.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2279)

3 files changed, 136 insertions, 6 deletions

diff --git a/doc/man3/SSL_CTX_set_early_cb.pod b/doc/man3/SSL_CTX_set_early_cb.pod
new file mode 100644
index 0000000000..b007292fdb
--- /dev/null
+++ b/doc/man3/SSL_CTX_set_early_cb.pod
@@ -0,0 +1,110 @@
+=pod
+
+=head1 NAME
+
+SSL_CTX_set_early_cb, SSL_early_cb_fn, SSL_early_isv2, SSL_early_get0_legacy_version, SSL_early_get0_random, SSL_early_get0_session_id, SSL_early_get0_ciphers, SSL_early_get0_compression_methods, SSL_early_get0_ext - callback functions for early server-side ClientHello processing
+
+=head1 SYNOPSIS
+
+ typedef int (*SSL_early_cb_fn)(SSL *s, int *al, void *arg);
+ void SSL_CTX_set_early_cb(SSL_CTX *c, SSL_early_cb_fn *f, void *arg);
+ int SSL_early_isv2(SSL *s);
+ unsigned int SSL_early_get0_legacy_version(SSL *s);
+ size_t SSL_early_get0_random(SSL *s, const unsigned char **out);
+ size_t SSL_early_get0_session_id(SSL *s, const unsigned char **out);
+ size_t SSL_early_get0_ciphers(SSL *s, const unsigned char **out);
+ size_t SSL_early_get0_compression_methods(SSL *s, const unsigned char **out);
+ int SSL_early_get0_ext(SSL *s, int type, const unsigned char **out,
+                        size_t *outlen);
+
+=head1 DESCRIPTION
+
+SSL_CTX_set_early_cb() sets the callback function, which is automatically
+called during the early stages of ClientHello processing on the server.
+The argument supplied when setting the callback is passed back to the
+callback at runtime.  A callback that returns failure (0) will cause the
+connection to terminate, and callbacks returning failure should indicate
+what alert value is to be sent in the B<al> parameter.  A callback may
+also return a negative value to suspend the handshake, and the handshake
+function will return immediately.  L<SSL_get_error(3)> will return
+SSL_ERROR_WANT_EARLY to indicate that the handshake was suspended.
+It is the job of the early callback to store information about the state
+of the last call if needed to continue.  On the next call into the handshake
+function, the early callback will be called again, and, if it returns
+success, normal handshake processing will continue from that point.
+
+SSL_early_isv2() indicates whether the ClientHello was carried in a
+SSLv2 record and is in the SSLv2 format.  The SSLv2 format has substantial
+differences from the normal SSLv3 format, including using three bytes per
+cipher suite, and not allowing extensions.  Additionally, the SSLv2 format
+'challenge' field is exposed via SSL_early_get0_random(), padded to
+SSL3_RANDOM_SIZE bytes with zeros if needed.  For SSLv2 format ClientHellos,
+SSL_early_get0_compression_methods() returns a dummy list that only includes
+the null compression method, since the SSLv2 format does not include a
+mechanism by which to negotiate compression.
+
+SSL_early_get0_random(), SSL_early_get0_session_id(), SSL_early_get0_ciphers(),
+and SSL_early_get0_compression_methods() provide access to the corresponding
+ClientHello fields, returning the field length and optionally setting an
+out pointer to the octets of that field.
+
+Similarly, SSL_early_get0_ext() provides access to individual extensions
+from the ClientHello on a per-extension basis.  For the provided wire
+protocol extension type value, the extension value and length are returned
+in the output parameters (if present).
+
+=head1 NOTES
+
+The early callback provides a vast window of possibilities for application
+code to affect the TLS handshake.  A primary use of the callback is to
+allow the server to examine the server name indication extension provided
+by the client in order to select an appropriate certificate to present,
+and make other configuration adjustments relevant to that server name
+and its configuration.  Such configuration changes can include swapping out
+the associated SSL_CTX pointer, modifying the server's list of permitted TLS
+versions, changing the server's cipher list, etc.
+
+It is also recommended that applications utilize an early callback and
+not use a servername callback, in order to avoid unexpected behavior that
+occurs due to the relative order of processing between things like session
+resumption and the historical servername callback.
+
+The SSL_early_* family of functions may only be called from code executing
+within an early callback.
+
+=head1 RETURN VALUES
+
+The application's supplied early callback returns 1 on success, 0 on failure,
+and a negative value to suspend processing.
+
+SSL_early_isv2() returns 1 for SSLv2-format ClientHellos and 0 otherwise.
+
+SSL_early_get0_random(), SSL_early_get0_session_id(), SSL_early_get0_ciphers(),
+and SSL_early_get0_compression_methods() return the length of the corresponding
+ClientHello fields.  If zero is returned, the ouput pointer should not be
+assumed to be valid.
+
+SSL_early_get0_ext() returns 1 if the extension of type 'type' is present, and
+0 otherwise.
+
+=head1 SEE ALSO
+
+L<ssl(7)>, L<SSL_CTX_set_tlsext_servername_callback(3)>
+
+=head1 HISTORY
+
+The SSL early callback, SSL_early_isv2(), SSL_early_get0_random(),
+SSL_early_get0_session_id(), SSL_early_get0_ciphers(),
+SSL_early_get0_compression_methods(), and SSL_early_get0_ext() were
+added in OpenSSL 1.1.1.
+
+=head1 COPYRIGHT
+
+Copyright 2017 The OpenSSL Project Authors. All Rights Reserved.
+
+Licensed under the OpenSSL license (the "License").  You may not use
+this file except in compliance with the License.  You can obtain a copy
+in the file LICENSE in the source distribution or at
+L<https://www.openssl.org/source/license.html>.
+
+=cut
diff --git a/doc/man3/SSL_get_error.pod b/doc/man3/SSL_get_error.pod
index db8f85c90a..e318de84b6 100644
--- a/doc/man3/SSL_get_error.pod
+++ b/doc/man3/SSL_get_error.pod
@@ -110,6 +110,13 @@ through a call to L<ASYNC_init_thread(3)>. The application should retry the
 operation after a currently executing asynchronous operation for the current
 thread has completed.
 
+=item SSL_ERROR_WANT_EARLY
+
+The operation did not complete because an application callback set by
+SSL_CTX_set_early_cb() has asked to be called again.
+The TLS/SSL I/O function should be called again later.
+Details depend on the application.
+
 =item SSL_ERROR_SYSCALL
 
 Some non-recoverable I/O error occurred.
@@ -130,10 +137,11 @@ L<ssl(7)>, L<err(7)>
 =head1 HISTORY
 
 SSL_ERROR_WANT_ASYNC was added in OpenSSL 1.1.0.
+SSL_ERROR_WANT_EARLY was added in OpenSSL 1.1.1.
 
 =head1 COPYRIGHT
 
-Copyright 2000-2016 The OpenSSL Project Authors. All Rights Reserved.
+Copyright 2000-2017 The OpenSSL Project Authors. All Rights Reserved.
 
 Licensed under the OpenSSL license (the "License").  You may not use
 this file except in compliance with the License.  You can obtain a copy
diff --git a/doc/man3/SSL_want.pod b/doc/man3/SSL_want.pod
index c86344eece..8efe50bcca 100644
--- a/doc/man3/SSL_want.pod
+++ b/doc/man3/SSL_want.pod
@@ -3,8 +3,8 @@
 =head1 NAME
 
 SSL_want, SSL_want_nothing, SSL_want_read, SSL_want_write, SSL_want_x509_lookup,
-SSL_want_async, SSL_want_async_job - obtain state information TLS/SSL I/O
-operation
+SSL_want_async, SSL_want_async_job, SSL_want_early - obtain state information
+TLS/SSL I/O operation
 
 =head1 SYNOPSIS
 
@@ -17,6 +17,7 @@ operation
  int SSL_want_x509_lookup(const SSL *ssl);
  int SSL_want_async(const SSL *ssl);
  int SSL_want_async_job(const SSL *ssl);
+ int SSL_want_early(const SSL *ssl);
 
 =head1 DESCRIPTION
 
@@ -81,19 +82,30 @@ The asynchronous job could not be started because there were no async jobs
 available in the pool (see ASYNC_init_thread(3)). A call to L<SSL_get_error(3)>
 should return SSL_ERROR_WANT_ASYNC_JOB.
 
+=item SSL_EARLY_WORK
+
+The operation did not complete because an application callback set by
+SSL_CTX_set_early_cb() has asked to be called again.
+A call to L<SSL_get_error(3)> should return
+SSL_ERROR_WANT_EARLY.
+
 =back
 
 SSL_want_nothing(), SSL_want_read(), SSL_want_write(), SSL_want_x509_lookup(),
-SSL_want_async() and SSL_want_async_job() return 1, when the corresponding
-condition is true or 0 otherwise.
+SSL_want_async(), SSL_want_async_job(), and SSL_want_early() return 1, when
+the corresponding condition is true or 0 otherwise.
 
 =head1 SEE ALSO
 
 L<ssl(7)>, L<err(7)>, L<SSL_get_error(3)>
 
+=head1 HISTORY
+
+SSL_want_early() and SSL_EARLY_WORK were added in OpenSSL 1.1.1.
+
 =head1 COPYRIGHT
 
-Copyright 2001-2016 The OpenSSL Project Authors. All Rights Reserved.
+Copyright 2001-2017 The OpenSSL Project Authors. All Rights Reserved.
 
 Licensed under the OpenSSL license (the "License").  You may not use
 this file except in compliance with the License.  You can obtain a copy