Refactor the async wait fd logic

Implementation experience has shown that the original plan for async wait
fds was too simplistic. Originally the async logic created a pipe internally
and user/engine code could then get access to it via API calls. It is more
flexible if the engine is able to create its own fd and provide it to the
async code.

Another issue is that there can be a lot of churn in the fd value within
the context of (say) a single SSL connection leading to continually adding
and removing fds from (say) epoll. It is better if we can provide some
stability of the fd value across a whole SSL connection. This is
problematic because an engine has no concept of an SSL connection.

This commit refactors things to introduce an ASYNC_WAIT_CTX which acts as a
proxy for an SSL connection down at the engine layer.

Reviewed-by: Richard Levitte <levitte@openssl.org>

2 files changed, 65 insertions, 45 deletions

diff --git a/doc/ssl/SSL_get_all_async_fds.pod b/doc/ssl/SSL_get_all_async_fds.pod
new file mode 100644
index 0000000000..a5064e213d
--- /dev/null
+++ b/doc/ssl/SSL_get_all_async_fds.pod
@@ -0,0 +1,65 @@
+=pod
+
+=head1 NAME
+
+SSL_waiting_for_async, SSL_get_all_async_fds, SSL_get_changed_async_fds - manage
+asynchronous operations
+
+=head1 SYNOPSIS
+
+ #include <openssl/ssl.h>
+
+ int SSL_waiting_for_async(SSL *s);
+ int SSL_get_all_async_fds(SSL *s, OSSL_ASYNC_FD *fd, size_t *numfds);
+ int SSL_get_changed_async_fds(SSL *s, OSSL_ASYNC_FD *addfd, size_t *numaddfds,
+                               OSSL_ASYNC_FD *delfd, size_t *numdelfds);
+
+=head1 DESCRIPTION
+
+SSL_waiting_for_async() determines whether an SSL connection is currently
+waiting for asynchronous operations to complete (see the SSL_MODE_ASYNC mode in
+L<SSL_CTX_set_mode(3)>).
+
+SSL_get_all_async_fds() returns a list of file descriptor which can be used in a
+call to select() or poll() to determine whether the current asynchronous
+operation has completed or not. A completed operation will result in data
+appearing as "read ready" on the file descriptor (no actual data should be read
+from the file descriptor). This function should only be called if the SSL object
+is currently waiting for asynchronous work to complete (i.e.
+SSL_ERROR_WANT_ASYNC has been received - see L<SSL_get_error(3)>). Typically the
+list will only contain one file descriptor. However if multiple asynchronous
+capable engines are in use then more than one is possible. The number of file
+descriptors returned is stored in B<*numfds> and the file descriptors themselves
+are in B<*fds>. The B<fds> parameter may be NULL in which case no file
+descriptors are returned but B<*numfds> is still populated. It is the callers
+responsibility to ensure sufficient memory is allocated at B<*fds> so typically
+this function is called twice (once with a NULL B<fds> parameter and once
+without).
+
+SSL_get_changed_async_fds() returns a list of the asynchronous file descriptors
+that have been added and a list that have been deleted since the last
+SSL_ERROR_WANT_ASYNC was received (or since the SSL object was created if no
+SSL_ERROR_WANT_ASYNC has been received). Similar to SSL_get_all_async_fds() it
+is the callers responsibility to ensure that B<*addfd> and B<*delfd> have
+sufficient memory allocated, although they may be NULL. The number of added fds
+and the number of deleted fds are stored in B<*numaddfds> and B<*numdelfds>
+respectively.
+
+=head1 RETURN VALUES
+
+SSL_waiting_for_async() will return 1 if the current SSL operation is waiting
+for an async operation to complete and 0 otherwise.
+
+SSL_get_all_async_fds() and SSL_get_changed_async_fds() return 1 on success or
+0 on error.
+
+=head1 SEE ALSO
+
+L<SSL_get_error(3)>, L<SSL_CTX_set_mode(3)>
+
+=head1 HISTORY
+
+SSL_waiting_for_async(), SSL_get_all_async_fds() and SSL_get_changed_async_fds()
+were first added to OpenSSL 1.1.0.
+
+=cut
diff --git a/doc/ssl/SSL_get_async_wait_fd.pod b/doc/ssl/SSL_get_async_wait_fd.pod
deleted file mode 100644
index 840c9c886e..0000000000
--- a/doc/ssl/SSL_get_async_wait_fd.pod
+++ /dev/null
@@ -1,45 +0,0 @@
-=pod
-
-=head1 NAME
-
-SSL_waiting_for_async, SSL_get_async_wait_fd - manage asynchronous operations
-
-=head1 SYNOPSIS
-
- #include <openssl/ssl.h>
-
- int SSL_waiting_for_async(SSL *s);
- int SSL_get_async_wait_fd(SSL *s);
-
-=head1 DESCRIPTION
-
-SSL_waiting_for_async() determines whether an SSL connection is currently
-waiting for asynchronous operations to complete (see the SSL_MODE_ASYNC mode in
-L<SSL_CTX_set_mode(3)>).
-
-SSL_get_async_wait_fd() returns a file descriptor which can be used in a call to
-select() or poll() to determine whether the current asynchronous operation has
-completed or not. A completed operation will result in data appearing as
-"read ready" on the file descriptor (no actual data should be read from the
-file descriptor). This function should only be called if the SSL object is
-currently waiting for asynchronous work to complete (i.e. SSL_ERROR_WANT_ASYNC
-has been received - see L<SSL_get_error(3)>).
-
-=head1 RETURN VALUES
-
-SSL_waiting_for_async() will return 1 if the current SSL operation is waiting
-for an async operation to complete and 0 otherwise.
-
-SSL_get_async_wait_fd() will return a file descriptor that can be used in a call
-to select() or poll() to wait for asynchronous work to complete, or -1 on error.
-
-=head1 SEE ALSO
-
-L<SSL_get_error(3)>, L<SSL_CTX_set_mode(3)>
-
-=head1 HISTORY
-
-SSL_waiting_for_async() and SSL_get_async_wait_fd() were first added to
-OpenSSL 1.1.0
-
-=cut