Document the issue with threads and dlopen()

If using threads and OpenSSL is loaded via dlopen(), and subsequently
closed again via dlclose() *before* the threads are destroyed, then
OpenSSL will not free up the per thread resources. We need to document
this restriction, and provide some guidance on what to do about it.

I did some testing and discovered/verified a few of things (at least
this is the behaviour on Linux):

- Using OpenSSL via dlopen in a mutli-threaded app does leak memory if
threads are destroyed after dlcose() is called.
- In a single threaded environment, or if threads are destroyed prior to
dlclose() being called, then no memory is leaked
- Using the RTLD_NODELETE flag to dlopen solves the above problem
- Interestingly the OpenSSL atexit() handler gets called when dlclose()
is called rather than at application exit (I was worred that it might crash
if there was an atexit() handler for a function that has been unloaded)
- RTLD_NODELETE is a non-standard flag - but it does seem to be fairly
widely supported. As far as I could determine (via google), at least Linux,
Solaris, OpenBSD, FreeBSD, HP-UX all seem to support it.

I also tested on Windows (using LoadLibrary instead of dlopen and
FreeLibrary instead of dlclose) and experienced similar behaviour, except
that (AFAIK) there is no equivalent of RTLD_NODELETE on Windows.

GitHub Issue #653

Reviewed-by: Richard Levitte <levitte@openssl.org>

1 files changed, 13 insertions, 3 deletions

diff --git a/doc/crypto/OPENSSL_init_crypto.pod b/doc/crypto/OPENSSL_init_crypto.pod
index 8caa36148c..a35325b3f2 100644
--- a/doc/crypto/OPENSSL_init_crypto.pod
+++ b/doc/crypto/OPENSSL_init_crypto.pod
@@ -203,9 +203,19 @@ platforms this is done in response to a DLL_THREAD_DETACH message being sent to
 the libcrypto32.dll entry point. Some windows functions may cause threads to exit
 without sending this message (for example ExitProcess()). If the application
 uses such functions, then the application must free up OpenSSL resources
-directly via a call to OPENSSL_thread_stop(). Similarly this message will
-also not be sent if OpenSSL is linked statically, and therefore applications
-using static linking should also call OPENSSL_thread_stop().
+directly via a call to OPENSSL_thread_stop() on each thread. Similarly this
+message will also not be sent if OpenSSL is linked statically, and therefore
+applications using static linking should also call OPENSSL_thread_stop() on each
+thread. Additionally if OpenSSL is loaded dynamically via LoadLibrary() and the
+threads are not destroyed until after FreeLibrary() is called then each thread
+should call OPENSSL_thread_stop() prior to the FreeLibrary() call.
+
+On Linux/Unix where OpenSSL has been loaded via dlopen() and the application is
+multi-threaded and if dlclose() is subsequently called prior to the threads
+being destroyed then OpenSSL will not be able to deallocate resources associated
+with those threads. The application should either call OPENSSL_thread_stop() on
+each thread prior to the dlclose() call, or alternatively the original dlopen()
+call should use the RTLD_NODELETE flag (where available on the platform).
 
 =head1 RETURN VALUES