Add OpenSSL.print_mem_leaks

Add a wrapper method for CRYPTO_mem_leaks_fp(stderr). Calling the method
at the end of programs helps debugging memory leak bugs in Ruby/OpenSSL.
This is defined only when --enable-debug option is given when building
Ruby/OpenSSL, and the OpenSSL version is capable.

The test suite recognizes 'OSSL_MDEBUG' environment variable. Set to '1'
to enable the memory leak checker. This would prevent creating another
memory leak problem at least on the success paths. Note that this may
print some false-positives with OpenSSL <= 1.0.2.

It was once introduced by f0754f0b2f33 ("test: add test/mdebug
extension", 2016-08-06) as a separate native extension, but reverted by
4c1ca7669180 ("Remove test/mdebug", 2016-08-26) because it didn't work
on Windows. Let's re-introduce as part of openssl.so.

1 files changed, 20 insertions, 0 deletions

diff --git a/test/utils.rb b/test/utils.rb
index 70bb5941..54f75f25 100644
--- a/test/utils.rb
+++ b/test/utils.rb
@@ -9,6 +9,26 @@ begin
 rescue LoadError
 end
 
+# Compile OpenSSL with crypto-mdebug and run this test suite with OSSL_MDEBUG=1
+# environment variable to enable memory leak check.
+if ENV["OSSL_MDEBUG"] == "1"
+  if OpenSSL.respond_to?(:print_mem_leaks)
+    OpenSSL.mem_check_start
+
+    END {
+      GC.start
+      case OpenSSL.print_mem_leaks
+      when nil
+        warn "mdebug: check what is printed"
+      when true
+        raise "mdebug: memory leaks detected"
+      end
+    }
+  else
+    warn "OSSL_MDEBUG=1 is specified but OpenSSL is not built with crypto-mdebug"
+  end
+end
+
 require "test/unit"
 require 'tempfile'
 require "rbconfig"