cipher: don't set dummy encryption key in Cipher#initializetopic/cipher-no-initialize-null-key

Remove the encryption key initialization from Cipher#initialize. This
is effectively a revert of r32723 ("Avoid possible SEGV from AES
encryption/decryption", 2011-07-28).

r32723, which added the key initialization, was a workaround for
Ruby Bug #2768. For some certain ciphers, calling EVP_CipherUpdate()
before setting an encryption key caused segfault. It was not a problem
until OpenSSL implemented GCM mode - the encryption key could be
overridden by repeated calls of EVP_CipherInit_ex(). But, it is not the
case for AES-GCM ciphers. Setting a key, an IV, a key, in this order
causes the IV to be reset to an all-zero IV.

The problem of Bug #2768 persists on the current versions of OpenSSL.
So, make Cipher#update raise an exception if a key is not yet set by the
user. Since encrypting or decrypting without key does not make any
sense, this should not break existing applications.

Users can still call Cipher#key= and Cipher#iv= multiple times with
their own responsibility.

Reference: https://bugs.ruby-lang.org/issues/2768
Reference: https://bugs.ruby-lang.org/issues/8221
Reference: https://github.com/ruby/openssl/issues/49

1 files changed, 23 insertions, 6 deletions

diff --git a/test/test_cipher.rb b/test/test_cipher.rb
index 74c5394f..015bb561 100644
--- a/test/test_cipher.rb
+++ b/test/test_cipher.rb
@@ -90,6 +90,7 @@ class OpenSSL::TestCipher < OpenSSL::TestCase
 
   def test_empty_data
     @c1.encrypt
+    @c1.random_key
     assert_raise(ArgumentError){ @c1.update("") }
   end
 
@@ -136,12 +137,10 @@ class OpenSSL::TestCipher < OpenSSL::TestCase
     }
   end
 
-  def test_AES_crush
-    500.times do
-      assert_nothing_raised("[Bug #2768]") do
-        # it caused OpenSSL SEGV by uninitialized key
-        OpenSSL::Cipher::AES128.new("ECB").update "." * 17
-      end
+  def test_update_raise_if_key_not_set
+    assert_raise(OpenSSL::Cipher::CipherError) do
+      # it caused OpenSSL SEGV by uninitialized key [Bug #2768]
+      OpenSSL::Cipher::AES128.new("ECB").update "." * 17
     end
   end
 
@@ -317,6 +316,24 @@ class OpenSSL::TestCipher < OpenSSL::TestCase
     }
   end if has_cipher?("aes-128-ocb")
 
+  def test_aes_gcm_key_iv_order_issue
+    pt = "[ruby/openssl#49]"
+    cipher = OpenSSL::Cipher.new("aes-128-gcm").encrypt
+    cipher.key = "x" * 16
+    cipher.iv = "a" * 12
+    ct1 = cipher.update(pt) << cipher.final
+    tag1 = cipher.auth_tag
+
+    cipher = OpenSSL::Cipher.new("aes-128-gcm").encrypt
+    cipher.iv = "a" * 12
+    cipher.key = "x" * 16
+    ct2 = cipher.update(pt) << cipher.final
+    tag2 = cipher.auth_tag
+
+    assert_equal ct1, ct2
+    assert_equal tag1, tag2
+  end if has_cipher?("aes-128-gcm")
+
   private
 
   def new_encryptor(algo)