diff options
author | Yusuke Endoh <mame@ruby-lang.org> | 2020-02-16 19:41:41 +0900 |
---|---|---|
committer | Yusuke Endoh <mame@ruby-lang.org> | 2020-02-16 19:53:39 +0900 |
commit | 36740fe410c27f8abdcef32ae40d4edef50c52ba (patch) | |
tree | ade567187eebef684f4f9af2b9985529a7f5c6bc /test | |
parent | f49e7110ca1e63f9f013297cb9b155b8b22f2f93 (diff) | |
download | ruby-openssl-36740fe410c27f8abdcef32ae40d4edef50c52ba.tar.gz |
Make OpenSSL::OSSL#test_memcmp_timing robust
The test was too fragile. Actually, it fails on one of our CIs
immediately after it was merged to ruby/ruby.
https://gist.github.com/ko1/7ea4a5826641f79e2f9e041d83e45dba#file-brlog-trunk_clang_40-20200216-101730-L532-L535
https://gist.github.com/ko1/1c657746092b871359d8bf9e0ad28921#file-brlog-trunk-test4-20200216-104518-L473-L476
* Two measurements, a-b and a-c, must be interative instead of
sequential; the execution time will be easily affected by disturbance
(say, cron job or some external process invoked during measurement)
* The comparison of the two results must be relative instead of
absolute; slow machine may take several tens of seconds for each
execution, and one delta second is too small. The test cases of a, b,
and c are very extreme, so if the target method has a bug, the two
execution times would be very different. So I think it is enough to
check if the difference is less than 10 times.
Diffstat (limited to 'test')
-rw-r--r-- | test/test_ossl.rb | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/test/test_ossl.rb b/test/test_ossl.rb index f517b1d8..f83e8136 100644 --- a/test/test_ossl.rb +++ b/test/test_ossl.rb @@ -52,10 +52,13 @@ class OpenSSL::OSSL < OpenSSL::SSLTestCase c = "y#{a}" a = "#{a}x" - n = 10_000 - a_b_time = Benchmark.measure { n.times { OpenSSL.fixed_length_secure_compare(a, b) } }.real - a_c_time = Benchmark.measure { n.times { OpenSSL.fixed_length_secure_compare(a, c) } }.real - assert_in_delta(a_b_time, a_c_time, 1, "fixed_length_secure_compare timing test failed") + a_b_time = a_c_time = 0 + 100.times do + a_b_time += Benchmark.measure { 100.times { OpenSSL.fixed_length_secure_compare(a, b) } }.real + a_c_time += Benchmark.measure { 100.times { OpenSSL.fixed_length_secure_compare(a, c) } }.real + end + assert_operator(a_b_time, :<, a_c_time * 10, "fixed_length_secure_compare timing test failed") + assert_operator(a_c_time, :<, a_b_time * 10, "fixed_length_secure_compare timing test failed") end end |