~(unsigned char) is not unsigned char

The unary ~ operator excercises integer promotion of the operand
_before_ actually applying bitwise complement (cf: ISO/IEC 9899:1990
section 6.3.3.3).  Which means `~buf[i]` is in fact
`(int)~(int)buf[i]`.

The problem is, when buf[i] is 0xFF:

      buf[i]        0xFF
 (int)buf[i] 0x0000_00FF
~(int)buf[i] 0xFFFF_FF00 This is -256, out of unsigned char range.

The proposed fix is to change the char signed.  By doing so,

                   buf[i]        0xFF
      (signed char)buf[i]        0xFF
 (int)(signed char)buf[i] 0xFFFF_FFFF
~(int)(signed char)buf[i] 0x0000_0000 This is 0, does not overflow.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65675 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

1 files changed, 6 insertions, 2 deletions

diff --git a/bignum.c b/bignum.c
index 8f9f2ce737..868c5dba88 100644
--- a/bignum.c
+++ b/bignum.c
@@ -616,8 +616,12 @@ static int
 bytes_2comp(unsigned char *buf, size_t len)
 {
     size_t i;
-    for (i = 0; i < len; i++)
-        buf[i] = ~buf[i];
+    for (i = 0; i < len; i++) {
+        signed char c = buf[i];
+        signed int d = ~c;
+        unsigned int e = d & 0xFF;
+        buf[i] = e;
+    }
     for (i = 0; i < len; i++) {
         buf[i]++;
         if (buf[i] != 0)