YJIT: Fix large ISeq rejection (#7576)

We crashed in some edge cases due to the recent change to not compile
encoded iseqs that are larger than `u16::MAX`.

- Match the C signature of rb_yjit_constant_ic_update() and clamp down
  to `IseqIdx` size
- Return failure instead of panicking with `unwrap()` in codegen when
  the iseq is too large

Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-authored-by: Noah Gibbs <noah.gibbs@shopify.com>

1 files changed, 13 insertions, 0 deletions

diff --git a/bootstraptest/test_yjit.rb b/bootstraptest/test_yjit.rb
index cab905a2af..64203322c6 100644
--- a/bootstraptest/test_yjit.rb
+++ b/bootstraptest/test_yjit.rb
@@ -3677,3 +3677,16 @@ assert_equal "foo", %q{
 
   literal("foo")
 }
+
+# regression test for accidentally having a parameter truncated
+# due to Rust/C signature mismatch. Used to crash with
+# > [BUG] rb_vm_insn_addr2insn: invalid insn address ...
+# or
+# > ... `Err` value: TryFromIntError(())'
+assert_normal_exit %q{
+  n = 16384
+  eval(
+    "def foo(arg); " + "_=arg;" * n + '_=1;' + "Object; end"
+  )
+  foo 1
+}