YJIT: Fix tailcall and JIT entry eating up FINISH frames (#9729)

Suppose YJIT runs a rb_vm_opt_send_without_block()
fallback and the control frame stack looks like:

```
will_tailcall_bar [FINISH]
caller_that_used_fallback
```

will_tailcall_bar() runs in the interpreter and sets up a tailcall.
Right before JIT_EXEC() in the `send` instruction, the stack will look like:

```
bar [FINISH]
caller_that_used_fallback
```

Previously, JIT_EXEC() ran bar() in JIT code, which caused the `FINISH`
flag to return to the interpreter instead of to the JIT code running
caller_that_used_fallback(), causing code to run twice and probably
crash. Recent flaky failures on CI about "each stub expects a particular
iseq" are probably due to leaving methods twice in
`test_optimizations.rb`.

Only run JIT code from the interpreter if a new frame is pushed.

2 files changed, 13 insertions, 1 deletions

diff --git a/test/ruby/test_optimization.rb b/test/ruby/test_optimization.rb
index 8d669e502c..70b6bde6ed 100644
--- a/test/ruby/test_optimization.rb
+++ b/test/ruby/test_optimization.rb
@@ -451,6 +451,17 @@ class TestRubyOptimization < Test::Unit::TestCase
     assert_equal(3, one_plus_two)
   end
 
+  def test_tailcall_and_post_arg
+    tailcall(<<~RUBY)
+      def ret_const = :ok
+
+      def post_arg(_a = 1, _b) = ret_const
+    RUBY
+
+    # YJIT probably uses a fallback on the call to post_arg
+    assert_equal(:ok, post_arg(0))
+  end
+
   def test_tailcall_interrupted_by_sigint
     bug12576 = 'ruby-core:76327'
     script = "#{<<-"begin;"}\n#{<<~'end;'}"
diff --git a/vm_exec.h b/vm_exec.h
index 152410a6a7..b1eeb50660 100644
--- a/vm_exec.h
+++ b/vm_exec.h
@@ -176,7 +176,8 @@ default:                        \
 // Run the JIT from the interpreter
 #define JIT_EXEC(ec, val) do { \
     rb_jit_func_t func; \
-    if (val == Qundef && (func = jit_compile(ec))) { \
+    /* don't run tailcalls since that breaks FINISH */ \
+    if (val == Qundef && GET_CFP() != ec->cfp && (func = jit_compile(ec))) { \
         val = func(ec, ec->cfp); \
         RESTORE_REGS(); /* fix cfp for tailcall */ \
         if (ec->tag->state) THROW_EXCEPTION(val); \