Fix use-after-free with interacting TracePoints

`vm_trace_hook()` runs global hooks before running local hooks.
Previously, we read the local hook list before running the global hooks
which led to use-after-free when a global hook frees the local hook
list. A global hook can do this by disabling a local TracePoint, for
example.

Delay local hook list loading until after running the global hooks.

Issue discovered by Jeremy Evans in GH-5862.

[Bug #18730]

1 files changed, 17 insertions, 0 deletions

diff --git a/test/ruby/test_gc_compact.rb b/test/ruby/test_gc_compact.rb
index da0023e6f3..be27199cdc 100644
--- a/test/ruby/test_gc_compact.rb
+++ b/test/ruby/test_gc_compact.rb
@@ -200,4 +200,21 @@ class TestGCCompact < Test::Unit::TestCase
     GC.compact
     assert_equal count + 1, GC.stat(:compact_count)
   end
+
+  def test_compacting_from_trace_point
+    obj = Object.new
+    def obj.tracee
+      :ret # expected to emit both line and call event from one instruction
+    end
+
+    results = []
+    TracePoint.new(:call, :line) do |tp|
+      results << tp.event
+      GC.verify_compaction_references
+    end.enable(target: obj.method(:tracee)) do
+      obj.tracee
+    end
+
+    assert_equal([:call, :line], results)
+  end
 end