Fix nested bmethod TracePoint and memory leak

df317151a5b4e0c5a30fcc321a9dc6abad63f7ed removed the code to free
rb_hook_list_t, so repeated targeting of the same bmethod started
to leak the hook list. You can observe how the maximum memory use
scales with input size in the following script with `/usr/bin/time -v`.

```ruby
o = Object.new
o.define_singleton_method(:foo) {}
trace = TracePoint.new(:return) {}

bmethod = o.method(:foo)

ARGV.first.to_i.times { trace.enable(target:bmethod){} }
4.times {GC.start}
```

After this change the maximum doesn't grow as quickly.

To plug the leak, check whether the hook list is already allocated
when enabling the targeting TracePoint for the bmethod. This fix
also allows multiple TracePoints to target the same bmethod, similar
to other valid TracePoint targets.

Finally, free the rb_hook_list_t struct when freeing the method
definition it lives on. Freeing in the GC is a good way to avoid
lifetime problems similar to the one fixed in df31715.

[Bug #18031]

1 files changed, 3 insertions, 1 deletions

diff --git a/vm_trace.c b/vm_trace.c
index 99d3104b40..7f65f98695 100644
--- a/vm_trace.c
+++ b/vm_trace.c
@@ -1226,7 +1226,9 @@ rb_tracepoint_enable_for_target(VALUE tpval, VALUE target, VALUE target_line)
         rb_method_definition_t *def = (rb_method_definition_t *)rb_method_def(target);
         if (def->type == VM_METHOD_TYPE_BMETHOD &&
             (tp->events & (RUBY_EVENT_CALL | RUBY_EVENT_RETURN))) {
-            def->body.bmethod.hooks = ZALLOC(rb_hook_list_t);
+            if (def->body.bmethod.hooks == NULL) {
+                def->body.bmethod.hooks = ZALLOC(rb_hook_list_t);
+            }
             rb_hook_list_connect_tracepoint(target, def->body.bmethod.hooks, tpval, 0);
             rb_hash_aset(tp->local_target_set, target, Qfalse);
             target_bmethod = true;