diff options
| author | k0kubun <k0kubun@b2dd03c8-39d4-4d8f-98ff-823fe69b080e> | 2018-02-04 11:22:28 +0000 |
|---|---|---|
| committer | k0kubun <k0kubun@b2dd03c8-39d4-4d8f-98ff-823fe69b080e> | 2018-02-04 11:22:28 +0000 |
| commit | ed935aa5be0e5e6b8d53c3e7d76a9ce395dfa18b (patch) | |
| tree | 04c71c64100ed07478bbaed89228c1323a34f2d0 | |
| parent | 56530b541b2601d2d8bfc4e53796faaba414ae2f (diff) | |
| download | ruby-ed935aa5be0e5e6b8d53c3e7d76a9ce395dfa18b.tar.gz | |
mjit_compile.c: merge initial JIT compiler
which has been developed by Takashi Kokubun <takashikkbn@gmail> as
YARV-MJIT. Many of its bugs are fixed by wanabe <s.wanabe@gmail.com>.
This JIT compiler is designed to be a safe migration path to introduce
JIT compiler to MRI. So this commit does not include any bytecode
changes or dynamic instruction modifications, which are done in original
MJIT.
This commit even strips off some aggressive optimizations from
YARV-MJIT, and thus it's slower than YARV-MJIT too. But it's still
fairly faster than Ruby 2.5 in some benchmarks (attached below).
Note that this JIT compiler passes `make test`, `make test-all`, `make
test-spec` without JIT, and even with JIT. Not only it's perfectly safe
with JIT disabled because it does not replace VM instructions unlike
MJIT, but also with JIT enabled it stably runs Ruby applications
including Rails applications.
I'm expecting this version as just "initial" JIT compiler. I have many
optimization ideas which are skipped for initial merging, and you may
easily replace this JIT compiler with a faster one by just replacing
mjit_compile.c. `mjit_compile` interface is designed for the purpose.
common.mk: update dependencies for mjit_compile.c.
internal.h: declare `rb_vm_insn_addr2insn` for MJIT.
vm.c: exclude some definitions if `-DMJIT_HEADER` is provided to
compiler. This avoids to include some functions which take a long time
to compile, e.g. vm_exec_core. Some of the purpose is achieved in
transform_mjit_header.rb (see `IGNORED_FUNCTIONS`) but others are
manually resolved for now. Load mjit_helper.h for MJIT header.
mjit_helper.h: New. This is a file used only by JIT-ed code. I'll
refactor `mjit_call_cfunc` later.
vm_eval.c: add some #ifdef switches to skip compiling some functions
like Init_vm_eval.
win32/mkexports.rb: export thread/ec functions, which are used by MJIT.
include/ruby/defines.h: add MJIT_FUNC_EXPORTED macro alis to clarify
that a function is exported only for MJIT.
array.c: export a function used by MJIT.
bignum.c: ditto.
class.c: ditto.
compile.c: ditto.
error.c: ditto.
gc.c: ditto.
hash.c: ditto.
iseq.c: ditto.
numeric.c: ditto.
object.c: ditto.
proc.c: ditto.
re.c: ditto.
st.c: ditto.
string.c: ditto.
thread.c: ditto.
variable.c: ditto.
vm_backtrace.c: ditto.
vm_insnhelper.c: ditto.
vm_method.c: ditto.
I would like to improve maintainability of function exports, but I
believe this way is acceptable as initial merging if we clarify the
new exports are for MJIT (so that we can use them as TODO list to fix)
and add unit tests to detect unresolved symbols.
I'll add unit tests of JIT compilations in succeeding commits.
Author: Takashi Kokubun <takashikkbn@gmail.com>
Contributor: wanabe <s.wanabe@gmail.com>
Part of [Feature #14235]
---
* Known issues
* Code generated by gcc is faster than clang. The benchmark may be worse
in macOS. Following benchmark result is provided by gcc w/ Linux.
* Performance is decreased when Google Chrome is running
* JIT can work on MinGW, but it doesn't improve performance at least
in short running benchmark.
* Currently it doesn't perform well with Rails. We'll try to fix this
before release.
---
* Benchmark reslts
Benchmarked with:
Intel 4.0GHz i7-4790K with 16GB memory under x86-64 Ubuntu 8 Cores
- 2.0.0-p0: Ruby 2.0.0-p0
- r62186: Ruby trunk (early 2.6.0), before MJIT changes
- JIT off: On this commit, but without `--jit` option
- JIT on: On this commit, and with `--jit` option
** Optcarrot fps
Benchmark: https://github.com/mame/optcarrot
| |2.0.0-p0 |r62186 |JIT off |JIT on |
|:--------|:--------|:--------|:--------|:--------|
|fps |37.32 |51.46 |51.31 |58.88 |
|vs 2.0.0 |1.00x |1.38x |1.37x |1.58x |
** MJIT benchmarks
Benchmark: https://github.com/benchmark-driver/mjit-benchmarks
(Original: https://github.com/vnmakarov/ruby/tree/rtl_mjit_branch/MJIT-benchmarks)
| |2.0.0-p0 |r62186 |JIT off |JIT on |
|:----------|:--------|:--------|:--------|:--------|
|aread |1.00 |1.09 |1.07 |2.19 |
|aref |1.00 |1.13 |1.11 |2.22 |
|aset |1.00 |1.50 |1.45 |2.64 |
|awrite |1.00 |1.17 |1.13 |2.20 |
|call |1.00 |1.29 |1.26 |2.02 |
|const2 |1.00 |1.10 |1.10 |2.19 |
|const |1.00 |1.11 |1.10 |2.19 |
|fannk |1.00 |1.04 |1.02 |1.00 |
|fib |1.00 |1.32 |1.31 |1.84 |
|ivread |1.00 |1.13 |1.12 |2.43 |
|ivwrite |1.00 |1.23 |1.21 |2.40 |
|mandelbrot |1.00 |1.13 |1.16 |1.28 |
|meteor |1.00 |2.97 |2.92 |3.17 |
|nbody |1.00 |1.17 |1.15 |1.49 |
|nest-ntimes|1.00 |1.22 |1.20 |1.39 |
|nest-while |1.00 |1.10 |1.10 |1.37 |
|norm |1.00 |1.18 |1.16 |1.24 |
|nsvb |1.00 |1.16 |1.16 |1.17 |
|red-black |1.00 |1.02 |0.99 |1.12 |
|sieve |1.00 |1.30 |1.28 |1.62 |
|trees |1.00 |1.14 |1.13 |1.19 |
|while |1.00 |1.12 |1.11 |2.41 |
** Discourse's script/bench.rb
Benchmark: https://github.com/discourse/discourse/blob/v1.8.7/script/bench.rb
NOTE: Rails performance was somehow a little degraded with JIT for now.
We should fix this.
(At least I know opt_aref is performing badly in JIT and I have an idea
to fix it. Please wait for the fix.)
*** JIT off
Your Results: (note for timings- percentile is first, duration is second in millisecs)
categories_admin:
50: 17
75: 18
90: 22
99: 29
home_admin:
50: 21
75: 21
90: 27
99: 40
topic_admin:
50: 17
75: 18
90: 22
99: 32
categories:
50: 35
75: 41
90: 43
99: 77
home:
50: 39
75: 46
90: 49
99: 95
topic:
50: 46
75: 52
90: 56
99: 101
*** JIT on
Your Results: (note for timings- percentile is first, duration is second in millisecs)
categories_admin:
50: 19
75: 21
90: 25
99: 33
home_admin:
50: 24
75: 26
90: 30
99: 35
topic_admin:
50: 19
75: 20
90: 25
99: 30
categories:
50: 40
75: 44
90: 48
99: 76
home:
50: 42
75: 48
90: 51
99: 89
topic:
50: 49
75: 55
90: 58
99: 99
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62197 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
50 files changed, 680 insertions, 101 deletions
diff --git a/Makefile.in b/Makefile.in index 128dffa597..e75cfe54e3 100644 --- a/Makefile.in +++ b/Makefile.in @@ -516,7 +516,7 @@ update-simplecov: update-coverage: update-simplecov update-simplecov-html update-doclie INSNS = opt_sc.inc optinsn.inc optunifs.inc insns.inc insns_info.inc \ - vmtc.inc vm.inc + vmtc.inc vm.inc mjit_compile.inc $(INSNS): $(srcdir)/insns.def vm_opts.h \ $(srcdir)/defs/opt_operand.def $(srcdir)/defs/opt_insn_unif.def \ @@ -506,7 +506,7 @@ VALUE return ary; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_ary_tmp_new_from_values(VALUE klass, long n, const VALUE *elts) { VALUE ary; @@ -640,7 +640,7 @@ rb_check_array_type(VALUE ary) return rb_check_convert_type_with_id(ary, T_ARRAY, "Array", idTo_ary); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_check_to_array(VALUE ary) { return rb_check_convert_type_with_id(ary, T_ARRAY, "Array", idTo_a); @@ -1297,7 +1297,7 @@ rb_ary_aref2(VALUE ary, VALUE b, VALUE e) return rb_ary_subseq(ary, beg, len); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_ary_aref1(VALUE ary, VALUE arg) { long beg, len; @@ -4490,7 +4490,7 @@ rb_uint128t2big(uint128_t n) return big; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_int128t2big(int128_t n) { int neg = 0; @@ -616,7 +616,7 @@ rb_define_class_id(ID id, VALUE super) * \return the value \c Class#inherited's returns * \pre Each of \a super and \a klass must be a \c Class object. */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_class_inherited(VALUE super, VALUE klass) { ID inherited; @@ -1773,7 +1773,7 @@ rb_define_attr(VALUE klass, const char *name, int read, int write) rb_attr(klass, rb_intern(name), read, write, FALSE); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_keyword_error_new(const char *error, VALUE keys) { const VALUE *ptr = RARRAY_CONST_PTR(keys); @@ -890,6 +890,7 @@ $(srcs_vpath)insns.inc: $(srcdir)/tool/ruby_vm/views/insns.inc.erb $(srcs_vpath)insns_info.inc: $(srcdir)/tool/ruby_vm/views/insns_info.inc.erb $(srcs_vpath)vmtc.inc: $(srcdir)/tool/ruby_vm/views/vmtc.inc.erb $(srcs_vpath)vm.inc: $(srcdir)/tool/ruby_vm/views/vm.inc.erb +$(srcs_vpath)mjit_compile.inc: $(srcdir)/tool/ruby_vm/views/mjit_compile.inc.erb $(srcdir)/tool/ruby_vm/views/_mjit_compile_insn.erb $(srcdir)/tool/ruby_vm/views/_mjit_compile_send.erb common-srcs: $(srcs_vpath)parse.c $(srcs_vpath)lex.c $(srcs_vpath)enc/trans/newline.c $(srcs_vpath)id.c \ srcs-lib srcs-ext incs @@ -2003,8 +2004,12 @@ mjit.$(OBJEXT): {$(VPATH)}mjit.h mjit.$(OBJEXT): {$(VPATH)}ruby_assert.h mjit.$(OBJEXT): {$(VPATH)}version.h mjit.$(OBJEXT): {$(VPATH)}vm_core.h +mjit_compile.$(OBJEXT): {$(VPATH)}insns.inc +mjit_compile.$(OBJEXT): {$(VPATH)}insns_info.inc mjit_compile.$(OBJEXT): {$(VPATH)}internal.h +mjit_compile.$(OBJEXT): {$(VPATH)}mjit.h mjit_compile.$(OBJEXT): {$(VPATH)}mjit_compile.c +mjit_compile.$(OBJEXT): {$(VPATH)}mjit_compile.inc mjit_compile.$(OBJEXT): {$(VPATH)}vm_core.h load.$(OBJEXT): $(CCAN_DIR)/check_type/check_type.h load.$(OBJEXT): $(CCAN_DIR)/container_of/container_of.h @@ -754,7 +754,7 @@ rb_iseq_translate_threaded_code(rb_iseq_t *iseq) } #if OPT_DIRECT_THREADED_CODE || OPT_CALL_THREADED_CODE -static int +int rb_vm_insn_addr2insn(const void *addr) /* cold path */ { int insn; @@ -1161,7 +1161,7 @@ exc_set_backtrace(VALUE exc, VALUE bt) return rb_ivar_set(exc, id_bt, rb_check_backtrace(bt)); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_exc_set_backtrace(VALUE exc, VALUE bt) { return exc_set_backtrace(exc, bt); @@ -2216,6 +2216,7 @@ obj_free(rb_objspace_t *objspace, VALUE obj) break; case T_MODULE: case T_CLASS: + mjit_remove_class_serial(RCLASS_SERIAL(obj)); rb_id_table_free(RCLASS_M_TBL(obj)); if (RCLASS_IV_TBL(obj)) { st_free_table(RCLASS_IV_TBL(obj)); @@ -4054,7 +4055,7 @@ stack_check(rb_execution_context_t *ec, int water_mark) #define STACKFRAME_FOR_CALL_CFUNC 838 -int +MJIT_FUNC_EXPORTED int rb_ec_stack_check(rb_execution_context_t *ec) { return stack_check(ec, STACKFRAME_FOR_CALL_CFUNC); @@ -6053,7 +6054,7 @@ rb_gc_writebarrier_unprotect(VALUE obj) /* * remember `obj' if needed. */ -void +MJIT_FUNC_EXPORTED void rb_gc_writebarrier_remember(VALUE obj) { rb_objspace_t *objspace = &rb_objspace; @@ -443,7 +443,7 @@ rb_hash_new_compare_by_id(void) return hash; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_hash_new_with_size(st_index_t size) { VALUE ret = rb_hash_new(); @@ -495,7 +495,7 @@ rb_hash_tbl(VALUE hash) return hash_tbl(hash); } -struct st_table * +MJIT_FUNC_EXPORTED struct st_table * rb_hash_tbl_raw(VALUE hash) { return hash_tbl(hash); @@ -2155,7 +2155,7 @@ keys_i(VALUE key, VALUE value, VALUE ary) * */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_hash_keys(VALUE hash) { VALUE keys; @@ -2243,7 +2243,7 @@ rb_hash_values(VALUE hash) * See also Enumerable#include? */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_hash_has_key(VALUE hash, VALUE key) { if (!RHASH(hash)->ntbl) @@ -2949,7 +2949,7 @@ rb_hash_compare_by_id(VALUE hash) * */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_hash_compare_by_id_p(VALUE hash) { if (!RHASH(hash)->ntbl) diff --git a/include/ruby/defines.h b/include/ruby/defines.h index 5006305059..b49fabe636 100644 --- a/include/ruby/defines.h +++ b/include/ruby/defines.h @@ -270,6 +270,10 @@ void xfree(void*); #define RUBY_FUNC_EXPORTED #endif +/* MJIT_FUNC_EXPORTED is used for functions which are exported only for MJIT + and NOT ensured to be exported in future versions. */ +#define MJIT_FUNC_EXPORTED RUBY_FUNC_EXPORTED + #ifndef RUBY_EXTERN #define RUBY_EXTERN extern #endif @@ -695,8 +695,7 @@ defineclass class_iseq->body->iseq_encoded, GET_SP(), class_iseq->body->local_table_size, class_iseq->body->stack_max); - RESTORE_REGS(); - NEXT_INSN(); + EXEC_EC_CFP(); } /**********************************************************/ @@ -823,8 +822,7 @@ invokeblock val = vm_invoke_block(ec, GET_CFP(), &calling, ci, block_handler); if (val == Qundef) { - RESTORE_REGS(); - NEXT_INSN(); + EXEC_EC_CFP(); } } @@ -1090,7 +1088,9 @@ opt_neq val = vm_opt_neq(ci, cc, ci_eq, cc_eq, recv, obj); if (val == Qundef) { +#ifndef MJIT_HEADER ADD_PC(2); /* !!! */ +#endif DISPATCH_ORIGINAL_INSN(opt_send_without_block); } } @@ -1206,9 +1206,11 @@ opt_aset_with val = tmp; } else { +#ifndef MJIT_HEADER TOPN(0) = rb_str_resurrect(key); PUSH(val); ADD_PC(1); /* !!! */ +#endif DISPATCH_ORIGINAL_INSN(opt_send_without_block); } } @@ -1223,8 +1225,10 @@ opt_aref_with val = vm_opt_aref_with(recv, key); if (val == Qundef) { +#ifndef MJIT_HEADER PUSH(rb_str_resurrect(key)); ADD_PC(1); /* !!! */ +#endif DISPATCH_ORIGINAL_INSN(opt_send_without_block); } } @@ -1339,8 +1343,7 @@ opt_call_c_function THROW_EXCEPTION(err); } - RESTORE_REGS(); - NEXT_INSN(); + EXEC_EC_CFP(); } /* BLT */ diff --git a/internal.h b/internal.h index 8b1ee228cc..2c65ebcbfc 100644 --- a/internal.h +++ b/internal.h @@ -1124,6 +1124,7 @@ int rb_dvar_defined(ID, const struct rb_block *); int rb_local_defined(ID, const struct rb_block *); const char * rb_insns_name(int i); VALUE rb_insns_name_array(void); +int rb_vm_insn_addr2insn(const void *); /* complex.c */ VALUE rb_complex_plus(VALUE, VALUE); @@ -1480,7 +1480,7 @@ rb_iseq_line_no(const rb_iseq_t *iseq, size_t pos) } } -rb_event_flag_t +MJIT_FUNC_EXPORTED rb_event_flag_t rb_iseq_event_flags(const rb_iseq_t *iseq, size_t pos) { const struct iseq_insn_info_entry *entry = get_insn_info(iseq, pos); @@ -83,6 +83,8 @@ #include "mjit.h" #include "version.h" #include "gc.h" +#include "constant.h" +#include "id_table.h" #include "ruby_assert.h" extern void rb_native_mutex_lock(rb_nativethread_lock_t *lock); @@ -194,6 +196,9 @@ static char *header_file; static char *pch_file; /* Path of "/tmp", which can be changed to $TMP in MinGW. */ static char *tmp_dir; +/* Hash like { 1 => true, 2 => true, ... } whose keys are valid `class_serial`s. + This is used to invalidate obsoleted CALL_CACHE. */ +static VALUE valid_class_serials; /* Ruby level interface module. */ VALUE rb_mMJIT; @@ -1081,6 +1086,19 @@ child_after_fork(void) /* TODO: Should we initiate MJIT in the forked Ruby. */ } +static enum rb_id_table_iterator_result +valid_class_serials_add_i(ID key, VALUE v, void *unused) +{ + rb_const_entry_t *ce = (rb_const_entry_t *)v; + VALUE value = ce->value; + + if (!rb_is_const_id(key)) return ID_TABLE_CONTINUE; + if (RB_TYPE_P(value, T_MODULE) || RB_TYPE_P(value, T_CLASS)) { + mjit_add_class_serial(RCLASS_SERIAL(value)); + } + return ID_TABLE_CONTINUE; +} + /* Default permitted number of units with a JIT code kept in memory. */ #define DEFAULT_CACHE_SIZE 1000 @@ -1149,6 +1167,14 @@ mjit_init(struct mjit_options *opts) rb_native_cond_initialize(&mjit_worker_wakeup, RB_CONDATTR_CLOCK_MONOTONIC); rb_native_cond_initialize(&mjit_gc_wakeup, RB_CONDATTR_CLOCK_MONOTONIC); + /* Initialize class_serials cache for compilation */ + valid_class_serials = rb_hash_new(); + rb_obj_hide(valid_class_serials); + rb_gc_register_mark_object(valid_class_serials); + if (RCLASS_CONST_TBL(rb_cObject)) { + rb_id_table_foreach(RCLASS_CONST_TBL(rb_cObject), valid_class_serials_add_i, NULL); + } + /* Initialize worker thread */ finish_worker_p = FALSE; worker_finished = FALSE; @@ -1233,3 +1259,39 @@ mjit_mark(void) CRITICAL_SECTION_FINISH(4, "mjit_mark"); RUBY_MARK_LEAVE("mjit"); } + +/* A hook to update valid_class_serials. This should NOT be used in MJIT worker. */ +void +mjit_add_class_serial(rb_serial_t class_serial) +{ + if (!mjit_init_p) + return; + + CRITICAL_SECTION_START(3, "in mjit_add_class_serial"); + rb_hash_aset(valid_class_serials, LONG2FIX(class_serial), Qtrue); + CRITICAL_SECTION_FINISH(3, "in mjit_add_class_serial"); +} + +/* A hook to update valid_class_serials. This should NOT be used in MJIT worker. */ +void +mjit_remove_class_serial(rb_serial_t class_serial) +{ + if (!mjit_init_p) + return; + + CRITICAL_SECTION_START(3, "in mjit_remove_class_serial"); + rb_hash_delete_entry(valid_class_serials, LONG2FIX(class_serial)); + CRITICAL_SECTION_FINISH(3, "in mjit_remove_class_serial"); +} + +/* Return TRUE if class_serial is not obsoleted. This can be used in MJIT worker. */ +int +mjit_valid_class_serial_p(rb_serial_t class_serial) +{ + int found_p; + + CRITICAL_SECTION_START(3, "in valid_class_serial_p"); + found_p = st_lookup(RHASH_TBL_RAW(valid_class_serials), LONG2FIX(class_serial), NULL); + CRITICAL_SECTION_FINISH(3, "in valid_class_serial_p"); + return found_p; +} @@ -80,6 +80,9 @@ extern void mjit_free_iseq(const rb_iseq_t *iseq); extern void mjit_mark(void); extern struct mjit_cont *mjit_cont_new(rb_execution_context_t *ec); extern void mjit_cont_free(struct mjit_cont *cont); +extern void mjit_add_class_serial(rb_serial_t class_serial); +extern void mjit_remove_class_serial(rb_serial_t class_serial); +extern int mjit_valid_class_serial_p(rb_serial_t class_serial); /* A threshold used to reject long iseqs from JITting as such iseqs takes too much time to be compiled. */ diff --git a/mjit_compile.c b/mjit_compile.c index a48ae84e4c..b323192fe9 100644 --- a/mjit_compile.c +++ b/mjit_compile.c @@ -8,11 +8,163 @@ #include "internal.h" #include "vm_core.h" +#include "vm_exec.h" +#include "mjit.h" +#include "insns.inc" +#include "insns_info.inc" +#include "vm_insnhelper.h" -/* Compile ISeq to C code in F. Return TRUE if it succeeds to compile. */ +/* Storage to keep compiler's status. This should have information + which is global during one `mjit_compile` call. Ones conditional + in each branch should be stored in `compile_branch`. */ +struct compile_status { + int success; /* has TRUE if compilation has had no issue */ + int *compiled_for_pos; /* compiled_for_pos[pos] has TRUE if the pos is compiled */ +}; + +/* Storage to keep data which is consistent in each conditional branch. + This is created and used for one `compile_insns` call and its values + should be copied for extra `compile_insns` call. */ +struct compile_branch { + unsigned int stack_size; /* this simulates sp (stack pointer) of YARV */ + int finish_p; /* if TRUE, compilation in this branch should stop and let another branch to be compiled */ +}; + +struct case_dispatch_var { + FILE *f; + unsigned int base_pos; + VALUE last_value; +}; + +/* Returns iseq from cc if it's available and still not obsoleted. */ +static const rb_iseq_t * +get_iseq_if_available(CALL_CACHE cc) +{ + if (GET_GLOBAL_METHOD_STATE() == cc->method_state + && mjit_valid_class_serial_p(cc->class_serial) + && cc->me && cc->me->def->type == VM_METHOD_TYPE_ISEQ) { + return rb_iseq_check(cc->me->def->body.iseq.iseqptr); + } + return NULL; +} + +/* TODO: move to somewhere shared with vm_args.c */ +#define IS_ARGS_SPLAT(ci) ((ci)->flag & VM_CALL_ARGS_SPLAT) +#define IS_ARGS_KEYWORD(ci) ((ci)->flag & VM_CALL_KWARG) + +/* Returns TRUE if iseq is inlinable, otherwise NULL. This becomes TRUE in the same condition + as CI_SET_FASTPATH (in vm_callee_setup_arg) is called from vm_call_iseq_setup. */ +static int +inlinable_iseq_p(CALL_INFO ci, CALL_CACHE cc, const rb_iseq_t *iseq) +{ + extern int simple_iseq_p(const rb_iseq_t *iseq); + return iseq != NULL + && simple_iseq_p(iseq) && !(ci->flag & VM_CALL_KW_SPLAT) /* top of vm_callee_setup_arg */ + && (!IS_ARGS_SPLAT(ci) && !IS_ARGS_KEYWORD(ci) && !(METHOD_ENTRY_VISI(cc->me) == METHOD_VISI_PROTECTED)); /* CI_SET_FASTPATH */ +} + +static int +compile_case_dispatch_each(VALUE key, VALUE value, VALUE arg) +{ + struct case_dispatch_var *var = (struct case_dispatch_var *)arg; + unsigned int offset; + + if (var->last_value != value) { + offset = FIX2INT(value); + var->last_value = value; + fprintf(var->f, " case %d:\n", offset); + fprintf(var->f, " goto label_%d;\n", var->base_pos + offset); + fprintf(var->f, " break;\n"); + } + return ST_CONTINUE; +} + +static void compile_insns(FILE *f, const struct rb_iseq_constant_body *body, unsigned int stack_size, + unsigned int pos, struct compile_status *status); + +/* Main function of JIT compilation, vm_exec_core counterpart for JIT. Compile one insn to `f`, may modify + b->stack_size and return next position. + + When you add a new instruction to insns.def, it would be nice to have JIT compilation support here but + it's optional. This JIT compiler just ignores ISeq which includes unknown instruction, and ISeq which + does not have it can be compiled as usual. */ +static unsigned int +compile_insn(FILE *f, const struct rb_iseq_constant_body *body, const int insn, const VALUE *operands, + const unsigned int pos, struct compile_status *status, struct compile_branch *b) +{ + unsigned int next_pos = pos + insn_len(insn); + +/*****************/ + #include "mjit_compile.inc" +/*****************/ + + return next_pos; +} + +/* Compile one conditional branch. If it has branchXXX insn, this should be + called multiple times for each branch. */ +static void +compile_insns(FILE *f, const struct rb_iseq_constant_body *body, unsigned int stack_size, + unsigned int pos, struct compile_status *status) +{ + int insn; + struct compile_branch branch; + + branch.stack_size = stack_size; + branch.finish_p = FALSE; + + while (pos < body->iseq_size && !status->compiled_for_pos[pos] && !branch.finish_p) { +#if OPT_DIRECT_THREADED_CODE || OPT_CALL_THREADED_CODE + insn = rb_vm_insn_addr2insn((void *)body->iseq_encoded[pos]); +#else + insn = (int)body->iseq_encoded[pos]; +#endif + status->compiled_for_pos[pos] = TRUE; + + fprintf(f, "\nlabel_%d: /* %s */\n", pos, insn_name(insn)); + pos = compile_insn(f, body, insn, body->iseq_encoded + (pos+1), pos, status, &branch); + if (status->success && branch.stack_size > body->stack_max) { + if (mjit_opts.warnings || mjit_opts.verbose) + fprintf(stderr, "MJIT warning: JIT stack exceeded its max\n"); + status->success = FALSE; + } + if (!status->success) + break; + } +} + +/* Compile ISeq to C code in F. It returns 1 if it succeeds to compile. */ int mjit_compile(FILE *f, const struct rb_iseq_constant_body *body, const char *funcname) { - /* TODO: Write your own JIT compiler here. */ - return FALSE; + struct compile_status status; + status.success = TRUE; + status.compiled_for_pos = ZALLOC_N(int, body->iseq_size); + + fprintf(f, "VALUE %s(rb_execution_context_t *ec, rb_control_frame_t *reg_cfp) {\n", funcname); + fprintf(f, " VALUE *stack = reg_cfp->sp;\n"); + + /* Simulate `opt_pc` in setup_parameters_complex */ + if (body->param.flags.has_opt) { + int i; + fprintf(f, "\n"); + fprintf(f, " switch (reg_cfp->pc - reg_cfp->iseq->body->iseq_encoded) {\n"); + for (i = 0; i <= body->param.opt_num; i++) { + VALUE pc_offset = body->param.opt_table[i]; + fprintf(f, " case %"PRIdVALUE":\n", pc_offset); + fprintf(f, " goto label_%"PRIdVALUE";\n", pc_offset); + } + fprintf(f, " }\n"); + } + + /* ISeq might be used for catch table too. For that usage, this code cancels JIT execution. */ + fprintf(f, " if (reg_cfp->pc != 0x%"PRIxVALUE") {\n", (VALUE)body->iseq_encoded); + fprintf(f, " return Qundef;\n"); + fprintf(f, " }\n"); + + compile_insns(f, body, 0, 0, &status); + fprintf(f, "}\n"); + + xfree(status.compiled_for_pos); + return status.success; } @@ -1157,7 +1157,7 @@ flodivmod(double x, double y, double *divp, double *modp) * An error will be raised if y == 0. */ -double +MJIT_FUNC_EXPORTED double ruby_float_mod(double x, double y) { double mod; @@ -1333,7 +1333,7 @@ num_equal(VALUE x, VALUE y) * so an implementation-dependent value is returned. */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_float_equal(VALUE x, VALUE y) { volatile double a, b; @@ -1436,7 +1436,7 @@ flo_cmp(VALUE x, VALUE y) return rb_dbl_cmp(a, b); } -int +MJIT_FUNC_EXPORTED int rb_float_cmp(VALUE x, VALUE y) { return NUM2INT(flo_cmp(x, y)); @@ -198,7 +198,7 @@ rb_eql(VALUE obj1, VALUE obj2) * \private *++ */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_obj_equal(VALUE obj1, VALUE obj2) { if (obj1 == obj2) return Qtrue; @@ -217,7 +217,7 @@ VALUE rb_obj_hash(VALUE obj); *++ */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_obj_not(VALUE obj) { return RTEST(obj) ? Qfalse : Qtrue; @@ -233,7 +233,7 @@ rb_obj_not(VALUE obj) *++ */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_obj_not_equal(VALUE obj1, VALUE obj2) { VALUE result = rb_funcall(obj1, id_eq, 1, obj2); @@ -304,7 +304,7 @@ rb_obj_singleton_class(VALUE obj) } /*! \private */ -void +MJIT_FUNC_EXPORTED void rb_obj_copy_ivar(VALUE dest, VALUE obj) { if (!(RBASIC(dest)->flags & ROBJECT_EMBED) && ROBJECT_IVPTR(dest)) { @@ -3019,7 +3019,7 @@ rb_check_convert_type(VALUE val, int type, const char *tname, const char *method } /*! \private */ -VALUE +MJIT_FUNC_EXPORTED VALUE rb_check_convert_type_with_id(VALUE val, int type, const char *tname, ID method) { VALUE v; @@ -677,7 +677,7 @@ rb_vm_ifunc_new(VALUE (*func)(ANYARGS), const void *data, int min_argc, int max_ return IFUNC_NEW(func, data, arity.packed); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_func_proc_new(rb_block_call_func_t func, VALUE val) { struct vm_ifunc *ifunc = rb_vm_ifunc_proc_new(func, (void *)val); @@ -1206,7 +1206,7 @@ rb_hash_proc(st_index_t hash, VALUE prc) return rb_hash_uint(hash, (st_index_t)proc->block.as.captured.ep >> 16); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_sym_to_proc(VALUE sym) { static VALUE sym_proc_cache = Qfalse; @@ -2891,7 +2891,7 @@ rb_reg_init_str_enc(VALUE re, VALUE s, rb_encoding *enc, int options) return re; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_reg_new_ary(VALUE ary, int opt) { return rb_reg_new_str(rb_reg_preprocess_dregexp(ary, opt), opt); @@ -2171,7 +2171,7 @@ st_insert_generic(st_table *tab, long argc, const VALUE *argv, VALUE hash) /* Mimics ruby's { foo => bar } syntax. This function is placed here because it touches table internals and write barriers at once. */ -void +MJIT_FUNC_EXPORTED void rb_hash_bulk_insert(long argc, const VALUE *argv, VALUE hash) { st_index_t n; @@ -373,7 +373,7 @@ rb_setup_fake_str(struct RString *fake_str, const char *name, long len, rb_encod return setup_fake_str(fake_str, name, len, rb_enc_to_index(enc)); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_fstring_new(const char *ptr, long len) { struct RString fake_str; @@ -1445,7 +1445,7 @@ rb_obj_as_string(VALUE obj) return rb_obj_as_string_result(str, obj); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_obj_as_string_result(VALUE str, VALUE obj) { if (!RB_TYPE_P(str, T_STRING)) @@ -2933,7 +2933,7 @@ rb_str_append(VALUE str, VALUE str2) #define MIN_PRE_ALLOC_SIZE 48 -VALUE +MJIT_FUNC_EXPORTED VALUE rb_str_concat_literals(size_t num, const VALUE *strary) { VALUE str; @@ -10391,7 +10391,7 @@ rb_str_quote_unprintable(VALUE str) return str; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_id_quote_unprintable(ID id) { return rb_str_quote_unprintable(rb_id2str(id)); @@ -10468,7 +10468,7 @@ sym_to_sym(VALUE sym) return sym; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_sym_proc_call(ID mid, int argc, const VALUE *argv, VALUE passed_proc) { VALUE obj; diff --git a/test/-ext-/thread_fd_close/test_thread_fd_close.rb b/test/-ext-/thread_fd_close/test_thread_fd_close.rb index e6c3895da7..90993cf63d 100644 --- a/test/-ext-/thread_fd_close/test_thread_fd_close.rb +++ b/test/-ext-/thread_fd_close/test_thread_fd_close.rb @@ -6,6 +6,8 @@ require 'io/wait' class TestThreadFdClose < Test::Unit::TestCase def test_thread_fd_close + skip "MJIT thread is unexpected for this" if RubyVM::MJIT.enabled? + IO.pipe do |r, w| th = Thread.new do begin diff --git a/test/lib/minitest/unit.rb b/test/lib/minitest/unit.rb index 89b6a8eaf4..aa9a09f049 100644 --- a/test/lib/minitest/unit.rb +++ b/test/lib/minitest/unit.rb @@ -956,7 +956,9 @@ module MiniTest puts if @verbose $stdout.flush - leakchecker.check("#{inst.class}\##{inst.__name__}") + unless RubyVM::MJIT.enabled? # compiler process is wrongly considered as leaked + leakchecker.check("#{inst.class}\##{inst.__name__}") + end inst._assertions } diff --git a/test/ruby/test_io.rb b/test/ruby/test_io.rb index 2f0a8a7025..f1a9453ae9 100644 --- a/test/ruby/test_io.rb +++ b/test/ruby/test_io.rb @@ -2135,6 +2135,12 @@ class TestIO < Test::Unit::TestCase end def test_autoclose_true_closed_by_finalizer + if RubyVM::MJIT.enabled? + # This is skipped but this test passes with AOT mode. + # At least it should not be a JIT compiler's bug. + skip "MJIT worker does IO which is unexpected for this test" + end + feature2250 = '[ruby-core:26222]' pre = 'ft2250' t = Tempfile.new(pre) @@ -2150,7 +2156,7 @@ class TestIO < Test::Unit::TestCase assert_raise(Errno::EBADF, feature2250) {t.close} end ensure - t.close! + t&.close! end def test_autoclose_false_closed_by_finalizer diff --git a/test/ruby/test_settracefunc.rb b/test/ruby/test_settracefunc.rb index 69ddc05979..bff3b8c75b 100644 --- a/test/ruby/test_settracefunc.rb +++ b/test/ruby/test_settracefunc.rb @@ -1819,6 +1819,9 @@ class TestSetTraceFunc < Test::Unit::TestCase } # it is dirty hack. usually we shouldn't use such technique Thread.pass until t.status == 'sleep' + # When MJIT thread exists, t.status becomes 'sleep' even if it does not reach m2t_q.pop. + # This sleep forces it to reach m2t_q.pop for --jit-wait. + sleep 1 if RubyVM::MJIT.enabled? t.add_trace_func proc{|ev, file, line, *args| if file == __FILE__ diff --git a/test/ruby/test_thread.rb b/test/ruby/test_thread.rb index 6171c3a19e..2d84ee3a6d 100644 --- a/test/ruby/test_thread.rb +++ b/test/ruby/test_thread.rb @@ -280,6 +280,7 @@ class TestThread < Test::Unit::TestCase s += 1 end Thread.pass until t.stop? + sleep 1 if RubyVM::MJIT.enabled? # t.stop? behaves unexpectedly with --jit-wait assert_equal(1, s) t.wakeup Thread.pass while t.alive? diff --git a/test/rubygems/test_gem_util.rb b/test/rubygems/test_gem_util.rb index b85db44d51..dc85d25586 100644 --- a/test/rubygems/test_gem_util.rb +++ b/test/rubygems/test_gem_util.rb @@ -5,6 +5,7 @@ require 'rubygems/util' class TestGemUtil < Gem::TestCase def test_class_popen + skip "MJIT executes process and it's caught by Process.wait(-1)" if RubyVM::MJIT.enabled? assert_equal "0\n", Gem::Util.popen(Gem.ruby, '-e', 'p 0') assert_raises Errno::ECHILD do diff --git a/test/thread/test_cv.rb b/test/thread/test_cv.rb index 1e15d2e9ec..70cf4483a3 100644 --- a/test/thread/test_cv.rb +++ b/test/thread/test_cv.rb @@ -33,6 +33,8 @@ class TestConditionVariable < Test::Unit::TestCase end def test_condvar_wait_exception_handling + skip "MJIT thread is unexpected for this test, especially with --jit-wait" if RubyVM::MJIT.enabled? + # Calling wait in the only thread running should raise a ThreadError of # 'stopping only thread' mutex = Mutex.new diff --git a/test/thread/test_sync.rb b/test/thread/test_sync.rb index 8241445faf..e3294ff824 100644 --- a/test/thread/test_sync.rb +++ b/test/thread/test_sync.rb @@ -59,6 +59,7 @@ class SyncTest < Test::Unit::TestCase } sleep 0.1 until t.stop? + sleep 1 if RubyVM::MJIT.enabled? # t.stop? behaves unexpectedly with --jit-wait t.raise t.join diff --git a/test/webrick/test_httpserver.rb b/test/webrick/test_httpserver.rb index daeb7b955e..5ae105dabe 100644 --- a/test/webrick/test_httpserver.rb +++ b/test/webrick/test_httpserver.rb @@ -253,6 +253,7 @@ class TestWEBrickHTTPServer < Test::Unit::TestCase server.virtual_host(WEBrick::HTTPServer.new(vhost_config)) Thread.pass while server.status != :Running + sleep 1 if RubyVM::MJIT.enabled? # server.status behaves unexpectedly with --jit-wait assert_equal(1, started, log.call) assert_equal(0, stopped, log.call) assert_equal(0, accepted, log.call) diff --git a/test/webrick/test_server.rb b/test/webrick/test_server.rb index 4d539d0368..5f7f3a0b58 100644 --- a/test/webrick/test_server.rb +++ b/test/webrick/test_server.rb @@ -65,6 +65,7 @@ class TestWEBrickServer < Test::Unit::TestCase } TestWEBrick.start_server(Echo, config){|server, addr, port, log| true while server.status != :Running + sleep 1 if RubyVM::MJIT.enabled? # server.status behaves unexpectedly with --jit-wait assert_equal(1, started, log.call) assert_equal(0, stopped, log.call) assert_equal(0, accepted, log.call) @@ -2089,7 +2089,7 @@ threadptr_get_interrupts(rb_thread_t *th) return interrupt & (rb_atomic_t)~ec->interrupt_mask; } -void +MJIT_FUNC_EXPORTED void rb_threadptr_execute_interrupts(rb_thread_t *th, int blocking_timing) { rb_atomic_t interrupt; diff --git a/tool/ruby_vm/views/_mjit_compile_insn.erb b/tool/ruby_vm/views/_mjit_compile_insn.erb new file mode 100644 index 0000000000..60ba2505d2 --- /dev/null +++ b/tool/ruby_vm/views/_mjit_compile_insn.erb @@ -0,0 +1,133 @@ +% # -*- mode:c; style:ruby; coding: utf-8; indent-tabs-mode: nil -*- +% # Copyright (c) 2018 Takashi Kokubun. All rights reserved. +% # +% # This file is a part of the programming language Ruby. Permission is hereby +% # granted, to either redistribute and/or modify this file, provided that the +% # conditions mentioned in the file COPYING are met. Consult the file for +% # details. +% +% trace_enablable_insns = [ +% 'opt_send_without_block', +% 'send', +% 'invokeblock', +% 'invokesuper', +% ] +% +% to_cstr = lambda do |line| +% normalized = line.gsub(/\t/, ' ' * 8) +% indented = normalized.sub(/\A(?!#)/, ' ') # avoid indenting preprocessor +% rstring2cstr(indented.rstrip).sub(/"\z/, '\\n"') +% end +% + fprintf(f, "{\n"); + { +% # compiler: Prepare operands which may be used by `insn.call_attribute` +% insn.opes.each_with_index do |ope, i| + MAYBE_UNUSED(<%= ope.fetch(:decl) %>) = (<%= ope.fetch(:type) %>)operands[<%= i %>]; +% end +% +% # JIT: Declare variables for operands, popped values and return values +% ret_decls = insn.rets.map { |r| "MAYBE_UNUSED(#{r.fetch(:type)}) #{r.fetch(:name)}"} # TODO: fix #declarations to return Hash... +% insn.declarations.each do |decl| +% next if dispatched && ret_decls.include?(decl) # return value should be propagated to dispatcher. TODO: assert it's the same as dispatcher + fprintf(f, " <%= decl %>;\n"); +% end + +% # JIT: Set const expressions for `RubyVM::OperandsUnifications` insn +% insn.preamble.each do |amble| + fprintf(f, "<%= amble.expr %>\n"); +% end +% +% # JIT: Initialize operands +% insn.opes.each_with_index do |ope, i| + fprintf(f, " <%= ope.fetch(:name) %> = (<%= ope.fetch(:type) %>)0x%"PRIxVALUE";\n", operands[<%= i %>]); +% end +% +% # JIT: Initialize popped values +% insn.pops.reverse_each.with_index.reverse_each do |pop, i| + fprintf(f, " <%= pop.fetch(:name) %> = stack[%d];\n", b->stack_size - <%= i + 1 %>); +% end +% +% # JIT: move sp and pc if necessary +% if insn.handles_frame? + fprintf(f, " reg_cfp->pc = (VALUE *)0x%"PRIxVALUE";\n", (VALUE)(body->iseq_encoded + next_pos)); /* ADD_PC(INSN_ATTR(width)); */ + fprintf(f, " reg_cfp->sp = reg_cfp->bp + %d;\n", b->stack_size + 1 - <%= insn.pops.size %>); /* POPN(INSN_ATTR(popn)); */ +% else + fprintf(f, " reg_cfp->pc = (VALUE *)0x%"PRIxVALUE";\n", (VALUE)(body->iseq_encoded + pos)); + fprintf(f, " reg_cfp->sp = reg_cfp->bp + %d;\n", b->stack_size + 1); +% end +% +% # JIT: Print insn body in insns.def +% insn.expr.expr.each_line do |line| +% # Special macro expansion for ones that can't be resolved by macro redefinition. +% if line =~ /\A\s+DISPATCH_ORIGINAL_INSN\((?<insn_name>[^)]+)\);\s+\z/ + fprintf(f, " return Qundef; /* cancel JIT */\n"); +% elsif line =~ /\A\s+JUMP\((?<dest>[^)]+)\);\s+\z/ +% # Before we `goto` next insn, we need to set return values, especially for getinlinecache +% insn.rets.reverse_each.with_index do |ret, i| +% # TOPN(n) = ... + fprintf(f, " stack[%d] = <%= ret.fetch(:name) %>;\n", b->stack_size + (int)<%= insn.call_attribute('sp_inc') %> - <%= i + 1 %>); +% end +% +% dest = Regexp.last_match[:dest] +% if insn.name == 'opt_case_dispatch' # special case... TODO: use another macro to avoid checking name + { + struct case_dispatch_var arg; + arg.f = f; + arg.base_pos = pos + insn_len(insn); + arg.last_value = Qundef; + + fprintf(f, " switch (<%= dest %>) {\n"); + st_foreach(RHASH_TBL_RAW(hash), compile_case_dispatch_each, (VALUE)&arg); + fprintf(f, " case %lu:\n", else_offset); + fprintf(f, " goto label_%lu;\n", arg.base_pos + else_offset); + fprintf(f, " }\n"); + } +% else + next_pos = pos + insn_len(insn) + (unsigned int)<%= dest %>; + fprintf(f, " goto label_%d;\n", next_pos); +% end +% elsif line =~ /\A\s+RESTORE_REGS\(\);\s+\z/ # for `leave` only +#if OPT_CALL_THREADED_CODE + fprintf(f, " rb_ec_thread_ptr(ec)->retval = val;\n"); + fprintf(f, " return 0;\n"); +#else + fprintf(f, " return val;\n"); +#endif +% else + fprintf(f, <%= to_cstr.call(line) %>); +% end +% end +% +% # JIT: Set return values +% unless dispatched +% insn.rets.reverse_each.with_index do |ret, i| +% # TOPN(n) = ... + fprintf(f, " stack[%d] = <%= ret.fetch(:name) %>;\n", b->stack_size + (int)<%= insn.call_attribute('sp_inc') %> - <%= i + 1 %>); +% end +% end +% +% # JIT: We should evaluate ISeq modified for TracePoint if it's enabled. Note: This is slow. +% if trace_enablable_insns.include?(insn.name) + fprintf(f, " if (UNLIKELY(ruby_vm_event_enabled_flags & ISEQ_TRACE_EVENTS)) {\n"); + fprintf(f, " reg_cfp->sp = reg_cfp->bp + %d;\n", b->stack_size + (int)<%= insn.call_attribute('sp_inc') %> + 1); + fprintf(f, " return Qundef; /* cancel JIT */\n"); + fprintf(f, " }\n"); +% end +% +% # compiler: Move JIT compiler's internal stack pointer +% unless dispatched + b->stack_size += <%= insn.call_attribute('sp_inc') %>; +% end + } + fprintf(f, "}\n"); +% +% # compiler: If insn has conditional JUMP, the branch which is not targeted by JUMP should be compiled too. +% if insn.expr.expr =~ /if\s+\([^{}]+\)\s+\{[^{}]+JUMP\([^)]+\);[^{}]+\}/ + compile_insns(f, body, b->stack_size, pos + insn_len(insn), status); +% end +% +% # compiler: If insn returns (leave) or does longjmp (throw), the branch should no longer be compieled. TODO: create attr for it? +% if insn.expr.expr =~ /\sTHROW_EXCEPTION\([^)]+\);/ || insn.expr.expr =~ /\sRESTORE_REGS\(\);/ + b->finish_p = TRUE; +% end diff --git a/tool/ruby_vm/views/_mjit_compile_send.erb b/tool/ruby_vm/views/_mjit_compile_send.erb new file mode 100644 index 0000000000..6624cfe7db --- /dev/null +++ b/tool/ruby_vm/views/_mjit_compile_send.erb @@ -0,0 +1,74 @@ +% # -*- mode:c; style:ruby; coding: utf-8; indent-tabs-mode: nil -*- +% # Copyright (c) 2018 Takashi Kokubun. All rights reserved. +% # +% # This file is a part of the programming language Ruby. Permission is hereby +% # granted, to either redistribute and/or modify this file, provided that the +% # conditions mentioned in the file COPYING are met. Consult the file for +% # details. +% +% # Optimized case of send / opt_send_without_block instructions. + { +% # compiler: Prepare operands which may be used by `insn.call_attribute` +% insn.opes.each_with_index do |ope, i| + MAYBE_UNUSED(<%= ope.fetch(:decl) %>) = (<%= ope.fetch(:type) %>)operands[<%= i %>]; +% end +% + const rb_iseq_t *iseq; + unsigned int argc = ci->orig_argc; /* unlike `ci->orig_argc`, `argc` may include blockarg */ +% if insn.name == 'send' + argc += ((ci->flag & VM_CALL_ARGS_BLOCKARG) ? 1 : 0); +% end + + if (inlinable_iseq_p(ci, cc, iseq = get_iseq_if_available(cc))) { + int param_size = iseq->body->param.size; /* TODO: check calling->argc for argument_arity_error */ + +% # JIT: move sp and pc if necessary + fprintf(f, " reg_cfp->pc = (VALUE *)0x%"PRIxVALUE";\n", (VALUE)(body->iseq_encoded + next_pos)); /* ADD_PC(INSN_ATTR(width)); */ + fprintf(f, " reg_cfp->sp = reg_cfp->bp + %d;\n", b->stack_size + 1 - <%= insn.pops.size %>); /* POPN(INSN_ATTR(popn)); */ + +% # JIT: Invalidate call cache if it requires vm_search_method. This allows to inline some of following things. + fprintf(f, " if (UNLIKELY(GET_GLOBAL_METHOD_STATE() != %llu || RCLASS_SERIAL(CLASS_OF(stack[%d])) != %llu)) {\n", cc->method_state, b->stack_size - 1 - argc, cc->class_serial); + fprintf(f, " reg_cfp->pc = (VALUE *)0x%"PRIxVALUE";\n", (VALUE)(body->iseq_encoded + pos)); + fprintf(f, " return Qundef; /* cancel JIT */\n"); + fprintf(f, " }\n"); + +% # JIT: Print insn body in insns.def + fprintf(f, " {\n"); + fprintf(f, " struct rb_calling_info calling;\n"); +% if insn.name == 'send' + fprintf(f, " vm_caller_setup_arg_block(ec, reg_cfp, &calling, 0x%"PRIxVALUE", 0x%"PRIxVALUE", FALSE);\n", operands[0], operands[2]); +% else + fprintf(f, " calling.block_handler = VM_BLOCK_HANDLER_NONE;\n"); +% end + fprintf(f, " calling.argc = %d;\n", ci->orig_argc); + fprintf(f, " calling.recv = stack[%d];\n", b->stack_size - 1 - argc); + +% # JIT: Special CALL_METHOD. Inline vm_call_iseq_setup_normal for vm_call_iseq_setup_func FASTPATH. TODO: modify VM to share code with here + fprintf(f, " {\n"); + fprintf(f, " VALUE v;\n"); + fprintf(f, " VALUE *argv = reg_cfp->sp - calling.argc;\n"); + fprintf(f, " reg_cfp->sp = argv - 1;\n"); /* recv */ + fprintf(f, " vm_push_frame(ec, 0x%"PRIxVALUE", VM_FRAME_MAGIC_METHOD | VM_ENV_FLAG_LOCAL, calling.recv, " + "calling.block_handler, 0x%"PRIxVALUE", 0x%"PRIxVALUE", argv + %d, %d, %d);\n", + (VALUE)iseq, (VALUE)cc->me, (VALUE)iseq->body->iseq_encoded, param_size, iseq->body->local_table_size - param_size, iseq->body->stack_max); + fprintf(f, " if ((v = mjit_exec(ec)) == Qundef) {\n"); + fprintf(f, " VM_ENV_FLAGS_SET(ec->cfp->ep, VM_FRAME_FLAG_FINISH);\n"); /* This is vm_call0_body's code after vm_call_iseq_setup */ + fprintf(f, " v = vm_exec(ec);\n"); + fprintf(f, " }\n"); + fprintf(f, " stack[%d] = v;\n", b->stack_size - argc - 1); + fprintf(f, " }\n"); + + fprintf(f, " }\n"); + +% # JIT: We should evaluate ISeq modified for TracePoint if it's enabled. Note: This is slow. + fprintf(f, " if (UNLIKELY(ruby_vm_event_enabled_flags & ISEQ_TRACE_EVENTS)) {\n"); + fprintf(f, " reg_cfp->sp = reg_cfp->bp + %d;\n", b->stack_size + (int)<%= insn.call_attribute('sp_inc') %> + 1); + fprintf(f, " return Qundef; /* cancel JIT */\n"); + fprintf(f, " }\n"); + +% # compiler: Move JIT compiler's internal stack pointer + b->stack_size += <%= insn.call_attribute('sp_inc') %>; + + break; + } + } diff --git a/tool/ruby_vm/views/mjit_compile.inc.erb b/tool/ruby_vm/views/mjit_compile.inc.erb new file mode 100644 index 0000000000..7437a44855 --- /dev/null +++ b/tool/ruby_vm/views/mjit_compile.inc.erb @@ -0,0 +1,66 @@ +/* -*- mode:c; style:ruby; coding: utf-8; indent-tabs-mode: nil -*- */ + +% # Copyright (c) 2018 Takashi Kokubun. All rights reserved. +% # +% # This file is a part of the programming language Ruby. Permission is hereby +% # granted, to either redistribute and/or modify this file, provided that the +% # conditions mentioned in the file COPYING are met. Consult the file for +% # details. +<%= render 'copyright' %> +% +% # This is an ERB template that generates Ruby code that generates C code that +% # generates JIT-ed C code. +<%= render 'notice', locals: { + this_file: 'is the main part of compile_insn() in mjit_compile.c', + edit: __FILE__, +} -%> +% +% unsupported_insns = [ +% 'getblockparamproxy', # TODO: support this +% 'defineclass', # low priority +% 'opt_call_c_function', # low priority +% ] +% +% # Available variables and macros in JIT-ed function: +% # ec: the first argument of _mjitXXX +% # reg_cfp: the second argument of _mjitXXX +% # GET_CFP(): refers to `reg_cfp` +% # GET_EP(): refers to `reg_cfp->ep` +% # GET_SP(): refers to `reg_cfp->sp` +% # INC_SP(): refers to `reg_cfp->sp` +% # SET_SV(): refers to `reg_cfp->sp` +% # PUSH(): refers to `SET_SV()`, `INC_SP()` +% # GET_SELF(): refers to `reg_cfp->self` +% # GET_LEP(): refers to `VM_EP_LEP(reg_cfp->ep)` +% # EXEC_EC_CFP(): refers to `val = vm_exec(ec)` with frame setup +% # CALL_METHOD(): using `GET_CFP()` and `EXEC_EC_CFP()` +% # TOPN(): refers to `reg_cfp->sp`, which needs to have correct sp (of course) +% # STACK_ADDR_FROM_TOP(): refers to `reg_cfp->sp`, same problem here +% # DISPATCH_ORIGINAL_INSN(): expanded in _mjit_compile_insn.erb +% # THROW_EXCEPTION(): specially defined for JIT +% # RESTORE_REGS(): specially defined for `leave` + +switch (insn) { +% (RubyVM::BareInstructions.to_a + RubyVM::OperandsUnifications.to_a).each do |insn| +% next if unsupported_insns.include?(insn.name) + case BIN(<%= insn.name %>): +% if %w[opt_send_without_block send].include?(insn.name) +<%= render 'mjit_compile_send', locals: { insn: insn } -%> +% end +<%= render 'mjit_compile_insn', locals: { insn: insn, dispatched: false } -%> + break; +% end +% # We don't support InstructionsUnifications yet because it's not used for now. +% # We don't support TraceInstructions yet. There is no blocker for it but it's just not implemented. + default: + if (mjit_opts.warnings || mjit_opts.verbose >= 3) + /* passing excessive arguments to suppress warning in insns_info.inc as workaround... */ + fprintf(stderr, "MJIT warning: Failed to compile instruction: %s (%s: %d...)\n", + insn_name(insn), insn_op_types(insn), insn_len(insn) > 0 ? insn_op_type(insn, 0) : 0); + status->success = FALSE; + break; +} + +/* if next_pos is already compiled, next instruction won't be compiled in C code and needs `goto`. */ +if ((next_pos < body->iseq_size && status->compiled_for_pos[next_pos])) + fprintf(f, " goto label_%d;\n", next_pos); diff --git a/tool/transform_mjit_header.rb b/tool/transform_mjit_header.rb index 308379123b..4c3377b18f 100644 --- a/tool/transform_mjit_header.rb +++ b/tool/transform_mjit_header.rb @@ -17,6 +17,7 @@ module MJITHeader ] IGNORED_FUNCTIONS = [ + 'vm_search_method_slowpath', # This increases the time to compile when inlined. So we use it as external function. 'rb_equal_opt', # Not used from VM and not compilable ] diff --git a/tool/update-deps b/tool/update-deps index ccfc160e42..200afe7ca0 100755 --- a/tool/update-deps +++ b/tool/update-deps @@ -120,6 +120,7 @@ FILES_NEED_VPATH = %w[ known_errors.inc lex.c miniprelude.c + mjit_compile.inc newline.c node_name.inc opt_sc.inc diff --git a/variable.c b/variable.c index ca7e4988a6..433f97e0af 100644 --- a/variable.c +++ b/variable.c @@ -480,7 +480,7 @@ struct rb_global_variable { struct trace_var *trace; }; -struct rb_global_entry* +MJIT_FUNC_EXPORTED struct rb_global_entry* rb_global_entry(ID id) { struct rb_global_entry *entry; @@ -790,7 +790,7 @@ rb_f_untrace_var(int argc, const VALUE *argv) return Qnil; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_gvar_get(struct rb_global_entry *entry) { struct rb_global_variable *var = entry->var; @@ -823,7 +823,7 @@ trace_en(struct rb_global_variable *var) return Qnil; /* not reached */ } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_gvar_set(struct rb_global_entry *entry, VALUE val) { struct trace_data trace; @@ -858,7 +858,7 @@ rb_gv_get(const char *name) return rb_gvar_get(entry); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_gvar_defined(struct rb_global_entry *entry) { if (entry->var->getter == rb_gvar_undef_getter) return Qfalse; @@ -2010,7 +2010,7 @@ check_autoload_required(VALUE mod, ID id, const char **loadingpath) return 0; } -int +MJIT_FUNC_EXPORTED int rb_autoloading_value(VALUE mod, ID id, VALUE* value) { VALUE load; @@ -2211,7 +2211,7 @@ rb_autoload_p(VALUE mod, ID id) return (ele = check_autoload_data(load)) ? ele->feature : Qnil; } -void +MJIT_FUNC_EXPORTED void rb_const_warn_if_deprecated(const rb_const_entry_t *ce, VALUE klass, ID id) { if (RB_CONST_DEPRECATED_P(ce)) { @@ -2294,7 +2294,7 @@ rb_const_get_at(VALUE klass, ID id) return rb_const_get_0(klass, id, TRUE, FALSE, FALSE); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_public_const_get_from(VALUE klass, ID id) { return rb_const_get_0(klass, id, TRUE, TRUE, TRUE); @@ -2306,7 +2306,7 @@ rb_public_const_get(VALUE klass, ID id) return rb_const_get_0(klass, id, FALSE, TRUE, TRUE); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_public_const_get_at(VALUE klass, ID id) { return rb_const_get_0(klass, id, TRUE, FALSE, TRUE); @@ -2544,7 +2544,7 @@ rb_const_defined_at(VALUE klass, ID id) return rb_const_defined_0(klass, id, TRUE, FALSE, FALSE); } -int +MJIT_FUNC_EXPORTED int rb_public_const_defined_from(VALUE klass, ID id) { return rb_const_defined_0(klass, id, TRUE, TRUE, TRUE); @@ -3123,7 +3123,7 @@ rb_st_copy(VALUE obj, struct st_table *orig_tbl) return new_tbl; } -rb_const_entry_t * +MJIT_FUNC_EXPORTED rb_const_entry_t * rb_const_lookup(VALUE klass, ID id) { struct rb_id_table *tbl = RCLASS_CONST_TBL(klass); @@ -293,7 +293,7 @@ static void vm_collect_usage_register(int reg, int isset); #endif static VALUE vm_make_env_object(const rb_execution_context_t *ec, rb_control_frame_t *cfp); -static VALUE vm_invoke_bmethod(rb_execution_context_t *ec, rb_proc_t *proc, VALUE self, int argc, const VALUE *argv, VALUE block_handler); +extern VALUE vm_invoke_bmethod(rb_execution_context_t *ec, rb_proc_t *proc, VALUE self, int argc, const VALUE *argv, VALUE block_handler); static VALUE vm_invoke_proc(rb_execution_context_t *ec, rb_proc_t *proc, VALUE self, int argc, const VALUE *argv, VALUE block_handler); static VALUE rb_block_param_proxy; @@ -302,17 +302,24 @@ static VALUE rb_block_param_proxy; #include "vm_insnhelper.h" #include "vm_exec.h" #include "vm_insnhelper.c" + +#ifndef MJIT_HEADER + #include "vm_exec.c" #include "vm_method.c" +#endif /* #ifndef MJIT_HEADER */ #include "vm_eval.c" +#ifndef MJIT_HEADER #define PROCDEBUG 0 rb_serial_t rb_next_class_serial(void) { - return NEXT_CLASS_SERIAL(); + rb_serial_t class_serial = NEXT_CLASS_SERIAL(); + mjit_add_class_serial(class_serial); + return class_serial; } VALUE rb_cRubyVM; @@ -339,7 +346,7 @@ rb_vm_inc_const_missing_count(void) VALUE rb_class_path_no_cache(VALUE _klass); -int +MJIT_FUNC_EXPORTED int rb_dtrace_setup(rb_execution_context_t *ec, VALUE klass, ID id, struct ruby_dtrace_method_hook_args *args) { @@ -499,7 +506,7 @@ rb_vm_get_binding_creatable_next_cfp(const rb_execution_context_t *ec, const rb_ return 0; } -rb_control_frame_t * +MJIT_FUNC_EXPORTED rb_control_frame_t * rb_vm_get_ruby_level_next_cfp(const rb_execution_context_t *ec, const rb_control_frame_t *cfp) { if (RUBY_VM_CONTROL_FRAME_STACK_OVERFLOW_P(ec, cfp)) bp(); @@ -512,6 +519,8 @@ rb_vm_get_ruby_level_next_cfp(const rb_execution_context_t *ec, const rb_control return 0; } +#endif /* #ifndef MJIT_HEADER */ + static rb_control_frame_t * vm_get_ruby_level_caller_cfp(const rb_execution_context_t *ec, const rb_control_frame_t *cfp) { @@ -546,6 +555,8 @@ rb_vm_pop_cfunc_frame(void) vm_pop_frame(ec, cfp, cfp->ep); } +#ifndef MJIT_HEADER + void rb_vm_rewind_cfp(rb_execution_context_t *ec, rb_control_frame_t *cfp) { @@ -880,7 +891,7 @@ rb_proc_dup(VALUE self) } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_vm_make_proc_lambda(const rb_execution_context_t *ec, const struct rb_captured_block *captured, VALUE klass, int8_t is_lambda) { VALUE procval; @@ -1157,14 +1168,14 @@ vm_invoke_proc(rb_execution_context_t *ec, rb_proc_t *proc, VALUE self, return invoke_block_from_c_proc(ec, proc, self, argc, argv, passed_block_handler, proc->is_lambda); } -static VALUE +MJIT_FUNC_EXPORTED VALUE vm_invoke_bmethod(rb_execution_context_t *ec, rb_proc_t *proc, VALUE self, int argc, const VALUE *argv, VALUE block_handler) { return invoke_block_from_c_proc(ec, proc, self, argc, argv, block_handler, TRUE); } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_vm_invoke_proc(rb_execution_context_t *ec, rb_proc_t *proc, int argc, const VALUE *argv, VALUE passed_block_handler) { @@ -1391,7 +1402,7 @@ make_localjump_error(const char *mesg, VALUE value, int reason) return exc; } -void +MJIT_FUNC_EXPORTED void rb_vm_localjump_error(const char *mesg, VALUE value, int reason) { VALUE exc = make_localjump_error(mesg, value, reason); @@ -1775,7 +1786,7 @@ hook_before_rewind(rb_execution_context_t *ec, const rb_control_frame_t *cfp, in }; */ -static VALUE +MJIT_FUNC_EXPORTED VALUE vm_exec(rb_execution_context_t *ec) { enum ruby_tag_type state; @@ -1789,8 +1800,8 @@ vm_exec(rb_execution_context_t *ec) if ((state = EC_EXEC_TAG()) == TAG_NONE) { result = mjit_exec(ec); vm_loop_start: - if (result == Qundef) - result = vm_exec_core(ec, initial); + if (result == Qundef) + result = vm_exec_core(ec, initial); VM_ASSERT(ec->tag == &_tag); if ((state = _tag.state) != TAG_NONE) { err = (struct vm_throw_data *)result; @@ -2000,6 +2011,7 @@ vm_exec(rb_execution_context_t *ec) state = 0; ec->tag->state = TAG_NONE; ec->errinfo = Qnil; + result = Qundef; goto vm_loop_start; } @@ -3424,4 +3436,6 @@ vm_collect_usage_register(int reg, int isset) } #endif +#endif /* #ifndef MJIT_HEADER */ + #include "vm_call_iseq_optimized.inc" /* required from vm_insnhelper.c */ diff --git a/vm_backtrace.c b/vm_backtrace.c index d6a8a1370e..59340292cf 100644 --- a/vm_backtrace.c +++ b/vm_backtrace.c @@ -520,7 +520,7 @@ bt_iter_cfunc(void *ptr, const rb_control_frame_t *cfp, ID mid) loc->body.cfunc.prev_loc = arg->prev_loc; } -VALUE +MJIT_FUNC_EXPORTED VALUE rb_ec_backtrace_object(const rb_execution_context_t *ec) { struct bt_iter_arg arg; @@ -595,7 +595,7 @@ rb_backtrace_to_str_ary(VALUE self) return bt->strary; } -void +MJIT_FUNC_EXPORTED void rb_backtrace_use_iseq_first_lineno_for_last_location(VALUE self) { const rb_backtrace_t *bt; @@ -686,9 +686,10 @@ typedef struct rb_control_frame_struct { VALUE self; /* cfp[3] / block[0] */ const VALUE *ep; /* cfp[4] / block[1] */ const void *block_code; /* cfp[5] / block[2] */ /* iseq or ifunc */ + const VALUE *bp; /* cfp[6] */ #if VM_DEBUG_BP_CHECK - VALUE *bp_check; /* cfp[6] */ + VALUE *bp_check; /* cfp[7] */ #endif } rb_control_frame_t; @@ -20,7 +20,7 @@ static inline VALUE vm_yield_with_cref(rb_execution_context_t *ec, int argc, con static inline VALUE vm_yield(rb_execution_context_t *ec, int argc, const VALUE *argv); static inline VALUE vm_yield_with_block(rb_execution_context_t *ec, int argc, const VALUE *argv, VALUE block_handler); static inline VALUE vm_yield_force_blockarg(rb_execution_context_t *ec, VALUE args); -static VALUE vm_exec(rb_execution_context_t *ec); +VALUE vm_exec(rb_execution_context_t *ec); static void vm_set_eval_stack(rb_execution_context_t * th, const rb_iseq_t *iseq, const rb_cref_t *cref, const struct rb_block *base_block); static int vm_collect_local_variables_in_heap(const VALUE *dfp, const struct local_var_list *vars); @@ -38,7 +38,9 @@ typedef enum call_type { static VALUE send_internal(int argc, const VALUE *argv, VALUE recv, call_type scope); static VALUE vm_call0_body(rb_execution_context_t* ec, struct rb_calling_info *calling, const struct rb_call_info *ci, struct rb_call_cache *cc, const VALUE *argv); -static VALUE +#ifndef MJIT_HEADER + +MJIT_FUNC_EXPORTED VALUE vm_call0(rb_execution_context_t *ec, VALUE recv, ID id, int argc, const VALUE *argv, const rb_callable_method_entry_t *me) { struct rb_calling_info calling_entry, *calling; @@ -252,6 +254,8 @@ rb_current_receiver(void) return cfp->self; } +#endif /* #ifndef MJIT_HEADER */ + static inline void stack_check(rb_execution_context_t *ec) { @@ -262,6 +266,8 @@ stack_check(rb_execution_context_t *ec) } } +#ifndef MJIT_HEADER + static inline const rb_callable_method_entry_t *rb_search_method_entry(VALUE recv, ID mid); static inline enum method_missing_reason rb_method_call_status(rb_execution_context_t *ec, const rb_callable_method_entry_t *me, call_type scope, VALUE self); @@ -633,7 +639,7 @@ rb_method_missing(int argc, const VALUE *argv, VALUE obj) UNREACHABLE; } -static VALUE +MJIT_FUNC_EXPORTED VALUE make_no_method_exception(VALUE exc, VALUE format, VALUE obj, int argc, const VALUE *argv, int priv) { @@ -659,6 +665,8 @@ make_no_method_exception(VALUE exc, VALUE format, VALUE obj, return rb_class_new_instance(n, args, exc); } +#endif /* #ifndef MJIT_HEADER */ + static void raise_method_missing(rb_execution_context_t *ec, int argc, const VALUE *argv, VALUE obj, enum method_missing_reason last_call_status) @@ -740,6 +748,8 @@ method_missing(VALUE obj, ID id, int argc, const VALUE *argv, enum method_missin return result; } +#ifndef MJIT_HEADER + /*! * Calls a method * \param recv receiver of the method @@ -2175,3 +2185,5 @@ Init_vm_eval(void) id_tag = rb_intern_const("tag"); id_value = rb_intern_const("value"); } + +#endif /* #ifndef MJIT_HEADER */ @@ -167,8 +167,26 @@ default: \ #endif +#ifdef MJIT_HEADER +#define EXEC_EC_CFP() do { \ + VM_ENV_FLAGS_SET(ec->cfp->ep, VM_FRAME_FLAG_FINISH); \ + val = vm_exec(ec); \ +} while (0) +#else +#define EXEC_EC_CFP() do { \ + RESTORE_REGS(); \ + NEXT_INSN(); \ +} while (0) +#endif + #define VM_SP_CNT(ec, sp) ((sp) - (ec)->vm_stack) +#ifdef MJIT_HEADER +#define THROW_EXCEPTION(exc) do { \ + ec->errinfo = (VALUE)(exc); \ + EC_JUMP_TAG(ec, ec->tag->state); \ +} while (0) +#else #if OPT_CALL_THREADED_CODE #define THROW_EXCEPTION(exc) do { \ ec->errinfo = (VALUE)(exc); \ @@ -177,6 +195,7 @@ default: \ #else #define THROW_EXCEPTION(exc) return (VALUE)(exc) #endif +#endif #define SCREG(r) (reg_##r) diff --git a/vm_insnhelper.c b/vm_insnhelper.c index acb7d7999d..bc54acb0a9 100644 --- a/vm_insnhelper.c +++ b/vm_insnhelper.c @@ -243,7 +243,8 @@ vm_push_frame(rb_execution_context_t *ec, *sp++ = specval /* ep[-1] / block handler or prev env ptr */; *sp = type; /* ep[-0] / ENV_FLAGS */ - cfp->ep = sp; + /* Store initial value of ep as bp to skip calculation cost of bp on JIT cancellation. */ + cfp->ep = cfp->bp = sp; cfp->sp = sp + 1; #if VM_DEBUG_BP_CHECK @@ -1295,6 +1296,18 @@ vm_expandarray(rb_control_frame_t *cfp, VALUE ary, rb_num_t num, int flag) static VALUE vm_call_general(rb_execution_context_t *ec, rb_control_frame_t *reg_cfp, struct rb_calling_info *calling, const struct rb_call_info *ci, struct rb_call_cache *cc); +MJIT_FUNC_EXPORTED void +vm_search_method_slowpath(const struct rb_call_info *ci, struct rb_call_cache *cc, VALUE klass) +{ + cc->me = rb_callable_method_entry(klass, ci->mid); + VM_ASSERT(callable_method_entry_p(cc->me)); + cc->call = vm_call_general; +#if OPT_INLINE_METHOD_CACHE + cc->method_state = GET_GLOBAL_METHOD_STATE(); + cc->class_serial = RCLASS_SERIAL(klass); +#endif +} + static void vm_search_method(const struct rb_call_info *ci, struct rb_call_cache *cc, VALUE recv) { @@ -1312,13 +1325,7 @@ vm_search_method(const struct rb_call_info *ci, struct rb_call_cache *cc, VALUE } RB_DEBUG_COUNTER_INC(mc_inline_miss); #endif - cc->me = rb_callable_method_entry(klass, ci->mid); - VM_ASSERT(callable_method_entry_p(cc->me)); - cc->call = vm_call_general; -#if OPT_INLINE_METHOD_CACHE - cc->method_state = GET_GLOBAL_METHOD_STATE(); - cc->class_serial = RCLASS_SERIAL(klass); -#endif + vm_search_method_slowpath(ci, cc, klass); } static inline int @@ -1458,7 +1465,7 @@ rb_eql_opt(VALUE obj1, VALUE obj2) return opt_eql_func(obj1, obj2, &ci, &cc); } -static VALUE vm_call0(rb_execution_context_t *ec, VALUE, ID, int, const VALUE*, const rb_callable_method_entry_t *); +extern VALUE vm_call0(rb_execution_context_t *ec, VALUE, ID, int, const VALUE*, const rb_callable_method_entry_t *); static VALUE check_match(rb_execution_context_t *ec, VALUE pattern, VALUE target, enum vm_check_match_type type) @@ -1562,9 +1569,9 @@ static inline VALUE vm_call_method(rb_execution_context_t *ec, rb_control_frame_ static vm_call_handler vm_call_iseq_setup_func(const struct rb_call_info *ci, const int param_size, const int local_size); -static rb_method_definition_t *method_definition_create(rb_method_type_t type, ID mid); -static void method_definition_set(const rb_method_entry_t *me, rb_method_definition_t *def, void *opts); -static int rb_method_definition_eq(const rb_method_definition_t *d1, const rb_method_definition_t *d2); +extern rb_method_definition_t *method_definition_create(rb_method_type_t type, ID mid); +extern void method_definition_set(const rb_method_entry_t *me, rb_method_definition_t *def, void *opts); +extern int rb_method_definition_eq(const rb_method_definition_t *d1, const rb_method_definition_t *d2); static const rb_iseq_t * def_iseq_ptr(rb_method_definition_t *def) @@ -1590,7 +1597,7 @@ vm_call_iseq_setup_normal_0start(rb_execution_context_t *ec, rb_control_frame_t return vm_call_iseq_setup_normal(ec, cfp, calling, ci, cc, 0, param, local); } -static inline int +int simple_iseq_p(const rb_iseq_t *iseq) { return iseq->body->param.flags.has_opt == FALSE && diff --git a/vm_insnhelper.h b/vm_insnhelper.h index 31dbdac3fa..1dd491cf7f 100644 --- a/vm_insnhelper.h +++ b/vm_insnhelper.h @@ -130,8 +130,7 @@ enum vm_regan_acttype { #define CALL_METHOD(calling, ci, cc) do { \ VALUE v = (*(cc)->call)(ec, GET_CFP(), (calling), (ci), (cc)); \ if (v == Qundef && (v = mjit_exec(ec)) == Qundef) { \ - RESTORE_REGS(); \ - NEXT_INSN(); \ + EXEC_EC_CFP(); \ } \ else { \ val = v; \ @@ -185,7 +184,7 @@ enum vm_regan_acttype { #define GET_GLOBAL_CONSTANT_STATE() (ruby_vm_global_constant_state) #define INC_GLOBAL_CONSTANT_STATE() (++ruby_vm_global_constant_state) -static VALUE make_no_method_exception(VALUE exc, VALUE format, VALUE obj, +extern VALUE make_no_method_exception(VALUE exc, VALUE format, VALUE obj, int argc, const VALUE *argv, int priv); static inline struct vm_throw_data * diff --git a/vm_method.c b/vm_method.c index 695a985704..1062b70959 100644 --- a/vm_method.c +++ b/vm_method.c @@ -62,6 +62,7 @@ static struct { static void rb_class_clear_method_cache(VALUE klass, VALUE arg) { + mjit_remove_class_serial(RCLASS_SERIAL(klass)); RCLASS_SERIAL(klass) = rb_next_class_serial(); if (RB_TYPE_P(klass, T_ICLASS)) { @@ -171,7 +172,7 @@ rb_free_method_entry(const rb_method_entry_t *me) } static inline rb_method_entry_t *search_method(VALUE klass, ID id, VALUE *defined_class_ptr); -static int rb_method_definition_eq(const rb_method_definition_t *d1, const rb_method_definition_t *d2); +extern int rb_method_definition_eq(const rb_method_definition_t *d1, const rb_method_definition_t *d2); static inline rb_method_entry_t * lookup_method_table(VALUE klass, ID id) @@ -222,7 +223,7 @@ setup_method_cfunc_struct(rb_method_cfunc_t *cfunc, VALUE (*func)(), int argc) cfunc->invoker = call_cfunc_invoker_func(argc); } -static void +MJIT_FUNC_EXPORTED void method_definition_set(const rb_method_entry_t *me, rb_method_definition_t *def, void *opts) { *(rb_method_definition_t **)&me->def = def; @@ -336,7 +337,7 @@ method_definition_reset(const rb_method_entry_t *me) } } -static rb_method_definition_t * +MJIT_FUNC_EXPORTED rb_method_definition_t * method_definition_create(rb_method_type_t type, ID mid) { rb_method_definition_t *def; @@ -401,7 +402,7 @@ rb_method_entry_clone(const rb_method_entry_t *src_me) return me; } -const rb_callable_method_entry_t * +MJIT_FUNC_EXPORTED const rb_callable_method_entry_t * rb_method_entry_complement_defined_class(const rb_method_entry_t *src_me, ID called_id, VALUE defined_class) { rb_method_definition_t *def = src_me->def; @@ -812,7 +813,7 @@ method_entry_get(VALUE klass, ID id, VALUE *defined_class_ptr) return method_entry_get_without_cache(klass, id, defined_class_ptr); } -const rb_method_entry_t * +MJIT_FUNC_EXPORTED const rb_method_entry_t * rb_method_entry(VALUE klass, ID id) { return method_entry_get(klass, id, NULL); @@ -853,7 +854,7 @@ prepare_callable_method_entry(VALUE defined_class, ID id, const rb_method_entry_ return cme; } -const rb_callable_method_entry_t * +MJIT_FUNC_EXPORTED const rb_callable_method_entry_t * rb_callable_method_entry(VALUE klass, ID id) { VALUE defined_class; @@ -886,7 +887,7 @@ method_entry_resolve_refinement(VALUE klass, ID id, int with_refinement, VALUE * return me; } -const rb_callable_method_entry_t * +MJIT_FUNC_EXPORTED const rb_callable_method_entry_t * rb_callable_method_entry_with_refinements(VALUE klass, ID id, VALUE *defined_class_ptr) { VALUE defined_class, *dcp = defined_class_ptr ? defined_class_ptr : &defined_class; @@ -900,7 +901,7 @@ rb_method_entry_without_refinements(VALUE klass, ID id, VALUE *defined_class_ptr return method_entry_resolve_refinement(klass, id, FALSE, defined_class_ptr); } -const rb_callable_method_entry_t * +MJIT_FUNC_EXPORTED const rb_callable_method_entry_t * rb_callable_method_entry_without_refinements(VALUE klass, ID id, VALUE *defined_class_ptr) { VALUE defined_class, *dcp = defined_class_ptr ? defined_class_ptr : &defined_class; @@ -1462,7 +1463,7 @@ original_method_definition(const rb_method_definition_t *def) return def; } -static int +MJIT_FUNC_EXPORTED int rb_method_definition_eq(const rb_method_definition_t *d1, const rb_method_definition_t *d2) { d1 = original_method_definition(d1); diff --git a/vm_trace.c b/vm_trace.c index 5734dba3e8..3b4f2231dc 100644 --- a/vm_trace.c +++ b/vm_trace.c @@ -325,7 +325,7 @@ exec_hooks_protected(rb_execution_context_t *ec, rb_vm_t *vm, rb_hook_list_t *li return state; } -void +MJIT_FUNC_EXPORTED void rb_exec_event_hooks(rb_trace_arg_t *trace_arg, int pop_p) { rb_execution_context_t *ec = trace_arg->ec; diff --git a/win32/Makefile.sub b/win32/Makefile.sub index d42625c692..3013fbe576 100644 --- a/win32/Makefile.sub +++ b/win32/Makefile.sub @@ -1192,7 +1192,7 @@ rb_mjit_header.h: PHONY probes.h $(Q) $(IFCHANGE) $@ vm.i INSNS = opt_sc.inc optinsn.inc optunifs.inc insns.inc insns_info.inc \ - vmtc.inc vm.inc + vmtc.inc vm.inc mjit_compile.inc !if [exit > insns_rules.mk] !else if [for %I in ($(INSNS)) do \ diff --git a/win32/mkexports.rb b/win32/mkexports.rb index 7672ac307c..887b5d909b 100755 --- a/win32/mkexports.rb +++ b/win32/mkexports.rb @@ -7,7 +7,7 @@ module RbConfig end class Exports - PrivateNames = /(?:Init_|ruby_static_id_|.*_threadptr_|rb_ec_|DllMain\b)/ + PrivateNames = /(?:Init_|ruby_static_id_|DllMain\b)/ @@subclass = [] def self.inherited(klass) |