| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
add state field.
(TRANSCODING_STATE): defined.
(rb_transcoder): add fields: state_size, state_init_func,
state_fini_func.
change rb_transcoding* argument to void*.
* transcode.c (transcode_restartable0): use TRANSCODING_STATE for
first arguments of transcoder functions.
(rb_transcoding_open_by_transcoder): initialize state field.
(rb_transcoding_close): finalize state field.
* tool/transcode-tblgen.rb: provide state size/init/fini.
* enc/trans/newline.trans (universal_newline_init): defined.
(fun_so_universal_newline): take void* as a state pointer.
(rb_universal_newline): provide state size/init/fini.
(rb_crlf_newline): ditto.
(rb_cr_newline): ditto.
* enc/trans/iso2022.trans (iso2022jp_init): defined.
(fun_si_iso2022jp_to_eucjp): take void* as a state pointer.
(fun_so_iso2022jp_to_eucjp): ditto.
(fun_so_eucjp_to_iso2022jp): ditto.
(iso2022jp_reset_sequence_size): ditto.
(finish_eucjp_to_iso2022jp): ditto.
(rb_ISO_2022_JP_to_EUC_JP): provide state size/init/fini.
(rb_EUC_JP_to_ISO_2022_JP): ditto.
* enc/trans/utf_16_32.trans (fun_so_from_utf_16be): take void* as a
state pointer.
(fun_so_to_utf_16be): ditto.
(fun_so_from_utf_16le): ditto.
(fun_so_to_utf_16le): ditto.
(fun_so_from_utf_32be): ditto.
(fun_so_to_utf_32be): ditto.
(fun_so_from_utf_32le): ditto.
(fun_so_to_utf_32le): ditto.
(rb_from_UTF_16BE): provide state size/init/fini.
(rb_to_UTF_16BE): ditto.
(rb_from_UTF_16LE): ditto.
(rb_to_UTF_16LE): ditto.
(rb_from_UTF_32BE): ditto.
(rb_to_UTF_32BE): ditto.
(rb_from_UTF_32LE): ditto.
(rb_to_UTF_32LE): ditto.
* enc/trans/japanese.trans (fun_so_eucjp2sjis): take void* as a state
pointer.
(fun_so_sjis2eucjp): ditto.
(rb_eucjp2sjis): provide state size/init/fini.
(rb_sjis2eucjp): provide state size/init/fini.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19096 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(WORDINDEX2INFO): defined.
(INFO2WORDINDEX): defined.
* tool/transcode-tblgen.rb: use WORDINDEX2INFO.
* transcode.c: use INFO2WORDINDEX.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19093 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
| |
word_array_length.
* tool/transcode-tblgen.rb (transcode_generated_code): generate
byte_array_length and word_array_length.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19088 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
| |
(ActionMap#gen_array_code): moved to ArrayCode.
(ActionMap#numelt_array_code): ditto.
(ActionMap#array_code_insert_at_last): ditto.
(TRANSCODE_GENERATED_BYTES_CODE): use ArrayCode.
(TRANSCODE_GENERATED_WORDS_CODE): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
| |
generate_lookup_node.
(ActionMap#numelt_array_code): ditto.
(ActionMap#array_code_insert_at_last): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(BYTE_LOOKUP): removed.
(BYTE_LOOKUP_BASE): don't cast.
(BYTE_LOOKUP_INFO): ditto.
(PType): unsigned int, instead of uintptr_t.
(rb_transcoding): change type of next_field, conv_tree_start and
word_array.
* tool/transcode-tblgen.rb: generate word_array as array of unsigned
int.
* transcode.c (transcode_restartable0): follow the above type change.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19070 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19047 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
code. use it in rb_transcoder.
* enc/trans/newline.trans: use TRANSCODE_TABLE_INFO.
* enc/trans/iso2022.trans: ditto.
* enc/trans/utf_16_32.trans: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
word_array to avoid relocation.
* transcode.c (transcode_restartable0): add word_array to get infos
and BYTE_LOOKUPs.
* transcode_data.h (BYTE_LOOKUP_INFO): change return type to
uintptr_t.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19044 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
byte_array to avoid relocation.
* transcode.c (transcode_restartable0): add byte_array to get offsets
array.
* transcode_data.h (BYTE_LOOKUP_BASE): change return type to
uintptr_t.
(rb_transcoder): add fields: byte_array, word_array and word_size.
* enc/trans/newline.trans: follow rb_transcoder change.
* enc/trans/iso2022.trans: ditto.
* enc/trans/utf_16_32.trans: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
| |
single array.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(BYTE_LOOKUP_BASE): follow the type change.
(BYTE_LOOKUP_INFO): ditto.
(PType): ditto.
(rb_transcoding): ditto.
* tool/transcode-tblgen.rb: follow the type change.
* transcode.c: ditto.
* enc/trans/newline.trans: ditto.
* enc/trans/iso2022.trans: ditto.
* enc/trans/utf_16_32.trans: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19036 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19032 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19031 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
| |
"\xA2\xAF".encode("utf-8", "eucJP-ms") should raise
Encoding::ConversionUndefined, not Encoding::InvalidByteSequence.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19029 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
| |
mapping when some mappings are given for a character.
[ruby-dev:36068]
* tool/transcode-tblgen.rb: expandtab.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19018 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19016 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19007 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
generating table at once.
(transcode_tblgen): returns an empty string.
(transcode_generate_node): ditto.
* enc/trans/newline.trans: use transcode_generated_code.
* enc/trans/iso2022.trans: ditto.
* enc/trans/single_byte.trans: ditto.
* enc/trans/utf_16_32.trans: ditto.
* enc/trans/japanese.trans: ditto.
* enc/trans/korean.trans: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19006 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
| |
message on STDERR.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19005 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* enc/trans/japanese.trans: use Citrus maps.
* enc/trans/CP: add maps from Citrus.
* enc/trans/JIS: ditto.
* test/ruby/test_transcode.rb: Shift_JIS and EUC-JP doesn't support
IBM extended characters.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19003 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
| |
* tool/make-snapshot (package): supported release numbers 1.9.0-x.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18857 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(rb_transcoder): add field: stateful_type.
* tool/transcode-tblgen.rb: generate stateful_type field as
stateless_converter.
* enc/trans/iso2022.trans: follow rb_transcoder change.
* enc/trans/newline.trans: ditto.
* enc/trans/utf_16_32.trans: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18650 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* transcode_data.h (rb_transcoder): add resetsize_func field.
* enc/trans/iso2022.trans (iso2022jp_reset_sequence_size): defined.
(rb_EUC_JP_to_ISO_2022_JP): provede resetsize_func.
* tool/transcode-tblgen.rb: set NULL for resetsize_func.
* transcode.c (rb_econv_output): new function for inserting output.
(output_replacement_character): use rb_econv_output.
(transcode_loop): check return value of
output_replacement_character.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18628 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18619 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18553 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18550 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18542 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
| |
improvement.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18529 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
| |
* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid
(but unique to ASCII-8BIT) character.
* transcode.c (rb_eConversionUndefined): new error.
(rb_eInvalidByteSequence): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
resetting a state of stateful encoding.
* enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify
finish_eucjp_to_iso2022jp for resetstate_func.
* tool/transcode-tblgen.rb: specify NULL for resetstate_func.
* transcode.c (output_replacement_character): call resetstate_func
before appending the replacement character.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* enc/trans/single_byte.trans: ditto.
* enc/trans/utf_16_32.trans: ditto.
* enc/trans/korean.trans: ditto.
* enc/trans/japanese.trans: ditto.
* enc/depend: follow the renaming.
* tool/build-transcode: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18488 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
| |
transcode-tblgen.rb itself.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18473 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
| |
* enc/depend: makes target directory before compile/link.
* tool/transcode-tblgen.rb: creates target directory.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18469 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
transcoding.
(rb_transcoder): add max_input field.
from_unit_length field is renamed to input_unit_length.
* tool/transcode-tblgen.rb: generate max_input field.
* enc/trans/iso2022.erb.c: follow rb_transcoder change.
* enc/trans/utf_16_32.erb.c: ditto.
* transcode.c (PARTIAL_INPUT): new constant.
(transcode_char_start): new function.
(transcode_result_t): new type.
(transcode_restartable): new function.
(more_output_buffer): new function.
(transcode_loop): use transcode_restartable.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18452 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
from_utf8 field removed.
* tool/transcode-tblgen.rb: generate offsets range.
follow rb_transcoder change.
* transcode.c (transcode_loop): don't use from_utf8.
make invalid region from_unit_length wise.
* enc/trans/iso2022.erb.c: follow rb_transcoder and
transcode_generate_node change.
* enc/trans/utf_16_32.erb.c: follow rb_transcoder and
transcode_generate_node change.
explicit :invalid map removed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(rb_transcoder): preprocessor and postprocessor field removed.
change arguments of func_ii, func_si, func_io and func_so.
new field "finish_func".
* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
generatable.
* transcode.c (transcoder_lib_table): removed.
(transcoder_table): change structure.
(transcoder_key): removed because the above structure change.
(make_transcoder_entry): new function.
(get_transcoder_entry): ditto.
(rb_register_transcoder): follow the structure change.
(declare_transcoder): ditto.
(transcode_search_path): new function for breadth first search to
find a list of converters.
(transcode_search_path_i): new function.
(transcode_dispatch_cb): ditto.
(transcode_dispatch): use transcode_search_path.
(transcode_loop): follow the argument change.
(str_transcode): preprocessor and postprocessor stuff removed.
* enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion
re-implemented.
* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.
nute(23:52:53)% head -40 ChangeLog
Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org>
* transcode_data.h (rb_transcoding): new field "stateful".
(rb_transcoder): preprocessor and postprocessor field removed.
change arguments of func_ii, func_si, func_io and func_so.
new field "finish_func".
* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
generatable.
* transcode.c (transcoder_lib_table): removed.
(transcoder_table): change structure.
(transcoder_key): removed because the above structure change.
(make_transcoder_entry): new function.
(get_transcoder_entry): ditto.
(rb_register_transcoder): follow the structure change.
(declare_transcoder): ditto.
(transcode_search_path): new function for breadth first search to
find a list of converters.
(transcode_search_path_i): new function.
(transcode_dispatch_cb): ditto.
(transcode_dispatch): use transcode_search_path.
(transcode_loop): follow the argument change.
(str_transcode): preprocessor and postprocessor stuff removed.
* enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion
re-implemented.
* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.
* enc/trans/utf_16_32.erb.c: follow argument change of FUNso.
[ruby-dev:35798]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
| |
argument removed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18395 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
| |
[ruby-dev:35709]
* transcode.c (transcode_loop): don't need rb_enc_mbclen now.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18383 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
| |
(transcode_generate_node): call ActionMap#generate_node with showing
table name.
* enc/trans/utf_16_32.erb.c: use transcode_generate_node.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18382 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
| |
sources.
* enc/trans/{japanese,korean,single_byte,utf_16_32}.c: to be
autogenerated now.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18376 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* tool/transcode-tblgen.rb: new file.
* enc/trans/make_transdb.rb: exclude *.erb.c.
* enc/depend: exclude *.erb.c.
* enc/trans/utf_16_32.erb.c: new file.
* enc/trans/single_byte.erb.c: new file.
* enc/trans/japanese.erb.c: new file.
* enc/trans/korean.erb.c: new file.
* enc/trans/iso-8859-2-tbl.rb: new file.
* enc/trans/iso-8859-3-tbl.rb: new file.
* enc/trans/iso-8859-4-tbl.rb: new file.
* enc/trans/iso-8859-5-tbl.rb: new file.
* enc/trans/iso-8859-6-tbl.rb: new file.
* enc/trans/iso-8859-7-tbl.rb: new file.
* enc/trans/iso-8859-8-tbl.rb: new file.
* enc/trans/iso-8859-9-tbl.rb: new file.
* enc/trans/iso-8859-10-tbl.rb: new file.
* enc/trans/iso-8859-11-tbl.rb: new file.
* enc/trans/iso-8859-13-tbl.rb: new file.
* enc/trans/iso-8859-14-tbl.rb: new file.
* enc/trans/iso-8859-15-tbl.rb: new file.
* enc/trans/eucjp-tbl.rb: new file.
* enc/trans/sjis-tbl.rb: new file.
* enc/trans/euckr-tbl.rb: new file.
* enc/trans/utf_16_32.c: regenerated.
* enc/trans/single_byte.c: regenerated.
* enc/trans/japanese.c: regenerated.
* enc/trans/korean.c: regenerated.
[ruby-dev:35730]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18373 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17776 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
|
|
|
|
|
|
|
|
| |
(undef, alias, definemethod).
Call RubyVM::FrozenCore's singleton method instead.
Add "putiseq" and "putspecialobject" instructions.
* id.c, id.h: add ids for above.
* tool/parse.rb: "VM" no longer exists. Use RubyVM instead.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17755 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17495 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17494 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|