aboutsummaryrefslogtreecommitdiffstats
path: root/enc/trans
Commit message (Collapse)AuthorAgeFilesLines
* * enc/trans/big5.trans, big5-hkscs-tbl.rb:duerst2009-07-242-0/+18389
| | | | | | | | | | | | | | | | new Chinese BIG5-HKSCS transcoding (with Tatsuya Mizuno) * test/ruby/test_transcode.rb: added tests for the above (with Tatsuya Mizuno) * enc/big5.c: Added BIG5-HKSCS as a replicate encoding of BIG5 (short term solution, needs more work; with Tatsuya Mizuno) * tool/transcode-tblgen.rb: made 'pat' directly accessible in class StrSet git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24264 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/utf8_mac.trans: remove wrong optimization.naruse2009-06-131-9/+0
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23686 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Fix: DON'T move in_p because before in_p is replaced by buffered data.naruse2009-05-051-7/+7
| | | | | | | | | | | | | | | | * transcode.c: NOMAP is now multibyte direct map. * transcode.c: remove ASIS. * transcode_data.h: ditto. * tool/transcode-tb (ActionMap#generate_info): remove :asis. * tool/transcode-tb (ActionMap#generate_info): add :nomap0. * enc/trans/utf8_mac.trans: replace :asis by :nomap0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23344 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/utf8_mac-tbl.rb: don't use Unicode escape.naruse2009-05-022-955/+949
| | | | | | * enc/trans/utf8_mac.trans: follow above. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23325 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/utf8_mac.trans: get rid of a 1.9 feature for crossnobu2009-04-301-2/+2
| | | | | | | compile. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23309 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Add new transcoder: CP51932 <-> CP50221.naruse2009-04-291-0/+183
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23307 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/utf8_mac.trans: Add converter for UTF8-MAC.naruse2009-04-262-0/+1202
| | | | | | | | * enc/trans/utf8_mac-tbl.rb: ditto. * test/ruby/test_econv.rb: tests for above. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23296 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/gb18030.trans: get rid of a 1.9 feature for crossnobu2009-01-141-5/+4
| | | | | | | compile. [ruby-core:21345] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21512 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/gb18030.trans, gb18030-tbl.rb:duerst2009-01-142-0/+63415
| | | | | | | | | | | | | | | new Chinese GB18030 transcoding (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) * transcode_data.h, transcode.c, tool/transcode_tblgen.rb: added support for GB18030-specific 4-byte sequences (with Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * template/{encdb,transdb}.h.tmpl: moved enc/make_encdb.rb andnobu2009-01-131-61/+0
| | | | | | | | | enc/trans/make_transdb.rb using tool/generic_erb.rb. * common.mk (encdb.h, transdb.h): generates from avobe template. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21490 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb (converters): should not depend on thenobu2009-01-131-1/+5
| | | | | | | hash order for cross compile. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21489 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/gbk.trans, gbk-tbl.rb:duerst2009-01-042-0/+21809
| | | | | | | | | | new Chinese GBK transcoding (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21315 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * test/ruby/test_transcode.rb: added tests for GB2312duerst2009-01-041-0/+3
| | | | | | | | | | (from Yoshihiro Kambayashi) * enc/trans/chinese.trans: set valid byte patterns for GB2312 and GB12345 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21314 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/big5.trans, big5-tbl.rb:duerst2009-01-042-0/+13721
| | | | | | | | | | new Chinese Big5 transcoding (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21313 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* change encoding name.naruse2009-01-031-2/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21285 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/chinese.trans: added for transcoding EUC-CN and GB12345.naruse2009-01-035-0/+30331
| | | | | | * enc/trans/GB/: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21283 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans, cp850-tbl.rb, cp852-tbl.rb,duerst2008-12-097-0/+745
| | | | | | | | | | | cp855-tbl.rb, koi8-r-tbl.rb, koi8-u-tbl.rb, tis-620-tbl.rb: new single-byte transcodings (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi), small cosmetic fixes git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20599 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans, macgreek-tbl.rb, macroman-tbl.rb,duerst2008-11-1118-0/+2210
| | | | | | | | | | | | | | macromania-tbl.rb, macturkish-tbl.rb, macukraine-tbl.rb, ibm437-tbl.rb, ibm852-tbl.rb, ibm855-tbl.rb, ibm857-tbl.rb, ibm860-tbl.rb, ibm861-tbl.rb, ibm862-tbl.rb, ibm863-tbl.rb, ibm865-tbl.rb, ibm866-tbl.rb, ibm869-tbl.rb, ibm775-tbl.rb: new single-byte transcodings (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20178 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans, maccroatioan-tbl.rb,duerst2008-10-314-0/+391
| | | | | | | | | | | maccyrillic-tbl.rb, maciceland-tbl.rb: new single-byte transcodings (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20075 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans: refactoring to make it easierduerst2008-10-302-54/+135
| | | | | | | | | | to add more transcodings (with Yoshihiro Kambayashi) * enc/trans/iso-8859-1-tbl.rb: new file to avoid having to treat ISO-8859-1 as special git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20054 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans: adding WINDOWS-wwww encodingsduerst2008-10-199-0/+994
| | | | | | | | | | | | | | (wwww = 874/1250/1251/1253/1254/1255/1256/1257) (contributed by Yoshihiro Kambayashi) * enc/trans/windows-wwww-tbl.rb: 8 new files (contributed by Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added test_windows_wwww (contributed by Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19846 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * tool/transcode-tblgen.rb: added set_valid_byte_patternduerst2008-10-181-0/+1
| | | | | | | | | | to reduce coupling between table generation script and specific encodings. * enc/trans/single_byte.trans: using set_valid_byte_pattern git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans (transcode_tblgen_singlebyte): renamedakr2008-10-141-11/+4
| | | | | | | | from transcode_tblgen_windows. (transcode_tblgen_iso8859): use transcode_tblgen_singlebyte. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans: added windows-1252duerst2008-10-142-0/+136
| | | | | | | | | | | | | * enc/trans/windows-1252-tbl.rb: new file (contributed by Yoshihiro Kambayashi) * tool/transcode-tblgen.rb: listed windows-1252 as '1byte' * test/ruby/test_transcode.rb: added test_windows_1252 (contributed by Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19778 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoder): resetsize_func and resetstate_funcakr2008-09-151-1/+1
| | | | | | | | | | also returns ssize_t. * enc/trans/iso2022.trans: follow the type change. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19354 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h: return output functions ssize_t.akr2008-09-145-19/+19
| | | | | | | | | | | | | | | | | | | * transcode.c (transcode_restartable0): don't need to cast the result of output functions. * enc/trans/newline.trans: follow the type change. * enc/trans/escape.trans: ditto. * enc/trans/utf_16_32.trans: ditto. * enc/trans/iso2022.trans: ditto. * enc/trans/japanese.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19351 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h: output function takes output buffer size.akr2008-09-145-19/+19
| | | | | | | | | | | | | | | | | | * transcode.c: give output buffer size for output functions. * enc/trans/newline.trans: follow the type change. * enc/trans/escape.trans: ditto. * enc/trans/utf_16_32.trans: ditto. * enc/trans/iso2022.trans: ditto. * enc/trans/japanese.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19350 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans: transcoder name renamed to use underscore.akr2008-09-121-4/+4
| | | | | | | | * transcode.c: follow the renaming. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans (rb_universal_newline): swap src_encodingakr2008-09-091-2/+1
| | | | | | | | | | | | and dst_encoding. * transcode.c (rb_econv_decorate_at): call get_transcoder_entry only once. (rb_econv_binmode): follow universal_newline change. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19276 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (rb_econv_asciicompat_encoding): renamedakr2008-09-085-18/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | from rb_econv_stateless_encoding to apply stateless ASCII incompatible encodings such as UTF-16BE. * io.c (make_writeconv): use rb_econv_asciicompat_encoding. * transcode_data.h (rb_transcoder_asciicompat_type_t): renamed from rb_transcoder_stateful_type_t. (rb_transcoder): use rb_transcoder_asciicompat_type_t. * transcode.c: follow the type change. (asciicompat_encoding_i): renamed from stateless_encoding_i. (rb_econv_asciicompat_encoding): renamed from rb_econv_stateless_encoding. (econv_s_asciicompat_encoding): method renamed. * tool/transcode-tblgen.rb: follow the type change. * enc/trans/utf_16_32.trans: follow the type change. rb_from_UTF_16BE to UTF-8 is asciicompat_decoder. rb_from_UTF_16LE to UTF-8 is asciicompat_decoder. rb_from_UTF_32BE to UTF-8 is asciicompat_decoder. rb_from_UTF_32LE to UTF-8 is asciicompat_decoder. UTF-8 to rb_to_UTF_16BE is asciicompat_encoder. UTF-8 to rb_to_UTF_16LE is asciicompat_encoder. UTF-8 to rb_to_UTF_32BE is asciicompat_encoder. UTF-8 to rb_to_UTF_32LE is asciicompat_encoder. * enc/trans/newline.trans: follow the type change. universal newline decoder is asciicompat_converter. * enc/trans/escape.trans: follow the type change. * enc/trans/iso2022.trans: ditto. * enc/trans/japanese.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19249 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/iso2022.trans: upcase to iso-2022-jp.akr2008-09-071-4/+4
| | | | | | | | * enc/emacs_mule.c: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19221 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/iso2022.trans: stateless-iso-2022-jp is defined to avoidakr2008-09-071-40/+122
| | | | | | | | | | | undefined conversion error between iso-2022-jp and the corresponding stateless encoding. * enc/emacs_mule.c: replicate emacs-mule as stateless-iso-2022-jp. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19220 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans (hexstr): renamed from str1.akr2008-09-071-9/+9
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19218 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans: use transcode_tblgen.akr2008-09-071-58/+20
| | | | | | | | * tool/transcode-tblgen.rb: generate an empty line after str1. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19217 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (STR1): defined for a string up to 255 bytes.akr2008-09-071-51/+15
| | | | | | | | | | | | | | | | (STR1_BYTEINDEX): defined. (makeSTR1): defined. * tool/transcode-tblgen.rb: generate STR1. * transcode.c (transcode_restartable0): interpret STR1. * enc/trans/escape.trans (fun_so_escape_xml_chref): removed. STR1 is used instead. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19214 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (ECONV_XML_ATTR_CONTENT_ENCODER): defined.akr2008-09-071-24/+35
| | | | | | | | | | | | | | | | | | | | (ECONV_STATEFUL_ENCODER_MASK): defined. (ECONV_XML_ATTR_QUOTE_ENCODER): defined. (ECONV_XML_ATTR_ENCODER): removed. * enc/trans/escape.trans (rb_escape_xml_attr_content): defined. (rb_escape_xml_attr_quote): defined. (rb_escape_xml_attr): removed. * io.c (NEED_WRITECONV): writeconv is required if supplemental converter is used. (make_writeconv): apply stateful encoder in writeconv. * transcode.c: follow the constant change. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19209 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (ECONV_XML_TEXT_ENCODER): renamed fromakr2008-09-061-25/+25
| | | | | | | | | | | | | ECONV_HTML_TEXT_ENCODER. (ECONV_XML_ATTR_ENCODER): renamed from ECONV_HTML_ATTR_ENCODER. * enc/trans/escape.trans: follow the renaming. * transcode.c: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19191 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans (fun_so_escape_html_attr): fix return type.akr2008-09-061-1/+1
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19177 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans (escape_html_attr_init): new function.akr2008-09-061-9/+57
| | | | | | | | | | (fun_so_escape_html_attr): new function. (escape_html_attr_finish): new function. (rb_escape_html_attr): use them to quote the converted result. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19173 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans: new file.akr2008-09-051-0/+109
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19165 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans (universal_newline_finish): new function.akr2008-09-051-2/+13
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19152 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans: record newline types met in universalakr2008-09-041-12/+27
| | | | | | | newline decoder. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19136 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): remove stateful field.akr2008-09-034-45/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | add state field. (TRANSCODING_STATE): defined. (rb_transcoder): add fields: state_size, state_init_func, state_fini_func. change rb_transcoding* argument to void*. * transcode.c (transcode_restartable0): use TRANSCODING_STATE for first arguments of transcoder functions. (rb_transcoding_open_by_transcoder): initialize state field. (rb_transcoding_close): finalize state field. * tool/transcode-tblgen.rb: provide state size/init/fini. * enc/trans/newline.trans (universal_newline_init): defined. (fun_so_universal_newline): take void* as a state pointer. (rb_universal_newline): provide state size/init/fini. (rb_crlf_newline): ditto. (rb_cr_newline): ditto. * enc/trans/iso2022.trans (iso2022jp_init): defined. (fun_si_iso2022jp_to_eucjp): take void* as a state pointer. (fun_so_iso2022jp_to_eucjp): ditto. (fun_so_eucjp_to_iso2022jp): ditto. (iso2022jp_reset_sequence_size): ditto. (finish_eucjp_to_iso2022jp): ditto. (rb_ISO_2022_JP_to_EUC_JP): provide state size/init/fini. (rb_EUC_JP_to_ISO_2022_JP): ditto. * enc/trans/utf_16_32.trans (fun_so_from_utf_16be): take void* as a state pointer. (fun_so_to_utf_16be): ditto. (fun_so_from_utf_16le): ditto. (fun_so_to_utf_16le): ditto. (fun_so_from_utf_32be): ditto. (fun_so_to_utf_32be): ditto. (fun_so_from_utf_32le): ditto. (fun_so_to_utf_32le): ditto. (rb_from_UTF_16BE): provide state size/init/fini. (rb_to_UTF_16BE): ditto. (rb_from_UTF_16LE): ditto. (rb_to_UTF_16LE): ditto. (rb_from_UTF_32BE): ditto. (rb_to_UTF_32BE): ditto. (rb_from_UTF_32LE): ditto. (rb_to_UTF_32LE): ditto. * enc/trans/japanese.trans (fun_so_eucjp2sjis): take void* as a state pointer. (fun_so_sjis2eucjp): ditto. (rb_eucjp2sjis): provide state size/init/fini. (rb_sjis2eucjp): provide state size/init/fini. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19096 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/japanese.trans: new file.akr2008-09-031-0/+96
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: check $(srcdir)/enc/trans beforenaruse2008-09-031-2/+3
| | | | | | | | enc/trans. * enc/trans/make_transdb.rb: keep names_t. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19081 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: check foo.c only if foo.trans exists.akr2008-09-021-2/+15
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19068 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: error message improved.akr2008-09-021-4/+5
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19067 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* revert last commit.akr2008-09-021-5/+4
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19066 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: error message improved.akr2008-09-021-4/+5
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19065 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/utf_16_32.trans (from_UTF_8): rename from to_UTF_16BEusa2008-09-021-15/+15
| | | | | | | | because it was not collect. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19063 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-27 12:31:00 +0000