aboutsummaryrefslogtreecommitdiffstats
path: root/enc
Commit message (Collapse)AuthorAgeFilesLines
* Embed the Emoji versionnobu2018-11-271-0/+9
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66023 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* deal with ONIGENC_CASE_IS_TITLECASE flag on lowercase charactersduerst2018-11-251-4/+9
| | | | | | | | | In the function onigenc_unicode_case_map() in enc/unicode.c, deal with the case that the ONIGENC_CASE_IS_TITLECASE flag is set on lowercase characters. This is in preparation for Georgian Mtavruli, which are uppercase but not titlecase, in Unicode 11.0.0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65971 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* prepare for Unicode 11.0.0 updateduerst2018-11-231-4/+3
| | | | | | | | | - enc/unicode/case-folding.rb: - Convert unpredicted case to actual flag setting - Eliminate an unused variable - Change a variable name to avoid a warning git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65933 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Make some internal functions staticnobu2018-11-161-2/+4
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65764 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/unicode.c: 'a' is bigger than 'A'shyouhei2018-11-161-2/+4
| | | | | | | | | | | | | | | | | | | | | In ASCII, 'a' is bigger than 'A'. Which means 'A' - 'a' is a negative number (-32, to be precise). In C, the type of 'a' and 'A' are signed int (cf: ISO/IEC 9899:1990 section 6.1.3.4). So 'A' - 'a' is also a signed int. It is `(signed int)-32`. The problem is, OnigCodePoint is unsigned int. Adding a negative number to a variable of OnigCodepoint (`code` here) introduces an unintentional cast of `(unsigned)(signed)-32`, which is 4,294,967,264. Adding this value to code then overflows, and the result eventually becomes normal codepoint. The series of operations are not a serious problem but because `code >= 'a'` holds, we can `(code - 'a') + 'A'` to reroute this. See also: https://github.com/k-takata/Onigmo/pull/107 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65752 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* revert r65091, r65090 because ci failsduerst2018-10-164-47339/+14
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65093 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update to Unicode 11.0.0 (basic step, not complete yet)duerst2018-10-164-5/+47330
| | | | | | | | | | | | | | | - common.mk: Change Unicode version to 11.0.0 - enc/unicode/case-folding.rb, enc/unicode.c: Initial changes to deal with Gregorian Mtavruli. This should bring us up to the same level as e.g. Python 3.7, by following the Unicode tables exactly. But it will produce undesirable (mixed-case) results for String#capitalize. This will be addressed in a later commit. - enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h: Add generated files. - lib/unicode_normalize/tables.rb: Updated table. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65091 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* add some comments to enc/unicode/case-folding.rb [ci skip]duerst2018-10-161-9/+9
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65090 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Removed data for old Unicode [ci skip]nobu2018-10-161-28/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65088 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode.c: moved addtional GCB rangesnobu2018-10-151-0/+52
| | | | | | | * enc/unicode.c: moved additional Grapheme Cluster Break ranges which depend on the Unicode version. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* regparse.c: Suppress duplicated range warning by mere \Xnobu2018-10-151-2/+0
| | | | | | | | * regparse.c (node_extended_grapheme_cluster): as Unicode 10 has added Grapheme_Cluster_Break properties to some characters, remove duplicated ranges for Unicode 9. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby tool/update-deps --fixnaruse2018-07-051-0/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63860 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Refactor ERB version checking for keyword argumentsk0kubun2018-02-271-2/+2
| | | | | | Improving code like r62590. See r62529 for details. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62594 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Judge ERB version not RUBY_VERSION but ERB.versionnaruse2018-02-241-2/+2
| | | | | | On cross compilation, ruby command uses fake RUBY_VERSION. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62560 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* erb.rb: deprecate safe_level of ERB.newk0kubun2018-02-221-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Also, as it's in the middle of the list of 4 arguments, 3rd and 4th arguments (trim_mode, eoutvar) are changed to keyword arguments. Old ways to specify arguments are deprecated and warned now. bin/erb: deprecate -S option. We'll remove all of deprecated ones at Ruby 2.7+. enc/make_encmake.rb: stopped using deprecated interface ext/etc/mkconstants.rb: ditto ext/socket/mkconstants.rb: ditto sample/ripper/ruby2html.rb: ditto spec/ruby/library/erb/defmethod/def_erb_method_spec.rb: ditto spec/ruby/library/erb/new_spec.rb: ditto test/erb/test_erb.rb: ditto test/erb/test_erb_command.rb: ditto tool/generic_erb.rb: ditto tool/ruby_vm/helpers/dumper.rb: ditto tool/transcode-tblgen.rb: ditto lib/rdoc/erbio.rb: ditto lib/rdoc/generator/darkfish.rb: ditto [Feature #14256] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62529 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update dependenciesnobu2018-01-131-2/+0
| | | | | | | | | | | | | * common.mk: enc/unicode.$(OBJEXT) depends on onigmo.h via oniguruma.h. * common.mk: dependencies of *prelude.$(OBJEXT) are defined for each generated C sources. * enc/depend: casefold.h and name2ctype.h are located under $(UNICODE_HDR_DIR). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61800 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update dependencies using `tool/update-deps`kazu2018-01-131-0/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61799 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update dependenciesnobu2018-01-091-0/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61715 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* gperf.sed: static declarationsnobu2017-12-153-3/+3
| | | | | | | * tool/gperf.sed: comment out arguments part only, to keep the following declarations static. [Feature #13883] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61282 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* support gperf 3.1nobu2017-12-084-42/+34
| | | | | | | | | | * tool/gperf.sed: extracted sed commands to a script. ANSI-C code produced by gperf 3.1 declares length arguments as `size_t`. it causes conflict with existing declarations, and needs casts for a local variable and return statements. [Feature #13883] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61076 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* fix for emoji-data.txtnobu2017-12-022-385/+653
| | | | | | | | | | | * common.mk: download emoji-data.txt. As emoji data files are located in a separate directory in Unicode.org site, reearranged Unicode data files directories same as the site. * tool/enc-unicode.rb (get_file): search emoji data files in the second argument path. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60977 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Add missing filenaruse2017-12-011-0/+546
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60967 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update to Onigmo 6.1.3-669ac9997619954c298da971fcfacccf36909d05.naruse2017-12-012-523/+6
| | | | | | [Bug #13892] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60966 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* remove Unicode 9.0.0-related filesduerst2017-09-062-43881/+0
| | | | | | We don't need these files anymore because we upgraded to Unicode 10.0.0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59760 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update Ruby to Unicode 10.0.0duerst2017-09-062-0/+45155
| | | | | | | | | - In common.mk, set UNICODE_VERSION to 10.0.0 - Generate and add enc/unicode/10.0.0/casefold.h and enc/unicode/10.0.0/name2ctype.h - Update lib/unicode_normalize/tables.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* replace copyrights by explanatory text in data files for GB2312/GB12345 mappingsduerst2017-09-014-252/+20
| | | | | | | Replace the copyrights and explanatory texts in the data files used for mapping GB2312/GB12345 to/from Unicode with short explanatory texts. [Bug #12598] [ci skip] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59715 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Change max byte length of UTF-8 to 4 bytesduerst2017-05-301-1/+1
| | | | | | | | In enc/utf_8.c, change maximum byte length of UTF-8 to 4 bytes (from 6) to conform to definition of UTF-8. This closes issue #13590. (This is a retry of r58954, after issue #13590 has been addressed.) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58965 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* revert r58954 temporarilyduerst2017-05-291-1/+1
| | | | | | | | Revert change to maximum of 4 bytes for UTF-8 characters at r58954 temporarily. This failed spec at https://travis-ci.org/ruby/ruby/builds/237086017, but it is totally unclear why. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58955 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Change max byte length of UTF-8 to 4 bytesduerst2017-05-291-1/+1
| | | | | | | In enc/utf_8.c, change maximum byte length of UTF-8 to 4 bytes (from 6) to conform to definition of UTF-8. This closes issue #13590. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58954 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* delete enc/prelude.rb, because no longer neededduerst2017-05-061-4/+0
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58579 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* clean autogenerated filesnobu2017-04-212-229/+2
| | | | | | | | | | * enc/depend (clean, clean-srcs): fix path of name2ctype.h, and remove casefold.h too. * enc/jis/props.h: autogenerated file. [ruby-core:80823] [Bug #13493] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58438 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/depend: remove Unicode versionsnobu2017-04-181-2/+0
| | | | | | | * enc/depend (enc/unicode.o): remove hardcoded Unicode versions. this object file must be compiled by toplevel make. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58386 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* fix ext/-test-/struct/ dependenciesnormal2017-04-151-0/+2
| | | | | | | | | | | | | | I started writing a template for auto-generation and let "tool/update-deps --fix" fill in the rest. Hopefully this fixes problems with some CI builds after r58359. Further changes to other ext/-test-/ files should probably add or update "depend" files, too. * ext/-test-/struct/depend: new file * enc/depend: auto-updated with unicode 9.0.0 headers (side-effect) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58364 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc-unicode.rb: uniname2ctype_offsetnobu2017-03-231-786/+787
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58065 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update name2ctype.hnobu2017-03-231-806/+979
| | | | | | | * enc/unicode/9.0.0/name2ctype.h: update due to merger of Onigmo 6.0.0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58064 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby tool/update-deps --fixshyouhei2017-03-221-2/+64
| | | | | | | | | | | | | | Onigumo 6 (r57045) introduced new onigumo.h header file, which is required from quite much everywhere. This commit adds necessary dependencies. Note: ruby/oniguruma.h now includes onigumo.h, ruby/io.h includes oniguruma.h, ruby/encoding.h also includes oniguruma.h, and internal.h includes encoding.h. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58054 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* fix UTF-32 valid_encoding?nobu2017-03-092-6/+30
| | | | | | | | | | | | * enc/utf_32be.c (utf32be_mbc_enc_len): check arguments precisely. [ruby-core:79966] [Bug #13292] * enc/utf_32le.c (utf32le_mbc_enc_len): ditto. * regenc.h (UNICODE_VALID_CODEPOINT_P): predicate for valid Unicode codepoints. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57816 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Merge Onigmo 6.0.0naruse2016-12-1042-527/+593
| | | | | | | | | | | * https://github.com/k-takata/Onigmo/blob/Onigmo-6.0.0/HISTORY * fix for ruby 2.4: https://github.com/k-takata/Onigmo/pull/78 * suppress warning: https://github.com/k-takata/Onigmo/pull/79 * include/ruby/oniguruma.h: include onigmo.h. * template/encdb.h.tmpl: ignore duplicated definition of EUC-CN in enc/euc_kr.c. It is defined in enc/gb2313.c with CRuby macro. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* remove special processing for U+03B9/U+03BC/U+A64Bduerst2016-12-042-18/+5
| | | | | | | | | | | | | * enc/unicode.c: Remove special processing for U+03B9/U+03BC/U+A64B (GREEK SMALL LETTERs IOTA/MU, CYRILLIC SMALL LETTER MONOGRAPH UK) from onigenc_unicode_case_map and simplify code. * enc/unicode/case-folding.rb: Remove check for U+03B9/U+03BC/U+A64B. This and the previous few related commits make sure that we won't hit the equivalent of bug #12990 anymore for future updates of Unicode versions. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56976 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Reorder codepoints in some entries of CaseUnfold_11_Tableduerst2016-12-042-8/+17
| | | | | | | | | * enc/unicode/case-folding.rb: Reorder codepoints so that the upper-case mapping comes first. * enc/unicode/9.0.0/casefold.h: Codepoints reordered, upper-case mapping flag added. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56975 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Use offsetof macro and shrink table sizenobu2016-12-011-786/+786
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56952 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* constify CaseMappingSpecialsnobu2016-12-013-3/+3
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56951 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Regexp supports Unicoe 9.0.0's \Xnaruse2016-11-301-1889/+3139
| | | | | | | | | | | | | | | | | | | | | | | | | * meta character \X matches Unicode 9.0.0 characters with some workarounds for UTR #51 Unicode Emoji, Version 4.0 emoji zwj sequences. [Feature #12831] [ruby-core:77586] The term "character" can have many meanings bytes, codepoints, combined characters, and so on. "grapheme cluster" is highest one of such words, which means user-perceived characters. Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION specifies how to handle grapheme clusters (extended grapheme cluster). But some specs aren't updated to current situation because Unicode Emoji is rapidly extended without well definition. It breaks the precondition of UTR#29 "Grapheme cluster boundaries can be easily tested by looking at immediately adjacent characters". (the sentence will be removed in the next version) Though some of its detail are described in Unicode Technical Report #51 UNICODE EMOJI but it is not merged into UTR#29 yet. http://unicode.org/reports/tr29/ http://unicode.org/reports/tr51/ http://unicode.org/Public/emoji/4.0/ git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56949 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* fix uppercasing for U+A64B, CYRILLIC SMALL LETTER MONOGRAPH UKduerst2016-11-302-6/+8
| | | | | | | | | | | | | | | | | * enc/unicode.c: Add U+A64B to the special cases 03B9 and 03BC at the end of onigenc_unicode_case_map (Bug #12990). * enc/unicode/case-folding.rb: Add U+A64B to the special cases 03B9 and 03BC. Add a comment pointing to enc/unicode.c. Change warnings to exceptions for unpredicted cases, because this would have been more easily noticed (the warning was not noticed when upgrading to Unicode 9.0.0). * test/ruby/enc/test_case_comprehensive.rb: Remove temporary exclusion of U+A64B from testing. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56941 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/windows_1254.c: Fix typo. Reported by k-takata atduerst2016-10-291-1/+1
| | | | | | | https://github.com/k-takata/Onigmo/commit/ceb59cc. Thanks! git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56523 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update windows-1255 tablenobu2016-10-281-1/+2
| | | | | | | * enc/trans/windows-1255-tbl.rb: update mapping from 0xCA to U+05BA. [Feature #12877] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56516 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/depend: downcasenobu2016-10-281-1/+1
| | | | | | * enc/depend: downcase table file names. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56515 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/depend: extract transcode_tblgennobu2016-10-281-1/+5
| | | | | | | * enc/depend: extract transcode_tblgen method calls for libraries loaded by dynamically generated names, in single_byte.trans. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56514 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* single_byte.trans: dead codenobu2016-10-281-6/+3
| | | | | | | * enc/trans/single_byte.trans (transcode_tblgen_singlebyte): remove useless code. returned value is not used. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56513 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/windows_1254.c, test/ruby/enc/test_case_comprehensive.rb:duerst2016-10-161-6/+62
| | | | | | | Implement non-ASCII case conversion for Windows-1254. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56433 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-24 14:53:20 +0000