aboutsummaryrefslogtreecommitdiffstats
path: root/lib/unicode_normalize
Commit message (Collapse)AuthorAgeFilesLines
* update to Unicode Version 12.1.0 (beta)duerst2019-04-051-2/+2
| | | | | | | | | | | | | | | | | | | | | | Unicode Version 12.1.0 adds one single character, U+32FF SQUARE ERA NAME REIWA, for the new Japanese era starting on May 1st. 12.1.0 will be finalized only on May 7th, so we go with the beta version because further changes in the data we need are highly unlikely, and we want to make sure Ruby is ready for the new era. * common.mk: change UNICODE_VERSION to 12.1.0, UNICODE_BETA to YES * enc/unicode/12.1.0, enc/unicode/12.1.0/casefold.h, enc/unicode/12.1.0/name2ctype.h: add directory and generated data files for new version * lib/unicode_normalize/tables.rb: update for new character * test/ruby/test_regexp.rb: add test for character property age=12.1 * test/test_unicode_normalize.rb: add test for NFKC decomposition of new character This (mostly) completes issue #15195. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67441 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* change lib/unicode_normalize/tables.rb to single item per line to make diffs ↵duerst2019-04-041-1127/+8991
| | | | | | | | | | | | shorter * template/unicode_norm_gen.tmpl: Change formatting of output to produce only a single item (or range) for each line to make future diffs shorter and easier to understand and check. * lib/unicode_normalize/tables.rb: output of the above git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67439 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update Unicode version (and Emoji version) to 12.0.0duerst2019-03-061-79/+80
| | | | | | | | | | | | | - common.mk: set UNICODE_VERSION and UNICODE_EMOJI_VERSION to 12.0.0 - lib/unicode_normalize/tables.rb: update table data to Unicode version 12.0.0 - enc/unicode/12.0.0/casefold.h, enc/unicode/12.0.0/name2ctype.h: add generated files for Unicode version 12.0.0 This is the main commit for #15321. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update to Unicode 11.0.0 (main step, not complete yet)duerst2018-12-051-94/+98
| | | | | | | | | | | | | | - common.mk: Change Unicode version to 11.0.0, and Emoji version to 11.0 - test/ruby/enc/test_emoji_breaks.rb: update hard-coded Emoji version - enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h: Add generated files. Files for Unicode 10.0.0 will be removed once we are sure 11.0.0 works. - lib/unicode_normalize/tables.rb: Updated table. - regparse.c: Almost completely reimplement grapheme cluster detection in function node_extended_grapheme_cluster(). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/*: Prefer require_relative over require, remove explicit extensionmarcandre2018-11-021-1/+1
| | | | | | [#15206] [Fix GH-1976] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65506 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* revert r65091, r65090 because ci failsduerst2018-10-161-98/+94
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65093 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update to Unicode 11.0.0 (basic step, not complete yet)duerst2018-10-161-94/+98
| | | | | | | | | | | | | | | - common.mk: Change Unicode version to 11.0.0 - enc/unicode/case-folding.rb, enc/unicode.c: Initial changes to deal with Gregorian Mtavruli. This should bring us up to the same level as e.g. Python 3.7, by following the Unicode tables exactly. But it will produce undesirable (mixed-case) results for String#capitalize. This will be addressed in a later commit. - enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h: Add generated files. - lib/unicode_normalize/tables.rb: Updated table. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65091 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* fix range check for Hangul jamo trailers in Unicode normalizationduerst2018-07-281-1/+1
| | | | | | | | | | | | | | | | | | | * lib/unicode_normalize/normalize.rb: Fix the range check for trailing Hangul jamo characters in Unicode normalization. Different from leading or vowel jamos, where LBASE and VBASE are actual characters, a value equal to TBASE expresses the absence of a trailing jamo. This fix is technically correct, but there was no bug because the regular expressions in lib/unicode_normalize/tables.rb eliminate jamos equal to TBASE from normalization processing. * test/test_unicode_normalize.rb: Add preventive test test_no_trailing_jamo based on https://github.com/python/cpython/commit/d134809cd3764c6a634eab7bb8995e3e2eff14d5 just for the case we ever get a regression. This closes issue #14934, thanks to MaLin (Lin Ma) for reporting. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update Ruby to Unicode 10.0.0duerst2017-09-061-77/+80
| | | | | | | | | - In common.mk, set UNICODE_VERSION to 10.0.0 - Generate and add enc/unicode/10.0.0/casefold.h and enc/unicode/10.0.0/name2ctype.h - Update lib/unicode_normalize/tables.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* add explanations about status of module UnicodeNormalizeduerst2017-05-091-0/+15
| | | | | | | | | In lib/unicode_normalize/normalize.rb, add explanations and clarifications about the status of the files and the module. [ci skip] This is in response to discussions at https://github.com/ruby/spec/pull/433 and https://bugs.ruby-lang.org/issues/5481#note-58. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58617 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* move definition of String#unicode_normalized? to C to make sure it is documentedduerst2017-05-041-6/+0
| | | | | | | | | | | | * lib/unicode_normalize.rb: Remove definition of String#unicode_normalized? (including documentation). Leave a comment explaining that the file is now empty. * string.c: Define String#unicode_normalized? in rb_str_unicode_normalized_p in C, (including documentation) * lib/unicode_normalize/normalize.rb: Remove (re)definition of String#unicode_normalized? to avoid warnings (when $VERBOSE==true) and problems when String is frozen git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58555 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* move definition of String#unicode_normalize! to C to make sure it is documentedduerst2017-05-041-4/+0
| | | | | | | | | | | | * lib/unicode_normalize.rb: Remove definition of String#unicode_normalize! (including documentation) * string.c: Define String#unicode_normalize! in rb_str_unicode_normalize_bang in C, (including documentation) * lib/unicode_normalize/normalize.rb: Remove (re)definition of String#unicode_normalize! to avoid warnings (when $VERBOSE==true) and problems when String is frozen git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58553 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* move definition of String#unicode_normalize to C to make sure it is documentedduerst2017-05-031-4/+0
| | | | | | | | | | | | * lib/unicode_normalize.rb: Remove definition of String#unicode_normalize (including documentation) * string.c: Define String#unicode_normalize in rb_str_unicode_normalize in C, (including documentation) * lib/unicode_normalize/normalize.rb: Remove (re)definition of String#unicode_normalize to avoid warnings (when $VERBOSE==true) and problems when String is frozen git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58550 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* rework definition of String#unicode_normalize! and #unicode_normalized?duerst2017-05-021-1/+8
| | | | | | | | simplify String#unicode_normalize! and #unicode_normalized? in lib/unicode_normalize.rb by redefining them in lib/unicode_normalize/normalize.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58538 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* rework definition of String#unicode_normalizeduerst2017-05-021-0/+6
| | | | | | | simplify String#unicode_normalize in lib/unicode_normalize.rb by redefining it in lib/unicode_normalize/normalize.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58537 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* nodoc UnicodeNormalize modulestomar2017-04-122-2/+2
| | | | | | | | | * lib/unicode_normalize/normalize.rb: [DOC] nodoc the internal UnicodeNormalize module. * lib/unicode_normalize/tables.rb: ditto. * template/unicode_norm_gen.tmpl: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58329 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * common.mk: Updated Unicode version to 9.0.0 [Feature #12513]duerst2016-09-071-84/+93
| | | | | | | | * unicode/9.0.0/casefold.h, name2ctype.h, unicode/data/9.0.0: new directories/files for Unicode version 9.0.0 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Fix commit misskazu2016-07-201-2/+0
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55713 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* fix typoskazu2016-07-201-0/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55711 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * lib/unicode_normalize/tables.rb: Removeduerst2016-07-191-2/+0
| | | | | | | UnicodeNormalize::UNICODE_VERSION (#12546). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55706 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Add frozen_string_literal: false for all filesnaruse2015-12-161-0/+1
| | | | | | When you change this to true, you may need to add more tests. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * lib/open-uri.rb: Remove indicator for "frozen_string_literal: true".akr2015-11-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * lib/pp.rb: Ditto. * lib/prettyprint.rb: Ditto. * lib/resolv.rb: Ditto. * lib/securerandom.rb: Ditto. * lib/tmpdir.rb: Ditto. * lib/unicode_normalize/tables.rb: Ditto. * test/net/ftp/test_buffered_socket.rb: Ditto. * test/net/ftp/test_mlsx_entry.rb: Ditto. * test/open-uri/test_open-uri.rb: Ditto. * test/open-uri/test_ssl.rb: Ditto. * test/pathname/test_pathname.rb: Ditto. * test/test_pp.rb: Ditto. * test/test_prettyprint.rb: Ditto. * tool/transcode-tblgen.rb: Ditto. * ext/pathname/lib/pathname.rb: Ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* common.mk, lib/unicode_normalize/tables.rb: Change Unicodeduerst2015-10-021-67/+67
| | | | | | Version for character normalization data from 7.0.0 to 8.0.0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52000 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode_norm_gen.tmpl: end markernobu2015-09-291-1/+1
| | | | | | | * template/unicode_norm_gen.tmpl: pragma needs the end marker too, not only the beginning marker. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51972 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* tool/unicode_norm_gen.tmpl, lib/unicode_normalize/tables.rb:duerst2015-09-291-845/+842
| | | | | | | get rid of many .freeze commands by using frozen_string_literal pragma. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51971 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* normalize.rb: remove redundant hashnobu2015-03-111-10/+2
| | | | | | | | | | | * lib/unicode_normalize/normalize.rb (UnicodeNormalize): REGEXP_K matches only single chars which are keys of KOMPATIBLE_TABLE, so string in nfkd_one is always single char and one of the key of KOMPATIBLE_TABLE, that is that the default proc of NF_HASH_K only copies a pair in KOMPATIBLE_TABLE. therefore NF_HASH_K is a part of KOMPATIBLE_TABLE always, and just redundant. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49929 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* tables.rb: addnobu2014-11-111-0/+1163
| | | | | | | * lib/unicode_normalize/tables.rb: commit not to download and convert Unicode data files every time. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48386 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* normalize.rb: fix syntax errornobu2014-11-091-1/+1
| | | | | | | * lib/unicode_normalize/normalize.rb (normalized): fix syntax error, `when` clase allows `*` but not `**`. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48340 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Replaced if-else by case in ↵duerst2014-11-091-3/+4
| | | | | | self.normalized? in parallel to r48309. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48338 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode_norm_gen.tmpl: expand kompatible_tablenobu2014-11-061-16/+2
| | | | | | | * template/unicode_norm_gen.tmpl: expand kompatible_table so that recursive expansion is not needed at runtime. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* normalize.rb: trivial optimizationsnobu2014-11-061-5/+6
| | | | | | | * lib/unicode_normalize/normalize.rb (nfc_one, normalize): trivial optimizations. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48309 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* normalize.rb: explicit separatornobu2014-11-061-1/+1
| | | | | | | * lib/unicode_normalize/normalize.rb (canonical_ordering_one): use explicit separator, not to depend on $,. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48308 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Comment clarification. [ci skip]duerst2014-11-051-1/+1
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48290 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: added US_ASCIIduerst2014-10-251-0/+4
| | | | | | | | as trivially supported encoding (is always normalized, and may appear mixed in with UTF-8 or other Unicode encodings). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48134 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/tables.rb: remove auto generated file.nobu2014-10-211-1154/+0
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48074 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/tables.rb: Committing to make versionduerst2014-10-211-0/+1154
| | | | | | | update easier and more predictable, and reducing compilation time. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Added comment to point toduerst2014-10-211-0/+2
| | | | | | | relevant portion of Unicode standard for Hangul (de)composition identifiers and algorithm. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48071 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode_normalize/normalize.rb: rename variablenobu2014-10-201-5/+5
| | | | | | | | * lib/unicode_normalize/normalize.rb (hangul_decomp_one): use more descriptive name. leave [SLVT]BASE and [LVTNS]COUNT as they are vague names. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48055 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize.rb: revert r48046. The s in sIndexduerst2014-10-201-5/+5
| | | | | | | | | | is not hungarian notation. The variable name sIndex is directly taken from the relevant part of the Unicode Standard, where it is written SIndex and stands for 'syllable index'. See pp. 144/145 of http://www.unicode.org/versions/Unicode7.0.0/ch03.pdf. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48052 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode_normalize/normalize.rb: remove prefixnobu2014-10-201-5/+5
| | | | | | | * lib/unicode_normalize/normalize.rb (hangul_decomp_one): remove system hungarian prefix, nonsense in ruby. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode_normalize/normalize.rb: simplifynobu2014-10-201-3/+3
| | | | | | | * lib/unicode_normalize/normalize.rb (NF_HASH_{D,C,K}): remove first element by Hash#shift. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode_normalize/normalize.rb: remove unnecessary module namesnobu2014-10-201-11/+11
| | | | | | | * lib/unicode_normalize/normalize.rb (UnicodeNormalize): use self instead of module name and remove module name if unnecessary. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48044 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize.rb: remove BOMsnobu2014-10-191-1/+1
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48028 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Added a missing file extension in ↵duerst2014-10-191-1/+1
| | | | | | require statement. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48022 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* tool/unicode_norm_gen.rb, lib/unicode_normalize.rb:duerst2014-10-191-1/+1
| | | | | | | File name change from lib/unicode_normalize/normalize_tables.rb to lib/unicode_normalize/tables.rb. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48015 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * remove trailing spaces.svn2014-10-191-12/+12
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48009 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Changed module name, adjusted copyright.duerst2014-10-191-15/+13
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48008 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * properties.svn2014-10-191-176/+176
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48007 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Importing fromduerst2014-10-191-0/+176
https://github.com/duerst/eprun/blob/master/lib/normalize.rb. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48005 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.39.1) at 2024-05-27 23:18:49 +0000