From 9a86fe15755c3f84b32fe54264be847e27ca2da6 Mon Sep 17 00:00:00 2001 From: Stephen Brennan Date: Fri, 12 Apr 2024 23:36:16 -0700 Subject: [PATCH] Add FAQ entry explaining string indexing --- FAQ.md | 42 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/FAQ.md b/FAQ.md index 5cb8a2c..64e0a27 100644 --- a/FAQ.md +++ b/FAQ.md @@ -43,6 +43,46 @@ The `--output=...` option belongs to the main program, and needs to be in front This is [Argparse4j](https://argparse4j.github.io/)'s behavior. - +https://github.com/AsamK/signal-cli/issues/1504 ### DBus errors when starting daemon See [Troubleshooting DBus](https://github.com/AsamK/signal-cli/wiki/DBus-service#user-content-troubleshooting). + +### String Indexing Units + +String indexing is required to properly interpret text formatting as well as mentions. Both involve specifying a substring using `start` and `length`. The units for string indexing are UTF-16 code units, _not_ Unicode code points! This comes from the Signal protocol and the behavior of Android/Java. + +Each Unicode character whose code point is within the [Basic Multilingual Plane](https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane) (that is, whose code point is less than `0x10000`) is represented by one UTF-16 code unit. Characters with code point greater or equal to `0x10000` (such as Emoji) are represented by _two_ UTF-16 code units. To illustrate, consider the string `0💩1💩2💩3`. This string consists of 7 Unicode code points, but 10 UTF-16 code units (because the emoji [U+1F4A9](https://www.fileformat.info/info/unicode/char/1f4a9/index.htm) is beyond the BMP, so each instance counts for two code units). So the following are substrings: + +* start: 0, length: 3 - `0💩` +* start: 1, length: 3 - `💩1` +* start: 2, length: 3 - _invalid_ + +For users of programming languages which index strings by Unicode code points (e.g. Python), you will need to carefully convert indices. For example, this Python function properly converts UTF-16 string indices to Unicode indices: + +```python +>>> def utf16_to_unicode(string: str, utf16_index: int) -> int: +... for unicode_index, c in enumerate(string): +... if utf16_index <= 0: +... break +... utf16_index -= 2 if ord(c) >= 0x10000 else 1 +... if utf16_index < 0: +... raise IndexError("UTF-16 index breaks surrogate pair") +... elif utf16_index > 0: +... raise IndexError("UTF-16 index past end of string") +... else: +... return unicode_index +>>> utf16_to_unicode("0💩1💩2💩3", 0) +0 +>>> utf16_to_unicode("0💩1💩2💩3", 9) +6 +>>> utf16_to_unicode("0💩1💩2💩3", 1) +1 +>>> utf16_to_unicode("0💩1💩2💩3", 2) +Traceback (most recent call last): + File "", line 1, in + File "", line 7, in utf16_to_unicode +IndexError: UTF-16 index breaks surrogate pair + +>>> utf16_to_unicode("0💩1💩2💩3", 3) +2 +``` \ No newline at end of file