nik/signal-cli
1
0
Fork 0
mirror of https://github.com/AsamK/signal-cli.git synced 2026-08-30 06:16:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add FAQ entry explaining string indexing

Stephen Brennan 2024-04-12 23:36:16 -07:00
parent 226ee32b27
commit 9a86fe1575
1 changed files with 41 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
FAQ.md

@ -43,6 +43,46 @@ The `--output=...` option belongs to the main program, and needs to be in front
This is [Argparse4j](https://argparse4j.github.io/)'s behavior.
https://github.com/AsamK/signal-cli/issues/1504
### DBus errors when starting daemon
See [Troubleshooting DBus](https://github.com/AsamK/signal-cli/wiki/DBus-service#user-content-troubleshooting).
### String Indexing Units
String indexing is required to properly interpret text formatting as well as mentions. Both involve specifying a substring using `start` and `length`. The units for string indexing are UTF-16 code units, _not_ Unicode code points! This comes from the Signal protocol and the behavior of Android/Java.
Each Unicode character whose code point is within the [Basic Multilingual Plane](https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane) (that is, whose code point is less than `0x10000`) is represented by one UTF-16 code unit. Characters with code point greater or equal to `0x10000` (such as Emoji) are represented by _two_ UTF-16 code units. To illustrate, consider the string `0💩1💩2💩3`. This string consists of 7 Unicode code points, but 10 UTF-16 code units (because the emoji [U+1F4A9](https://www.fileformat.info/info/unicode/char/1f4a9/index.htm) is beyond the BMP, so each instance counts for two code units). So the following are substrings:
* start: 0, length: 3 - `0💩`
* start: 1, length: 3 - `💩1`
* start: 2, length: 3 - _invalid_
For users of programming languages which index strings by Unicode code points (e.g. Python), you will need to carefully convert indices. For example, this Python function properly converts UTF-16 string indices to Unicode indices:
```python
>>> def utf16_to_unicode(string: str, utf16_index: int) -> int:
... for unicode_index, c in enumerate(string):
... if utf16_index <= 0:
... break
... utf16_index -= 2 if ord(c) >= 0x10000 else 1
... if utf16_index < 0:
... raise IndexError("UTF-16 index breaks surrogate pair")
... elif utf16_index > 0:
... raise IndexError("UTF-16 index past end of string")
... else:
... return unicode_index
>>> utf16_to_unicode("0💩1💩2💩3", 0)
0
>>> utf16_to_unicode("0💩1💩2💩3", 9)
6
>>> utf16_to_unicode("0💩1💩2💩3", 1)
1
>>> utf16_to_unicode("0💩1💩2💩3", 2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in utf16_to_unicode
IndexError: UTF-16 index breaks surrogate pair
>>> utf16_to_unicode("0💩1💩2💩3", 3)
2
```