nik/chorus
1
0
Fork 0
mirror of https://github.com/mikedilger/chorus.git synced 2026-08-31 07:31:01 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Compare commits

base: nik:master
Branches Tags
nik:master
nik:latest
nik:profile
nik:v2.0.2
nik:v2.0.1
nik:v2.0.0
nik:v1.7.2
nik:v1.7.1
nik:v1.7.0
nik:v1.6.1
nik:v1.6.0
nik:v1.5.3
nik:v1.5.2
nik:v1.5.1
nik:v1.5.0
nik:v1.4.0
nik:v1.3.0
nik:v1.2.0
nik:v1.1.1
nik:v1.1.0
nik:v1.0.7
nik:v1.0.6
nik:v1.0.5
nik:v1.0.4
nik:v1.0.3
nik:v1.0.2
nik:v1.0.1
nik:v1.0
..
compare: nik:v1.0.1
Branches Tags
nik:latest
nik:master
nik:profile
nik:v2.0.2
nik:v2.0.1
nik:v2.0.0
nik:v1.7.2
nik:v1.7.1
nik:v1.7.0
nik:v1.6.1
nik:v1.6.0
nik:v1.5.3
nik:v1.5.2
nik:v1.5.1
nik:v1.5.0
nik:v1.4.0
nik:v1.3.0
nik:v1.2.0
nik:v1.1.1
nik:v1.1.0
nik:v1.0.7
nik:v1.0.6
nik:v1.0.5
nik:v1.0.4
nik:v1.0.3
nik:v1.0.2
nik:v1.0.1
nik:v1.0

No commits in common. "master" and "v1.0.1" have entirely different histories.
master ... v1.0.1

86 changed files with 5443 additions and 7965 deletions
Show Stats Expand all files Collapse all files

182
CHANGELOG.md
View File

@ -1,182 +0,0 @@
# v2.0.0
- IMPORTANT: You need to manually [Migrate](docs/MIGRATION.md) your users and moderators.
- Management commands have been added: listadmins, listmoderators, grantmoderator, revokemoderator,
listusers, grantuser, revokeuser, stats, listeventsneedingmoderation, clearevent, clearpubkey,
listrole, grantrole, revokerole, fetchbannedevents
- fix: subscriptions will be CLOSED: auth-required if any matching event requires auth. Chorus
will serve the redacted results first, but will not EOSE.
- fix: subscriptions will be CLOSED when completed, without EOSE, if the filter has any ids set.
# v1.7.2
- Support for NIP-62 (PR #1256) Right to Vanish
- Fix NIP-86 management request header determination
- Management adds: numconnections, uptime
- chorus_cmd: fetch_by_id
# v1.7.1
- Config setting allow_scrape_if_negentropy=true
- CLOSED no longer sent after CLOSE
# v1.7.0
- Negentropy support (you must set enable_negentropy=true)
- Blossom extensions are better
# v1.6.1
- base_url added (you should set this if you are behind a proxy like nginx)
- NIP-45 COUNT support
- NIP-45 PR #1561 HyperLogLog count support
# v1.6.0
- Support for running as a blossom server
- Websocket close frames provided on close
- Better pre-flight checks
- server-wide OPTIONS requests now handled
# v1.5.3
- Performance improvements (streaming body)
- Approval not required for kinds 0, 3, and reactions (as well as prior list)
- FIX honor enable_ip_blocking = false
- NEW `chorus_cmd` with `delete_by_id` and `delete_by_pubkey`
- optional `icon_url` for NIP-11
# version 1.5.2 (2024-07-31, d8689540)
- NEW CONFIG: `throttling_bytes_per_second` how many bytes are allowed per second
- NEW CONFIG: `throttling_burst` how many bytes can be accepted per connection in a burst
- Don't wait more than 5 seconds for connections to close when shutting down
# version 1.5.1 (2024-07-14, c9c71311)
- FIX: large non-utf8 messages were attempted to be logged causing a panic
- FIX: parse errors were not being punished so a nasty client could do a DoS of chorus with
simple junk.
# version 1.5.0 (2024-07-13, 870e470d)
- BREAKING: If you run chorus behind a proxy like nginx, you MUST set the new `chorus_is_behind_a_proxy`
config variable to true, and your proxy MUST set the `X-Real-IP` header. If the header is missing,
connections will not be served. If you fail to set the `chorus_is_behind_a_proxy` setting, the proxy
IP address will be used directly, generally causing all connections to quickly become banned due to
the bad behavior of just one client, or due to too many connections from a single IP.
- NEW CONFIG: `chorus_is_behind_a_proxy` (please set to true or false)
- NEW CONFIG: `max_connections_per_ip` (defaults to 5)
- NEW CONFIG: `moderator_hex_keys` (see next bullet point)
- A rudimentary Management API is now available using https://github.com/nostr-protocol/nips/pull/1325
To use management front ends against chorus, you must add hex pubkeys to `moderator_hex_keys`.
- Errors about DM kinds are much less common now, as we don't explicitly error unless they specify some set
of kinds (we implicitly filter out the DMs still)
- Accurate count of bytes sent/received (SSL header data is now counted)
- Kind 10050 dm relay list events are now treated the same way as kind 10002 relay list events.
- Error message detail (e.g. source code line numbers) is now no longer sent to clients.
- Some mild errors are now swallowed.
- Updates of many dependencies, some updates were very large jumps and may change network/http behavior.
- Receipt of a deleted EVENT now returns OK false (was OK true)
# v1.4.0 (2024-05-07, 25058ef4)
- Origin header logged
- CLOSED: auth-required sent if DMs requested and not authenticated yet
- config: minimum_ban_seconds, timeout_seconds, enable_ip_blocking
- default for allow_scrape_if_max_seconds raised from 3600 to 7200
- default for max_subscriptions raised from 32 to 128
- timeouts no longer affect ban seconds
- Internal: switched to pocket for the backend storage
- creates lmdb subdirectory if missing
- several bugfixes: filter parsing, empty tags, event ordering, chorus_compress was fixed
# v1.3.0 (2024-04-10, 7883d754)
- `chorus_moderate` tool to approve events or pubkeys
- `chorus_dump_approvals` tool to view prior approvals
- `chorus_compress` to backup and rebuild your data, compressing as it goes.
- NIP-40 event expiration support
- extended NIP-11 document, with updates to supported NIPs
- Traffic bytes counted and logged on exit (or HUP)
- Logging improvements
- Data migrations. Less space consumed.
- performance improved via 8-byte alignment of events
# v1.2.0 (2024-03-22, a701e148)
- Config option to run as an open relay
- Config variable for scraper behavior
- More efficient scraping
- Reload config on HUP without dropping connections
- IP addresses no longer logged. They are hashed and tracked by their hash.
# v1.1.1 (2024-03-02, fda607a6)
- FIX some LIMIT related bugs
- FIX some indexing bugs that cuased some events that should have been returned to not be.
- Allow scrape up to limit=100, or time range of 1 hour
# v1.1.0 (2024-02-21, ec315d98)
- Update docs
# v1.0.7 (2024-02-21, 4ebe2545)
- BREAKING: Config changed to TOML format
- Logging improvements
- Error handling improvements
- Ban time improvements
- Documentation updates
- Restructured into a `bin` and a `lib`
- Tool to dump all events as JSON to stdout
# v1.0.6 (2024-02-21, 50995a75)
- FIX: REQ errors now properly CLOSED
- Logging improvements
- Persistent IP reputation data for adjusting ban times
- Config settings for `serve_ephmeral` and `serve_relay_lists`
- Uses 'X-Real-Ip' header if behind a proxy
# v1.0.5 (2024-02-20, 5e7c1c38)
- FIX: significant performance problem addressed
- Logging improvements
- EVENTs submitted that are known to be deleted are now responded to with 'blocked:'
- Longer filters now accepted
# v1.0.4 (2024-02-20, 9c7aa299)
- FIX: after EOSE, outbound events were not being screened properly
- FIX: shutdown issue
- NIP-11 updated
- Idle connections with no REQs now timed out at 5 seconds
# v1.0.3 (2024-02-20, 015d847f)
- FIX: limits were not working properly
- Update IP banning logic and timing
- Improve logging
- Allow more REQ filters (with low limits)
- Better performance when filters are more open
# v1.0.2 (2024-02-19, b85cd929)
- FIX: deadlock (nested transaction issue)
- Logging changes
- Documentation updated
# v1.0.1 (2024-02-19, 9d65d773)
- FIX: AUTH was failing due to a bad time difference comparison
- AUTH failures now include detail as to why
- documentation updated
# v1.0.0 (2024-02-19, 7ed36b95)
- Initial release.
- Works as a personal relay
- Supported NIPs: 1, 4, 9, 11, 42, 59, 65
- Bans IPs to prevent abuse
- Limits to 32 subscriptions by default, configurable

1818
Cargo.lock generated
View File

File diff suppressed because it is too large Load Diff

38
Cargo.toml
View File

@ -1,6 +1,6 @@
[package]
name = "chorus"
version = "2.0.2"
version = "1.0.1"
description = "A personal relay for nostr"
authors = ["Mike Dilger <mike@mikedilger.com>"]
license = "MIT"
@ -8,37 +8,21 @@ repository = "https://github.com/mikedilger/chorus"
edition = "2021"
[dependencies]
base64 = "0.22"
bitcoin_hashes = "0.19"
dashmap = "6"
env_logger = "0.11"
env_logger = "0.10"
futures = "0.3"
hex = "0.4"
http = "1.3"
http-body-util = "0.1"
hyper = { version = "1.6", features = [ "http1", "server" ] }
hyper-tungstenite = "0.17"
hyper-util = "0.1"
lazy_static = "1.5"
heed = { git = "https://github.com/meilisearch/heed", rev = "64fd6fec293c0dee94855b8267557ce03e7ce5d8" }
hyper = { version = "0.14", features = [ "http1", "server", "runtime", "stream" ] }
hyper-tungstenite = "0.11"
lazy_static = "1.4"
log = "0.4"
mime-sniffer = "0.1"
mime2ext = "0.1"
negentropy = "0.5"
pocket-types = { git = "https://github.com/mikedilger/pocket", ref = "329334f20948c796c6016b673b92551ac4855ad7" }
pocket-db = { git = "https://github.com/mikedilger/pocket", ref = "329334f20948c796c6016b673b92551ac4855ad7" }
parking_lot = "0.12"
rustls-pki-types = "1.11"
rustls-pemfile = "2.2"
secp256k1 = { version = "0.31", features = [ "hashes", "global-context" ] }
mmap-append = { git = "https://github.com/mikedilger/mmap-append", rev = "0d20e193e7f13a442865a4a40a5da9a120e87411" }
ron = "0.8"
rustls-pemfile = "1.0"
secp256k1 = { version = "0.28", features = [ "hashes", "global-context", "rand-std" ] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
speedy = "0.8"
textnonce = "1"
tokio = { version = "1", features = [ "full" ] }
tokio-rustls = "0.26"
tokio-util = { version = "0.7", features = [ "io" ] }
tokio-stream = "0.1"
toml = "0.8"
tokio-rustls = "0.24"
url = "2.5"
[dev-dependencies]

50
README.md
View File

@ -4,9 +4,7 @@ Chorus is a nostr relay.
It is pretty fast: [docs/PERFORMANCE.md](docs/PERFORMANCE.md).
It can work as your personal relay (or as an open relay): [docs/PERSONAL_RELAY.md](docs/PERSONAL_RELAY.md)
It can serve as a blossom server.
It can work as your personal relay: [docs/PERSONAL_RELAY.md](docs/PERSONAL_RELAY.md).
To deploy chorus, read [docs/DEPLOYING.md](docs/DEPLOYING.md)
@ -14,50 +12,8 @@ To configure chorus read [docs/CONFIG.md](docs/CONFIG.md)
To undertand the relay's behavior, read [docs/BEHAVIOR.md](docs/BEHAVIOR.md)
To understand command line tools, read [docs/TOOLS.md](docs/TOOLS.md)
To understand the management API, read [docs/MANAGEMENT.md](docs/MANAGEMENT.md)
Chorus does not have any provisions for charging users.
Chorus does not have any provisions for synchronizing events with other relays outside of the nostr protocol.
## Status
## Alternatives
### strfry
[strfry](https://github.com/hoytech/strfry) is a more mature relay that offers additional features including:
- Synchronizing events with other relays efficiently (negentropy)
- Zero-downtime restarts
- Websocket permessage-deflate
- Plugins for event sifting
However:
- Chorus is probably faster (more efficient)
- Chorus has personal relay rules by default
- Chorus has extensive IP banning and rate limiting to protect your relay from abuse
- Chorus supports NIP-42 (AUTH), NIP-59 (GiftWrap), NIP-65 (Relay Lists) and PR 1030 and PR 1325.
- Chorus has a moderation cmd line tool and a moderation API (PR 1325)
- Chorus can act as a blossom server
### nostream
[nostream](https://github.com/Cameri/nostream)
### nostr-rs-relay
[nostr-rs-relay](https://git.sr.ht/~gheartsfield/nostr-rs-relay)
### khatru
[khatru](https://github.com/fiatjaf/khatru)
## Git branches
Use the branch `latest`.
Do not run off of the `master` branch. Master is updated with breaking changes that are
not only unstable, but which may require you to update your configuration. I will not
announce upgrade instructions until release.
Chorus is now released.

3
contrib/chorus-direct.service
View File

@ -12,8 +12,7 @@ AmbientCapabilities=CAP_NET_BIND_SERVICE
ExecStartPre=+cp -H /etc/letsencrypt/live/chorus.example.com/fullchain.pem /opt/chorus/etc/tls/
ExecStartPre=+cp -H /etc/letsencrypt/live/chorus.example.com/privkey.pem /opt/chorus/etc/tls/
ExecStartPre=+chown chorus /opt/chorus/etc/tls/fullchain.pem /opt/chorus/etc/tls/privkey.pem
ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.toml
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.ron
Restart=always
RestartSec=5

3
contrib/chorus-proxied.service
View File

@ -8,8 +8,7 @@ Environment="RUST_BACKTRACE=1"
Environment="RUST_LOG=info"
WorkingDirectory=/opt/chorus
User=chorus
ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.toml
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.ron
Restart=always
RestartSec=5

2
contrib/chorus.nginx.conf → contrib/chorus.nginx.com
View File

@ -23,7 +23,7 @@ server {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-Ip $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_read_timeout 1d;
proxy_send_timeout 1d;
}

16
contrib/chorus.ron Normal file
View File

@ -0,0 +1,16 @@
FriendlyConfig(
data_directory: "/opt/chorus/var/chorus",
ip_address: "127.0.0.1",
port: 443,
use_tls: true,
certchain_pem_path: "/opt/chorus/etc/tls/fullchain.pem",
key_pem_path: "/opt/chorus/etc/tls/privkey.pem",
name: Some("Chorus Default"),
description: Some("A default config of the Chorus relay"),
contact: None,
public_key_hex: None,
user_hex_keys: [
],
verify_events: true,
allow_scraping: false,
)

354
contrib/chorus.toml
View File

@ -1,354 +0,0 @@
# This is a config file for the Chorus nostr relay
# Refer to https://github.com/mikedilger/chorus
# This is the directory where chorus stores data.
#
# Default is "/tmp".
#
# If deployed according to docs/DEPLOYING.md, is "/opt/chorus/var/chorus".
#
data_directory = "/opt/chorus/var/chorus"
# This is the IP address that chorus listens on. If deployed directly on the
# Internet, this should be an Internet globally accessible IP address.
# If proxied or if testing locally, this can be a localhost address.
#
# Default is "127.0.0.1".
#
ip_address = "127.0.0.1"
# This is the port that chorus listens on. If deployed directly on the Internet,
# this should probably be 443 which is the expected default port for the
# "wss://" protocol.
#
# Default is 443.
#
port = 443
# This is the DNS hostname of your relay.
# This is used for verifying AUTH events, which specify your relay host name.
#
hostname = "localhost"
# If chorus is behind a proxy like nginx, set this to true. In this case chorus will look for and
# trust the `X-Real-Ip` HTTP request header to get the real IP of the client. This header MUST exist
# or the connection will not be served.
#
# Default is false.
#
chorus_is_behind_a_proxy = false
# If chorus is behing a proxy, it can't compute it's Internet-visible URL. So set it here.
# This is used currently by web management and blossom and maybe more in the future.
#
# Default is not set.
#
# base_url =
# If true, chorus will handle TLS, running over HTTPS. If false, chorus run over HTTP.
#
# If you are proxying via nginx, normally you will set this to false and allow nginx to handle TLS.
#
use_tls = true
# This is the path to your TLS certificate chain file.
#
# If use_tls is false, this value is irrelevant.
#
# Default is "./tls/fullchain.pem".
#
# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
# "/opt/chorus/etc/tls/fullchain.pem" and the systemd service copies letsencrypt TLS
# certificates into this position on start.
#
certchain_pem_path = "/opt/chorus/etc/tls/fullchain.pem"
# This is the path to yoru TLS private key file.
#
# If use_tls is false, this value is irrelevant.
#
# Default is "./tls/privkey.pem".
#
# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
# "/opt/chorus/etc/tls/privkey.pem" and the systemd service copies letsencrypt TLS
# certificates into this position on start.
#
key_pem_path = "/opt/chorus/etc/tls/privkey.pem"
# This is a name for your relay, displayed in the NIP-11 response.
#
# Default is "Chorus Default"
#
# name = "Chorus Default"
# This is a description for your relay, displayed in the NIP-11 response.
#
# Default is "A default config of the Chorus relay"
#
# description = "A default config of the Chorus relay"
# This is a banner URL pointing to an image representing your relay, displayed in the NIP-11
# response.
#
# Default is not set
#
# banner_url =
# This is an icon URL pointing to an image representing your relay, displayed in the NIP-11
# response.
#
# Default is not set
#
# icon_url =
# This is an optional privacy policy as a blob of text (not a URL, not HTML).
#
# Default is not set
#
# privacy_policy = ""
# This is an optional terms of service document as a blob of text (not a URL, not HTML).
#
# Default is not set
#
# terms_of_service = ""
# This is an optional contact for your relay, displayed in the NIP-11 response.
#
# Default is not set
#
# contact =
# This is an optional public key (hex format) for your relay, displayed in the NIP-11 response.
#
# Default is not set
#
# contact_public_key_hex =
# If open_relay is true, the relay behaves as an open public relay.
#
# Default is false.
#
# open_relay = false
# These are the public keys (hex format) of your relay's administrators. This does NOT
# automatically make them a relay user, but it will eventually allow them to add/remove users
# and moderators.
#
# Default is []
#
admin_hex_keys = []
# This is a boolean indicating whether or not chorus verifies incoming events.
#
# This setting only skips verification of events that are submitted by AUTHed and
# authorized users. Chorus always verifies incoming AUTH events, and any event that
# is not submitted by an AUTHed and authorized relay user.
#
# Default is true.
#
verify_events = true
# This is a boolean indicating whether or not scraping is allowed. Scraping is any filter
# where all of the following are true:
#
# - `ids` is missing or empty
# - `authors` is missing or empty
# - There are no `#X` tag filters
#
# Filter that fail to match these conditions will be rejected if allow_scraping is false.
#
# If allow_scraping is true, be aware that filters that don't match any of these conditions
# (and are not bound by since/until/limit) have no indexes to speed up their query, so they
# scan through every single event on the relay.
#
# See also `allow_scrape_if_limited_to` and `allow_scrape_if_max_seconds`.
#
# Default is false.
#
allow_scraping = false
# This is a u32 count of events indicating a filter `limit` value under which a scrape is
# allowed, irrespective of the `allow_scraping` setting. Such scrapes are not expensive due
# to the limit. The limit must be specified in the filter; it does not do the scrape and
# then count to see if it was under the limit.
#
# See `allow_scraping` to learn the definition of a scrape.
#
# The default is 100.
#
allow_scrape_if_limited_to = 100
# This is a u64 number of seconds indicating a filter time range under which a scrape is
# allowed, irrespective of the `allow_scraping` setting. Such scrapes are rarely expensive
# due to the short time period.
#
# See `allow_scraping` to learn the definition of a scrape.
#
# The default is 7200.
#
allow_scrape_if_max_seconds = 7200
# Within negentropy synchronization, allow scraping
#
# The default is true
#
allow_scrape_if_negentropy = true
# This is an integer indicating the maximum number of subscriptions a connection can have open
# at a given time.
#
# If you set this too low, clients will be incentivised to resubmit updated subscriptions which
# will pull down the same events over again, instead of submitting a new subscription that only
# gets the additional events that the client wants. It may seem intuitive that setting this to
# a low value like 10 will decrease server load, but it will probably increase server load.
#
# It is strongly recommended to not go below 16.
#
# Default is 128.
#
max_subscriptions = 128
# Whether or not to accept and serve all ephemeral events to everybody.
#
# Default is true.
#
serve_ephemeral = true
# Whether or not to accept and serve kind 10002 Relay List Metadata (NIP-65) events to everybody.
#
# Default is true.
#
serve_relay_lists = true
# How verbose to log issues with the main server code.
#
# Possible values are: Trace, Debug, Info, Warn, Error
#
# Default is Info
#
server_log_level = "Info"
# How verbose to log library issues and other general issues
#
# Possible values are: Trace, Debug, Info, Warn, Error
#
# Default is Info
#
library_log_level = "Info"
# How verbose to log issues with client requests
#
# Possible values are: Trace, Debug, Info, Warn, Error
#
# Default is Info
#
client_log_level = "Info"
# Whether to block incoming connections based on recent prior behavior
#
# Chorus normally blocks IP addresses for a short period preventing quick reconnections,
# and for a longer period if the previous connection ended in some error condition.
#
# By setting this variable to false, it will allow all connections, incluing poorly behaving
# clients that reconnect over and over in a tight loop.
#
# Default is true
#
enable_ip_blocking = true
# Number of seconds to ban an IP address after disconnection.
#
# Enforcing this minimum ban prevents clients from immediately reconnecting which can cause tight loops.
# Only relevant if enable_ip_blocking is true.
#
# Default is 1
#
minimum_ban_seconds = 1
# Number of seconds beyond which chorus times out a client that has no open subscriptions.
#
# Default is 60
#
timeout_seconds = 60
# Maximum number of websocket connections per IP address
#
# Default is 5
#
max_connections_per_ip = 5
# The maximum rate (excluding bursts) of data that will be transmitted over a websocket connection
# (both directions, per connection). Beyond this rate (in a sustained way) the connection will be
# closed.
#
# Default is 1024576 bytes per second.
#
throttling_bytes_per_second = 1024576
# The allowable bursts of data beyond the normal rate.
#
# We keep a count of how many bytes they are allowed, and that count starts at this number.
# As bytes are consumed the count goes down, but we refund throttling_bytes_per_second every
# second. If that bucket doesn't have enough, the burst won't be allowed and the connection
# will be closed.
#
# Default is 16777216 bytes.
#
throttling_burst = 16777216
# Blossom server directory
#
# Set to a filesystem directory where you want chorus to store files.
#
# Blossom allows clients to upload files making them available for the public to download.
# Our implementation makes all files publicly readable, but only chorus users can upload
# or delete. See https://github.com/hzrd149/blossom
#
# Default is not set
#
# blossom_directory =
# Whether to allow negentropy syncing or not
#
# If this is enabled, keep in mind that some negentropy syncs are scrapes and if you want to
# allow those, you should also enable allow_scraping. But this will require scans of the entire
# database since scrapes have no indexes.
#
# Default is false
#
enable_negentropy = false

1
docker/.env
View File

@ -1 +0,0 @@
PODMAN_USERNS=keep-id

35
docker/Dockerfile
View File

@ -1,35 +0,0 @@
FROM rust:alpine as builder
RUN apk add --no-cache git
WORKDIR /root
RUN git clone https://github.com/mikedilger/chorus
WORKDIR /root/chorus
RUN git checkout latest
RUN cargo build --release
RUN strip ./target/release/* || true
FROM alpine:latest
# Install system packages
# RUN apk add --no-cache curl gcc musl-dev openssl-dev pkgconfig git make cmake
# Setup chorus user and directories
RUN mkdir -p /opt/chorus/etc /opt/chorus/src/chorus /opt/chorus/var /opt/chorus/sbin /opt/chorus/lib && \
mkdir -p /opt/chorus/var/chorus /opt/chorus/var/www && \
mkdir -p /opt/chorus/lib/systemd/system
COPY --from=builder /root/chorus/target/release/chorus /opt/chorus/sbin/chorus
COPY --from=builder /root/chorus/target/release/chorus_cmd /opt/chorus/sbin/chorus_cmd
COPY --from=builder /root/chorus/target/release/chorus_compress /opt/chorus/sbin/chorus_compress
COPY --from=builder /root/chorus/target/release/chorus_dump /opt/chorus/sbin/chorus_dump
COPY --from=builder /root/chorus/target/release/chorus_dump_approvals /opt/chorus/sbin/chorus_dump_approvals
COPY --from=builder /root/chorus/target/release/chorus_moderate /opt/chorus/sbin/chorus_moderate
RUN chmod 0700 /opt/chorus/sbin/*
VOLUME /opt/chorus/etc
VOLUME /opt/chorus/var
WORKDIR /opt/chorus
ENV RUST_BACKTRACE=1
ENV RUST_LOG=info
ENTRYPOINT ["/opt/chorus/sbin/chorus", "/opt/chorus/etc/chorus.toml"]

6
docker/README.md
View File

@ -1,6 +0,0 @@
# Docker install of Chorus
You need to `cp chorus.toml.in chorus.toml` and then edit this file
to set up your configuration
Then you can build the container: `docker build .`

12
docker/docker-compose.yml
View File

@ -1,12 +0,0 @@
services:
chorus:
build:
context: .
dockerfile: Dockerfile
container_name: chorus
restart: unless-stopped
volumes:
- ./etc:/opt/chorus/etc
- ./var:/opt/chorus/var
ports:
- 443:1444

1
docker/etc/.gitignore vendored
View File

@ -1 +0,0 @@
/chorus.toml

354
docker/etc/chorus.toml.in
View File

@ -1,354 +0,0 @@
# This is a config file for the Chorus nostr relay
# Refer to https://github.com/mikedilger/chorus
# This is the directory where chorus stores data.
#
# Default is "/tmp".
#
# If deployed according to docs/DEPLOYING.md, is "/opt/chorus/var/chorus".
#
data_directory = "/opt/chorus/var/chorus"
# This is the IP address that chorus listens on. If deployed directly on the
# Internet, this should be an Internet globally accessible IP address.
# If proxied or if testing locally, this can be a localhost address.
#
# Default is "127.0.0.1".
#
ip_address = "0.0.0.0"
# This is the port that chorus listens on. If deployed directly on the Internet,
# this should probably be 443 which is the expected default port for the
# "wss://" protocol.
#
# Default is 443.
#
port = 443
# This is the DNS hostname of your relay.
# This is used for verifying AUTH events, which specify your relay host name.
#
hostname = "localhost"
# If chorus is behind a proxy like nginx, set this to true. In this case chorus will look for and
# trust the `X-Real-Ip` HTTP request header to get the real IP of the client. This header MUST exist
# or the connection will not be served.
#
# Default is false.
#
chorus_is_behind_a_proxy = true
# If chorus is behing a proxy, it can't compute it's Internet-visible URL. So set it here.
# This is used currently by web management and blossom and maybe more in the future.
#
# Default is not set.
#
# base_url =
# If true, chorus will handle TLS, running over HTTPS. If false, chorus run over HTTP.
#
# If you are proxying via nginx, normally you will set this to false and allow nginx to handle TLS.
#
use_tls = false
# This is the path to your TLS certificate chain file.
#
# If use_tls is false, this value is irrelevant.
#
# Default is "./tls/fullchain.pem".
#
# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
# "/opt/chorus/etc/tls/fullchain.pem" and the systemd service copies letsencrypt TLS
# certificates into this position on start.
#
certchain_pem_path = "/use/an/nginx/proxy"
# This is the path to yoru TLS private key file.
#
# If use_tls is false, this value is irrelevant.
#
# Default is "./tls/privkey.pem".
#
# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
# "/opt/chorus/etc/tls/privkey.pem" and the systemd service copies letsencrypt TLS
# certificates into this position on start.
#
key_pem_path = "/use/an/nginx/proxy"
# This is a name for your relay, displayed in the NIP-11 response.
#
# Default is "Chorus Default"
#
# name = "Chorus Default"
# This is a description for your relay, displayed in the NIP-11 response.
#
# Default is "A default config of the Chorus relay"
#
# description = "A default config of the Chorus relay"
# This is a banner URL pointing to an image representing your relay, displayed in the NIP-11
# response.
#
# Default is not set
#
# banner_url =
# This is an icon URL pointing to an image representing your relay, displayed in the NIP-11
# response.
#
# Default is not set
#
# icon_url =
# This is an optional privacy policy as a blob of text (not a URL, not HTML).
#
# Default is not set
#
# privacy_policy = ""
# This is an optional terms of service document as a blob of text (not a URL, not HTML).
#
# Default is not set
#
# terms_of_service = ""
# This is an optional contact for your relay, displayed in the NIP-11 response.
#
# Default is not set
#
# contact =
# This is an optional public key (hex format) for your relay, displayed in the NIP-11 response.
#
# Default is not set
#
# contact_public_key_hex =
# If open_relay is true, the relay behaves as an open public relay.
#
# Default is false.
#
# open_relay = false
# These are the public keys (hex format) of your relay's administrators. This does NOT
# automatically make them a relay user, but it will eventually allow them to add/remove users
# and moderators.
#
# Default is []
#
admin_hex_keys = []
# This is a boolean indicating whether or not chorus verifies incoming events.
#
# This setting only skips verification of events that are submitted by AUTHed and
# authorized users. Chorus always verifies incoming AUTH events, and any event that
# is not submitted by an AUTHed and authorized relay user.
#
# Default is true.
#
verify_events = true
# This is a boolean indicating whether or not scraping is allowed. Scraping is any filter
# where all of the following are true:
#
# - `ids` is missing or empty
# - `authors` is missing or empty
# - There are no `#X` tag filters
#
# Filter that fail to match these conditions will be rejected if allow_scraping is false.
#
# If allow_scraping is true, be aware that filters that don't match any of these conditions
# (and are not bound by since/until/limit) have no indexes to speed up their query, so they
# scan through every single event on the relay.
#
# See also `allow_scrape_if_limited_to` and `allow_scrape_if_max_seconds`.
#
# Default is false.
#
allow_scraping = false
# This is a u32 count of events indicating a filter `limit` value under which a scrape is
# allowed, irrespective of the `allow_scraping` setting. Such scrapes are not expensive due
# to the limit. The limit must be specified in the filter; it does not do the scrape and
# then count to see if it was under the limit.
#
# See `allow_scraping` to learn the definition of a scrape.
#
# The default is 100.
#
allow_scrape_if_limited_to = 500
# This is a u64 number of seconds indicating a filter time range under which a scrape is
# allowed, irrespective of the `allow_scraping` setting. Such scrapes are rarely expensive
# due to the short time period.
#
# See `allow_scraping` to learn the definition of a scrape.
#
# The default is 7200.
#
allow_scrape_if_max_seconds = 14400
# Within negentropy synchronization, allow scraping
#
# The default is true
#
allow_scrape_if_negentropy = true
# This is an integer indicating the maximum number of subscriptions a connection can have open
# at a given time.
#
# If you set this too low, clients will be incentivised to resubmit updated subscriptions which
# will pull down the same events over again, instead of submitting a new subscription that only
# gets the additional events that the client wants. It may seem intuitive that setting this to
# a low value like 10 will decrease server load, but it will probably increase server load.
#
# It is strongly recommended to not go below 16.
#
# Default is 128.
#
max_subscriptions = 128
# Whether or not to accept and serve all ephemeral events to everybody.
#
# Default is true.
#
serve_ephemeral = true
# Whether or not to accept and serve kind 10002 Relay List Metadata (NIP-65) events to everybody.
#
# Default is true.
#
serve_relay_lists = true
# How verbose to log issues with the main server code.
#
# Possible values are: Trace, Debug, Info, Warn, Error
#
# Default is Info
#
server_log_level = "Info"
# How verbose to log library issues and other general issues
#
# Possible values are: Trace, Debug, Info, Warn, Error
#
# Default is Info
#
library_log_level = "Info"
# How verbose to log issues with client requests
#
# Possible values are: Trace, Debug, Info, Warn, Error
#
# Default is Info
#
client_log_level = "Info"
# Whether to block incoming connections based on recent prior behavior
#
# Chorus normally blocks IP addresses for a short period preventing quick reconnections,
# and for a longer period if the previous connection ended in some error condition.
#
# By setting this variable to false, it will allow all connections, incluing poorly behaving
# clients that reconnect over and over in a tight loop.
#
# Default is true
#
enable_ip_blocking = true
# Number of seconds to ban an IP address after disconnection.
#
# Enforcing this minimum ban prevents clients from immediately reconnecting which can cause tight loops.
# Only relevant if enable_ip_blocking is true.
#
# Default is 1
#
minimum_ban_seconds = 1
# Number of seconds beyond which chorus times out a client that has no open subscriptions.
#
# Default is 60
#
timeout_seconds = 60
# Maximum number of websocket connections per IP address
#
# Default is 5
#
max_connections_per_ip = 5
# The maximum rate (excluding bursts) of data that will be transmitted over a websocket connection
# (both directions, per connection). Beyond this rate (in a sustained way) the connection will be
# closed.
#
# Default is 1024576 bytes per second.
#
throttling_bytes_per_second = 1024576
# The allowable bursts of data beyond the normal rate.
#
# We keep a count of how many bytes they are allowed, and that count starts at this number.
# As bytes are consumed the count goes down, but we refund throttling_bytes_per_second every
# second. If that bucket doesn't have enough, the burst won't be allowed and the connection
# will be closed.
#
# Default is 16777216 bytes.
#
throttling_burst = 16777216
# Blossom server directory
#
# Set to a filesystem directory where you want chorus to store files.
#
# Blossom allows clients to upload files making them available for the public to download.
# Our implementation makes all files publicly readable, but only chorus users can upload
# or delete. See https://github.com/hzrd149/blossom
#
# Default is not set
#
# blossom_directory =
# Whether to allow negentropy syncing or not
#
# If this is enabled, keep in mind that some negentropy syncs are scrapes and if you want to
# allow those, you should also enable allow_scraping. But this will require scans of the entire
# database since scrapes have no indexes.
#
# Default is false
#
enable_negentropy = false

1
docker/var/.gitignore vendored
View File

@ -1 +0,0 @@

31
docs/BEHAVIOR.md
View File

@ -4,11 +4,7 @@
If `verify_events` is set in the configuration, chorus rejects invalid events in all cases.
If an event has a '-' tag, it must be submitted by an AUTHed user that matches the pubkey of the event, else it is rejected.
If `open_relay` is true, all other events are accepted. If false, the remaining rules apply.
Chorus accepts all events submitted by AUTHed authorized users.
Chorus accepts all events submitted by authorized users.
Chorus accepts relay list metadata (kind 10002) from anybody.
@ -22,41 +18,33 @@ If you wish to change these rules, change the source code at `nostr.rs:screen_in
## REQ Read permissions and behavior
Chorus does not serve any DM (kind 4) or GiftWrap (kind 1059) unless the connection is AUTHed and the user matches either a tagged person or the author of the event.
If `open_relay` is true, all other events are served. If false, the remaining rules apply.
Chorus serves all relay list metadata (kind 10002) events queried.
Chorus serves all ephemeral events.
Chorus serves all events to AUTHed authorized users.
Chorus does not serve any DM (kind 4) or GiftWrap (kind 1059) unless the connection is AUTHed and the user matches either a tagged person or the author of the event.
Chorus serves all events to AUTHed and authorized users.
Chorus serves all events which were authored by an authorized user.
Filters which are broad are considered scrapers and are not serviced. Scraping is any filter where all of the following are true:
- `ids` is missing or empty
- `authors` is missing or empty
- There are no `#X` tag filters
If you wish to change these rules, change the source code at `nostr.rs:screen_outgoing_event()`
## Abuse, Banning, Throttling, and the like
WebSocket frames and messages are limited to 1 MB (slightly less for messages due to some overhead).
WebSocket frames and messages are limited to 1 MB.
Each connection has a memory buffer used for JSON deserialization that is no larger
than the WebSocket message. No more than one such buffer exists per connection, and memory
allocation is generally tightly controlled.
Every connection is IP banned for 2 seconds after disconnection, whether or not the connection
Every connection is IP banned for 4 seconds after disconnection, whether or not the connection
was well behaved. We don't think clients should ever reconnect immediately. If chorus is
run directly (not behind an nginx proxy), this IP banning is more efficient because it happens
prior to SSL setup.
A maximum of 32 subscriptions are allowed by default (per connection), although this is
configurable with the `max_subscriptions` configuration setting.
A maximum of 32 subscriptions are allowed by default, although this is configurable with the
`max_subscriptions` configuration setting.
## NIP Support
@ -85,6 +73,8 @@ as well as remembering these (id,pubkey) pairs to reject such events subsequentl
Chorus fully complies with NIP-11.
Chorus returns a brief result without much detail.
### NIP-26 Delegated Event Signing
Chorus does not support NIP-26.
@ -134,3 +124,4 @@ Chorus does not support NIP-94.
### NIP-96 HTTP File Storage Integration
Chorus does not support NIP-96.

221
docs/CONFIG.md
View File

@ -2,7 +2,7 @@
The chorus binary requires one command line parameter which specifies the config file path.
The config file must be in TOML format. See the [TOML documentation](https://github.com/toml-lang/toml)
The config file must be in RON format. See the [RON documentation](https://docs.rs/ron/latest/ron/)
## Configuration Variables
@ -10,7 +10,7 @@ The config file must be in TOML format. See the [TOML documentation](https://git
This is the directory where chorus stores data.
Default is "/opt/chorus/var/chorus".
Default is "/tmp".
If deployed according to [docs/DEPLOYING.md](docs/DEPLOYING.md), is "/opt/chorus/var/chorus".
@ -34,23 +34,6 @@ Default is 443.
This is the DNS hostname of your relay. This is used for verifying AUTH events, which specify
your relay host name.
Default is localhost
### chorus_is_behind_a_proxy
If chorus is behind a proxy like nginx, set this to true. In this case chorus will look for and
trust the `X-Real-Ip` HTTP request header to get the real IP of the client. This header MUST exist
or the connection will not be served.
Default is false.
### base_url
If chorus is behing a proxy, it can't compute it's Internet-visible URL. So set it here.
This is used currently by web management and blossom and maybe more in the future.
Default is not set.
### use_tls
If true, chorus will handle TLS, running over HTTPS. If false, chorus run over HTTP.
@ -58,81 +41,57 @@ If true, chorus will handle TLS, running over HTTPS. If false, chorus run over
If you are proxying via nginx, normally you will set this to false and allow nginx to handle
TLS.
Default is true
### certchain_pem_path
This is the path to your TLS certificate chain file.
If `use_tls` is false, this value is irrelevant.
Default is "/opt/chorus/etc/tls/fullchain.pem"
Default is "./tls/fullchain.pem".
If deployed according to [docs/DEPLOYING.md](docs/DEPLOYING.md) using the direct method,
systemd service copies letsencrypt TLS certificates into this position on start.
this is set to "/opt/chorus/etc/tls/fullchain.pem" and the systemd service copies letsencrypt
TLS certificates into this position on start.
### key_pem_path
This is the path to your TLS private key file.
This is the path to yoru TLS private key file.
If `use_tls` is false, this value is irrelevant.
Default is "/opt/chorus/etc/tls/privkey.pem"
Default is "./tls/privkey.pem".
If deployed according to [docs/DEPLOYING.md](docs/DEPLOYING.md) using the direct method,
systemd service copies letsencrypt TLS certificates into this position on start.
this is set to "/opt/chorus/etc/tls/privkey.pem" and the systemd service copies letsencrypt
TLS certificates into this position on start.
### name
This is an optional name for your relay, displayed in the NIP-11 response.
Default is "Chorus Default"
Default is None.
### description
This is an optional description for your relay, displayed in the NIP-11 response.
Default is "A default config of the Chorus relay".
### banner_url
This is an optional URL for an graphical banner representing your relay, displayed in the NIP-11 response.
### icon_url
This is an optional URL for an graphical icon representing your relay, displayed in the NIP-11 response.
### privacy_policy
This is an optional privacy policy as a blob of text (not a URL, not HTML).
### terms_of_service
This is an optional terms of service document as a blob of text (not a URL, not HTML).
Default is None.
### contact
This is an optional administrative contact for your relay, displayed in the NIP-11 response.
This is an optional contact for your relay, displayed in the NIP-11 response.
Default is None.
### contact_public_key_hex
### public_key_hex
This is an optional public key (hex format) for your relay's administrative contact, displayed in the NIP-11 response.
Deprecated "public_key_hex" also works.
This is an optional public key (hex format) for your relay, displayed in the NIP-11 response.
Default is None.
### open_relay
### user_hex_keys
If open_relay true, the relay behaves as an open public relay.
Default is false.
### admin_hex_keys
These are the public keys (hex format) of your relay's administrators. This does NOT automatically make them a relay user, but it will eventually allow them to add/remove users and moderators.
These are the public keys (hex format) of your relay's authorized users. See [BEHAVIOR.md](BEHAVIOR.md) to understand how chorus uses these.
Default is `[]`
@ -146,157 +105,27 @@ Default is true.
### allow_scraping
This is a boolean indicating whether or not scraping is allowed. Scraping is any filter where all of the following are true:
This is a boolean indicating whether or not scraping is allowed. Scraping is any filter that does not match one of the following conditions:
- `ids` is missing or empty
- `authors` is missing or empty
- There are no `#X` tag filters
- A non-empty `id` list is set
- A non-empty `authors` list is set and a non-empty `kinds` list is set
- A non-empty `authors` list is set and at least one tag is set.
- A non-empty `kinds` list is set and at least one tag is set.
Filter that fail to match these conditions will be rejected if `allow_scraping` is false.
If `allow_scraping` is true, be aware that filters that don't match any of these conditions (and are not bound by since/until/limit) have no indexes to speed up their query, so they scan through every single event on the relay.
If `allow_scraping` is true, be aware that filters that don't match any of these conditions have no indexes to speed up their query, so they scan through every single event on the relay.
See also `allow_scrape_if_limited_to` and # `allow_scrape_if_max_seconds`.
The purpose of this setting is as a temporary setting that allows you to dump every single event on your relay.
Default is false.
### allow_scrape_if_limited_to
This is a u32 count of events indicating a filter `limit` value under which a scrape is allowed, irrespective of the `allow_scraping` setting. Such scrapes are not expensive due to the limit. The limit must be specified in the filter; it does not do the scrape and then count to see if it was under the limit.
See `allow_scraping` to learn the definition of a scrape.
The default is 100.
### allow_scrape_if_max_seconds
This is a u64 number of seconds indicating a filter time range under which a scrape is allowed, irrespective of the `allow_scraping` setting. Such scrapes are rarely expensive due to the short time period.
See `allow_scraping` to learn the definition of a scrape.
The default is 7200.
### allow_scrape_if_negentropy
Within negentropy synchronization, allow scraping.
The default is true
### max_subscriptions
This is an integer indicating the maximum number of subscriptions a connection can have open at a given time.
This is a usize indicating the maximum number of subscriptions a connection can have open at a given time.
If you set this too low, clients will be incentivised to resubmit updated subscriptions which will pull down the same events over again, instead of submitting a new subscription that only gets the additional events that the client wants. It may seem intuitive that setting this to a low value like 10 will decrease server load, but it will probably increase server load.
It is strongly recommended to not go below 16.
Default is 128.
### serve_ephemeral
Whether or not to accept and serve all ephemeral events to everybody.
Default is true.
### serve_relay_lists
Whether or not to accept and serve kind 10002 Relay List Metadata (NIP-65) events to everybody.
Default is true.
### server_log_level
How verbose to log issues with the main server code.
Possible values are: Trace, Debug, Info, Warn, Error
Default is Info
### library_log_level
How verbose to log library issues and other general issues
Possible values are: Trace, Debug, Info, Warn, Error
Default is Info
### client_log_level
How verbose to log issues with client requests
Possible values are: Trace, Debug, Info, Warn, Error
Default is Info
### enable_ip_blocking
Whether to block incoming connections based on recent prior behavior
Chorus normally blocks IP addresses for a short period preventing quick reconnections, and for a longer period if the previous connection ended in some error condition.
By setting this variable to false, it will allow all connections, incluing poorly behaving clients that reconnect over and over in a tight loop.
Default is true
### minimum_ban_seconds
Number of seconds to ban an IP address after disconnection.
Enforcing this minimum ban prevents clients from immediately reconnecting which can cause tight loops.
Only relevant if enable_ip_blocking is true.
Default is 1
### timeout_seconds
Number of seconds beyond which chorus times out a client that has no open subscriptions.
Default is 60
### max_connections_per_ip
Maximum number of websocket connections per IP address
Default is 5
### throttling_bytes_per_second
The maximum rate (excluding bursts) of data that will be transmitted over a websocket connection
(both directions, per connection). Beyond this rate (in a sustained way) the connection will be
closed.
Default is 1024576 bytes per second.
### throttling_burst
The allowable bursts of data beyond the normal rate.
We keep a count of how many bytes they are allowed, and that count starts at this number.
As bytes are consumed the count goes down, but we refund throttling_bytes_per_second every
second. If that bucket doesn't have enough, the burst won't be allowed and the connection
will be closed.
Default is 16777216 bytes.
### blossom_directory
Blossom server directory
Set to a filesystem directory where you want chorus to store files.
Blossom allows clients to upload files making them available for the public to download.
Our implementation makes all files publicly readable, but only chorus users can upload
or delete. See https://github.com/hzrd149/blossom
Default is None
### enable_negentropy
Whether to allow negentropy syncing or not
If this is enabled, keep in mind that some negentropy syncs are scrapes and if you want to
allow those, you should also enable allow_scraping. But this will require scans of the entire
database since scrapes have no indexes.
Default is false
Default is 32.

45
docs/DEPLOYING.md
View File

@ -8,15 +8,6 @@ Generally these are servers in data centres, but you might be able to make a por
to the Internet on a home machine if your ISP doesn't use CGNAT and you know how to
configure your firewall/router for this. We leave this up to you.
## Operating System
Chorus was developed on and for linux. It might work on other operating systems, but I
won't spend any time getting it working on non-linux systems.
In particular, I wrote [mmap-append](https://github.com/mikedilger/mmap-append) to use
linux specific mmap functionality. Ashkan Kiani came later and generalized the code but
I haven't tried it elsewhere and so I can't vouch for it elsewhere.
## Deploying the files
You'll want to create a `chorus` user. Here is an example for debian based systems:
@ -43,7 +34,6 @@ sudo chown $(id -u) /opt/chorus/src
cd /opt/chorus/src
git clone https://github.com/mikedilger/chorus
cd chorus
git checkout latest
```
Check if you have `rustc` and `cargo` installed. If so, you can skip this part.
@ -66,16 +56,16 @@ Now let's continue by building chorus:
cargo build --release
```
Ok now let's install that (along with the utility binaries):
Ok now let's install that:
```bash
sudo install --mode=0700 --owner=chorus ./target/release/{chorus,chorus_compress,chorus_dump,chorus_dump_approvals,chorus_moderate,chorus_cmd} /opt/chorus/sbin/
sudo install --mode=0700 --owner=chorus ./target/release/chorus /opt/chorus/sbin/chorus
```
Now let's create our config file
```bash
sudo -u chorus cp /opt/chorus/src/chorus/contrib/chorus.toml /opt/chorus/etc/
sudo -u chorus cp /opt/chorus/src/chorus/contrib/chorus.ron /opt/chorus/etc/
```
Go ahead and edit that file to your liking. In particular:
@ -83,9 +73,8 @@ Go ahead and edit that file to your liking. In particular:
- Change the `ip_address` to your internet-accessible IP address (if you are running directly)
or to 127.0.0.1 with a local port like 8080 (if you are proxying behind nginx)
- Change the port if necessary
- Change the name, description, banner_url, icon_url, privacy_policy, terms_of_service and
contact (e.g. your email address) as desired
- Set your contact_public_key_hex (it is an option, so use `Some()`)
- Change the name, description, and contact (e.g. your email address) as desired
- Set your public_key_hex (it is an option, so use `Some()`)
- Set hex keys of users for which this relay will act as a personal relay
@ -167,10 +156,6 @@ Restart nginx
sudo systemctl restart nginx.service
```
## Adding users and moderators
See [MANAGEMENT](MANAGEMENT.md) for how to add users and moderators.
## Monitoring the service
You can watch the logs with a command like this
@ -179,23 +164,3 @@ You can watch the logs with a command like this
sudo journalctl -f -u chorus.service
```
## Updating
````bash
cd /opt/chorus/src/chorus
git pull
cargo build --release
sudo install --mode=0700 --owner=chorus ./target/release/{chorus,chorus_compress,chorus_dump,chorus_dump_approvals,chorus_moderate,chorus_cmd} /opt/chorus/sbin/
sudo systemctl restart chorus.service
````
## Uninstalling
```bash
sudo systemctl disable chorus.service
sudo rm -f /etc/nginx/sites-available/chorus.nginx.conf
sudo rm -f /etc/nginx/sites-enabled/chorus.nginx.conf
sudo systemctl restart nginx.service
sudo rm -rf /opt/chorus
sudo userdel chorus
```

42
docs/MANAGEMENT.md
View File

@ -1,42 +0,0 @@
# Chorus Management
This page is about using the online Management API or command line [TOOLS.md](TOOLS.md)
to manage users and moderate events.
## Managing users
To list users: `chorus_cmd <configtoml> dump_users`
To add a user: `chorus_cmd <configtoml> add_user <pubkeyhex> 0`
To add a moderator: `chorus_cmd <configtoml> add_user <pubkeyhex> 1`
To remove moderator flag, just add the user again with 0.
To remove a user: `chorus_cmd <configtoml> rm_user <pubkeyhex>`
## Managing events
To cycle through all events needing moderation, and address each one, use `chorus_moderate <configtoml>`
To remove an event by id: `chorus_cmd <configtoml> delete_by_id <idhex>`
To remove multiple events by pubkey: `chorus_cmd <configtoml> delete_by_pubkey <pubkeyhex>`
## Relay Management NIP
The Relay Management API is in flux currently. It is still a pull request on the NIPs repo: [PR 1325](https://github.com/nostr-protocol/nips/pull/1325).
This document may go out of date as things are changing rapidly.
## The status of a pubkey (user)
Users can be in one of four moderation states: Authorized, Approved, Banned, and Default.
**Authorized**: These are the users you have added as authorized (whether or not they are a moderator). These user's events are always accepted and these users can read everything (except other user's DMs for example).
**Approved**: These are users who are not authorized, but which via moderation have been approved. Approved user's posts are publically available to anybody to read. Because they are not authorized, they can only post replies to authorized users.
**Banned**: These are users who cannot make any posts at all to the relay.
**Default**: All pubkeys not explicitly put into any of the other three categories default to this category. Because they are not authorized, they can only post replies to authorized users. Because they are not approved, these replies are only visible to authorized users and are not publicly visible (unless and until a moderator approves the specific post).

25
docs/MIGRATION.md
View File

@ -1,25 +0,0 @@
# Migration
## From 1.0 to 2.0
1) Add to your config file `admin_hex_keys` to include the nostr hex keys of administrators.
These will (eventually) be allowed to manage users and moderators remotely via the management
interface. Note that being an admin does NOT automatically grant user or moderator rights,
it ONLY grants the right to administer users.
2) Users and moderators are now dynamically configured in the database. Use `chorus_cmd` to
manage them from the command line:
* Adding a user: `chorus_cmd <chorus.toml> add_user <pubkey> 0`
* Adding a moderator: `chorus_cmd <chorus.toml> add_user <pubkey> 1`
* Removing a user or moderator: `chorus_cmd <chorus.toml> rm_user <pubkey>`
* Listing users and moderators: `chorus_cmd <chorus.toml> dump_users`
3) Remove the following from your config file as these are no longer used:
* `user_hex_keys` - users are now dynamically configured and configuration is in the database.
* `moderator_hex_keys` - moderators are now dynamically configured and configuration is in the database.
4) Configuration setting `public_key_kex` has been renamed `contact_public_key_hex` and is
used only for the NIP-11 data.

12
docs/PERSONAL_RELAY.md
View File

@ -5,14 +5,14 @@ users, perhaps just yourself.
It hosts your own events and makes them available to the public.
It accepts events that tag your users, but doesn't allow the public to read them back until
they have passed moderation and until then, only your users (after being authenticated) can see
them.
It accepts events that tag your users, but doesn't allow the public to read them back. Only
your users (after being authenticated) can see them.
One nice thing about a personal relay is that you are in control, and you have a record
of your events in your possession.
Another nice thing is that you don't have to moderate content if you don't care to.
Another nice thing is that you don't have to moderate content, since the only content
that is served to the public is content that your trusted users created.
## The Dark Side
@ -23,7 +23,3 @@ they will need to setup SSL connections to far too many relays.
It may be better for others if people were to use a smaller number of larger relay services.
But who am I to say? You are the sovereign. Here is your personal relay.
## Open Relay
You can run chorus as an open public relay. Just set `open_relay` to true in the config.

35
docs/TOOLS.md
View File

@ -1,35 +0,0 @@
# Tools
Chorus comes with four binaries other than the main `chorus` binary.
## chorus_dump
Usage: **chorus_dump** *<path_to_config_file\>*
This dumps every event to STDOUT in JSON format.
## chorus_compress
Usage: **chorus_compress** *<path_to_config_file\>*
This compresses the data by rewriting it entirely.
This leaves the old data under `.bak` extensions (`event.map.bak` and `lmdb.bak`) If these arready exist, the compress command will fail. You are responsible for deleting or saving this files.
## chorus_dump_approvals
Usage: **chorus_dump_approvals** *<path_to_config_file\>*
This shows the public keys you have approved.
## chorus_moderate
Usage: **chorus_moderate** *<path_to_config_file\>*
This is an interactive tool to moderate events which have been accepted, but which are not authored by a chorus user. If you chose to approve them, they will become available to the public.
## chorus_cmd
Usage: **chorus_cmd** *<path_to_config_file\>* *<command\>* *[args...]*
Commands available: delete_by_id (specify the ID in hex), delete_by_pubkey (specify the pubkey in hex)

4
run_sample.sh
View File

@ -1,4 +1,6 @@
#!/bin/bash
export RUST_LOG='info,chorus=debug'
cargo build --release && \
./target/release/chorus ./sample/sample.config.toml
./target/release/chorus ./sample/sample.config.ron

4
sample/.gitignore vendored
View File

@ -1,4 +1,2 @@
lmdb/
lmdb.bak/
event.map
event.map.bak
event.map

22
sample/sample.config.ron Normal file
View File

@ -0,0 +1,22 @@
FriendlyConfig(
data_directory: "./sample",
ip_address: "127.0.0.1",
port: 8080,
hostname: "localhost",
use_tls: false,
certchain_pem_path: "tls/fullchain.pem",
key_pem_path: "tls/privkey.pem",
name: Some("Chorus Sample"),
description: Some("A sample run of the Chorus relay"),
contact: None,
public_key_hex: None,
user_hex_keys: [
"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"
],
verify_events: true,
// This is a bad idea in production, but useful for testing or for dumping
// your entire relay
allow_scraping: true,
max_subscriptions: 32,
)

35
sample/sample.config.toml
View File

@ -1,35 +0,0 @@
# See contrib/chorus.toml for a documented config file
data_directory = "./sample"
ip_address = "127.0.0.1"
port = 8081
hostname = "localhost"
chorus_is_behind_a_proxy = false
use_tls = false
certchain_pem_path = "tls/fullchain.pem"
key_pem_path = "tls/privkey.pem"
name = "Chorus Sample"
description = "A sample run of the Chorus relay"
privacy_policy = "This relay guarantees nothing. Privacy is your concern, not ours."
terms_of_service = "This relay guarantees nothing. Use at your own risk."
# icon_url =
open_relay = false
admin_hex_keys = [
"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"
]
verify_events = true
allow_scraping = false
allow_scrape_if_limited_to = 100
allow_scrape_if_max_seconds = 7200
max_subscriptions = 128
serve_ephemeral = true
serve_relay_lists = true
server_log_level = "Info"
library_log_level = "Info"
client_log_level = "Warn"
enable_ip_blocking = true
minimum_ban_seconds = 1
timeout_seconds = 60
max_connections_per_ip = 5
throttling_bytes_per_second = 131072
throttling_burst = 4194304

192
src/bin/chorus.rs
View File

@ -1,192 +0,0 @@
use chorus::config::{Config, FriendlyConfig};
use chorus::counting_stream::CountingStream;
use chorus::error::Error;
use chorus::globals::GLOBALS;
use chorus::ip::HashedPeer;
use std::env;
use std::fs::OpenOptions;
use std::io::Read;
use std::sync::atomic::Ordering;
use std::time::Duration;
use tokio::net::TcpListener;
use tokio::signal::unix::{signal, SignalKind};
#[tokio::main]
async fn main() -> Result<(), Error> {
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus <config_path>");
}
let _ = args.next(); // ignore program name
let config_path = args.next().unwrap();
let config = chorus::load_config(&config_path)?;
chorus::setup_logging(&config);
// Log host name
log::info!(target: "Server", "HOSTNAME = {}", config.hostname);
chorus::setup_store(&config)?;
if let Some(ref blossom_directory) = config.blossom_directory {
let filestore = chorus::filestore::FileStore::new(blossom_directory).await?;
let _ = GLOBALS.filestore.set(filestore);
}
// TLS setup
let maybe_tls_acceptor = if config.use_tls {
log::info!(target: "Server", "Using TLS");
Some(chorus::tls::tls_acceptor(&config)?)
} else {
log::info!(target: "Server", "Not using TLS");
None
};
// Bind listener to port
let listener = TcpListener::bind((&*config.ip_address, config.port)).await?;
log::info!(target: "Server", "Running on {}:{}", config.ip_address, config.port);
// Store config into GLOBALS
*GLOBALS.config.write() = config;
let mut interrupt_signal = signal(SignalKind::interrupt())?;
let mut quit_signal = signal(SignalKind::quit())?;
let mut terminate_signal = signal(SignalKind::terminate())?;
let mut hup_signal = signal(SignalKind::hangup())?;
loop {
tokio::select! {
// Exits gracefully upon exit-type signals
v = interrupt_signal.recv() => if v.is_some() {
log::info!(target: "Server", "SIGINT");
break;
},
v = quit_signal.recv() => if v.is_some() {
log::info!(target: "Server", "SIGQUIT");
break;
},
v = terminate_signal.recv() => if v.is_some() {
log::info!(target: "Server", "SIGTERM");
break;
},
// Reload config on HUP
v = hup_signal.recv() => if v.is_some() {
log::info!(target: "Server", "SIGHUP: Reloading configuration");
// Reload the config file
let mut file = OpenOptions::new().read(true).open(config_path.clone())?;
let mut contents = String::new();
file.read_to_string(&mut contents)?;
let friendly_config: FriendlyConfig = toml::from_str(&contents)?;
let config: Config = friendly_config.into_config()?;
*GLOBALS.config.write() = config;
chorus::print_stats();
},
// Accepts network connections and spawn a task to serve each one
v = listener.accept() => {
let (tcp_stream, hashed_peer) = {
let (tcp_stream, peer_addr) = v?;
let hashed_peer = HashedPeer::new(peer_addr);
(tcp_stream, hashed_peer)
};
// Possibly IP block early
if ! GLOBALS.config.read().chorus_is_behind_a_proxy
&& GLOBALS.config.read().enable_ip_blocking
{
let ip_data = chorus::get_ip_data(hashed_peer.ip())?;
if ip_data.is_banned() {
log::debug!(target: "Client",
"{}: Blocking reconnection until {}",
hashed_peer.ip(),
ip_data.ban_until);
// note: no need to shutdown() which only drops the write half.
// the whole thing gets dropped when we continue.
continue;
}
}
let counting_stream = CountingStream(tcp_stream);
let maybe_tls_acceptor_clone = maybe_tls_acceptor.clone();
tokio::spawn(async move {
match maybe_tls_acceptor_clone {
Some(tls_acceptor) => {
match tls_acceptor.accept(counting_stream).await {
Ok(stream) => {
let io = hyper_util::rt::TokioIo::new(stream);
chorus::serve(io, hashed_peer).await;
},
Err(e) => {
log::error!(
target: "Client",
"{}: TLS accept: {}", hashed_peer, e
);
}
}
},
None => {
let io = hyper_util::rt::TokioIo::new(counting_stream);
chorus::serve(io, hashed_peer).await;
}
};
});
}
};
}
// Pre-sync in case something below hangs up
let _ = GLOBALS.store.get().unwrap().sync();
// Set the shutting down signal
let _ = GLOBALS.shutting_down.send(true);
// Wait for active websockets to shutdown gracefully
let mut num_connections = GLOBALS.num_connections.load(Ordering::Relaxed);
if num_connections != 0 {
log::info!(target: "Server", "Waiting for {num_connections} websockets to shutdown...");
// We will check if all clients have shutdown every 50ms
let interval = tokio::time::interval(Duration::from_millis(50));
tokio::pin!(interval);
let mut ms = 0;
while num_connections != 0 {
// If we get another shutdown signal, stop waiting for websockets
tokio::select! {
v = interrupt_signal.recv() => if v.is_some() {
break;
},
v = quit_signal.recv() => if v.is_some() {
break;
},
v = terminate_signal.recv() => if v.is_some() {
break;
},
_instant = interval.tick() => {
ms += 50;
if ms > 5_000 {
log::info!(target: "Server", "Some connections were hung.");
break;
}
num_connections = GLOBALS.num_connections.load(Ordering::Relaxed);
continue;
}
}
}
}
chorus::print_stats();
log::info!(target: "Server", "Syncing and shutting down.");
let _ = GLOBALS.store.get().unwrap().sync();
Ok(())
}

107
src/bin/chorus_cmd.rs
View File

@ -1,107 +0,0 @@
use chorus::error::{ChorusError, Error};
use chorus::globals::GLOBALS;
use pocket_db::ScreenResult;
use pocket_types::{Filter, Id, Pubkey, Tags};
use std::env;
const USAGE: &str = "usage: chorus_cmd <config_path> <command> [args...]";
fn main() -> Result<(), Error> {
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus_cmd <config_path> <command> [args...]");
}
let _ = args.next(); // ignore program name
// Load config
let config_path = args
.next()
.ok_or::<Error>(ChorusError::General(USAGE.to_owned()).into())?;
let mut config = chorus::load_config(config_path)?;
// Force allow of scraping (this program is a scraper)
config.allow_scraping = true;
chorus::setup_logging(&config);
chorus::setup_store(&config)?;
// Handle command
let command = args
.next()
.ok_or::<Error>(ChorusError::General(USAGE.to_owned()).into())?;
match &*command {
"delete_by_id" => {
let idstr = args
.next()
.ok_or::<Error>(ChorusError::General("ID argument missing".to_owned()).into())?;
let id: Id = Id::read_hex(idstr.as_bytes())?;
GLOBALS.store.get().unwrap().remove_event(id)?;
println!("Done.");
}
"delete_by_pubkey" => {
let pubstr = args.next().ok_or::<Error>(
ChorusError::General("Pubkey argument missing".to_owned()).into(),
)?;
let pk: Pubkey = Pubkey::read_hex(pubstr.as_bytes())?;
let mut tags_buffer: [u8; 128] = [0; 128];
let (_, tags) = Tags::from_json(b"[]", &mut tags_buffer)?;
let mut filter_buffer: [u8; 128] = [0; 128];
let filter =
Filter::from_parts(&[], &[pk], &[], tags, None, None, None, &mut filter_buffer)?;
let (events, _redacted) =
GLOBALS
.store
.get()
.unwrap()
.find_events(filter, true, 0, 0, |_| ScreenResult::Match)?;
for event in events.iter() {
GLOBALS.store.get().unwrap().remove_event(event.id())?;
}
println!("Done.");
}
"fetch_by_id" => {
let idstr = args
.next()
.ok_or::<Error>(ChorusError::General("ID argument missing".to_owned()).into())?;
let id: Id = Id::read_hex(idstr.as_bytes())?;
if let Some(event) = GLOBALS.store.get().unwrap().get_event_by_id(id)? {
println!("{event}");
} else {
println!("Not found.");
}
}
"dump_users" => {
let users = chorus::dump_authorized_users()?;
for (pubkey, moderator) in users.iter() {
println!("{} {}", pubkey, if *moderator { "moderator" } else { "" });
}
}
"add_user" => {
let pubstr = args.next().ok_or::<Error>(
ChorusError::General("Pubkey argument missing".to_owned()).into(),
)?;
let pk: Pubkey = Pubkey::read_hex(pubstr.as_bytes())?;
let moderator = args.next().ok_or::<Error>(
ChorusError::General("Moderator argument missing".to_owned()).into(),
)?;
let moderator: bool = moderator == "1";
chorus::add_authorized_user(pk, moderator)?;
}
"rm_user" => {
let pubstr = args.next().ok_or::<Error>(
ChorusError::General("Pubkey argument missing".to_owned()).into(),
)?;
let pk: Pubkey = Pubkey::read_hex(pubstr.as_bytes())?;
chorus::rm_authorized_user(pk)?;
}
_ => {
return Err(ChorusError::General("Unknown command.".to_owned()).into());
}
}
Ok(())
}

33
src/bin/chorus_compress.rs
View File

@ -1,33 +0,0 @@
use chorus::error::Error;
use std::env;
use std::io::BufRead;
fn main() -> Result<(), Error> {
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus_compress <config_path>");
}
let _ = args.next(); // ignore program name
let config_path = args.next().unwrap();
let config = chorus::load_config(config_path)?;
chorus::setup_logging(&config);
println!("Chorus must NOT be running when you do this.");
println!("Proceed? (break out with ^C, or press <ENTER> to proceed)");
let stdin = std::io::stdin();
let _ = stdin.lock().lines().next().unwrap().unwrap();
let store = chorus::setup_store_and_return(&config)?;
let pre_stats = store.stats()?;
println!("{:?}", pre_stats);
let new_store = unsafe { store.rebuild()? };
let post_stats = new_store.stats()?;
println!("{:?}", post_stats);
Ok(())
}

43
src/bin/chorus_dump.rs
View File

@ -1,43 +0,0 @@
use chorus::error::Error;
use chorus::globals::GLOBALS;
use pocket_db::ScreenResult;
use pocket_types::{Event, Filter};
use std::env;
fn main() -> Result<(), Error> {
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus_dump <chorus_config_path>");
}
let _ = args.next(); // ignore program name
let config_path = args.next().unwrap();
let mut config = chorus::load_config(config_path)?;
// Force allow of scraping (this program is a scraper)
config.allow_scraping = true;
chorus::setup_logging(&config);
chorus::setup_store(&config)?;
let mut buffer: [u8; 128] = [0; 128];
let (_incount, _outcount, filter) = Filter::from_json(b"{}", &mut buffer)?;
let screen = |_: &Event| -> ScreenResult { ScreenResult::Match };
let (mut events, _redacted) = GLOBALS.store.get().unwrap().find_events(
filter,
config.allow_scraping,
config.allow_scrape_if_limited_to,
config.allow_scrape_if_max_seconds,
screen,
)?;
for event in events.drain(..) {
let bytes = event.as_json()?;
let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
println!("{s}");
}
Ok(())
}

30
src/bin/chorus_dump_approvals.rs
View File

@ -1,30 +0,0 @@
use chorus::error::Error;
use std::env;
fn main() -> Result<(), Error> {
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus_moderate <config_path>");
}
let _ = args.next(); // ignore program name
let config_path = args.next().unwrap();
let mut config = chorus::load_config(config_path)?;
// Force allow of scraping (this program is a scraper)
config.allow_scraping = true;
chorus::setup_logging(&config);
chorus::setup_store(&config)?;
for (id, approved) in chorus::dump_event_approvals()? {
println!("ID {} = {}", id, approved);
}
for (pubkey, approved) in chorus::dump_pubkey_approvals()? {
println!("PUBKEY {} = {}", pubkey, approved);
}
Ok(())
}

27
src/bin/chorus_init.rs
View File

@ -1,27 +0,0 @@
use chorus::error::Error;
use chorus::globals::GLOBALS;
use std::env;
#[tokio::main]
async fn main() -> Result<(), Error> {
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus <config_path>");
}
let _ = args.next(); // ignore program name
let config_path = args.next().unwrap();
let config = chorus::load_config(&config_path)?;
chorus::setup_logging(&config);
// Log host name
log::info!(target: "Server", "HOSTNAME = {}", config.hostname);
chorus::setup_store(&config)?;
let _ = GLOBALS.store.get().unwrap().sync();
Ok(())
}

140
src/bin/chorus_moderate.rs
View File

@ -1,140 +0,0 @@
use chorus::error::Error;
use chorus::globals::GLOBALS;
use pocket_db::ScreenResult;
use pocket_types::{Event, Filter, Kind};
use std::env;
use std::io::Write;
fn main() -> Result<(), Error> {
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus_moderate <config_path>");
}
let _ = args.next(); // ignore program name
let config_path = args.next().unwrap();
let mut config = chorus::load_config(config_path)?;
// Force allow of scraping (this program is a scraper)
config.allow_scraping = true;
chorus::setup_logging(&config);
// These kinds don't need approval:
let allowed_kinds = [
Kind::from(4), // Encrypted Direct Message
Kind::from(1059), // Giftwrap
Kind::from(10002), // Relay list
Kind::from(10050), // DM Relay list
Kind::from(0), // Metadata
Kind::from(3), // Following list
Kind::from(7), // Reaction
];
chorus::setup_store(&config)?;
let mut buffer: [u8; 128] = [0; 128];
let (_incount, _outcount, filter) = Filter::from_json(b"{}", &mut buffer)?;
let screen = |_: &Event| -> ScreenResult { ScreenResult::Match };
let (mut events, _redacted) = GLOBALS.store.get().unwrap().find_events(
filter,
config.allow_scraping,
config.allow_scrape_if_limited_to,
config.allow_scrape_if_max_seconds,
screen,
)?;
let mut stdout = std::io::stdout();
let stdin = std::io::stdin();
let mut input = String::new();
'eventloop: for event in events.drain(..) {
if allowed_kinds.contains(&event.kind()) {
continue;
}
// Skip ephemeral events which don't need approval
if event.kind().is_ephemeral() {
continue;
}
// Skip if the author is authorized user
if chorus::is_authorized_user(event.pubkey()) {
continue;
}
//let bytes = event.as_json()?;
//let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
//println!("{s}");
// Skip if event marked approved
if matches!(chorus::get_event_approval(event.id()), Ok(Some(true))) {
continue;
}
// Skip if pubkey marked approved
if matches!(chorus::get_pubkey_approval(event.pubkey()), Ok(Some(true))) {
continue;
}
// Delete if pubkey marked banned
if matches!(chorus::get_pubkey_approval(event.pubkey()), Ok(Some(false))) {
GLOBALS.store.get().unwrap().remove_event(event.id())?;
continue;
}
println!("---------------------------------------------------------------");
println!("kind={} id={}", event.kind(), event.id());
println!("{}", String::from_utf8_lossy(event.content()));
println!("---------------------------------------------------------------");
println!(" Pubkey: (p) approve, (P) ban and delete");
println!(" Id: (i) approve, (I) ban and delete");
println!(" other: (s) skip, (q) quit");
loop {
print!(">> ");
let _ = stdout.flush();
input.clear();
stdin.read_line(&mut input)?;
if input.is_empty() {
continue;
}
match input.bytes().next().unwrap() {
b'p' => {
chorus::mark_pubkey_approval(event.pubkey(), true)?;
println!("User approved.");
break;
}
b'P' => {
chorus::mark_pubkey_approval(event.pubkey(), false)?;
GLOBALS.store.get().unwrap().remove_event(event.id())?;
println!("User banned.");
break;
}
b'i' => {
chorus::mark_event_approval(event.id(), true)?;
println!("Event approved.");
break;
}
b'I' => {
chorus::mark_event_approval(event.id(), false)?;
GLOBALS.store.get().unwrap().remove_event(event.id())?;
println!("Event banned.");
break;
}
b's' => {
println!("Skipped.");
break;
}
b'q' => break 'eventloop,
_ => {
println!("?");
}
}
}
}
Ok(())
}

217
src/config.rs
View File

@ -1,95 +1,45 @@
use crate::error::Error;
use hyper::http::uri::{Authority, Scheme, Uri};
use pocket_types::Pubkey;
use crate::types::Pubkey;
use serde::{Deserialize, Serialize};
use std::str::FromStr;
use url::Host;
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default)]
pub struct FriendlyConfig {
pub data_directory: String,
pub ip_address: String,
pub port: u16,
pub hostname: String,
pub chorus_is_behind_a_proxy: bool,
pub base_url: Option<String>,
pub use_tls: bool,
pub certchain_pem_path: String,
pub key_pem_path: String,
pub name: Option<String>,
pub description: Option<String>,
pub banner_url: Option<String>,
pub icon_url: Option<String>,
pub privacy_policy: Option<String>,
pub terms_of_service: Option<String>,
pub contact: Option<String>,
#[serde(alias = "public_key_hex")]
pub contact_public_key_hex: Option<String>,
pub open_relay: bool,
pub admin_hex_keys: Vec<String>,
pub public_key_hex: Option<String>,
pub user_hex_keys: Vec<String>,
pub verify_events: bool,
pub allow_scraping: bool,
pub allow_scrape_if_limited_to: u32,
pub allow_scrape_if_max_seconds: u64,
pub allow_scrape_if_negentropy: bool,
pub max_subscriptions: usize,
pub serve_ephemeral: bool,
pub serve_relay_lists: bool,
pub server_log_level: String,
pub library_log_level: String,
pub client_log_level: String,
pub enable_ip_blocking: bool,
pub minimum_ban_seconds: u64,
pub timeout_seconds: u64,
pub max_connections_per_ip: usize,
pub throttling_bytes_per_second: usize,
pub throttling_burst: usize,
pub blossom_directory: Option<String>,
pub enable_negentropy: bool,
}
impl Default for FriendlyConfig {
fn default() -> FriendlyConfig {
FriendlyConfig {
data_directory: "/opt/chorus/var/chorus".to_string(),
data_directory: "/tmp".to_string(),
ip_address: "127.0.0.1".to_string(),
port: 443,
port: 80,
hostname: "localhost".to_string(),
chorus_is_behind_a_proxy: false,
base_url: None,
use_tls: true,
certchain_pem_path: "/opt/chorus/etc/tls/fullchain.pem".to_string(),
key_pem_path: "/opt/chorus/etc/tls/privkey.pem".to_string(),
name: Some("Chorus Default".to_string()),
description: Some("A default config of the Chorus relay".to_string()),
banner_url: None,
icon_url: None,
privacy_policy: None,
terms_of_service: None,
use_tls: false,
certchain_pem_path: "./tls/fullchain.pem".to_string(),
key_pem_path: "./tls/privkey.pem".to_string(),
name: None,
description: None,
contact: None,
contact_public_key_hex: None,
open_relay: false,
admin_hex_keys: vec![],
public_key_hex: None,
user_hex_keys: vec![],
verify_events: true,
allow_scraping: false,
allow_scrape_if_limited_to: 100,
allow_scrape_if_max_seconds: 7200,
allow_scrape_if_negentropy: true,
max_subscriptions: 128,
serve_ephemeral: true,
serve_relay_lists: true,
server_log_level: "Info".to_string(),
library_log_level: "Info".to_string(),
client_log_level: "Info".to_string(),
enable_ip_blocking: true,
minimum_ban_seconds: 1,
timeout_seconds: 60,
max_connections_per_ip: 5,
throttling_bytes_per_second: 1024 * 1024,
throttling_burst: 1024 * 1024 * 16,
blossom_directory: None,
enable_negentropy: false,
max_subscriptions: 32,
}
}
}
@ -101,101 +51,48 @@ impl FriendlyConfig {
ip_address,
port,
hostname,
chorus_is_behind_a_proxy,
base_url,
use_tls,
certchain_pem_path,
key_pem_path,
name,
description,
banner_url,
icon_url,
privacy_policy,
terms_of_service,
contact,
contact_public_key_hex,
open_relay,
admin_hex_keys,
public_key_hex,
user_hex_keys,
verify_events,
allow_scraping,
allow_scrape_if_limited_to,
allow_scrape_if_max_seconds,
allow_scrape_if_negentropy,
max_subscriptions,
serve_ephemeral,
serve_relay_lists,
server_log_level,
library_log_level,
client_log_level,
enable_ip_blocking,
minimum_ban_seconds,
timeout_seconds,
max_connections_per_ip,
throttling_bytes_per_second,
throttling_burst,
blossom_directory,
enable_negentropy,
} = self;
let mut contact_public_key: Option<Pubkey> = None;
if let Some(pkh) = contact_public_key_hex {
contact_public_key = Some(Pubkey::read_hex(pkh.as_bytes())?);
let mut public_key: Option<Pubkey> = None;
if let Some(pkh) = public_key_hex {
public_key = Some(Pubkey::read_hex(pkh.as_bytes())?);
};
let mut admin_keys: Vec<Pubkey> = Vec::with_capacity(admin_hex_keys.len());
for pkh in admin_hex_keys.iter() {
admin_keys.push(Pubkey::read_hex(pkh.as_bytes())?);
let mut user_keys: Vec<Pubkey> = Vec::with_capacity(user_hex_keys.len());
for pkh in user_hex_keys.iter() {
user_keys.push(Pubkey::read_hex(pkh.as_bytes())?);
}
let hostname = Host::parse(&hostname)?;
let server_log_level =
log::LevelFilter::from_str(&server_log_level).unwrap_or(log::LevelFilter::Info);
let library_log_level =
log::LevelFilter::from_str(&library_log_level).unwrap_or(log::LevelFilter::Info);
let client_log_level =
log::LevelFilter::from_str(&client_log_level).unwrap_or(log::LevelFilter::Info);
Ok(Config {
data_directory,
ip_address,
port,
hostname,
chorus_is_behind_a_proxy,
base_url,
use_tls,
certchain_pem_path,
key_pem_path,
name,
description,
banner_url,
icon_url,
privacy_policy,
terms_of_service,
contact,
contact_public_key,
open_relay,
admin_keys,
admin_hex_keys,
public_key,
user_keys,
user_hex_keys,
verify_events,
allow_scraping,
allow_scrape_if_limited_to,
allow_scrape_if_max_seconds,
allow_scrape_if_negentropy,
max_subscriptions,
serve_ephemeral,
serve_relay_lists,
server_log_level,
library_log_level,
client_log_level,
enable_ip_blocking,
minimum_ban_seconds,
timeout_seconds,
max_connections_per_ip,
throttling_bytes_per_second,
throttling_burst,
blossom_directory,
enable_negentropy,
})
}
}
@ -206,76 +103,16 @@ pub struct Config {
pub ip_address: String,
pub port: u16,
pub hostname: Host,
pub chorus_is_behind_a_proxy: bool,
pub base_url: Option<String>,
pub use_tls: bool,
pub certchain_pem_path: String,
pub key_pem_path: String,
pub name: Option<String>,
pub description: Option<String>,
pub banner_url: Option<String>,
pub icon_url: Option<String>,
pub privacy_policy: Option<String>,
pub terms_of_service: Option<String>,
pub contact: Option<String>,
pub contact_public_key: Option<Pubkey>,
pub open_relay: bool,
pub admin_keys: Vec<Pubkey>,
pub admin_hex_keys: Vec<String>,
pub public_key: Option<Pubkey>,
pub user_keys: Vec<Pubkey>,
pub user_hex_keys: Vec<String>,
pub verify_events: bool,
pub allow_scraping: bool,
pub allow_scrape_if_limited_to: u32,
pub allow_scrape_if_max_seconds: u64,
pub allow_scrape_if_negentropy: bool,
pub max_subscriptions: usize,
pub serve_ephemeral: bool,
pub serve_relay_lists: bool,
pub server_log_level: log::LevelFilter,
pub library_log_level: log::LevelFilter,
pub client_log_level: log::LevelFilter,
pub enable_ip_blocking: bool,
pub minimum_ban_seconds: u64,
pub timeout_seconds: u64,
pub max_connections_per_ip: usize,
pub throttling_bytes_per_second: usize,
pub throttling_burst: usize,
pub blossom_directory: Option<String>,
pub enable_negentropy: bool,
}
impl Default for Config {
fn default() -> Config {
let friendly = FriendlyConfig::default();
// We know the default config passes into_config without error:
friendly.into_config().unwrap()
}
}
impl Config {
/// Get the URI for our server matching the inner Uri, overridden with either
/// our base_url parts or our hostname/port.
pub fn uri_parts(&self, inner: Uri, http: bool) -> Result<http::uri::Parts, Error> {
let mut uri_parts = inner.into_parts();
if let Some(s) = &self.base_url {
let base_uri = s.parse::<Uri>()?;
let base_uri_parts = base_uri.into_parts();
uri_parts.scheme = base_uri_parts.scheme;
uri_parts.authority = base_uri_parts.authority;
} else {
let scheme = match (self.use_tls, http) {
(false, false) => Scheme::from_str("ws").unwrap(),
(true, false) => Scheme::from_str("wss").unwrap(),
(false, true) => Scheme::HTTP,
(true, true) => Scheme::HTTPS,
};
uri_parts.scheme = Some(scheme);
let authority = Authority::from_str(&format!("{}:{}", self.hostname, self.port))?;
uri_parts.authority = Some(authority);
}
Ok(uri_parts)
}
}

53
src/counting_stream.rs
View File

@ -1,53 +0,0 @@
use crate::globals::GLOBALS;
use std::pin::Pin;
use std::sync::atomic::Ordering;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
#[derive(Debug)]
pub struct CountingStream<S>(pub S);
impl<S: AsyncRead + Unpin> AsyncRead for CountingStream<S> {
fn poll_read(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &mut ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
// Count bytes for statistics
let pre = buf.filled().len();
let result = Pin::new(&mut self.get_mut().0).poll_read(cx, buf);
let post = buf.filled().len();
let count = post - pre;
if count > 0 {
let _ = GLOBALS
.bytes_inbound
.fetch_add(count as u64, Ordering::SeqCst);
}
result
}
}
impl<S: AsyncWrite + Unpin> AsyncWrite for CountingStream<S> {
fn poll_write(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &[u8],
) -> Poll<Result<usize, std::io::Error>> {
// Count bytes for statistics
let _ = GLOBALS
.bytes_outbound
.fetch_add(buf.len() as u64, Ordering::SeqCst);
Pin::new(&mut self.get_mut().0).poll_write(cx, buf)
}
fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), std::io::Error>> {
Pin::new(&mut self.get_mut().0).poll_flush(cx)
}
fn poll_shutdown(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Result<(), std::io::Error>> {
Pin::new(&mut self.get_mut().0).poll_shutdown(cx)
}
}

432
src/error.rs
View File

@ -1,4 +1,3 @@
use std::convert::Infallible;
use std::error::Error as StdError;
use std::panic::Location;
@ -23,150 +22,93 @@ impl std::fmt::Display for Error {
/// Errors that can occur in the chorus crate
#[derive(Debug)]
pub enum ChorusError {
// Nostr AUTH failure
// Auth failure
AuthFailure(String),
// Auth required
AuthRequired,
// Bad request
BadRequest(&'static str),
// Bad event id
BadEventId,
// Bad X-Real-Ip header
BadRealIpHeader(String),
// Bad hex input
BadHexInput,
// Bad X-Real-Ip header characters
BadRealIpHeaderCharacters,
// Event is banned
BannedEvent,
// User is banned
BannedUser,
// Base64 Decode Error
Base64Decode(base64::DecodeError), // 24b
// Blocked IP
BlockedIp,
// Blossom Authorization failure
BlossomAuthFailure(String),
// Output buffer too small
BufferTooSmall,
// Channel Recv
ChannelRecv(tokio::sync::broadcast::error::RecvError), // 24b
ChannelRecv(tokio::sync::broadcast::error::RecvError),
// Channel Send
ChannelSend(tokio::sync::broadcast::error::SendError<u64>), // 16b
ChannelSend(tokio::sync::broadcast::error::SendError<usize>),
// Config
Config(Box<toml::de::Error>), // 16b
Config(ron::error::SpannedError),
// Crypto
Crypto(secp256k1::Error), // 16b
Crypto(secp256k1::Error),
// Closing on error(s)
ErrorClose,
// Deleted event
Deleted,
// Duplicate event
Duplicate,
// End of Input
EndOfInput,
// Event is Invalid
EventIsInvalid(String),
// From hex
FromHex(hex::FromHexError), // 24b
// From UTF8
FromUtf8(Box<std::string::FromUtf8Error>),
// General
General(String),
// Http
Http(Box<hyper::http::Error>),
Http(hyper::http::Error),
// Hyper
Hyper(hyper::Error),
// Infallible
Infallible,
// Invalid URI
InvalidUri(hyper::http::uri::InvalidUri),
// Invalid URI Parts
InvalidUriParts(hyper::http::uri::InvalidUriParts),
// I/O
Io(std::io::Error),
// Management Authorization failure
ManagementAuthFailure(String),
// JSON Bad (general)
JsonBad(&'static str, usize),
// Missing Table
MissingTable(&'static str),
// JSON Bad Character
JsonBadCharacter(char, usize, char),
// Negentropy error
Negentropy(negentropy::Error),
// JSON Bad Event
JsonBadEvent(&'static str, usize),
// Non-ASCII HTTP header value
NonAsciiHttpHeaderValue(http::header::ToStrError),
// JSON Bad Filter
JsonBadFilter(&'static str, usize),
// JSON Bad String Character
JsonBadStringChar(u32),
// JSON Escape
JsonEscape,
// JSON Escape Surrogate
JsonEscapeSurrogate,
// LMDB
Lmdb(heed::Error),
// No private key
NoPrivateKey,
// Not Implemented
NotImplemented,
// No such subscription
NoSuchSubscription,
// Protected Event
ProtectedEvent,
// Pocket Db Error
PocketDb(pocket_db::Error),
// Pocket Db Heed Error
PocketDbHeed(Box<pocket_db::heed::Error>),
// Pocket Types Error
PocketType(pocket_types::Error),
// Rate limit exceeded
RateLimitExceeded,
// X-Real-Ip header is missing
RealIpHeaderMissing,
// Restricted
Restricted,
// Rustls
Rustls(Box<tokio_rustls::rustls::Error>),
Rustls(tokio_rustls::rustls::Error),
// Tungstenite
Tungstenite(hyper_tungstenite::tungstenite::error::Error),
// Filter is underspecified
Scraper,
// Serde JSON
SerdeJson(serde_json::Error),
// Shutting Down
ShuttingDown,
// Signal - Not Blossom Request
SignalNotBlossom,
// Speedy
Speedy(speedy::Error),
// Timed Out
TimedOut,
// Too many subscriptions
TooManySubscriptions,
// Tungstenite
Tungstenite(Box<hyper_tungstenite::tungstenite::error::Error>),
// URL Parse
UrlParse(url::ParseError),
@ -185,56 +127,45 @@ impl std::fmt::Display for ChorusError {
match self {
ChorusError::AuthFailure(s) => write!(f, "AUTH failure: {s}"),
ChorusError::AuthRequired => write!(f, "AUTH required"),
ChorusError::BadRequest(s) => write!(f, "Bad Request: {s}"),
ChorusError::BadRealIpHeader(s) => write!(f, "Bad X-Real-Ip header: {s}"),
ChorusError::BadRealIpHeaderCharacters => {
write!(f, "Bad X-Real-Ip header (non utf-8 characters)")
}
ChorusError::BannedEvent => write!(f, "Event is banned"),
ChorusError::BannedUser => write!(f, "User is banned"),
ChorusError::Base64Decode(e) => write!(f, "{e}"),
ChorusError::BlockedIp => write!(f, "IP is temporarily blocked"),
ChorusError::BlossomAuthFailure(s) => write!(f, "Authorization failure: {s}"),
ChorusError::BadEventId => write!(f, "Bad event id, does not match hash"),
ChorusError::BadHexInput => write!(f, "Bad hex input"),
ChorusError::BufferTooSmall => write!(f, "Output buffer too small"),
ChorusError::ChannelRecv(e) => write!(f, "{e}"),
ChorusError::ChannelSend(e) => write!(f, "{e}"),
ChorusError::Config(e) => write!(f, "{e}"),
ChorusError::Crypto(e) => write!(f, "{e}"),
ChorusError::ErrorClose => write!(f, "Closing due to error(s)"),
ChorusError::Deleted => write!(f, "Event was previously deleted"),
ChorusError::Duplicate => write!(f, "Duplicate event"),
ChorusError::EndOfInput => write!(f, "End of input"),
ChorusError::EventIsInvalid(s) => write!(f, "Event is invalid: {s}"),
ChorusError::FromHex(e) => write!(f, "{e}"),
ChorusError::FromUtf8(e) => write!(f, "{e}"),
ChorusError::General(s) => write!(f, "{s}"),
ChorusError::Http(e) => write!(f, "{e}"),
ChorusError::Hyper(e) => write!(f, "{e}"),
ChorusError::Infallible => panic!("INFALLIBLE"),
ChorusError::InvalidUri(e) => write!(f, "{e}"),
ChorusError::InvalidUriParts(e) => write!(f, "{e}"),
ChorusError::Io(e) => write!(f, "{e}"),
ChorusError::ManagementAuthFailure(s) => write!(f, "Authorization failure: {s}"),
ChorusError::MissingTable(t) => write!(f, "Missing table: {t}"),
ChorusError::Negentropy(e) => write!(f, "Negentropy: {e}"),
ChorusError::NonAsciiHttpHeaderValue(e) => {
write!(f, "Non ASCII HTTP header value: {e}")
ChorusError::JsonBad(err, pos) => write!(f, "JSON bad: {err} at position {pos}"),
ChorusError::JsonBadCharacter(c, pos, ec) => write!(
f,
"JSON bad character: {c} at position {pos}, {ec} was expected"
),
ChorusError::JsonBadEvent(err, pos) => {
write!(f, "JSON bad event: {err} at position {pos}")
}
ChorusError::JsonBadFilter(err, pos) => {
write!(f, "JSON bad filter: {err} at position {pos}")
}
ChorusError::JsonBadStringChar(ch) => {
write!(f, "JSON string bad character: codepoint {ch}")
}
ChorusError::JsonEscape => write!(f, "JSON string escape error"),
ChorusError::JsonEscapeSurrogate => write!(
f,
"JSON string escape surrogate (ancient style) is not supported"
),
ChorusError::Lmdb(e) => write!(f, "{e}"),
ChorusError::NoPrivateKey => write!(f, "Private Key Not Found"),
ChorusError::NotImplemented => write!(f, "Not implemented"),
ChorusError::NoSuchSubscription => write!(f, "No such subscription"),
ChorusError::PocketDb(e) => write!(f, "{e}"),
ChorusError::PocketDbHeed(e) => write!(f, "{e}"),
ChorusError::PocketType(e) => write!(f, "{e}"),
ChorusError::RateLimitExceeded => write!(f, "Rate limit exceeded"),
ChorusError::ProtectedEvent => write!(f, "Protected event"),
ChorusError::RealIpHeaderMissing => write!(f, "X-Real-Ip header is missing"),
ChorusError::Restricted => write!(f, "Restricted"),
ChorusError::Rustls(e) => write!(f, "{e}"),
ChorusError::Scraper => write!(f, "Filter is underspecified. Scrapers are not allowed"),
ChorusError::SerdeJson(e) => write!(f, "{e}"),
ChorusError::ShuttingDown => write!(f, "Shutting down"),
ChorusError::SignalNotBlossom => write!(f, "internal-signal-not-blossom"),
ChorusError::Speedy(e) => write!(f, "{e}"),
ChorusError::TimedOut => write!(f, "Timed out"),
ChorusError::TooManySubscriptions => write!(f, "Too many subscriptions"),
ChorusError::Tungstenite(e) => write!(f, "{e}"),
ChorusError::Scraper => write!(f, "Filter is underspecified. Scrapers are not allowed"),
ChorusError::UrlParse(e) => write!(f, "{e}"),
ChorusError::Utf8(e) => write!(f, "{e}"),
ChorusError::Utf8Error => write!(f, "UTF-8 error"),
@ -246,25 +177,15 @@ impl std::fmt::Display for ChorusError {
impl StdError for ChorusError {
fn source(&self) -> Option<&(dyn StdError + 'static)> {
match self {
ChorusError::Base64Decode(e) => Some(e),
ChorusError::ChannelRecv(e) => Some(e),
ChorusError::ChannelSend(e) => Some(e),
ChorusError::Config(e) => Some(e),
ChorusError::Crypto(e) => Some(e),
ChorusError::FromHex(e) => Some(e),
ChorusError::FromUtf8(e) => Some(e),
ChorusError::Http(e) => Some(e),
ChorusError::Hyper(e) => Some(e),
ChorusError::InvalidUri(e) => Some(e),
ChorusError::InvalidUriParts(e) => Some(e),
ChorusError::Io(e) => Some(e),
ChorusError::NonAsciiHttpHeaderValue(e) => Some(e),
ChorusError::PocketDb(e) => Some(e),
ChorusError::PocketDbHeed(e) => Some(e),
ChorusError::PocketType(e) => Some(e),
ChorusError::Lmdb(e) => Some(e),
ChorusError::Rustls(e) => Some(e),
ChorusError::SerdeJson(e) => Some(e),
ChorusError::Speedy(e) => Some(e),
ChorusError::Tungstenite(e) => Some(e),
ChorusError::UrlParse(e) => Some(e),
ChorusError::Utf8(e) => Some(e),
@ -274,73 +195,6 @@ impl StdError for ChorusError {
}
}
impl ChorusError {
#[track_caller]
pub fn into_err(self) -> Error {
Error {
inner: self,
location: std::panic::Location::caller(),
}
}
pub fn punishment(&self) -> f32 {
match self {
ChorusError::AuthFailure(_) => 0.25,
ChorusError::AuthRequired => 0.0,
ChorusError::BadRequest(_) => 0.1,
ChorusError::BadRealIpHeader(_) => 0.0,
ChorusError::BadRealIpHeaderCharacters => 0.0,
ChorusError::BannedEvent => 0.1,
ChorusError::BannedUser => 0.2,
ChorusError::Base64Decode(_) => 0.0,
ChorusError::BlockedIp => 0.0,
ChorusError::BlossomAuthFailure(_) => 0.0,
ChorusError::ChannelRecv(_) => 0.0,
ChorusError::ChannelSend(_) => 0.0,
ChorusError::Config(_) => 0.0,
ChorusError::Crypto(_) => 0.1,
ChorusError::ErrorClose => 1.0,
ChorusError::EventIsInvalid(_) => 0.2,
ChorusError::FromHex(_) => 0.2,
ChorusError::FromUtf8(_) => 0.2,
ChorusError::General(_) => 0.0,
ChorusError::Http(_) => 0.0,
ChorusError::Hyper(_) => 0.0,
ChorusError::Infallible => panic!("INFALLIBLE"),
ChorusError::InvalidUri(_) => 0.0,
ChorusError::InvalidUriParts(_) => 0.0,
ChorusError::Io(_) => 0.0,
ChorusError::ManagementAuthFailure(_) => 0.0,
ChorusError::MissingTable(_) => 0.0,
ChorusError::Negentropy(_) => 0.1,
ChorusError::NonAsciiHttpHeaderValue(_) => 0.2,
ChorusError::NoPrivateKey => 0.0,
ChorusError::NotImplemented => 0.0,
ChorusError::NoSuchSubscription => 0.05,
ChorusError::PocketDb(_) => 0.0,
ChorusError::PocketDbHeed(_) => 0.0,
ChorusError::PocketType(_) => 0.25,
ChorusError::RateLimitExceeded => 1.0,
ChorusError::ProtectedEvent => 0.35,
ChorusError::RealIpHeaderMissing => 0.0,
ChorusError::Restricted => 0.1,
ChorusError::Rustls(_) => 0.0,
ChorusError::Scraper => 0.4,
ChorusError::SerdeJson(_) => 0.0,
ChorusError::ShuttingDown => 0.0,
ChorusError::SignalNotBlossom => 0.0,
ChorusError::Speedy(_) => 0.0,
ChorusError::TimedOut => 0.1,
ChorusError::TooManySubscriptions => 0.1,
ChorusError::Tungstenite(_) => 0.0,
ChorusError::UrlParse(_) => 0.1,
ChorusError::Utf8(_) => 0.1,
ChorusError::Utf8Error => 0.1,
ChorusError::WebsocketProtocol(_) => 0.1,
}
}
}
// Note: we impl Into because our typical pattern is ChorusError::Variant.into()
// when we tried implementing From, the location was deep in rust code's
// blanket into implementation, which wasn't the line number we wanted.
@ -368,9 +222,9 @@ impl From<tokio::sync::broadcast::error::RecvError> for Error {
}
}
impl From<tokio::sync::broadcast::error::SendError<u64>> for Error {
impl From<tokio::sync::broadcast::error::SendError<usize>> for Error {
#[track_caller]
fn from(err: tokio::sync::broadcast::error::SendError<u64>) -> Self {
fn from(err: tokio::sync::broadcast::error::SendError<usize>) -> Self {
Error {
inner: ChorusError::ChannelSend(err),
location: std::panic::Location::caller(),
@ -378,11 +232,11 @@ impl From<tokio::sync::broadcast::error::SendError<u64>> for Error {
}
}
impl From<toml::de::Error> for Error {
impl From<ron::error::SpannedError> for Error {
#[track_caller]
fn from(err: toml::de::Error) -> Self {
fn from(err: ron::error::SpannedError) -> Self {
Error {
inner: ChorusError::Config(Box::new(err)),
inner: ChorusError::Config(err),
location: std::panic::Location::caller(),
}
}
@ -402,7 +256,7 @@ impl From<hyper::http::Error> for Error {
#[track_caller]
fn from(err: hyper::http::Error) -> Self {
Error {
inner: ChorusError::Http(Box::new(err)),
inner: ChorusError::Http(err),
location: std::panic::Location::caller(),
}
}
@ -418,26 +272,6 @@ impl From<hyper::Error> for Error {
}
}
impl From<hyper::http::uri::InvalidUri> for Error {
#[track_caller]
fn from(err: hyper::http::uri::InvalidUri) -> Self {
Error {
inner: ChorusError::InvalidUri(err),
location: std::panic::Location::caller(),
}
}
}
impl From<hyper::http::uri::InvalidUriParts> for Error {
#[track_caller]
fn from(err: hyper::http::uri::InvalidUriParts) -> Self {
Error {
inner: ChorusError::InvalidUriParts(err),
location: std::panic::Location::caller(),
}
}
}
impl From<std::io::Error> for Error {
#[track_caller]
fn from(err: std::io::Error) -> Self {
@ -448,41 +282,11 @@ impl From<std::io::Error> for Error {
}
}
impl From<http::header::ToStrError> for Error {
impl From<heed::Error> for Error {
#[track_caller]
fn from(err: http::header::ToStrError) -> Self {
fn from(err: heed::Error) -> Self {
Error {
inner: ChorusError::NonAsciiHttpHeaderValue(err),
location: std::panic::Location::caller(),
}
}
}
impl From<pocket_db::Error> for Error {
#[track_caller]
fn from(err: pocket_db::Error) -> Self {
Error {
inner: ChorusError::PocketDb(err),
location: std::panic::Location::caller(),
}
}
}
impl From<pocket_db::heed::Error> for Error {
#[track_caller]
fn from(err: pocket_db::heed::Error) -> Self {
Error {
inner: ChorusError::PocketDbHeed(Box::new(err)),
location: std::panic::Location::caller(),
}
}
}
impl From<pocket_types::Error> for Error {
#[track_caller]
fn from(err: pocket_types::Error) -> Self {
Error {
inner: ChorusError::PocketType(err),
inner: ChorusError::Lmdb(err),
location: std::panic::Location::caller(),
}
}
@ -492,7 +296,7 @@ impl From<tokio_rustls::rustls::Error> for Error {
#[track_caller]
fn from(err: tokio_rustls::rustls::Error) -> Self {
Error {
inner: ChorusError::Rustls(Box::new(err)),
inner: ChorusError::Rustls(err),
location: std::panic::Location::caller(),
}
}
@ -502,7 +306,7 @@ impl From<hyper_tungstenite::tungstenite::error::Error> for Error {
#[track_caller]
fn from(err: hyper_tungstenite::tungstenite::error::Error) -> Self {
Error {
inner: ChorusError::Tungstenite(Box::new(err)),
inner: ChorusError::Tungstenite(err),
location: std::panic::Location::caller(),
}
}
@ -537,75 +341,3 @@ impl From<url::ParseError> for Error {
}
}
}
impl From<speedy::Error> for Error {
#[track_caller]
fn from(err: speedy::Error) -> Self {
Error {
inner: ChorusError::Speedy(err),
location: std::panic::Location::caller(),
}
}
}
impl From<serde_json::Error> for Error {
#[track_caller]
fn from(err: serde_json::Error) -> Self {
Error {
inner: ChorusError::SerdeJson(err),
location: std::panic::Location::caller(),
}
}
}
impl From<hex::FromHexError> for Error {
#[track_caller]
fn from(err: hex::FromHexError) -> Self {
Error {
inner: ChorusError::FromHex(err),
location: std::panic::Location::caller(),
}
}
}
impl From<std::string::FromUtf8Error> for Error {
#[track_caller]
fn from(err: std::string::FromUtf8Error) -> Self {
Error {
inner: ChorusError::FromUtf8(Box::new(err)),
location: std::panic::Location::caller(),
}
}
}
impl From<base64::DecodeError> for Error {
#[track_caller]
fn from(err: base64::DecodeError) -> Self {
Error {
inner: ChorusError::Base64Decode(err),
location: std::panic::Location::caller(),
}
}
}
impl From<negentropy::Error> for Error {
#[track_caller]
fn from(e: negentropy::Error) -> Error {
Error {
inner: ChorusError::Negentropy(e),
location: std::panic::Location::caller(),
}
}
}
impl From<Infallible> for Error {
fn from(_: Infallible) -> Self {
panic!("INFALLIBLE")
}
}
impl From<Error> for std::io::Error {
fn from(e: Error) -> std::io::Error {
std::io::Error::other(e)
}
}

65
src/filestore/hash_output.rs
View File

@ -1,65 +0,0 @@
use crate::{ChorusError, Error};
use std::fmt;
use std::path::{Path, PathBuf};
/// A simple type for a SHA-256 hash output of 32 bytes
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct HashOutput([u8; 32]);
impl HashOutput {
pub fn from_engine(engine: bitcoin_hashes::sha256::HashEngine) -> HashOutput {
use bitcoin_hashes::sha256;
let hashvalue = sha256::Hash::from_engine(engine);
HashOutput(hashvalue.as_byte_array()[0..32].try_into().unwrap())
}
pub fn from_hex(input: &str) -> Result<HashOutput, Error> {
let bytes = hex::decode(input)?;
if bytes.len() == 32 {
Ok(HashOutput(bytes.try_into().unwrap()))
} else {
Err(
ChorusError::General("HashOutput::from_hex() got wrong length string".to_string())
.into(),
)
}
}
pub fn from_bytes(bytes: [u8; 32]) -> HashOutput {
HashOutput(bytes)
}
pub fn to_pathbuf<P: AsRef<Path>>(&self, base: P) -> PathBuf {
let s = hex::encode(self.0);
let mut output: PathBuf = PathBuf::new();
output.push(base);
output.push(&s[0..=1]);
output.push(&s[2..=3]);
output.push(&s[4..]);
output
}
}
impl fmt::Display for HashOutput {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", hex::encode(self.0))
}
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn test_hash_output_to_pathbuf() {
let hash = HashOutput::from_hex(
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
)
.unwrap();
assert_eq!(
&format!("{}", hash.to_pathbuf("/tmp").display()),
"/tmp/e3/b0/c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
);
}
}

173
src/filestore/mod.rs
View File

@ -1,173 +0,0 @@
use crate::error::{ChorusError, Error};
use futures::TryStreamExt;
use http_body_util::combinators::BoxBody;
use http_body_util::{BodyDataStream, BodyExt, StreamBody};
use hyper::body::{Bytes, Frame};
use std::fs::Metadata;
use std::path::{Path, PathBuf};
use tokio::fs;
use tokio::fs::File;
use tokio_util::io::{InspectReader, ReaderStream, StreamReader};
mod hash_output;
pub use hash_output::HashOutput;
pub struct FileStore {
pub base: PathBuf,
pub temp: PathBuf,
}
impl FileStore {
pub async fn new<P: AsRef<Path>>(base: P) -> Result<FileStore, Error> {
let base = base.as_ref().to_owned();
let temp = {
let mut temp = base.clone();
temp.push("temp");
temp
};
if !fs::try_exists(&temp).await? {
fs::create_dir_all(&temp).await?;
}
Ok(FileStore { base, temp })
}
fn tmpfile(&self) -> PathBuf {
let mut tf = self.temp.clone();
let nonce = textnonce::TextNonce::sized_urlsafe(32).unwrap();
tf.push(&nonce.0);
tf
}
/// Store a file in storage, streamed from a hyper BoxBody
///
/// Returns it's HashOutput by which it can be later retrieved or deleted.
pub async fn store(
&self,
data: BoxBody<Bytes, Error>,
expected_hash: Option<HashOutput>,
) -> Result<(u64, HashOutput, Option<String>), Error> {
use bitcoin_hashes::sha256;
use std::io::Write; // for hash_engine.write_all()
// We will download into a temporary file (as we don't know the hash yet)
let temppathbuf = self.tmpfile();
let mut tempfile = File::options()
.create(true)
.truncate(true)
.write(true)
.open(&temppathbuf)
.await?;
// Convert the BoxBody into a Data Stream
let body_stream = BodyDataStream::new(data);
// Convert the Data Stream into something that is AsyncRead (over bytes)
let stream_reader = StreamReader::new(body_stream);
// Wrap this in something that lets us inspect the content so we can
// hash it as it comes in, as well as count the size
let mut size: u64 = 0;
let mut hash_engine = sha256::HashEngine::default();
let mut inspect_reader = InspectReader::new(stream_reader, |bytes: &[u8]| {
size += bytes.len() as u64;
hash_engine.write_all(bytes).unwrap(); // I don't think hashing will fail
});
// Copy the data into the tempfile (hashing and counting as we go)
let count = tokio::io::copy(&mut inspect_reader, &mut tempfile).await?;
drop(tempfile);
// Verify our code was correct
if count != size {
return Err(ChorusError::General("INTERNAL COUNT MISMATCH".to_string()).into());
}
// Finish the hash
let hash = HashOutput::from_engine(hash_engine);
// Verify the expected hash matches
if let Some(expected) = expected_hash {
if hash != expected {
// Remove the temporary file
fs::remove_file(&temppathbuf).await?;
// And complain
return Err(ChorusError::BlossomAuthFailure(
"File hash does not match authorized hash".to_string(),
)
.into());
}
}
// Sniff the mime-type
let maybe_mime_string = {
use mime_sniffer::MimeTypeSniffer;
use tokio::io::AsyncReadExt;
let mut readtempfile = File::open(&temppathbuf).await?;
let mut buffer: Vec<u8> = vec![0; 128];
let _ = readtempfile.read(&mut buffer).await?;
buffer.sniff_mime_type().map(|s| s.to_string())
};
// Compute the proper path
let pathbuf = hash.to_pathbuf(&self.base);
// If it already exists, trust the existing copy
if fs::try_exists(&pathbuf).await? {
// Just clean up
fs::remove_file(&temppathbuf).await?;
return Ok((size, hash, maybe_mime_string));
}
// Make the parent directory
fs::create_dir_all(pathbuf.parent().unwrap()).await?;
// Move the file
fs::rename(&temppathbuf, &pathbuf).await?;
Ok((size, hash, maybe_mime_string))
}
/// Retrieve a file from storage by its HashOutput, streamed to a hyper BoxBoxy
pub async fn retrieve(&self, hash: HashOutput) -> Result<BoxBody<Bytes, Error>, Error> {
// Compute the path
let pathbuf = hash.to_pathbuf(&self.base);
// Open the file
let file = File::open(&pathbuf).await?;
// Convert the AsyncRead file into a Stream
let reader_stream = ReaderStream::new(file);
// Convert the Stream into a Body
let stream_body = StreamBody::new(reader_stream.map_ok(Frame::data));
// Box the body, mapping the error
let boxed_body = BodyExt::map_err(stream_body, |e| e.into()).boxed();
Ok(boxed_body)
}
/// Check if a file exists and provide it's metadata (including .len())
pub async fn metadata(&self, hash: HashOutput) -> Result<Metadata, Error> {
// Compute the path
let pathbuf = hash.to_pathbuf(&self.base);
Ok(tokio::fs::metadata(&pathbuf).await?)
}
/// Delete a file from storage by its HashOutput
pub async fn delete(&self, hash: HashOutput) -> Result<(), Error> {
// Compute the path
let pathbuf = hash.to_pathbuf(&self.base);
// Delete the file
tokio::fs::remove_file(&pathbuf).await?;
Ok(())
}
}

52
src/globals.rs
View File

@ -1,63 +1,51 @@
use crate::config::Config;
use crate::filestore::FileStore;
use crate::ip::HashedIp;
use dashmap::DashMap;
use hyper::server::conn::http1;
use hyper_util::rt::tokio::TokioTimer;
use crate::config::{Config, FriendlyConfig};
use crate::store::Store;
use crate::types::Time;
use hyper::server::conn::Http;
use lazy_static::lazy_static;
use parking_lot::RwLock;
use pocket_db::Store;
use std::sync::atomic::{AtomicU64, AtomicUsize};
use std::collections::HashMap;
use std::net::IpAddr;
use std::sync::atomic::AtomicUsize;
use std::sync::OnceLock;
use std::time::{Duration, Instant};
use tokio::sync::broadcast::Sender as BroadcastSender;
use tokio::sync::watch::Sender as WatchSender;
use tokio::sync::RwLock;
pub struct Globals {
pub start_time: Instant,
pub bytes_inbound: AtomicU64,
pub bytes_outbound: AtomicU64,
pub config: RwLock<Config>,
pub store: OnceLock<Store>,
pub filestore: OnceLock<FileStore>,
pub http1builder: http1::Builder,
pub http_server: Http,
pub rid: OnceLock<String>,
/// This is a broadcast channel where new incoming events are advertised by their offset.
/// Every handler needs to listen to it and check if the incoming event matches any
/// subscribed fitlers for their client, and if so, send the event to their client under
/// that subscription.
pub new_events: BroadcastSender<u64>,
pub new_events: BroadcastSender<usize>,
pub num_connections: AtomicUsize,
pub num_connections_per_ip: DashMap<HashedIp, usize>,
pub num_clients: AtomicUsize,
pub shutting_down: WatchSender<bool>,
pub banlist: RwLock<HashMap<IpAddr, Time>>,
}
lazy_static! {
pub static ref GLOBALS: Globals = {
let mut http_server = hyper::server::conn::Http::new();
http_server.http1_only(true);
http_server.http1_keep_alive(true);
let (new_events, _) = tokio::sync::broadcast::channel(512);
let (shutting_down, _) = tokio::sync::watch::channel(false);
let mut http1builder = http1::Builder::new();
http1builder.ignore_invalid_headers(true);
http1builder.keep_alive(true);
http1builder.timer(TokioTimer::new());
http1builder.header_read_timeout(Duration::from_secs(5));
Globals {
start_time: Instant::now(),
bytes_inbound: AtomicU64::new(0),
bytes_outbound: AtomicU64::new(0),
config: RwLock::new(Default::default()),
config: RwLock::new(FriendlyConfig::default().into_config().unwrap()),
store: OnceLock::new(),
filestore: OnceLock::new(),
http1builder,
http_server,
rid: OnceLock::new(),
new_events,
num_connections: AtomicUsize::new(0),
num_connections_per_ip: DashMap::new(),
num_clients: AtomicUsize::new(0),
shutting_down,
banlist: RwLock::new(HashMap::new()),
}
};
}

172
src/ip.rs
View File

@ -1,172 +0,0 @@
use pocket_types::Time;
use speedy::{Readable, Writable};
use std::net::{IpAddr, SocketAddr};
#[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
pub struct HashedIp(pub [u8; 20], bool);
impl std::fmt::Display for HashedIp {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
unsafe { write!(f, "{}", std::str::from_utf8_unchecked(self.0.as_slice())) }
}
}
impl HashedIp {
pub fn new(ip_addr: IpAddr) -> HashedIp {
use base64::prelude::*;
use secp256k1::hashes::{sha256, Hash};
let bytes = ip_addr.write_to_vec().unwrap();
let hashvalue: sha256::Hash = Hash::hash(&bytes);
let tag = BASE64_STANDARD.encode(&hashvalue.as_byte_array()[0..16]);
HashedIp(
tag.as_bytes()[..20].try_into().unwrap(),
ip_addr.is_loopback(),
)
}
pub fn from_bytes(bytes: &[u8]) -> HashedIp {
HashedIp(bytes[0..20].try_into().unwrap(), false)
}
pub fn is_loopback(&self) -> bool {
self.1
}
}
#[derive(Debug, Clone, Copy)]
pub struct HashedPeer(pub HashedIp, pub u16);
impl std::fmt::Display for HashedPeer {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}:{}", self.0, self.1)
}
}
impl HashedPeer {
pub fn new(peer_addr: SocketAddr) -> HashedPeer {
let hashed_ip = HashedIp::new(peer_addr.ip());
HashedPeer(hashed_ip, peer_addr.port())
}
pub fn from_parts(hashed_ip: HashedIp, port: u16) -> HashedPeer {
HashedPeer(hashed_ip, port)
}
pub fn ip(&self) -> HashedIp {
self.0
}
pub fn port(&self) -> u16 {
self.1
}
}
// Single-session exit condition
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum SessionExit {
// No problems
Ok,
// Session exited with an error
ErrorExit,
// Session exited because of too many nostr command errors
// (based on a per-session command punishment value that we don't track long term)
TooManyErrors,
// Session timed out without an active subscription
Timeout,
}
// Long term reputation of an IP address
// The values are running totals, updating with (9/10) of old and (1/10) of new.
//
// Used to determine ban time multiplier from short-term violations
#[derive(Debug, Clone, Default, Readable, Writable)]
pub struct IpReputation {
pub good: f32,
pub errored: f32,
pub too_many_errors: f32,
pub timed_out: f32,
}
impl IpReputation {
pub fn update(&mut self, session_exit: SessionExit) {
// Lessen the running totals
self.good *= 9.0 / 10.0;
self.errored *= 9.0 / 10.0;
self.too_many_errors *= 9.0 / 10.0;
self.timed_out *= 9.0 / 10.0;
match session_exit {
SessionExit::Ok => self.good += 1.0,
SessionExit::ErrorExit => self.errored += 1.0,
SessionExit::TooManyErrors => self.too_many_errors += 1.0,
SessionExit::Timeout => self.timed_out += 1.0,
};
}
pub fn ban_multiplier(&self) -> f32 {
let good_endings = 1.0 + self.good + (self.errored / 2.0);
let bad_endings = 1.0 + self.timed_out + self.too_many_errors + (self.errored / 2.0);
bad_endings / good_endings
}
}
// Memory-only short-term record of IP handling
#[derive(Debug, Clone, Default, Readable, Writable)]
pub struct IpData {
pub ban_until: u64,
pub reputation: IpReputation,
}
impl IpData {
pub fn update_on_session_close(
&mut self,
session_exit: SessionExit,
minimum_ban_seconds: u64,
) -> u64 {
// Update reputation
self.reputation.update(session_exit);
// Compute ban_until
let mut until = Time::now();
let seconds = self.ban_seconds(session_exit, minimum_ban_seconds);
until = until + seconds;
self.ban_until = self.ban_until.max(until.as_u64());
seconds
}
pub fn is_banned(&self) -> bool {
Time::from_u64(self.ban_until) > Time::now()
}
fn ban_seconds(&self, session_exit: SessionExit, minimum_ban_seconds: u64) -> u64 {
let multiplier = self.reputation.ban_multiplier();
match session_exit {
SessionExit::Ok => minimum_ban_seconds,
SessionExit::Timeout => minimum_ban_seconds,
SessionExit::ErrorExit => minimum_ban_seconds + (2.0 * multiplier) as u64,
SessionExit::TooManyErrors => minimum_ban_seconds + (5.0 * multiplier) as u64,
}
}
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn test_hashed_ip() {
let ipaddr: std::net::IpAddr = "127.0.0.1".parse().unwrap();
println!("HashedIP={}", HashedIp::new(ipaddr));
let socketaddr = std::net::SocketAddr::new(ipaddr, 80);
println!("HashedPEER={}", HashedPeer::new(socketaddr));
}
}

942
src/lib.rs
View File

@ -1,942 +0,0 @@
pub mod config;
pub mod counting_stream;
pub mod error;
pub mod filestore;
pub mod globals;
pub mod ip;
mod neg_storage;
pub mod nostr;
pub mod reply;
pub mod tls;
pub mod web;
use crate::config::{Config, FriendlyConfig};
use crate::error::{ChorusError, Error};
use crate::globals::GLOBALS;
use crate::ip::{HashedIp, HashedPeer, IpData, SessionExit};
use crate::reply::NostrReply;
use futures::{sink::SinkExt, stream::StreamExt};
use http_body_util::combinators::BoxBody;
use http_body_util::{BodyExt, Empty};
use hyper::body::{Bytes, Incoming};
use hyper::service::Service;
use hyper::upgrade::Upgraded;
use hyper::StatusCode;
use hyper::{Request, Response};
use hyper_tungstenite::tungstenite;
use hyper_tungstenite::{HyperWebsocket, WebSocketStream};
use hyper_util::rt::TokioIo;
use neg_storage::NegentropyStorageVector;
use pocket_db::{ScreenResult, Store};
use pocket_types::{Id, OwnedFilter, Pubkey};
use speedy::{Readable, Writable};
use std::collections::HashMap;
use std::error::Error as StdError;
use std::fs::OpenOptions;
use std::future::Future;
use std::io::Read;
use std::net::IpAddr;
use std::path::Path;
use std::pin::Pin;
use std::sync::atomic::Ordering;
use std::time::Duration;
use textnonce::TextNonce;
use tokio::io::{AsyncRead, AsyncWrite};
use tokio::time::Instant;
use tungstenite::protocol::frame::Utf8Bytes;
use tungstenite::protocol::WebSocketConfig;
use tungstenite::Message;
/// Serve a single network connection
pub async fn serve<T>(stream: TokioIo<T>, peer: HashedPeer)
where
T: AsyncRead + AsyncWrite + Unpin + Send + 'static,
{
// Serve the network stream with our http server and our ChorusService
let service = ChorusService { peer };
let http1builder = GLOBALS.http1builder.clone();
let connection = http1builder
.serve_connection(stream, service)
.with_upgrades();
// If our service exits with an error, log the error
if let Err(he) = connection.await {
if let Some(src) = he.source() {
if &*format!("{}", src) == "Transport endpoint is not connected (os error 107)" {
// do nothing
} else {
// Print in detail
log::error!(target: "Client", "{}: {:?}", peer, src);
}
} else {
// Print in less detail
let e: Error = he.into();
log::error!(target: "Client", "{}: {}", peer, e);
}
}
}
// This is our per-connection HTTP service
struct ChorusService {
peer: HashedPeer,
}
impl Service<Request<Incoming>> for ChorusService {
type Response = Response<BoxBody<Bytes, Self::Error>>;
type Error = Error;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
// This is called for each HTTP request made by the client
// NOTE: it is not called for each websocket message once upgraded.
fn call(&self, req: Request<Incoming>) -> Self::Future {
let mut hashed_peer = self.peer;
let failvalue =
|c: ChorusError| -> Self::Future { Box::pin(futures::future::ready(Err(c.into()))) };
if GLOBALS.config.read().chorus_is_behind_a_proxy {
// If chorus is behind a proxy that sets an "X-Real-Ip" header, we use
// that ip address instead (otherwise their log file will just give the proxy IP
// for every peer)
//
// This header must be found and be valid for us to proceed
if let Some(rip) = req.headers().get("x-real-ip") {
if let Ok(ripstr) = rip.to_str() {
if let Ok(ipaddr) = ripstr.parse::<IpAddr>() {
let hashed_ip = HashedIp::new(ipaddr);
hashed_peer = HashedPeer::from_parts(hashed_ip, hashed_peer.port());
} else {
return failvalue(ChorusError::BadRealIpHeader(ripstr.to_owned()));
}
} else {
return failvalue(ChorusError::BadRealIpHeaderCharacters);
}
} else {
return failvalue(ChorusError::RealIpHeaderMissing);
}
// Possibly IP block late (if behind a proxy)
if GLOBALS.config.read().enable_ip_blocking {
if let Ok(ip_data) = crate::get_ip_data(hashed_peer.ip()) {
if ip_data.is_banned() {
log::debug!(target: "Client",
"{}: Blocking reconnection until {}",
hashed_peer.ip(),
ip_data.ban_until);
return failvalue(ChorusError::BlockedIp);
}
}
}
}
Box::pin(async move { handle_http_request(hashed_peer, req).await })
}
}
async fn handle_http_request(
peer: HashedPeer,
mut request: Request<Incoming>,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
let ua = match request.headers().get("user-agent") {
Some(ua) => ua.to_str().unwrap_or("NON-UTF8-HEADER").to_owned(),
None => "(no user-agent)".to_owned(),
};
let origin = match request.headers().get("origin") {
Some(o) => o.to_str().unwrap_or("NON-UTF8-HEADER").to_owned(),
None => "(no origin)".to_owned(),
};
let max_conn = GLOBALS.config.read().max_connections_per_ip;
if let Some(cur) = GLOBALS.num_connections_per_ip.get(&peer.ip()) {
if *cur.value() >= max_conn {
return Ok(Response::builder()
.status(StatusCode::TOO_MANY_REQUESTS)
.body(Empty::new().map_err(|e| e.into()).boxed())?);
}
}
if hyper_tungstenite::is_upgrade_request(&request) {
// If the client asks for a Sec-Websocket-Protocol that we don't understand,
// Respond with 501 Not Implemented
let maybe_protocol: Option<String> = match request.headers().get("sec-websocket-protocol") {
None => None,
Some(hv) => hv.to_str().ok().map(|s| s.to_owned()),
};
if let Some(ref protocol) = maybe_protocol {
let mut we_can_do_nostr: bool = false;
for option in protocol.split(',') {
if option.trim() == "nostr" {
we_can_do_nostr = true;
break;
}
}
if !we_can_do_nostr {
return Ok(Response::builder()
.status(StatusCode::NOT_IMPLEMENTED)
.body(Empty::new().map_err(|e| e.into()).boxed())?);
}
}
let mut web_socket_config = WebSocketConfig::default();
web_socket_config.max_write_buffer_size = 1024 * 1024; // 1 MB
web_socket_config.max_message_size = Some(1024 * 1024); // 1 MB
web_socket_config.max_frame_size = Some(1024 * 1024); // 1 MB
let (mut response, websocket) =
hyper_tungstenite::upgrade(&mut request, Some(web_socket_config))?;
// If the client asked for Sec-Websocket-Protocol, then we already checked it must
// have asked for 'nostr', so send that as a response header
if maybe_protocol.is_some() {
response.headers_mut().insert(
http::header::SEC_WEBSOCKET_PROTOCOL,
http::header::HeaderValue::from_static("nostr"),
);
}
// Start the websocket thread
tokio::spawn(async move { websocket_thread(peer, websocket, origin, ua).await });
Ok(response.map(|body| body.map_err(|e| e.into()).boxed()))
} else {
web::serve_http(peer, request).await
}
}
async fn websocket_thread(peer: HashedPeer, websocket: HyperWebsocket, origin: String, ua: String) {
// Await the websocket upgrade process
match websocket.await {
Ok(websocket) => {
// Build a websocket service
let mut ws_service = WebSocketService {
peer,
subscriptions: HashMap::new(),
neg_subscriptions: HashMap::new(),
// We start with a 1-page buffer, and grow it if needed.
buffer: vec![0; 4096],
websocket,
last_message: Instant::now(),
burst_tokens: GLOBALS.config.read().throttling_burst,
challenge: TextNonce::new().into_string(),
authed_as: Vec::new(),
error_punishment: 0.0,
replied: false,
negentropy_sub: None,
};
// Increment connection count
let old_num_websockets = GLOBALS.num_connections.fetch_add(1, Ordering::SeqCst);
// Increment per-ip connection count
GLOBALS
.num_connections_per_ip
.entry(peer.ip())
.and_modify(|count| *count += 1)
.or_insert(1);
// we cheat somewhat and log these websocket open and close messages
// as server messages
log::info!(
target: "Server",
"{}: TOTAL={}, New Connection: {}, {}",
peer,
old_num_websockets + 1,
origin,
ua,
);
// Everybody gets a ban on disconnect to prevent rapid reconnection
let mut session_exit: SessionExit = SessionExit::Ok;
let mut msg = "Closed";
// Handle the websocket
if let Err(e) = ws_service.handle_websocket_stream().await {
match e.inner {
ChorusError::Tungstenite(ref t) => {
match *t.as_ref() {
tungstenite::error::Error::Protocol(
tungstenite::error::ProtocolError::ResetWithoutClosingHandshake,
) => {
// So they disconnected ungracefully.
// No big deal, still SessionExit::Ok
msg = "Reset";
}
tungstenite::error::Error::Io(ref ioerror) => {
match ioerror.kind() {
std::io::ErrorKind::ConnectionReset
| std::io::ErrorKind::ConnectionAborted
| std::io::ErrorKind::UnexpectedEof => {
// no biggie.
msg = "Reset";
}
_ => {
log::error!(target: "Client", "{}: {}", peer, e);
session_exit = SessionExit::ErrorExit;
msg = "Error Exited";
}
}
}
_ => {
log::error!(target: "Client", "{}: {}", peer, e);
session_exit = SessionExit::ErrorExit;
msg = "Error Exited";
}
}
}
ChorusError::ErrorClose => {
session_exit = SessionExit::TooManyErrors;
msg = "Errored Out";
}
ChorusError::RateLimitExceeded => {
session_exit = SessionExit::TooManyErrors; // close enough for now.
msg = "Rate Limit Exceeded";
}
ChorusError::TimedOut => {
session_exit = SessionExit::Timeout;
msg = "Timed Out (with no subscriptions)";
}
ChorusError::Io(_) => {
// Usually "Connection reset by peer" but any I/O error
// isn't a big deal.
msg = "Reset";
}
_ => {
log::error!(target: "Client", "{}: {}", peer, e);
session_exit = SessionExit::ErrorExit;
msg = "Error Exited";
}
}
}
// Decrement connection count
let old_num_websockets = GLOBALS.num_connections.fetch_sub(1, Ordering::SeqCst);
// Decrement per-ip connection count
match GLOBALS.num_connections_per_ip.get_mut(&peer.ip()) {
Some(mut refmut) => {
if *refmut.value_mut() > 0 {
*refmut.value_mut() -= 1;
} else {
unreachable!("The connection should be in the map")
}
}
None => unreachable!("The connection count should be greater than zero"),
};
// Update ip data (including ban time)
let minimum_ban_seconds = GLOBALS.config.read().minimum_ban_seconds;
let ban_seconds = if GLOBALS.config.read().enable_ip_blocking {
let mut ban_seconds = 0;
if let Ok(mut ip_data) = get_ip_data(peer.ip()) {
ban_seconds =
ip_data.update_on_session_close(session_exit, minimum_ban_seconds);
let _ = update_ip_data(peer.ip(), &ip_data);
}
ban_seconds
} else {
minimum_ban_seconds
};
// we cheat somewhat and log these websocket open and close messages
// as server messages
log::info!(
target: "Server",
"{}: TOTAL={}, {}, ban={}s",
peer,
old_num_websockets - 1,
msg,
ban_seconds
);
}
Err(e) => {
log::error!(target: "Client", "{}: {}", peer, e);
}
}
}
struct WebSocketService {
pub peer: HashedPeer,
pub subscriptions: HashMap<String, Vec<OwnedFilter>>,
pub neg_subscriptions: HashMap<String, NegentropyStorageVector>,
pub buffer: Vec<u8>,
pub websocket: WebSocketStream<TokioIo<Upgraded>>,
pub last_message: Instant,
pub burst_tokens: usize,
pub challenge: String,
pub authed_as: Vec<Pubkey>,
pub error_punishment: f32,
pub replied: bool,
pub negentropy_sub: Option<String>,
}
impl WebSocketService {
async fn send(&mut self, m: Message) -> Result<(), Error> {
log::trace!(target: "Client", "{}: {}", self.peer, m);
// Throttling: we consume burst tokens, but we do not throttle on output
if m.len() > self.burst_tokens {
log::info!(target: "Client", "{}: Rate limited exceeded", self.peer);
let reply = NostrReply::Notice("Rate limit exceeded.".into());
self.websocket.send(Message::text(reply.as_json()?)).await?;
let error = ChorusError::RateLimitExceeded;
self.error_punishment += error.punishment();
return Err(error.into());
} else {
self.burst_tokens -= m.len();
}
self.replied = true;
Ok(self.websocket.send(m).await?)
}
async fn wsclose(&mut self, error: Error) -> Result<(), Error> {
use tungstenite::protocol::frame::coding::CloseCode;
use tungstenite::protocol::frame::CloseFrame;
let (code, reason) = match &error.inner {
ChorusError::TimedOut => (CloseCode::Policy, Utf8Bytes::from_static("timed out")),
ChorusError::ShuttingDown => (CloseCode::Restart, Utf8Bytes::from_static("restarting")),
ChorusError::BannedUser | ChorusError::BlockedIp => {
(CloseCode::Policy, Utf8Bytes::from_static("banned"))
}
e => (CloseCode::Error, format!("{}", e).into()),
};
let close_frame = CloseFrame { code, reason };
// NOTE: This is the same as sending Message::Close(..)
self.websocket.close(Some(close_frame)).await?;
// Drive to completion
while (self.websocket.next().await).is_some() {}
Err(error)
}
async fn handle_websocket_stream(&mut self) -> Result<(), Error> {
// Subscribe to the shutting down channel
let mut shutting_down = GLOBALS.shutting_down.subscribe();
// Subscribe to the new_events broadcast channel
let mut new_events = GLOBALS.new_events.subscribe();
// Offer AUTH to clients right off the bat
let reply = NostrReply::Auth(self.challenge.clone());
self.send(Message::text(reply.as_json()?)).await?;
let mut last_message_at = Instant::now();
let timeout_seconds = GLOBALS.config.read().timeout_seconds;
let mut interval = tokio::time::interval(Duration::from_secs(1));
let _ = interval.tick().await; // consume the first tick
tokio::pin!(interval);
loop {
tokio::select! {
instant = interval.tick() => {
// Drop them if they have no subscriptions
if self.subscriptions.is_empty() && self.neg_subscriptions.is_empty() {
// And they are idle for timeout_seconds with no subscriptions
if last_message_at + Duration::from_secs(timeout_seconds) < instant {
self.wsclose(ChorusError::TimedOut.into()).await?;
}
}
}
message_option = self.websocket.next() => {
last_message_at = Instant::now();
match message_option {
Some(message) => {
let message = message?;
if let Err(e) = self.handle_websocket_message(message).await {
self.wsclose(e).await?;
}
},
None => break, // the websocket is closed
}
},
offset_result = new_events.recv() => {
let offset = offset_result?;
self.handle_new_event(offset).await?;
},
_r = shutting_down.changed() => {
self.wsclose(ChorusError::ShuttingDown.into()).await?;
},
}
}
Ok(())
}
// If the event matches a subscription they have open, send them the event
async fn handle_new_event(&mut self, new_event_offset: u64) -> Result<(), Error> {
if self.subscriptions.is_empty() {
return Ok(());
}
let event = GLOBALS
.store
.get()
.unwrap()
.get_event_by_offset(new_event_offset)?;
let event_flags = nostr::event_flags(event, self.authed_as.as_slice());
let authorized_user = self.authed_as.iter().any(|u| is_authorized_user(*u));
'subs: for (subid, filters) in self.subscriptions.iter() {
for filter in filters.iter() {
if filter.event_matches(event)? {
let screen_result =
nostr::screen_outgoing_event(event, &event_flags, authorized_user);
if screen_result == ScreenResult::Redacted {
// TBD: Update subscription so the final close can
// let them know there were redactions from
// the post-EOSE data
} else if screen_result == ScreenResult::Match {
let message = NostrReply::Event(subid, event);
// note, this is not currently counted in throttling
self.websocket
.send(Message::text(message.as_json()?))
.await?;
continue 'subs;
}
}
}
}
Ok(())
}
async fn handle_websocket_message(&mut self, message: Message) -> Result<(), Error> {
// Throttling
{
let (throttling_burst, throttling_bytes_per_second) = {
let config = GLOBALS.config.read();
(config.throttling_burst, config.throttling_bytes_per_second)
};
// Get (and update) timing
let elapsed = self.last_message.elapsed();
self.last_message = Instant::now();
// Grant new tokens
let new_tokens = throttling_bytes_per_second * elapsed.as_millis() as usize / 1_000;
self.burst_tokens += new_tokens;
// Cap tokens to a maximum
if self.burst_tokens > throttling_burst {
self.burst_tokens = throttling_burst;
}
// Consume tokens, possibly closing the connection if there are not enough
if message.len() > self.burst_tokens {
log::info!(target: "Client", "{}: Rate limited exceeded", self.peer);
let reply = NostrReply::Notice("Rate limit exceeded.".into());
self.websocket.send(Message::text(reply.as_json()?)).await?;
let error = ChorusError::RateLimitExceeded;
self.error_punishment += error.punishment();
return Err(error.into());
} else {
self.burst_tokens -= message.len();
}
}
match message {
Message::Text(msg) => {
log::trace!(target: "Client", "{}: <= {}", self.peer, msg);
self.replied = false;
// This is defined in nostr.rs
if let Err(e) = self.handle_nostr_message(&msg).await {
self.error_punishment += e.inner.punishment();
log::error!(target: "Client", "{}: {e}", self.peer);
if !matches!(e.inner, ChorusError::AuthRequired) {
if msg.len() < 2048 {
log::warn!(target: "Client", "{}: msg was {}", self.peer, msg);
} else {
log::warn!(target: "Client", "{}: msg > 2048 (not shown)", self.peer);
}
}
if !self.replied {
if let Some(subid) = &self.negentropy_sub {
let reply = NostrReply::NegErr(subid, format!("error: {e}"));
self.send(Message::text(reply.as_json()?)).await?;
} else {
let reply = NostrReply::Notice(format!("error: {}", e.inner));
self.send(Message::text(reply.as_json()?)).await?;
}
}
if self.error_punishment >= 1.0 {
let reply = NostrReply::Notice("Closing due to error(s)".into());
self.send(Message::text(reply.as_json()?)).await?;
return Err(ChorusError::ErrorClose.into());
}
}
}
Message::Binary(msg) => {
let reply = NostrReply::Notice(
"binary messages are not processed by this relay".to_owned(),
);
self.send(Message::text(reply.as_json()?)).await?;
log::info!(target: "Client",
"{}: Received unhandled binary message: {:02X?}",
self.peer,
msg
);
}
Message::Ping(msg) => {
// No need to send a reply: tungstenite takes care of this for you.
log::debug!(target: "Client", "{}: Received ping message: {:02X?}", self.peer, msg);
}
Message::Pong(msg) => {
log::debug!(target: "Client", "{}: Received pong message: {:02X?}", self.peer, msg);
}
Message::Close(msg) => {
// No need to send a reply: tungstenite takes care of this for you.
if let Some(msg) = &msg {
log::debug!(target: "Client",
"{}: Received websocket close message with code {} and message: {}",
self.peer,
msg.code,
msg.reason
);
} else {
log::debug!(
target: "Client",
"{}: Received websocket close message",
self.peer,
);
}
}
Message::Frame(_msg) => {
unreachable!();
}
}
Ok(())
}
}
/// Print statistics
pub fn print_stats() {
let mut runtime: u64 = GLOBALS.start_time.elapsed().as_secs();
if runtime < 1 {
runtime = 1;
}
log::info!(
target: "Server",
"Runtime: {} seconds", runtime
);
log::info!(
target: "Server",
"Inbound: {} bytes ({} B/s)",
GLOBALS.bytes_inbound.load(Ordering::Relaxed),
(GLOBALS.bytes_inbound.load(Ordering::Relaxed) as f32) / (runtime as f32)
);
log::info!(
target: "Server",
"Outbound: {} bytes ({} B/s)",
GLOBALS.bytes_outbound.load(Ordering::Relaxed),
(GLOBALS.bytes_outbound.load(Ordering::Relaxed) as f32) / (runtime as f32)
);
if let Ok(status) = GLOBALS.store.get().unwrap().stats() {
log::info!(
target: "Server",
"Store: {} event bytes in {} events, {} bytes for the indexes",
status.event_bytes,
status.index_stats.i_index_entries,
status.index_stats.disk_usage
);
}
}
/// Load config file
pub fn load_config<P: AsRef<Path>>(config_path: P) -> Result<Config, Error> {
// Read config file
let mut file = OpenOptions::new().read(true).open(config_path)?;
let mut contents = String::new();
file.read_to_string(&mut contents)?;
let friendly_config: FriendlyConfig = toml::from_str(&contents)?;
let config: Config = friendly_config.into_config()?;
Ok(config)
}
/// Setup logging
pub fn setup_logging(config: &Config) {
env_logger::Builder::new()
.filter_level(config.library_log_level)
.filter(Some("Server"), config.server_log_level)
.filter(Some("Client"), config.client_log_level)
.format_target(true)
.format_module_path(false)
.format_timestamp_millis()
.init();
log::debug!(target: "Server", "Loaded config file.");
}
/// Setup storage
pub fn setup_store(config: &Config) -> Result<(), Error> {
let store = setup_store_and_return(config)?;
let _ = GLOBALS.store.set(store);
Ok(())
}
/// Setup storage and return it
pub fn setup_store_and_return(config: &Config) -> Result<Store, Error> {
let store = Store::new(
&config.data_directory,
vec![
"approved-events", // id.as_slice() -> u8(bool)
"approved-pubkeys", // pubkey.as_slice() -> u8(bool)
"ip_data", // HashedIp.0 -> IpData
"users", // pubkey.as_slice() -> u8(bool) true if moderator
],
)?;
Ok(store)
}
/// Get IpData from storage about this remote HashedIp
pub fn get_ip_data(ip: HashedIp) -> Result<IpData, Error> {
let store = GLOBALS.store.get().unwrap();
let ip_data = store
.extra_table("ip_data")
.ok_or(Into::<Error>::into(ChorusError::MissingTable("ip_data")))?;
let txn = store.read_txn()?;
let key = &ip.0;
let bytes = match ip_data.get(&txn, key)? {
Some(b) => b,
None => return Ok(Default::default()),
};
Ok(IpData::read_from_buffer(bytes)?)
}
/// Get IpData in storage about this remote HashedIp
pub fn update_ip_data(ip: HashedIp, data: &IpData) -> Result<(), Error> {
let store = GLOBALS.store.get().unwrap();
let ip_data = store
.extra_table("ip_data")
.ok_or(Into::<Error>::into(ChorusError::MissingTable("ip_data")))?;
let mut txn = store.write_txn()?;
let key = &ip.0;
let bytes = data.write_to_vec()?;
ip_data.put(&mut txn, key, &bytes)?;
txn.commit()?;
Ok(())
}
/// Dump all IpData from storage
pub fn dump_ip_data() -> Result<Vec<(HashedIp, IpData)>, Error> {
let store = GLOBALS.store.get().unwrap();
let ip_data = store
.extra_table("ip_data")
.ok_or(Into::<Error>::into(ChorusError::MissingTable("ip_data")))?;
let txn = store.read_txn()?;
let mut output: Vec<(HashedIp, IpData)> = Vec::new();
for i in ip_data.iter(&txn)? {
let (key, val) = i?;
let hashedip = HashedIp::from_bytes(key);
let data = IpData::read_from_buffer(val)?;
output.push((hashedip, data));
}
Ok(output)
}
/// Mark an event as approved or not
pub fn mark_event_approval(id: Id, approval: bool) -> Result<(), Error> {
let store = GLOBALS.store.get().unwrap();
let approved_events = store
.extra_table("approved-events")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-events",
)))?;
let mut txn = store.write_txn()?;
approved_events.put(&mut txn, id.as_slice(), &[approval as u8])?;
txn.commit()?;
Ok(())
}
/// Clear an event approval status
pub fn clear_event_approval(id: Id) -> Result<(), Error> {
let store = GLOBALS.store.get().unwrap();
let approved_events = store
.extra_table("approved-events")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-events",
)))?;
let mut txn = store.write_txn()?;
approved_events.delete(&mut txn, id.as_slice())?;
txn.commit()?;
Ok(())
}
/// Fetch an event approval status
pub fn get_event_approval(id: Id) -> Result<Option<bool>, Error> {
let store = GLOBALS.store.get().unwrap();
let approved_events = store
.extra_table("approved-events")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-events",
)))?;
let txn = store.read_txn()?;
Ok(approved_events
.get(&txn, id.as_slice())?
.map(|u| !u.is_empty() && u[0] != 0))
}
/// Dump all event approval statuses
pub fn dump_event_approvals() -> Result<Vec<(Id, bool)>, Error> {
let store = GLOBALS.store.get().unwrap();
let mut output: Vec<(Id, bool)> = Vec::new();
let approved_events = store
.extra_table("approved-events")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-events",
)))?;
let txn = store.read_txn()?;
for i in approved_events.iter(&txn)? {
let (key, val) = i?;
let id = Id::from_bytes(key.try_into().unwrap());
let approval: bool = !val.is_empty() && val[0] != 0;
output.push((id, approval));
}
Ok(output)
}
/// Mark a pubkey as approved or not
pub fn mark_pubkey_approval(pubkey: Pubkey, approval: bool) -> Result<(), Error> {
let store = GLOBALS.store.get().unwrap();
let approved_pubkeys = store
.extra_table("approved-pubkeys")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-pubkeys",
)))?;
let mut txn = store.write_txn()?;
approved_pubkeys.put(&mut txn, pubkey.as_slice(), &[approval as u8])?;
txn.commit()?;
Ok(())
}
/// Clear a pubkey approval status
pub fn clear_pubkey_approval(pubkey: Pubkey) -> Result<(), Error> {
let store = GLOBALS.store.get().unwrap();
let approved_pubkeys = store
.extra_table("approved-pubkeys")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-pubkeys",
)))?;
let mut txn = store.write_txn()?;
approved_pubkeys.delete(&mut txn, pubkey.as_slice())?;
txn.commit()?;
Ok(())
}
/// Fetch a pubkey approval status
pub fn get_pubkey_approval(pubkey: Pubkey) -> Result<Option<bool>, Error> {
let store = GLOBALS.store.get().unwrap();
let approved_pubkeys = store
.extra_table("approved-pubkeys")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-pubkeys",
)))?;
let txn = store.read_txn()?;
Ok(approved_pubkeys
.get(&txn, pubkey.as_slice())?
.map(|u| !u.is_empty() && u[0] != 0))
}
/// Dump all pubkey approval statuses
pub fn dump_pubkey_approvals() -> Result<Vec<(Pubkey, bool)>, Error> {
let store = GLOBALS.store.get().unwrap();
let mut output: Vec<(Pubkey, bool)> = Vec::new();
let approved_pubkeys = store
.extra_table("approved-pubkeys")
.ok_or(Into::<Error>::into(ChorusError::MissingTable(
"approved-pubkeys",
)))?;
let txn = store.read_txn()?;
for i in approved_pubkeys.iter(&txn)? {
let (key, val) = i?;
let pubkey = Pubkey::from_bytes(key.try_into().unwrap());
let approval: bool = !val.is_empty() && val[0] != 0;
output.push((pubkey, approval));
}
Ok(output)
}
/// Add authorized user (or change moderator flag)
pub fn add_authorized_user(pubkey: Pubkey, moderator: bool) -> Result<(), Error> {
let store = GLOBALS.store.get().unwrap();
let users = store
.extra_table("users")
.ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
let mut txn = store.write_txn()?;
users.put(&mut txn, pubkey.as_slice(), &[moderator as u8])?;
txn.commit()?;
Ok(())
}
/// Remove authorized user
pub fn rm_authorized_user(pubkey: Pubkey) -> Result<(), Error> {
let store = GLOBALS.store.get().unwrap();
let users = store
.extra_table("users")
.ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
let mut txn = store.write_txn()?;
users.delete(&mut txn, pubkey.as_slice())?;
txn.commit()?;
Ok(())
}
/// Get authorized user
pub fn get_authorized_user(pubkey: Pubkey) -> Result<Option<bool>, Error> {
let store = GLOBALS.store.get().unwrap();
let users = store
.extra_table("users")
.ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
let txn = store.read_txn()?;
Ok(users
.get(&txn, pubkey.as_slice())?
.map(|u| !u.is_empty() && u[0] != 0))
}
/// Dump all authorized users
pub fn dump_authorized_users() -> Result<Vec<(Pubkey, bool)>, Error> {
let store = GLOBALS.store.get().unwrap();
let mut output: Vec<(Pubkey, bool)> = Vec::new();
let users = store
.extra_table("users")
.ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
let txn = store.read_txn()?;
for i in users.iter(&txn)? {
let (key, val) = i?;
let pubkey = Pubkey::from_bytes(key.try_into().unwrap());
let moderator: bool = !val.is_empty() && val[0] != 0;
output.push((pubkey, moderator));
}
Ok(output)
}
/// Is the pubkey an authorized user?
pub fn is_authorized_user(pubkey: Pubkey) -> bool {
match get_authorized_user(pubkey) {
Err(_) => false,
Ok(None) => false,
Ok(Some(_)) => true,
}
}
/// Is the pubkey a moderator?
pub fn is_moderator(pubkey: Pubkey) -> bool {
match get_authorized_user(pubkey) {
Err(_) => false,
Ok(None) => false,
Ok(Some(moderator)) => moderator,
}
}
/// Is the pubkey an admin?
pub fn is_admin(pubkey: Pubkey) -> bool {
GLOBALS.config.read().admin_keys.contains(&pubkey)
}

75
src/macros.rs Normal file
View File

@ -0,0 +1,75 @@
static HEX_CHARS: &[u8; 16] = b"0123456789abcdef";
#[allow(clippy::zero_prefixed_literal)]
static HEX_INVERSE: [u8; 128] = {
const __: u8 = 255;
[
// 1 2 3 4 5 6 7 8 9 A B C D E F
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 0
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 1
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 2
00, 01, 02, 03, 04, 05, 06, 07, 08, 09, __, __, __, __, __, __, // 3
__, 10, 11, 12, 13, 14, 15, __, __, __, __, __, __, __, __, __, // 4
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 5
__, 10, 11, 12, 13, 14, 15, __, __, __, __, __, __, __, __, __, // 6
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 7
]
};
macro_rules! write_hex {
($input:expr, $output:expr, $bytelen:expr) => {{
assert_eq!($input.len(), $bytelen);
if $output.len() != $bytelen * 2 {
Err(crate::error::ChorusError::BufferTooSmall.into())
} else {
for (i, byte) in $input.iter().enumerate() {
$output[i * 2] = crate::HEX_CHARS[((byte & 0xF0) >> 4) as usize];
$output[i * 2 + 1] = crate::HEX_CHARS[(byte & 0x0F) as usize];
}
Ok(())
}
}};
}
macro_rules! read_hex {
($input:expr, $output:expr, $bytelen:expr) => {{
assert_eq!($output.len(), $bytelen);
if $input.len() != $bytelen * 2 {
Err(Into::<crate::error::Error>::into(crate::error::ChorusError::EndOfInput))
} else {
let mut i = 0;
loop {
let high = crate::HEX_INVERSE[$input[i * 2] as usize];
if high == 255 {
break Err(crate::error::ChorusError::BadHexInput.into());
}
let low = crate::HEX_INVERSE[$input[i * 2 + 1] as usize];
if low == 255 {
break Err(crate::error::ChorusError::BadHexInput.into());
}
$output[i] = high * 16 + low;
i += 1;
if i == $bytelen {
break Ok(());
}
}
}
}};
}
macro_rules! parse_u16 {
($input:expr, $start:expr) => {
u16::from_ne_bytes($input[$start .. $start+2].try_into().unwrap())
}
}
macro_rules! parse_u32 {
($input:expr, $start:expr) => {
u32::from_ne_bytes($input[$start .. $start+4].try_into().unwrap())
}
}
macro_rules! parse_u64 {
($input:expr, $start:expr) => {
u64::from_ne_bytes($input[$start .. $start+8].try_into().unwrap())
}
}

441
src/main.rs Normal file
View File

@ -0,0 +1,441 @@
include!("macros.rs");
pub mod config;
pub mod error;
pub mod globals;
pub mod nostr;
pub mod reply;
pub mod store;
pub mod tls;
pub mod types;
pub mod web;
use crate::config::{Config, FriendlyConfig};
use crate::error::{ChorusError, Error};
use crate::globals::GLOBALS;
use crate::reply::NostrReply;
use crate::store::Store;
use crate::tls::MaybeTlsStream;
use crate::types::{OwnedFilter, Pubkey, Time};
use futures::{sink::SinkExt, stream::StreamExt};
use hyper::service::Service;
use hyper::upgrade::Upgraded;
use hyper::{Body, Request, Response};
use hyper_tungstenite::{tungstenite, WebSocketStream};
use std::collections::HashMap;
use std::env;
use std::error::Error as StdError;
use std::fs::OpenOptions;
use std::future::Future;
use std::io::Read;
use std::net::SocketAddr;
use std::pin::Pin;
use std::sync::atomic::Ordering;
use std::task::{Context, Poll};
use std::time::Duration;
use textnonce::TextNonce;
use tokio::net::{TcpListener, TcpStream};
use tokio::signal::unix::{signal, SignalKind};
use tungstenite::protocol::WebSocketConfig;
use tungstenite::Message;
#[tokio::main]
async fn main() -> Result<(), Error> {
env_logger::init();
// Get args (config path)
let mut args = env::args();
if args.len() <= 1 {
panic!("USAGE: chorus <config_path>");
}
let _ = args.next(); // ignore program name
let config_path = args.next().unwrap();
// Read config file
let mut file = OpenOptions::new().read(true).open(config_path)?;
let mut contents = String::new();
file.read_to_string(&mut contents)?;
let friendly_config: FriendlyConfig = ron::from_str(&contents)?;
let config: Config = friendly_config.into_config()?;
log::debug!("Loaded config file.");
// Setup store
let store = Store::new(&config.data_directory, config.allow_scraping)?;
let _ = GLOBALS.store.set(store);
// TLS setup
let maybe_tls_acceptor = if config.use_tls {
log::info!("Using TLS");
Some(tls::tls_acceptor(&config)?)
} else {
log::info!("Not using TLS");
None
};
// Bind listener to port
let listener = TcpListener::bind((&*config.ip_address, config.port)).await?;
log::info!("Running on {}:{}", config.ip_address, config.port);
// Store config into GLOBALS
*GLOBALS.config.write().await = config;
let mut interrupt_signal = signal(SignalKind::interrupt())?;
let mut quit_signal = signal(SignalKind::quit())?;
let mut terminate_signal = signal(SignalKind::terminate())?;
loop {
tokio::select! {
// Exits gracefully upon exit-type signals
v = interrupt_signal.recv() => if v.is_some() {
log::info!("SIGINT");
break;
},
v = quit_signal.recv() => if v.is_some() {
log::info!("SIGQUIT");
break;
},
v = terminate_signal.recv() => if v.is_some() {
log::info!("SIGTERM");
break;
},
// Accepts network connections and spawn a task to serve each one
v = listener.accept() => {
let (tcp_stream, peer_addr) = v?;
let ipaddr = peer_addr.ip();
if let Some(ban_until) = GLOBALS.banlist.read().await.get(&ipaddr) {
let now = Time::now();
if *ban_until > now {
log::debug!("{peer_addr}: Blocking reconnection until {ban_until}");
continue;
}
}
if let Some(tls_acceptor) = &maybe_tls_acceptor {
let tls_acceptor_clone = tls_acceptor.clone();
tokio::spawn(async move {
match tls_acceptor_clone.accept(tcp_stream).await {
Err(e) => log::error!("{}", e),
Ok(tls_stream) => {
if let Err(e) = serve(MaybeTlsStream::Rustls(tls_stream), peer_addr).await {
log::error!("{}", e);
}
}
}
});
} else {
serve(MaybeTlsStream::Plain(tcp_stream), peer_addr).await?;
}
}
};
}
// Pre-sync in case something below hangs up
let _ = GLOBALS.store.get().unwrap().sync();
// Set the shutting down signal
let _ = GLOBALS.shutting_down.send(true);
// Wait for active websockets to shutdown gracefully
let mut num_clients = GLOBALS.num_clients.load(Ordering::Relaxed);
if num_clients != 0 {
log::info!("Waiting for {num_clients} websockets to shutdown...");
// We will check if all clients have shutdown every 25ms
let sleep = tokio::time::sleep(Duration::from_millis(25));
tokio::pin!(sleep);
while num_clients != 0 {
// If we get another shutdown signal, stop waiting for websockets
tokio::select! {
v = interrupt_signal.recv() => if v.is_some() {
break;
},
v = quit_signal.recv() => if v.is_some() {
break;
},
v = terminate_signal.recv() => if v.is_some() {
break;
},
() = &mut sleep => {
num_clients = GLOBALS.num_clients.load(Ordering::Relaxed);
continue;
}
}
}
}
log::info!("Syncing and shutting down.");
let _ = GLOBALS.store.get().unwrap().sync();
Ok(())
}
// Serve a single network connection
async fn serve(stream: MaybeTlsStream<TcpStream>, peer_addr: SocketAddr) -> Result<(), Error> {
// Serve the network stream with our http server and our HttpService
let service = HttpService { peer: peer_addr };
let connection = GLOBALS
.http_server
.serve_connection(stream, service)
.with_upgrades();
tokio::spawn(async move {
// If our service exits with an error, log the error
if let Err(he) = connection.await {
if let Some(src) = he.source() {
if &*format!("{}", src) == "Transport endpoint is not connected (os error 107)" {
// do nothing
} else {
// Print in detail
log::error!("{:?}", src);
}
} else {
// Print in less detail
let e: Error = he.into();
log::error!("{}", e);
}
}
});
Ok(())
}
// This is our per-connection HTTP service
struct HttpService {
peer: SocketAddr,
}
impl Service<Request<Body>> for HttpService {
type Response = Response<Body>;
type Error = Error;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, _: &mut Context) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
// This is called for each HTTP request made by the client
// NOTE: it is not called for each websocket message once upgraded.
fn call(&mut self, req: Request<Body>) -> Self::Future {
let peer = self.peer;
Box::pin(async move { handle_http_request(peer, req).await })
}
}
async fn handle_http_request(
peer: SocketAddr,
mut request: Request<Body>,
) -> Result<Response<Body>, Error> {
if hyper_tungstenite::is_upgrade_request(&request) {
let web_socket_config = WebSocketConfig {
max_write_buffer_size: 1024 * 1024, // 1 MB
max_message_size: Some(1024 * 1024), // 1 MB
max_frame_size: Some(1024 * 1024), // 1 MB
..Default::default()
};
let (response, websocket) =
hyper_tungstenite::upgrade(&mut request, Some(web_socket_config))?;
tokio::spawn(async move {
// Await the websocket upgrade process
match websocket.await {
Ok(websocket) => {
// Build a websocket service
let mut ws_service = WebSocketService {
peer,
subscriptions: HashMap::new(),
// We start with a 1-page buffer, and grow it if needed.
buffer: vec![0; 4096],
websocket,
challenge: TextNonce::new().into_string(),
user: None,
};
// Increment count of active websockets
let old_num_websockets = GLOBALS.num_clients.fetch_add(1, Ordering::SeqCst);
log::info!(
"{}: websocket started (making {} active websockets)",
peer,
old_num_websockets + 1
);
// Handle the websocket
if let Err(e) = ws_service.handle_websocket_stream().await {
if matches!(
e.inner,
ChorusError::Tungstenite(tungstenite::error::Error::Protocol(
tungstenite::error::ProtocolError::ResetWithoutClosingHandshake
))
) {
// Swallow the boring error
} else {
log::error!("{}: {}", peer, e);
}
}
// Decrement count of active websockets
let old_num_websockets = GLOBALS.num_clients.fetch_sub(1, Ordering::SeqCst);
log::info!(
"{}: websocket ended (making {} active websockets)",
peer,
old_num_websockets - 1
);
// Everybody gets a 4 second ban on disconnect, to prevent rapid reconnection
let ipaddr = peer.ip();
let mut until = Time::now();
until.0 += 4;
if let Some(current_ban) = GLOBALS.banlist.read().await.get(&ipaddr) {
until.0 = current_ban.0.max(until.0);
}
GLOBALS.banlist.write().await.insert(ipaddr, until);
}
Err(e) => {
log::error!("{}", e);
}
}
});
Ok(response)
} else {
// check for Accept header of application/nostr+json
if let Some(accept) = request.headers().get("Accept") {
if let Ok(s) = accept.to_str() {
if s == "application/nostr+json" {
return web::serve_nip11(peer).await;
}
}
}
web::serve_http(peer, request).await
}
}
struct WebSocketService {
pub peer: SocketAddr,
pub subscriptions: HashMap<String, Vec<OwnedFilter>>,
pub buffer: Vec<u8>,
pub websocket: WebSocketStream<Upgraded>,
pub challenge: String,
pub user: Option<Pubkey>,
}
impl WebSocketService {
async fn handle_websocket_stream(&mut self) -> Result<(), Error> {
// Subscribe to the shutting down channel
let mut shutting_down = GLOBALS.shutting_down.subscribe();
// Subscribe to the new_events broadcast channel
let mut new_events = GLOBALS.new_events.subscribe();
// Offer AUTH to clients right off the bat
let reply = NostrReply::Auth(self.challenge.clone());
self.websocket.send(Message::text(reply.as_json())).await?;
loop {
tokio::select! {
message_option = self.websocket.next() => {
match message_option {
Some(message) => {
let message = message?;
self.handle_websocket_message(message).await?;
},
None => break, // the websocket is closed
}
},
offset_result = new_events.recv() => {
let offset = offset_result?;
self.handle_new_event(offset).await?;
},
_r = shutting_down.changed() => {
// Shutdown the websocket gracefully
self.websocket.send(Message::Close(None)).await?;
break;
},
}
}
Ok(())
}
// If the event matches a subscription they have open, send them the event
async fn handle_new_event(&mut self, new_event_offset: usize) -> Result<(), Error> {
if self.subscriptions.is_empty() {
return Ok(());
}
if let Some(event) = GLOBALS
.store
.get()
.unwrap()
.get_event_by_offset(new_event_offset)?
{
'subs: for (subid, filters) in self.subscriptions.iter() {
for filter in filters.iter() {
if filter.as_filter()?.event_matches(&event)? {
let message = NostrReply::Event(subid, event.clone());
self.websocket
.send(Message::text(message.as_json()))
.await?;
continue 'subs;
}
}
}
}
Ok(())
}
async fn handle_websocket_message(&mut self, message: Message) -> Result<(), Error> {
match message {
Message::Text(msg) => {
log::debug!("{}: <= {}", self.peer, msg);
// This is defined in nostr.rs
if let Err(e) = self.handle_nostr_message(msg).await {
log::error!("{e}");
let reply = NostrReply::Notice(format!("error: {}", e));
self.websocket.send(Message::text(reply.as_json())).await?;
}
}
Message::Binary(msg) => {
let reply = NostrReply::Notice(
"binary messages are not processed by this relay".to_owned(),
);
self.websocket.send(Message::text(reply.as_json())).await?;
log::info!(
"{}: Received unhandled binary message: {:02X?}",
self.peer,
msg
);
}
Message::Ping(msg) => {
// No need to send a reply: tungstenite takes care of this for you.
log::debug!("{}: Received ping message: {:02X?}", self.peer, msg);
}
Message::Pong(msg) => {
log::debug!("{}: Received pong message: {:02X?}", self.peer, msg);
}
Message::Close(msg) => {
// No need to send a reply: tungstenite takes care of this for you.
if let Some(msg) = &msg {
log::debug!(
"{}: Received close message with code {} and message: {}",
self.peer,
msg.code,
msg.reason
);
} else {
log::debug!("{}: Received close message", self.peer);
}
}
Message::Frame(_msg) => {
unreachable!();
}
}
Ok(())
}
}

123
src/neg_storage.rs
View File

@ -1,123 +0,0 @@
use negentropy::{Bound, Error, Id, Item, NegentropyStorageBase};
// We construct our own NegentropyStorageVector since NegentropyStorageBase
// is not implemented for a &NegentropyStorageVector upstream
/// Negentropy Storage Vector
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct NegentropyStorageVector {
items: Vec<Item>,
sealed: bool,
}
impl NegentropyStorageBase for &NegentropyStorageVector {
fn size(&self) -> Result<usize, Error> {
self.check_sealed()?;
Ok(self.items.len())
}
fn get_item(&self, i: usize) -> Result<Option<Item>, Error> {
self.check_sealed()?;
Ok(self.items.get(i).copied())
}
fn iterate(
&self,
begin: usize,
end: usize,
cb: &mut dyn FnMut(Item, usize) -> Result<bool, Error>,
) -> Result<(), Error> {
self.check_sealed()?;
self.check_bounds(begin, end)?;
for i in begin..end {
if !cb(self.items[i], i)? {
break;
}
}
Ok(())
}
fn find_lower_bound(&self, mut first: usize, last: usize, value: &Bound) -> usize {
let mut count: usize = last - first;
while count > 0 {
let mut it: usize = first;
let step: usize = count / 2;
it += step;
if self.items[it] < value.item {
it += 1;
first = it;
count -= step + 1;
} else {
count = step;
}
}
first
}
}
impl NegentropyStorageVector {
/// Create new storage
#[inline]
pub fn new() -> Self {
Self::default()
}
/// Create new storage with capacity
#[inline]
pub fn with_capacity(capacity: usize) -> Self {
Self {
items: Vec::with_capacity(capacity),
sealed: false,
}
}
/// Insert item
pub fn insert(&mut self, created_at: u64, id: Id) -> Result<(), Error> {
if self.sealed {
return Err(Error::AlreadySealed);
}
let elem: Item = Item::with_timestamp_and_id(created_at, id);
self.items.push(elem);
Ok(())
}
/// Seal
pub fn seal(&mut self) -> Result<(), Error> {
if self.sealed {
return Err(Error::AlreadySealed);
}
self.sealed = true;
self.items.sort();
self.items.dedup();
Ok(())
}
/// Unseal
pub fn unseal(&mut self) -> Result<(), Error> {
self.sealed = false;
Ok(())
}
fn check_sealed(&self) -> Result<(), Error> {
if !self.sealed {
return Err(Error::NotSealed);
}
Ok(())
}
fn check_bounds(&self, begin: usize, end: usize) -> Result<(), Error> {
if begin > end || end > self.items.len() {
return Err(Error::BadRange);
}
Ok(())
}
}

871
src/nostr.rs
View File

File diff suppressed because it is too large Load Diff

72
src/reply.rs
View File

@ -1,8 +1,6 @@
use crate::Error;
use pocket_types::{write_hex, Event, Hll8, Id};
use crate::types::{Event, Id};
use std::fmt;
#[derive(Debug, Clone, Copy)]
pub enum NostrReplyPrefix {
None,
AuthRequired,
@ -31,74 +29,32 @@ impl fmt::Display for NostrReplyPrefix {
}
}
#[derive(Debug, Clone)]
#[allow(clippy::large_enum_variant)]
pub enum NostrReply<'a> {
Auth(String),
Event(&'a str, &'a Event),
Event(&'a str, Event<'a>),
Ok(Id, bool, NostrReplyPrefix, String),
Eose(&'a str),
Closed(&'a str, NostrReplyPrefix, String),
Notice(String),
Count(&'a str, usize, Option<Hll8>),
NegErr(&'a str, String),
NegMsg(&'a str, Vec<u8>),
}
impl NostrReply<'_> {
pub fn as_json(&self) -> Result<String, Error> {
Ok(match self {
NostrReply::Auth(challenge) => {
let esc_challenge = escape(challenge)?;
format!(r#"["AUTH","{esc_challenge}"]"#)
}
NostrReply::Event(subid, event) => {
let esc_subid = escape(subid)?;
format!(r#"["EVENT","{esc_subid}",{event}]"#)
}
NostrReply::Ok(id, ok, prefix, msg) => {
let esc_msg = escape(msg)?;
format!(r#"["OK","{id}",{ok},"{prefix}{esc_msg}"]"#)
}
NostrReply::Eose(subid) => {
let esc_subid = escape(subid)?;
format!(r#"["EOSE","{esc_subid}"]"#)
}
pub fn as_json(&self) -> String {
match self {
NostrReply::Auth(challenge) => format!(r#"["AUTH", "{challenge}"]"#),
NostrReply::Event(subid, event) => format!(r#"["EVENT", "{subid}", {}]"#, event),
NostrReply::Ok(id, ok, prefix, msg) => format!(r#"["OK","{id}",{ok},"{prefix}{msg}"]"#),
NostrReply::Eose(subid) => format!(r#"["EOSE","{subid}"]"#),
NostrReply::Closed(subid, prefix, msg) => {
format!(r#"["CLOSED","{subid}","{prefix}{msg}"]"#)
}
NostrReply::Notice(msg) => {
let esc_msg = escape(msg)?;
format!(r#"["NOTICE","{esc_msg}"]"#)
}
NostrReply::Count(subid, c, opthll) => {
let esc_subid = escape(subid)?;
if let Some(hll) = opthll {
let hll = hll.to_hex_string();
format!(r#"["COUNT","{esc_subid}",{{"count":{c}, "hll":"{hll}"}}]"#)
} else {
format!(r#"["COUNT","{esc_subid}",{{"count":{c}}}]"#)
}
}
NostrReply::NegErr(subid, reason) => {
let esc_subid = escape(subid)?;
let esc_reason = escape(reason)?;
format!(r#"["NEG-ERR","{esc_subid}","{esc_reason}"]"#)
}
NostrReply::NegMsg(subid, msg) => {
let esc_subid = escape(subid)?;
// write msg as hex
let mut buf: Vec<u8> = vec![0; msg.len() * 2];
write_hex!(msg, &mut buf, msg.len()).unwrap();
let msg_hex = unsafe { std::str::from_utf8_unchecked(&buf) };
format!(r#"["NEG-MSG","{esc_subid}","{}"]"#, msg_hex)
}
})
NostrReply::Notice(msg) => format!(r#"["NOTICE","{msg}"]"#),
}
}
}
fn escape(s: &str) -> Result<String, Error> {
let v: Vec<u8> = Vec::with_capacity(256);
let e = pocket_types::json::json_escape(s.as_bytes(), v)?;
Ok(unsafe { String::from_utf8_unchecked(e) })
impl fmt::Display for NostrReply<'_> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.as_json())
}
}

183
src/store/event_store.rs Normal file
View File

@ -0,0 +1,183 @@
use crate::error::Error;
use crate::types::Event;
use mmap_append::MmapAppend;
use std::fs::{File, OpenOptions};
use std::mem;
use std::path::Path;
use std::sync::atomic::{AtomicUsize, Ordering};
/// This is the size of the initial event map, and also how large it grows by
/// when we need to grow it. This should be a multiple of the page size (4096)
// While debugging, we like to use a small value to increase the frequency of
// resizing, to help detect if there are problems in the algorithm.
#[cfg(debug_assertions)]
const EVENT_MAP_CHUNK: usize = 2048;
#[cfg(not(debug_assertions))]
const EVENT_MAP_CHUNK: usize = 4096 * 1024; // grow by 4 megabytes at a time
/// An EventStore is a fast storage facility for events.
#[derive(Debug)]
pub struct EventStore {
// the Mmap doesn't need us to keep the file, but we keep it for resizing.
event_map_file: File,
event_map_file_len: AtomicUsize,
// This is a linear sequence of events in an append-only memory mapped file which
// internally remembers the 'end' pointer and internally prevents multiple writers.
event_map: MmapAppend,
}
impl EventStore {
/// Create a new `EventStore`. The `event_map_file` is the eventually large file
/// that holds all the events.
pub fn new<P: AsRef<Path>>(event_map_file: P) -> Result<EventStore, Error> {
// Open the event map file, possibly creating if it isn't there
let event_map_file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open(event_map_file)?;
// Get it's size
let metadata = event_map_file.metadata()?;
let mut len = metadata.len() as usize;
// Determine if we just created it
// (not long enough for the required end offset)
let new = len < mem::size_of::<usize>();
// If brand new:
if new {
// grow to initial size
len = EVENT_MAP_CHUNK;
event_map_file.set_len(EVENT_MAP_CHUNK as u64)?;
}
// Memory map it
let event_map = unsafe { MmapAppend::new(&event_map_file, new)? };
log::info!(
"Event Store: new={:?} end={} len={}",
new,
event_map.get_end(),
len
);
Ok(EventStore {
event_map_file,
event_map_file_len: AtomicUsize::new(len),
event_map,
})
}
/// Get the number of bytes used in the event map
#[inline]
pub fn read_event_map_end(&self) -> usize {
self.event_map.get_end()
}
/// Get an event by its offset in the map
pub fn get_event_by_offset(&self, offset: usize) -> Result<Option<Event>, Error> {
// deserialize event
let event = Event::delineate(&self.event_map[offset..])?;
Ok(Some(event))
}
// This stores an event
// It does NOT validate the event.
// It does NOT check first if the event is already stored, so it could store a duplicate
// It does NOT record the event into any indexes
// But it does grow the file if needed and returns the offset where it was stored
pub fn store_event(&self, event: &Event) -> Result<usize, Error> {
let event_size = event.length();
loop {
let result = self.event_map.append(event_size, |dst| event.macopy(dst));
match result {
Ok(offset) => return Ok(offset),
Err(e) => {
if e.kind() == std::io::ErrorKind::Other {
if e.to_string() == "Out of space" {
// Determine the new size
let new_file_len = {
let file_len = self.event_map_file_len.load(Ordering::Relaxed);
file_len + EVENT_MAP_CHUNK
};
// Grow the file
self.event_map_file.set_len(new_file_len as u64)?;
// Resize the memory map
self.event_map.resize(new_file_len)?;
// Save this new length
self.event_map_file_len
.store(new_file_len, Ordering::Relaxed);
// Try again
continue;
} else {
return Err(e.into());
}
} else {
return Err(e.into());
}
}
}
}
}
}
#[cfg(test)]
mod tests {
use crate::store::EventStore;
use crate::types::Event;
#[test]
fn test_event_store() {
let tempdir = tempfile::tempdir().unwrap();
let path = tempdir.path().join("mmap");
let store = EventStore::new(&path).unwrap();
println!("Event map has {} used bytes", store.read_event_map_end());
let e1str = br#"{"id":"000000005ccb8402fe9af2ecc72ca1dbbf2dbeb9a0c6353b7f8198a65106f04a","pubkey":"7bdef7be22dd8e59f4600e044aa53a1cf975a9dc7d27df5833bc77db784a5805","created_at":1677930312,"kind":1,"tags":[["p","fe1d10131ca6103715d261f1615a8cd31f6b68a3d1a3272aab8e9e83f787126e"],["e","f76a6f60df029c388e98d4e97c67424cb71e34455302580d3aa48d0be96dfaef","wss://nos.lol","root"],["e","f529676213b123f55117ae9b93ca7dd6c85c872b7c260da7dbf60d71875879f6","wss://brb.io","reply"],["nonce","12912720851603613541","25"]],"content":"https://www.nostr.guru/p/fe1d10131ca6103715d261f1615a8cd31f6b68a3d1a3272aab8e9e83f787126e\nHere I can see only the today contact list.\n\nI tried to query relay.damus.io and nostr.wine but I can get only this last contact list, perhaps they have a time limit; you could ask some relay owner for a backup.\nIf can be useful this is the REQ filter to use:\n{\"kinds\": [3], \"authors\": [\"fe1d10131ca6103715d261f1615a8cd31f6b68a3d1a3272aab8e9e83f787126e\"], \"since\": 1672569529}","sig":"7c72d15cf9b4244cb8c49564d3735d0ac620dca69253956975def12054bc50217307f062f3e9ea879de8bae756c844a9315f03aabbb5f5fd94f9fc9beb76f457"}"#;
let mut buffer1: Vec<u8> = Vec::with_capacity(4096);
buffer1.resize(4096, 0);
let (_insize, event1) = Event::from_json(&e1str[..], &mut buffer1).unwrap();
let offset1 = store.store_event(&event1).unwrap();
let e2str = br#"{"id":"00000000a6fa8ee15b17fcc5bb49f09f85c15cddf5349986db09fddc0a123f7d","pubkey":"7cc328a08ddb2afdf9f9be77beff4c83489ff979721827d628a542f32a247c0e","created_at":1678835260,"kind":1,"tags":[["p","0000000033f569c7069cdec575ca000591a31831ebb68de20ed9fb783e3fc287"],["e","52338357568d06379ad2412a2a2033f23f224754d74148bbfc659b872c9477c4","wss://relay.damus.io/","root"],["nonce","9223372036941030112","32"]],"content":"oh shit, it got released?! no wayyy","sig":"ec9bcdcaa843a2a275857999556136d2b3cc47dac52c899d313dd70291b9661590d4c10b13680bbfa685d67d319798cdfa3ca58f61af5f8acffd0e25bca5ab95"}"#;
let mut buffer2: Vec<u8> = Vec::with_capacity(4096);
buffer2.resize(4096, 0);
let (_insize, event2) = Event::from_json(&e2str[..], &mut buffer2).unwrap();
let offset2 = store.store_event(&event2).unwrap();
let e3str = br#"{"id":"00000000ad0efde5b63e9b24b12a586dc98df372e1fd6f96ac6ad24ea2ed1350","pubkey":"c5fb6ecc876e0458e3eca9918e370cbcd376901c58460512fe537a46e58c38bb","created_at":1681739201,"kind":7,"tags":[["e","193bd20beb8fc13f4218ea106928c3be81ee3b2ad2b1bdbdd2c55efd859a195a","wss://eden.nostr.land/"],["p","3f770d65d3a764a9c5cb503ae123e62ec7598ad035d836e2a810f3877a745b24"],["client","gossip"],["nonce","2305843009213833122","22"]],"content":"+","sig":"f344111c221d2fea5f006865b98b0767b40ed1cc2907d8a325a8dea4b98414d008296ff4f4bd4666d52ec86ffd4739e807c6655ce43de98b473326e30957fcb2"}"#;
let mut buffer3: Vec<u8> = Vec::with_capacity(4096);
buffer3.resize(4096, 0);
let (_insize, event3) = Event::from_json(&e3str[..], &mut buffer3).unwrap();
let offset3 = store.store_event(&event3).unwrap();
println!("Event map has {} used bytes", store.read_event_map_end());
if let Some(event) = store.get_event_by_offset(offset1).unwrap() {
assert_eq!(event, event1);
} else {
panic!("EVENT 1 IS WRONG");
}
if let Some(event) = store.get_event_by_offset(offset2).unwrap() {
assert_eq!(event, event2);
} else {
panic!("EVENT 2 IS WRONG");
}
if let Some(event) = store.get_event_by_offset(offset3).unwrap() {
assert_eq!(event, event3);
} else {
panic!("EVENT 3 IS WRONG");
}
}
}

641
src/store/mod.rs Normal file
View File

@ -0,0 +1,641 @@
pub mod event_store;
pub use event_store::EventStore;
use crate::error::{ChorusError, Error};
use crate::types::{Event, Filter, Id, Kind, Pubkey, Time};
use heed::byteorder::BigEndian;
use heed::types::{OwnedType, UnalignedSlice, Unit, U64};
use heed::{Database, Env, EnvFlags, EnvOpenOptions, RwTxn};
use std::fs;
use std::ops::Bound;
#[derive(Debug)]
pub struct Store {
events: EventStore,
env: Env,
ids: Database<UnalignedSlice<u8>, OwnedType<usize>>,
akci: Database<UnalignedSlice<u8>, OwnedType<usize>>,
atci: Database<UnalignedSlice<u8>, OwnedType<usize>>,
ktci: Database<UnalignedSlice<u8>, OwnedType<usize>>,
deleted_offsets: Database<U64<BigEndian>, Unit>,
deleted_events: Database<UnalignedSlice<u8>, Unit>,
allow_scraping: bool,
}
impl Store {
/// Setup persistent storage
pub fn new(data_directory: &str, allow_scraping: bool) -> Result<Store, Error> {
let mut builder = EnvOpenOptions::new();
unsafe {
builder.flags(EnvFlags::NO_TLS);
}
builder.max_dbs(32);
builder.map_size(1048576 * 1024 * 24); // 24 GB
let dir = format!("{}/lmdb", data_directory);
fs::create_dir_all(&dir)?;
let env = match builder.open(&dir) {
Ok(env) => env,
Err(e) => {
log::error!("Unable to open LMDB at {}", dir);
return Err(e.into());
}
};
// Open/Create maps
let mut txn = env.write_txn()?;
let ids = env
.database_options()
.types::<UnalignedSlice<u8>, OwnedType<usize>>()
.name("ids")
.create(&mut txn)?;
let akci = env
.database_options()
.types::<UnalignedSlice<u8>, OwnedType<usize>>()
.name("akci")
.create(&mut txn)?;
let atci = env
.database_options()
.types::<UnalignedSlice<u8>, OwnedType<usize>>()
.name("atci")
.create(&mut txn)?;
let ktci = env
.database_options()
.types::<UnalignedSlice<u8>, OwnedType<usize>>()
.name("ktci")
.create(&mut txn)?;
let deleted_offsets = env
.database_options()
.types::<U64<BigEndian>, Unit>()
.name("deleted_offsets")
.create(&mut txn)?;
let deleted_events = env
.database_options()
.types::<UnalignedSlice<u8>, Unit>()
.name("deleted-events")
.create(&mut txn)?;
txn.commit()?;
log::info!("Store is setup");
let event_map_file = format!("{}/event.map", data_directory);
Ok(Store {
events: EventStore::new(event_map_file)?,
env,
ids,
akci,
atci,
ktci,
deleted_offsets,
deleted_events,
allow_scraping,
})
}
/// Sync the data to disk. This happens periodically, but sometimes it's useful to force
/// it.
pub fn sync(&self) -> Result<(), Error> {
self.env.force_sync()?;
Ok(())
}
/// Store an event.
///
/// Returns the offset where the event is stored at, which can be used to fetch
/// the event via get_event_by_offset().
///
/// If the event already exists, you will get a ChorusError::Duplicate
///
/// If the event is ephemeral, it will be stored and you will get an offset, but
/// it will not be indexed.
pub fn store_event(&self, event: &Event) -> Result<usize, Error> {
// TBD: should we validate the event?
let mut txn = self.env.write_txn()?;
let offset;
// Only if it doesn't already exist
if self.ids.get(&txn, event.id().0.as_slice())?.is_none() {
// Reject event if it was deleted
{
let deleted_key = Self::key_deleted_events(event.id(), event.pubkey());
if self.deleted_events.get(&txn, &deleted_key)?.is_some() {
return Err(ChorusError::Deleted.into());
}
}
// Store the event
offset = self.events.store_event(event)?;
if event.kind().is_ephemeral() {
// Do not index ephemeral events, not even by id.
// But save them in the deleted table
let offset_u64 = offset as u64;
self.deleted_offsets.put(&mut txn, &offset_u64, &())?;
} else {
// Index the event
self.index(&mut txn, event, offset)?;
}
// If replaceable or parameterized replaceable,
// find and delete all but the first one in the group
if event.kind().is_replaceable() || event.kind().is_parameterized_replaceable() {
self.delete_replaced(&mut txn, event)?;
}
// Handle deletion events
if event.kind() == Kind(5) {
self.handle_deletion_event(&mut txn, event)?;
}
txn.commit()?;
} else {
return Err(ChorusError::Duplicate.into());
}
Ok(offset)
}
pub fn handle_deletion_event(&self, txn: &mut RwTxn<'_>, event: &Event) -> Result<(), Error> {
for mut tag in event.tags()?.iter() {
if let Some(tagname) = tag.next() {
if tagname == b"e" {
if let Some(id_hex) = tag.next() {
if let Ok(id) = Id::read_hex(id_hex) {
// Add deletion pair to the event_deleted table
let deleted_key = Self::key_deleted_events(id, event.pubkey());
self.deleted_events.put(txn, &deleted_key, &())?;
// Delete pair
if let Some(target) = self.get_event_by_id(id)? {
if target.pubkey() == event.pubkey() {
self.delete(txn, id)?;
}
}
}
}
}
}
}
Ok(())
}
/// Get an event by its offset.
pub fn get_event_by_offset(&self, offset: usize) -> Result<Option<Event>, Error> {
self.events.get_event_by_offset(offset)
}
/// Get an event by Id
pub fn get_event_by_id(&self, id: Id) -> Result<Option<Event>, Error> {
let txn = self.env.read_txn()?;
if let Some(offset) = self.ids.get(&txn, id.0.as_slice())? {
self.events.get_event_by_offset(offset)
} else {
Ok(None)
}
}
/// Find all events that match the filter
pub fn find_events(&self, filter: Filter) -> Result<Vec<Event>, Error> {
let mut output: Vec<Event> = Vec::new();
if filter.num_ids() > 0 {
// Fetch by id
for id in filter.ids() {
if let Some(event) = self.get_event_by_id(id)? {
// and check each against the rest of the filter
if filter.event_matches(&event)? {
output.push(event);
}
}
// Stop if limited
if output.len() >= filter.limit() as usize {
return Ok(output);
}
}
} else if filter.num_authors() > 0 && filter.num_kinds() > 0 {
for author in filter.authors() {
'kind: for kind in filter.kinds() {
let start_prefix = Self::key_akci(
author,
kind,
filter.until(), // scan goes backwards in time
Id([0; 32]),
);
let end_prefix = Self::key_akci(
author,
kind,
filter.since(), // scan goes backwards in time
Id([255; 32]),
);
let range = (
Bound::Included(&*start_prefix),
Bound::Excluded(&*end_prefix),
);
let txn = self.env.read_txn()?;
let iter = self.akci.range(&txn, &range)?;
for result in iter {
let (_key, offset) = result?;
if let Some(event) = self.events.get_event_by_offset(offset)? {
// check against the rest of the filter
if filter.event_matches(&event)? {
output.push(event);
// If kind is replaceable (and not parameterized)
// then don't take any more events for this author-kind
// pair.
// NOTE that this optimization is difficult to implement
// for other replaceable event situations
if kind.is_replaceable() {
continue 'kind;
}
}
}
// Stop if limited
if output.len() >= filter.limit() as usize {
return Ok(output);
}
}
}
}
} else if filter.num_authors() > 0 && !filter.tags()?.is_empty() {
for author in filter.authors() {
let tags = filter.tags()?;
for mut tag in tags.iter() {
if let Some(tag0) = tag.next() {
if let Some(tagvalue) = tag.next() {
let start_prefix = Self::key_atci(
author,
tag0[0],
tagvalue,
filter.until(), // scan goes backwards in time
Id([0; 32]),
);
let end_prefix = Self::key_atci(
author,
tag0[0],
tagvalue,
filter.since(), // scan goes backwards in time
Id([255; 32]),
);
let range = (
Bound::Included(&*start_prefix),
Bound::Excluded(&*end_prefix),
);
let txn = self.env.read_txn()?;
let iter = self.akci.range(&txn, &range)?;
for result in iter {
let (_key, offset) = result?;
if let Some(event) = self.events.get_event_by_offset(offset)? {
// check against the rest of the filter
if filter.event_matches(&event)? {
output.push(event);
}
}
// Stop if limited
if output.len() >= filter.limit() as usize {
return Ok(output);
}
}
}
}
}
}
} else if filter.num_kinds() > 0 && !filter.tags()?.is_empty() {
for kind in filter.kinds() {
let tags = filter.tags()?;
for mut tag in tags.iter() {
if let Some(tag0) = tag.next() {
if let Some(tagvalue) = tag.next() {
let start_prefix = Self::key_ktci(
kind,
tag0[0],
tagvalue,
filter.until(), // scan goes backwards in time
Id([0; 32]),
);
let end_prefix = Self::key_ktci(
kind,
tag0[0],
tagvalue,
filter.since(), // scan goes backwards in time
Id([255; 32]),
);
let range = (
Bound::Included(&*start_prefix),
Bound::Excluded(&*end_prefix),
);
let txn = self.env.read_txn()?;
let iter = self.akci.range(&txn, &range)?;
for result in iter {
let (_key, offset) = result?;
if let Some(event) = self.events.get_event_by_offset(offset)? {
// check against the rest of the filter
if filter.event_matches(&event)? {
output.push(event);
}
}
// Stop if limited
if output.len() >= filter.limit() as usize {
return Ok(output);
}
}
}
}
}
}
} else if self.allow_scraping {
// This is INEFFICIENT as it scans through EVERY EVENT
// but the filter is a scraper and we don't have a lot of support
// for scrapers.
let txn = self.env.read_txn()?;
let iter = self.ids.iter(&txn)?;
for result in iter {
let (_key, offset) = result?;
if let Some(event) = self.events.get_event_by_offset(offset)? {
if filter.event_matches(&event)? {
output.push(event);
}
}
}
} else {
return Err(ChorusError::Scraper.into());
}
Ok(output)
}
/// Delete an event by id
fn delete(&self, txn: &mut RwTxn<'_>, id: Id) -> Result<(), Error> {
if let Some(offset) = self.ids.get(txn, id.0.as_slice())? {
self.set_offset_as_deleted(offset)?;
// Also remove from the id index
self.ids.delete(txn, id.0.as_slice())?;
}
Ok(())
}
// Index the event
fn index(&self, txn: &mut RwTxn<'_>, event: &Event, offset: usize) -> Result<(), Error> {
// Index by id
self.ids.put(txn, event.id().0.as_slice(), &offset)?;
// Index by author and kind (with created_at and id)
self.akci.put(
txn,
&Self::key_akci(event.pubkey(), event.kind(), event.created_at(), event.id()),
&offset,
)?;
for mut tsi in event.tags()?.iter() {
if let Some(tagname) = tsi.next() {
// FIXME make sure it is a letter too
if tagname.len() == 1 {
if let Some(tagvalue) = tsi.next() {
// Index by author and tag (with created_at and id)
self.atci.put(
txn,
&Self::key_atci(
event.pubkey(),
tagname[0],
tagvalue,
event.created_at(),
event.id(),
),
&offset,
)?;
// Index by kind and tag (with created_at and id)
self.ktci.put(
txn,
&Self::key_ktci(
event.kind(),
tagname[0],
tagvalue,
event.created_at(),
event.id(),
),
&offset,
)?;
}
}
}
}
Ok(())
}
// Remove the event from all indexes (except the 'id' index)
fn deindex(&self, txn: &mut RwTxn<'_>, event: &Event) -> Result<(), Error> {
for mut tsi in event.tags()?.iter() {
if let Some(tagname) = tsi.next() {
// FIXME make sure it is a letter too
if tagname.len() == 1 {
if let Some(tagvalue) = tsi.next() {
// Index by author and tag (with created_at and id)
self.atci.delete(
txn,
&Self::key_atci(
event.pubkey(),
tagname[0],
tagvalue,
event.created_at(),
event.id(),
),
)?;
// Index by kind and tag (with created_at and id)
self.ktci.delete(
txn,
&Self::key_ktci(
event.kind(),
tagname[0],
tagvalue,
event.created_at(),
event.id(),
),
)?;
}
}
}
}
// Index by author and kind (with created_at and id)
self.akci.delete(
txn,
&Self::key_akci(event.pubkey(), event.kind(), event.created_at(), event.id()),
)?;
// We leave it in the id map. If someone wants to load the replaced event by id
// they can still do it.
// self.ids.delete(&mut txn, event.id().0.as_slice())?;
Ok(())
}
// Set an event as deleted
// This removes it from indexes (except the id index) and adds it to the deleted table
fn set_offset_as_deleted(&self, offset: usize) -> Result<(), Error> {
let mut txn = self.env.write_txn()?;
let offset_u64 = offset as u64;
// Check if it is already deleted
if self.deleted_offsets.get(&txn, &offset_u64)?.is_some() {
return Ok(());
}
// Add to deleted database in case we need to get at it in the future.
self.deleted_offsets.put(&mut txn, &offset_u64, &())?;
// Get event
let event = match self.events.get_event_by_offset(offset)? {
Some(event) => event,
None => return Ok(()),
};
// Remove from indexes
self.deindex(&mut txn, &event)?;
txn.commit()?;
Ok(())
}
// If the event is replaceable or parameterized replaceable
// this deletes all the events in that group except the most recent one.
fn delete_replaced(&self, txn: &mut RwTxn<'_>, event: &Event) -> Result<(), Error> {
if event.kind().is_replaceable() {
let start_prefix = Self::key_akci(
event.pubkey(),
event.kind(),
Time::max(), // database is ordered in reverse time
Id([0; 32]),
);
let end_prefix =
Self::key_akci(event.pubkey(), event.kind(), Time::min(), Id([255; 32]));
let range = (
Bound::Included(&*start_prefix),
Bound::Excluded(&*end_prefix),
);
let iter = self.akci.range(txn, &range)?;
let mut first = true;
for result in iter {
// Keep the first result
if first {
first = false;
continue;
}
let (_key, offset) = result?;
// Delete the event
self.set_offset_as_deleted(offset)?;
}
} else if event.kind().is_parameterized_replaceable() {
let tags = event.tags()?;
if let Some(identifier) = tags.get_value(b"d") {
let start_prefix =
Self::key_atci(event.pubkey(), b'd', identifier, Time::max(), Id([0; 32]));
let end_prefix =
Self::key_atci(event.pubkey(), b'd', identifier, Time::min(), Id([255; 32]));
let range = (
Bound::Included(&*start_prefix),
Bound::Excluded(&*end_prefix),
);
let iter = self.akci.range(txn, &range)?;
let mut first = true;
for result in iter {
// Keep the first result
if first {
first = false;
continue;
}
let (_key, offset) = result?;
// Delete the event
self.set_offset_as_deleted(offset)?;
}
}
}
Ok(())
}
// For looking up event by Author and Kind
// author(32) + kind(2) + reversecreatedat(8) + id(32)
#[allow(dead_code)]
fn key_akci(author: Pubkey, kind: Kind, created_at: Time, id: Id) -> Vec<u8> {
let mut key: Vec<u8> = Vec::with_capacity(
std::mem::size_of::<Pubkey>()
+ std::mem::size_of::<Kind>()
+ std::mem::size_of::<Time>()
+ std::mem::size_of::<Id>(),
);
key.extend(author.as_slice());
key.extend(kind.0.to_be_bytes());
key.extend((u64::MAX - created_at.0).to_be_bytes().as_slice());
key.extend(id.as_slice());
key
}
// For looking up event by Author and Tag
// author(32) + tagletter(1) + fixlentag(182) + reversecreatedat(8) + id(32)
#[allow(dead_code)]
fn key_atci(author: Pubkey, letter: u8, tag_value: &[u8], created_at: Time, id: Id) -> Vec<u8> {
const PADLEN: usize = 182;
let mut key: Vec<u8> = Vec::with_capacity(
std::mem::size_of::<Pubkey>()
+ PADLEN
+ std::mem::size_of::<Time>()
+ std::mem::size_of::<Id>(),
);
key.extend(author.as_slice());
key.push(letter);
if tag_value.len() <= PADLEN {
key.extend(tag_value);
key.extend(core::iter::repeat(0).take(PADLEN - tag_value.len()));
} else {
key.extend(&tag_value[..PADLEN]);
}
key.extend((u64::MAX - created_at.0).to_be_bytes().as_slice());
key.extend(id.as_slice());
key
}
// For looking up event by Kind and Tag
// kind(2) + tagletter(1) + fixlentag(182) + reversecreatedat(8) + id(32)
#[allow(dead_code)]
fn key_ktci(kind: Kind, letter: u8, tag_value: &[u8], created_at: Time, id: Id) -> Vec<u8> {
const PADLEN: usize = 182;
let mut key: Vec<u8> = Vec::with_capacity(
std::mem::size_of::<Kind>()
+ PADLEN
+ std::mem::size_of::<Time>()
+ std::mem::size_of::<Id>(),
);
key.extend(kind.0.to_be_bytes());
key.push(letter);
if tag_value.len() <= PADLEN {
key.extend(tag_value);
key.extend(core::iter::repeat(0).take(PADLEN - tag_value.len()));
} else {
key.extend(&tag_value[..PADLEN]);
}
key.extend((u64::MAX - created_at.0).to_be_bytes().as_slice());
key.extend(id.as_slice());
key
}
fn key_deleted_events(id: Id, pubkey: Pubkey) -> Vec<u8> {
let mut key: Vec<u8> =
Vec::with_capacity(std::mem::size_of::<Id>() + std::mem::size_of::<Pubkey>());
key.extend(id.as_slice());
key.extend(pubkey.as_slice());
key
}
}

85
src/tls.rs
View File

@ -1,26 +1,27 @@
use crate::config::Config;
use crate::error::{ChorusError, Error};
use rustls_pki_types::{CertificateDer, PrivateKeyDer};
use rustls::{Certificate, PrivateKey};
use std::fs::File;
use std::io::BufReader;
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
use tokio_rustls::{rustls, TlsAcceptor};
pub fn tls_acceptor(config: &Config) -> Result<TlsAcceptor, Error> {
let cert_file = File::open(&config.certchain_pem_path)?;
let mut certificates: Vec<CertificateDer<'static>> = Vec::new();
for maybe_cert in rustls_pemfile::certs(&mut BufReader::new(cert_file)) {
let cert = maybe_cert?;
certificates.push(cert);
}
let certs: Vec<Certificate> =
rustls_pemfile::certs(&mut BufReader::new(File::open(&config.certchain_pem_path)?))?
.drain(..)
.map(Certificate)
.collect();
let key_file = File::open(&config.key_pem_path)?;
let mut keys: Vec<PrivateKeyDer> = Vec::new();
for maybe_key in rustls_pemfile::pkcs8_private_keys(&mut BufReader::new(key_file)) {
let key = maybe_key?;
keys.push(PrivateKeyDer::Pkcs8(key));
}
keys.reverse();
let mut keys: Vec<PrivateKey> =
rustls_pemfile::pkcs8_private_keys(&mut BufReader::new(File::open(&config.key_pem_path)?))?
.drain(..)
.rev()
.map(PrivateKey)
.collect();
let key = match keys.pop() {
Some(k) => k,
@ -28,8 +29,62 @@ pub fn tls_acceptor(config: &Config) -> Result<TlsAcceptor, Error> {
};
let tls_config = rustls::ServerConfig::builder()
.with_safe_defaults()
.with_no_client_auth()
.with_single_cert(certificates, key)?;
.with_single_cert(certs, key)?;
Ok(TlsAcceptor::from(Arc::new(tls_config)))
}
/// A stream that might be protected with TLS.
#[allow(clippy::large_enum_variant)] // not great though
#[derive(Debug)]
pub enum MaybeTlsStream<S> {
/// Unencrypted socket stream.
Plain(S),
/// Encrypted socket stream using `rustls`.
Rustls(tokio_rustls::server::TlsStream<S>),
}
impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
fn poll_read(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &mut ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
match self.get_mut() {
MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_read(cx, buf),
MaybeTlsStream::Rustls(s) => Pin::new(s).poll_read(cx, buf),
}
}
}
impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
fn poll_write(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &[u8],
) -> Poll<Result<usize, std::io::Error>> {
match self.get_mut() {
MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_write(cx, buf),
MaybeTlsStream::Rustls(s) => Pin::new(s).poll_write(cx, buf),
}
}
fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), std::io::Error>> {
match self.get_mut() {
MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_flush(cx),
MaybeTlsStream::Rustls(s) => Pin::new(s).poll_flush(cx),
}
}
fn poll_shutdown(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Result<(), std::io::Error>> {
match self.get_mut() {
MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_shutdown(cx),
MaybeTlsStream::Rustls(s) => Pin::new(s).poll_shutdown(cx),
}
}
}

318
src/types/event/json_event.rs Normal file
View File

@ -0,0 +1,318 @@
use crate::error::{ChorusError, Error};
use crate::types::parse::json_parse::*;
/// Parses a JSON event from the `input` buffer. Places the parsed event into the `output` buffer.
/// Returns the count of consumed bytes and output bytes
pub fn parse_json_event(input: &[u8], output: &mut [u8]) -> Result<(usize, usize), Error> {
// Minimum-sized JSON event is 204 characters long
if input.len() < 204 {
return Err(ChorusError::JsonBadEvent("Too Short", 0).into());
}
// NOTE: 152 is the minimum binary event
if output.len() < 152 {
return Err(ChorusError::BufferTooSmall.into());
}
// This tracks where we are currently looking in the input as we scan forward.
// It is short for INput POSition.
let mut inpos = 0;
// If tags comes before content, content can use this to know where to put itself.
// This is the length of the tags output section. 0 means it hasn't been written yet.
let mut tags_size: usize = 0;
// If content comes before tags, we cannot write it because we don't know how much
// space Tags will take. So we instead just remember where the content string
// begins so we can write it later.
let mut content_input_start: usize = 0;
// Remember which fields we have read using bit flags.
// We must get all seven of these fields for an event to be valid.
const HAVE_ID: u8 = 0x1 << 0;
const HAVE_PUBKEY: u8 = 0x1 << 1;
const HAVE_SIG: u8 = 0x1 << 2;
const HAVE_CREATED_AT: u8 = 0x1 << 3;
const HAVE_KIND: u8 = 0x1 << 4;
const HAVE_CONTENT: u8 = 0x1 << 5;
const HAVE_TAGS: u8 = 0x1 << 6;
let mut complete: u8 = 0;
eat_whitespace(input, &mut inpos);
verify_char(input, b'{', &mut inpos)?;
loop {
eat_whitespace(input, &mut inpos);
// Presuming that we must have at least one field, we don't have to look
// for the end of the object yet.
// Move to the start of the field name
verify_char(input, b'"', &mut inpos)?;
// No matter which field is next, we need at least 7 bytes for the smallest
// field and value: kind":1
// This allows us to skip length tests below that are shorter than inpos+7
if inpos + 7 > input.len() {
return Err(ChorusError::JsonBadEvent("Too Short or Missing Fields", inpos).into());
}
if &input[inpos..inpos + 3] == b"id\"" {
if complete & HAVE_ID == HAVE_ID {
return Err(ChorusError::JsonBadEvent("Duplicate id field", inpos).into());
}
inpos += 3;
eat_colon_with_whitespace(input, &mut inpos)?;
read_id(input, &mut inpos, &mut output[16..48])?;
complete |= HAVE_ID;
} else if &input[inpos..inpos + 4] == b"sig\"" {
if complete & HAVE_SIG == HAVE_SIG {
return Err(ChorusError::JsonBadEvent("Duplicate sig field", inpos).into());
}
inpos += 4;
eat_colon_with_whitespace(input, &mut inpos)?;
read_sig(input, &mut inpos, output)?;
complete |= HAVE_SIG;
} else if &input[inpos..inpos + 5] == b"kind\"" {
if complete & HAVE_KIND == HAVE_KIND {
return Err(ChorusError::JsonBadEvent("Duplicate kind field", inpos).into());
}
inpos += 5;
eat_colon_with_whitespace(input, &mut inpos)?;
let kind = read_kind(input, &mut inpos)?;
output[4..6].copy_from_slice(kind.to_ne_bytes().as_slice());
complete |= HAVE_KIND;
} else if &input[inpos..inpos + 5] == b"tags\"" {
if complete & HAVE_TAGS == HAVE_TAGS {
return Err(ChorusError::JsonBadEvent("Duplicate tags field", inpos).into());
}
inpos += 5;
eat_colon_with_whitespace(input, &mut inpos)?;
tags_size = read_tags_array(input, &mut inpos, &mut output[144..])?;
complete |= HAVE_TAGS;
if content_input_start != 0 {
// Content was found earlier than tags.
// Now that tags have been read, we should read the content
read_content(input, &mut content_input_start, output, 144 + tags_size)?;
complete |= HAVE_CONTENT;
}
} else if &input[inpos..inpos + 7] == b"pubkey\"" {
if complete & HAVE_PUBKEY == HAVE_PUBKEY {
return Err(ChorusError::JsonBadEvent("Duplicate pubkey field", inpos).into());
}
inpos += 7;
eat_colon_with_whitespace(input, &mut inpos)?;
read_pubkey(input, &mut inpos, &mut output[48..80])?;
complete |= HAVE_PUBKEY;
} else if inpos + 8 <= input.len() && &input[inpos..inpos + 8] == b"content\"" {
if complete & HAVE_CONTENT == HAVE_CONTENT {
return Err(ChorusError::JsonBadEvent("Duplicate pubkey field", inpos).into());
}
inpos += 8;
eat_colon_with_whitespace(input, &mut inpos)?;
if tags_size == 0 {
// Oops, we haven't read the tags yet. That means we don't yet know where
// to place the content. In this case we just remember the offset where
// this needs to be done, so we can do this later.
content_input_start = inpos;
// skip past it so we can read the subsequent fields
verify_char(input, b'"', &mut inpos)?;
burn_string(input, &mut inpos)?;
} else {
read_content(input, &mut inpos, output, 144 + tags_size)?;
complete |= HAVE_CONTENT;
}
} else if inpos + 11 <= input.len() && &input[inpos..inpos + 11] == b"created_at\"" {
if complete & HAVE_CREATED_AT == HAVE_CREATED_AT {
return Err(ChorusError::JsonBadEvent("Duplicate created_at field", inpos).into());
}
inpos += 11;
eat_colon_with_whitespace(input, &mut inpos)?;
let u = read_u64(input, &mut inpos)?;
output[8..16].copy_from_slice(u.to_ne_bytes().as_slice());
complete |= HAVE_CREATED_AT;
} else {
burn_key_and_value(input, &mut inpos)?;
}
// get past the comma, or detect the close brace and exit
if next_object_field(input, &mut inpos)? {
break;
}
}
if complete == 0b0111_1111 {
Ok((
inpos,
u32::from_ne_bytes(output[0..4].try_into().unwrap()) as usize,
))
} else {
Err(ChorusError::JsonBadEvent("Missing Fields", inpos).into())
}
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn test_parse_json_event() {
if 256_u16.to_ne_bytes() == [1, 0] {
test_parse_json_event_big_endian();
} else {
test_parse_json_event_little_endian();
}
}
fn test_parse_json_event_little_endian() {
let json = br#"{"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"kind":1,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","content":"He got snowed in","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]]}"#;
let mut buffer: Vec<u8> = Vec::with_capacity(4096);
buffer.resize(4096, 0);
let (_insize, size) = parse_json_event(&json[..], &mut buffer).unwrap();
assert_eq!(size, 372);
assert_eq!(
&buffer[0..size],
&[
116, 1, 0, 0, // 372 bytes long
1, 0, // kind 1
0, 0, // padding
102, 232, 61, 100, 0, 0, 0, 0, // created at 1681778790
169, 102, 48, 85, 22, 74, 184, 179, 13, 149, 36, 101, 99, 112, 196, 191, 147, 57,
59, 176, 81, 183, 237, 244, 85, 111, 64, 197, 41, 141, 192, 199, // id
238, 17, 165, 223, 244, 12, 25, 165, 85, 244, 31, 228, 43, 72, 240, 14, 97, 140,
145, 34, 86, 34, 174, 55, 182, 194, 187, 103, 183, 108, 78, 73, // pubkey
77, 254, 161, 166, 247, 49, 65, 213, 105, 30, 67, 175, 195, 35, 77, 190, 115, 1,
109, 176, 251, 32, 124, 242, 71, 224, 18, 124, 194, 89, 30, 230, 180, 190, 91, 70,
34, 114, 3, 10, 155, 222, 117, 136, 42, 174, 129, 15, 53, 150, 130, 177, 182, 206,
108, 187, 151, 32, 17, 65, 197, 118, 219, 66, // sig
// 144:
208, 0, // tags section is 208 bytes long
3, 0, // there are three tags
10, 0, // first tag is at offset 10
28, 0, // second tag is at offset 28
99, 0, // third tag is at offset 99
// 154: (144+10)
2, 0, // the first tag has 2 strings
6, 0, // the first string is 6 bytes long
99, 108, 105, 101, 110, 116, // "client"
6, 0, // the second string is 6 bytes long
103, 111, 115, 115, 105, 112, // "gossip"
// 172: (144+28)
2, 0, // the second tag has two strings
1, 0, // the first string is 1 char long
112, // "p"
64, 0, // the second string is 64 bytes long
101, 50, 99, 99, 102, 55, 99, 102, 50, 48, 52, 48, 51, 102, 51, 102, 50, 97, 52,
97, 53, 53, 98, 51, 50, 56, 102, 48, 100, 101, 51, 98, 101, 51, 56, 53, 53, 56, 97,
55, 100, 53, 102, 51, 51, 54, 51, 50, 102, 100, 97, 97, 101, 102, 99, 55, 50, 54,
99, 49, 99, 56, 101,
98, // "e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
// 243: (144+99)
4, 0, // the third tag has 4 strings
1, 0, // the first string is 1 char long
101, // "e"
64, 0, // the second string is 64 bytes long
50, 99, 56, 54, 97, 98, 99, 99, 57, 56, 102, 55, 102, 100, 56, 97, 54, 55, 53, 48,
97, 97, 98, 56, 100, 102, 54, 99, 49, 56, 54, 51, 57, 48, 51, 102, 49, 48, 55, 50,
48, 54, 99, 99, 50, 100, 55, 50, 101, 56, 97, 102, 101, 98, 54, 99, 51, 56, 51, 53,
55, 97, 101,
100, // "2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"
30, 0, // the third string is 30 bytes long
119, 115, 115, 58, 47, 47, 110, 111, 115, 116, 114, 45, 112, 117, 98, 46, 119, 101,
108, 108, 111, 114, 100, 101, 114, 46, 110, 101, 116,
47, // "wss://nostr-pub.wellorder.net/"
4, 0, // the fourth string is 4 bytes long
114, 111, 111, 116, // "root"
// 352: (144+208)
16, 0, 0, 0, // the content is 16 bytes long
72, 101, 32, 103, 111, 116, 32, 115, 110, 111, 119, 101, 100, 32, 105,
110, // "He got snowed in"
// 372:
]
);
// Same event in a different order
let json2 = br#"{"kind":1,"pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]],"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","content":"He got snowed in"}"#;
let mut buffer2: Vec<u8> = Vec::with_capacity(4096);
buffer2.resize(4096, 0);
let (_insize, size) = parse_json_event(&json2[..], &mut buffer2).unwrap();
assert_eq!(size, 372);
assert_eq!(&buffer[..372], &buffer2[..372]);
}
fn test_parse_json_event_big_endian() {
let json = br#"{"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"kind":1,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","content":"He got snowed in","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]]}"#;
let mut buffer: Vec<u8> = Vec::with_capacity(4096);
buffer.resize(4096, 0);
let (_insize, size) = parse_json_event(&json[..], &mut buffer).unwrap();
assert_eq!(size, 372);
assert_eq!(
&buffer[0..size],
&[
0, 0, 1, 116, // 372 bytes long
0, 1, // kind 1
0, 0, // padding
0, 0, 0, 0, 100, 61, 232, 102, // created at 1681778790
169, 102, 48, 85, 22, 74, 184, 179, 13, 149, 36, 101, 99, 112, 196, 191, 147, 57,
59, 176, 81, 183, 237, 244, 85, 111, 64, 197, 41, 141, 192, 199, // id
238, 17, 165, 223, 244, 12, 25, 165, 85, 244, 31, 228, 43, 72, 240, 14, 97, 140,
145, 34, 86, 34, 174, 55, 182, 194, 187, 103, 183, 108, 78, 73, // pubkey
77, 254, 161, 166, 247, 49, 65, 213, 105, 30, 67, 175, 195, 35, 77, 190, 115, 1,
109, 176, 251, 32, 124, 242, 71, 224, 18, 124, 194, 89, 30, 230, 180, 190, 91, 70,
34, 114, 3, 10, 155, 222, 117, 136, 42, 174, 129, 15, 53, 150, 130, 177, 182, 206,
108, 187, 151, 32, 17, 65, 197, 118, 219, 66, // sig
// 144:
0, 208, // tags section is 208 bytes long
3, 0, // there are three tags
0, 10, // first tag is at offset 10
0, 28, // second tag is at offset 28
0, 99, // third tag is at offset 99
// 154: (144+10)
0, 2, // the first tag has 2 strings
0, 6, // the first string is 6 bytes long
99, 108, 105, 101, 110, 116, // "client"
0, 6, // the second string is 6 bytes long
103, 111, 115, 115, 105, 112, // "gossip"
// 172: (144+28)
0, 2, // the second tag has two strings
0, 1, // the first string is 1 char long
112, // "p"
0, 64, // the second string is 64 bytes long
101, 50, 99, 99, 102, 55, 99, 102, 50, 48, 52, 48, 51, 102, 51, 102, 50, 97, 52,
97, 53, 53, 98, 51, 50, 56, 102, 48, 100, 101, 51, 98, 101, 51, 56, 53, 53, 56, 97,
55, 100, 53, 102, 51, 51, 54, 51, 50, 102, 100, 97, 97, 101, 102, 99, 55, 50, 54,
99, 49, 99, 56, 101,
98, // "e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
// 243: (144+99)
0, 4, // the third tag has 4 strings
0, 1, // the first string is 1 char long
101, // "e"
0, 64, // the second string is 64 bytes long
50, 99, 56, 54, 97, 98, 99, 99, 57, 56, 102, 55, 102, 100, 56, 97, 54, 55, 53, 48,
97, 97, 98, 56, 100, 102, 54, 99, 49, 56, 54, 51, 57, 48, 51, 102, 49, 48, 55, 50,
48, 54, 99, 99, 50, 100, 55, 50, 101, 56, 97, 102, 101, 98, 54, 99, 51, 56, 51, 53,
55, 97, 101,
100, // "2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"
0, 30, // the third string is 30 bytes long
119, 115, 115, 58, 47, 47, 110, 111, 115, 116, 114, 45, 112, 117, 98, 46, 119, 101,
108, 108, 111, 114, 100, 101, 114, 46, 110, 101, 116,
47, // "wss://nostr-pub.wellorder.net/"
0, 4, // the fourth string is 4 bytes long
114, 111, 111, 116, // "root"
// 352: (144+208)
0, 0, 0, 16, // the content is 16 bytes long
72, 101, 32, 103, 111, 116, 32, 115, 110, 111, 119, 101, 100, 32, 105,
110, // "He got snowed in"
// 372:
]
);
// Same event in a different order
let json2 = br#"{"kind":1,"pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]],"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","content":"He got snowed in"}"#;
let mut buffer2: Vec<u8> = Vec::with_capacity(4096);
buffer2.resize(4096, 0);
let (_insize, size) = parse_json_event(&json2[..], &mut buffer2).unwrap();
assert_eq!(size, 372);
assert_eq!(&buffer[..372], &buffer2[..372]);
}
}

188
src/types/event/mod.rs Normal file
View File

@ -0,0 +1,188 @@
use super::{Id, Kind, Pubkey, Sig, Tags, Time};
use crate::error::{ChorusError, Error};
use crate::types::parse::json_escape::json_escape;
use std::fmt;
mod json_event;
use json_event::parse_json_event;
/*
* 0 [4 bytes] length of the event structure
* 4 [2 bytes] kind
* 6 [2 bytes] PADDING
* 8 [8 bytes] created_at
* 16 [32 bytes] id
* 48 [32 bytes] pubkey
* 80 [64 bytes] sig
* 144 [T bytes] Tags
* 144+T [4 bytes] content length
* 144+T+4 [C bytes] content
* 144+T+4+C <--- beginning of region beyond the event
*/
#[derive(Debug, Clone, PartialEq)]
pub struct Event<'a>(&'a [u8]);
impl<'a> Event<'a> {
// Parse json into an Event. Returns the count of consumed input bytes and the Event
pub fn from_json(
json: &[u8],
output_buffer: &'a mut [u8],
) -> Result<(usize, Event<'a>), Error> {
let (incount, outcount) = parse_json_event(json, output_buffer)?;
Ok((incount, Event(&output_buffer[..outcount])))
}
// this marks off the slice of bytes that represent an event from a potentially longer input
pub fn delineate(input: &'a [u8]) -> Result<Event<'a>, Error> {
if input.len() < 144 + 4 + 4 {
return Err(ChorusError::EndOfInput.into());
}
let len = parse_u32!(input, 0) as usize;
if input.len() < len {
return Err(ChorusError::EndOfInput.into());
}
Ok(Event(&input[0..len]))
}
// This copies
pub fn copy(&self, output: &mut [u8]) -> Result<(), Error> {
if output.len() < self.0.len() {
return Err(ChorusError::BufferTooSmall.into());
}
output[..self.0.len()].copy_from_slice(self.0);
Ok(())
}
// This copies, using the event_store mmap-append api
pub fn macopy(&self, output: &mut [u8]) -> Result<usize, std::io::Error> {
if output.len() < self.0.len() {
return Err(std::io::Error::other(ChorusError::BufferTooSmall));
}
output[..self.0.len()].copy_from_slice(self.0);
Ok(self.0.len())
}
pub fn as_bytes(&self) -> &[u8] {
self.0
}
pub fn length(&self) -> usize {
self.0.len()
}
pub fn kind(&self) -> Kind {
Kind(parse_u16!(self.0, 4))
}
pub fn created_at(&self) -> Time {
Time(parse_u64!(self.0, 8))
}
pub fn id(&self) -> Id {
Id(self.0[16..16 + 32].try_into().unwrap())
}
pub fn pubkey(&self) -> Pubkey {
Pubkey(self.0[48..48 + 32].try_into().unwrap())
}
pub fn sig(&self) -> Sig {
Sig(self.0[80..80 + 64].try_into().unwrap())
}
pub fn tags(&'a self) -> Result<Tags<'a>, Error> {
Tags::delineate(&self.0[144..])
}
pub fn content(&'a self) -> &'a [u8] {
let t = parse_u16!(self.0, 144) as usize;
let c = parse_u32!(self.0, 144 + t) as usize;
&self.0[144 + t + 4..144 + t + 4 + c]
}
pub fn as_json(&self) -> Result<Vec<u8>, Error> {
let mut output: Vec<u8> = Vec::with_capacity(256);
output.extend(br#"{"id":""#);
let pos = output.len();
output.resize(pos + 64, 0);
self.id().write_hex(&mut output[pos..]).unwrap();
output.extend(br#"","pubkey":""#);
let pos = output.len();
output.resize(pos + 64, 0);
self.pubkey().write_hex(&mut output[pos..]).unwrap();
output.extend(br#"","kind":"#);
output.extend(format!("{}", self.kind().0).as_bytes());
output.extend(br#","created_at":"#);
output.extend(format!("{}", self.created_at().0).as_bytes());
output.extend(br#","tags":"#);
output.extend(self.tags()?.as_json());
output.extend(br#","content":""#);
// This is okay if it is not accurate. It generally avoids
// lots of little mallocs when the capacity is already allocated
output.reserve(self.content().len() * 7 / 6);
let mut output = json_escape(self.content(), output)?;
output.extend(br#"","sig":""#);
let pos = output.len();
output.resize(pos + 128, 0);
self.sig().write_hex(&mut output[pos..]).unwrap();
output.extend(br#""}"#);
Ok(output)
}
pub fn verify(&self) -> Result<(), Error> {
use secp256k1::hashes::{sha256, Hash};
use secp256k1::schnorr::Signature;
use secp256k1::{Message, XOnlyPublicKey};
// This is okay if it is not accurate. It generally avoids
// lots of little mallocs when the capacity is already allocated
let escaped_content = Vec::with_capacity(self.content().len() * 7 / 6);
let escaped_content = json_escape(self.content(), escaped_content)?;
let signable = format!(
r#"[0,"{}",{},{},{},"{}"]"#,
self.pubkey(),
self.created_at(),
self.kind(),
self.tags()?,
unsafe { std::str::from_utf8_unchecked(&escaped_content[..]) },
);
drop(escaped_content);
let hash = sha256::Hash::hash(signable.as_bytes());
let hashref = <sha256::Hash as AsRef<[u8]>>::as_ref(&hash);
if hashref != self.id().as_slice() {
return Err(ChorusError::BadEventId.into());
}
let pubkey = XOnlyPublicKey::from_slice(self.pubkey().as_slice())?;
let sig = Signature::from_slice(self.sig().as_slice())?;
let message = Message::from_digest_slice(hashref)?;
sig.verify(&message, &pubkey)?;
Ok(())
}
}
impl fmt::Display for Event<'_> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
if let Ok(bytes) = self.as_json() {
let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
write!(f, "{s}")
} else {
write!(f, "{{Corrupted Event}}")
}
}
}
#[derive(Debug, Clone)]
pub struct OwnedEvent(pub Vec<u8>);
impl OwnedEvent {
pub fn as_event(&self) -> Result<Event<'_>, Error> {
Event::delineate(&self.0)
}
}

540
src/types/filter/json_filter.rs Normal file
View File

@ -0,0 +1,540 @@
use crate::error::{ChorusError, Error};
use crate::types::parse::json_escape::json_unescape;
use crate::types::parse::json_parse::*;
use crate::types::parse::put;
/// Parses a JSON filter from the `input` buffer. Places the parsed filter into the `output` buffer.
/// Returns the count of consumed bytes and output bytes
pub fn parse_json_filter(input: &[u8], output: &mut [u8]) -> Result<(usize, usize), Error> {
if input.len() < 2 {
return Err(ChorusError::JsonBadFilter("Too short", 0).into());
}
// This tracks where we are currently looking in the input as we scan forward.
// It is short for INput POSition.
let mut inpos = 0;
// Remember which fields we have read using bit flags.
// We should only get at maximum one of each
const HAVE_IDS: u8 = 0x1 << 0;
const HAVE_AUTHORS: u8 = 0x1 << 1;
const HAVE_KINDS: u8 = 0x1 << 2;
const HAVE_LIMIT: u8 = 0x1 << 3;
const HAVE_SINCE: u8 = 0x1 << 4;
const HAVE_UNTIL: u8 = 0x1 << 5;
let mut found: u8 = 0;
// Remember which tags we have seen
// We track A-Z in the lower 26 bits, and a-z in the next 26 bits
let mut found_tags: u64 = 0;
let letter_to_tag_bit = |letter: u8| -> Option<u64> {
match letter {
65..=90 => Some(letter as u64 - 65),
97..=122 => Some(letter as u64 - 97 + 26),
_ => None,
}
};
// Start structure with that of an empty filter
put(
output,
0,
&[
0, 0, // length (we will fill it in later)
0, 0, // 0 ids
0, 0, // 0 authors
0, 0, // 0 kinds
255, 255, 255, 255, // max limit
0, 0, 0, 0, // padding
0, 0, 0, 0, 0, 0, 0, 0, // since 1970
255, 255, 255, 255, 255, 255, 255, 255, // until max unixtime
],
)?;
let mut end: usize = 32;
// We just store the position of ids, authors, kinds, and tags
// and come back to parse them properly again at the end,
// since we need to write them in a particular order.
let mut start_ids: Option<usize> = None;
let mut start_authors: Option<usize> = None;
let mut start_kinds: Option<usize> = None;
// Allowing up to 32 tag filter fields (plenty!)
// (we are not differentiating letters yet, just collecting offsets)
// (we make the array to avoid allocation)
let mut num_tag_fields = 0;
let mut start_tags: [usize; 32] = [usize::MAX; 32];
eat_whitespace(input, &mut inpos);
verify_char(input, b'{', &mut inpos)?;
loop {
eat_whitespace_and_commas(input, &mut inpos);
// Check for end
if input[inpos] == b'}' {
inpos += 1;
break;
}
verify_char(input, b'"', &mut inpos)?;
if inpos + 4 <= input.len() && &input[inpos..inpos + 4] == b"ids\"" {
// Check for duplicate
if found & HAVE_IDS == HAVE_IDS {
return Err(ChorusError::JsonBadFilter("Duplicate id field", inpos).into());
}
inpos += 4;
eat_colon_with_whitespace(input, &mut inpos)?;
verify_char(input, b'[', &mut inpos)?;
// Record for later
start_ids = Some(inpos);
// Burn the field
while inpos < input.len() && input[inpos] != b']' {
inpos += 1;
}
verify_char(input, b']', &mut inpos)?;
// Mark as found 'ids' FIXME this dups `start_ids`
found |= HAVE_IDS;
} else if inpos + 8 <= input.len() && &input[inpos..inpos + 8] == b"authors\"" {
// Check for duplicate
if found & HAVE_AUTHORS == HAVE_AUTHORS {
return Err(ChorusError::JsonBadFilter("Duplicate authors field", inpos).into());
}
inpos += 8;
eat_colon_with_whitespace(input, &mut inpos)?;
verify_char(input, b'[', &mut inpos)?;
// Save the input offset for post-processing
start_authors = Some(inpos);
// Burn the field
while inpos < input.len() && input[inpos] != b']' {
inpos += 1;
}
verify_char(input, b']', &mut inpos)?;
found |= HAVE_AUTHORS;
} else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"kinds\"" {
// Check for duplicate
if found & HAVE_KINDS == HAVE_KINDS {
return Err(ChorusError::JsonBadFilter("Duplicate kinds field", inpos).into());
}
inpos += 6;
eat_colon_with_whitespace(input, &mut inpos)?;
verify_char(input, b'[', &mut inpos)?;
// Mark this position and bypass this field
start_kinds = Some(inpos);
// Burn the field
while inpos < input.len() && input[inpos] != b']' {
inpos += 1;
}
verify_char(input, b']', &mut inpos)?;
found |= HAVE_KINDS;
} else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"since\"" {
// Check for duplicate
if found & HAVE_SINCE == HAVE_SINCE {
return Err(ChorusError::JsonBadFilter("Duplicate since field", inpos).into());
}
inpos += 6;
eat_colon_with_whitespace(input, &mut inpos)?;
let since = read_u64(input, &mut inpos)?;
put(output, 16, since.to_ne_bytes().as_slice())?;
found |= HAVE_SINCE;
} else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"until\"" {
// Check for duplicate
if found & HAVE_UNTIL == HAVE_UNTIL {
return Err(ChorusError::JsonBadFilter("Duplicate until field", inpos).into());
}
inpos += 6;
eat_colon_with_whitespace(input, &mut inpos)?;
let until = read_u64(input, &mut inpos)?;
put(output, 24, until.to_ne_bytes().as_slice())?;
found |= HAVE_UNTIL;
} else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"limit\"" {
// Check for duplicate
if found & HAVE_LIMIT == HAVE_LIMIT {
return Err(ChorusError::JsonBadFilter("Duplicate limit field", inpos).into());
}
inpos += 6;
eat_colon_with_whitespace(input, &mut inpos)?;
let limit = read_u64(input, &mut inpos)?;
let limit: u32 = limit as u32;
put(output, 8, limit.to_ne_bytes().as_slice())?;
found |= HAVE_LIMIT;
} else if inpos + 3 <= input.len()
&& input[inpos] == b'#'
&& ((input[inpos + 1] >= 65 && input[inpos + 1] <= 90)
|| (input[inpos + 1] >= 97 && input[inpos + 1] <= 122))
&& input[inpos + 2] == b'"'
{
inpos += 1; // pass the hash
// Mark this position (on the letter itself)
start_tags[num_tag_fields] = inpos;
num_tag_fields += 1;
let letter = input[inpos];
inpos += 2; // pass the letter and quote
// Remember we found this tag in the `found_tags` bitfield
if let Some(bit) = letter_to_tag_bit(letter) {
if found_tags & bit == bit {
return Err(ChorusError::JsonBadFilter("Duplicate tag", inpos).into());
}
found_tags |= bit;
}
// Burn the rest
eat_colon_with_whitespace(input, &mut inpos)?;
verify_char(input, b'[', &mut inpos)?;
burn_array(input, &mut inpos)?;
} else {
burn_key_and_value(input, &mut inpos)?;
}
}
// Copy ids
if let Some(mut inpos) = start_ids {
let mut num_ids: u16 = 0;
// `inpos` is right after the open bracket of the array
loop {
eat_whitespace_and_commas(input, &mut inpos);
if input[inpos] == b']' {
break;
}
read_id(input, &mut inpos, &mut output[end..])?;
num_ids += 1;
end += 32;
}
// Write num_ids
put(output, 2, num_ids.to_ne_bytes().as_slice())?;
}
// Copy authors
if let Some(mut inpos) = start_authors {
let mut num_authors: u16 = 0;
// `inpos` is right after the open bracket of the array
loop {
eat_whitespace_and_commas(input, &mut inpos);
if input[inpos] == b']' {
break;
}
read_pubkey(input, &mut inpos, &mut output[end..])?;
num_authors += 1;
end += 32;
}
// write num_authors
put(output, 4, num_authors.to_ne_bytes().as_slice())?;
}
// Copy kinds
if let Some(mut inpos) = start_kinds {
let mut num_kinds: u16 = 0;
// `inpos` is right after the open bracket of the array
loop {
eat_whitespace_and_commas(input, &mut inpos);
if input[inpos] == b']' {
break;
}
let u = read_u64(input, &mut inpos)?;
if u > 65535 {
return Err(
ChorusError::JsonBadFilter("Filter has kind number too large", inpos).into(),
);
}
put(output, end, (u as u16).to_ne_bytes().as_slice())?;
num_kinds += 1;
end += 2;
}
// write num_kinds
put(output, 6, num_kinds.to_ne_bytes().as_slice())?;
}
// Copy tags
{
let write_tags_start = end;
// write number of tags
put(
output,
write_tags_start + 2,
(num_tag_fields as u16).to_ne_bytes().as_slice(),
)?;
// bump end past offset fields
end += 4 + 2 * num_tag_fields;
// Now pull in each tag
#[allow(clippy::needless_range_loop)]
for w in 0..num_tag_fields {
// Write it's offset
put(
output,
write_tags_start + 4 + (2 * w),
((end - write_tags_start) as u16).to_ne_bytes().as_slice(),
)?;
let mut inpos = start_tags[w];
let letter = input[inpos];
// bump past count output and write letter
let countindex = end;
end += 2;
put(output, end, 1_u16.to_ne_bytes().as_slice())?;
if output.len() < end + 2 {
return Err(crate::error::ChorusError::BufferTooSmall.into());
}
output[end + 2] = letter;
// bump past what we just wrote
end += 3;
// scan further in input
inpos += 1; // move off letter
verify_char(input, b'"', &mut inpos)?;
eat_colon_with_whitespace(input, &mut inpos)?;
verify_char(input, b'[', &mut inpos)?;
let mut count: u16 = 1; // the tag letter itself counts
loop {
eat_whitespace_and_commas(input, &mut inpos);
if input[inpos] == b']' {
break;
}
verify_char(input, b'"', &mut inpos)?;
// copy data
let (inlen, outlen) = json_unescape(&input[inpos..], &mut output[end + 2..])?;
// write len
put(output, end, (outlen as u16).to_ne_bytes().as_slice())?;
end += 2 + outlen;
inpos += inlen + 1;
count += 1;
}
// write count
put(output, countindex, count.to_ne_bytes().as_slice())?;
}
// write length of tags section
put(
output,
write_tags_start,
((end - write_tags_start) as u16).to_ne_bytes().as_slice(),
)?;
}
if end > 65535 {
return Err(ChorusError::JsonBadFilter("Filter is too long", end).into());
}
// Write length of filter
put(output, 0, (end as u16).to_ne_bytes().as_slice())?;
Ok((inpos, end))
}
#[cfg(test)]
mod test {
use super::*;
use crate::types::{Filter, Kind, Pubkey, Tags, TagsIter, TagsStringIter, Time};
#[test]
fn test_parse_json_empty_filter() {
let json = br##"{}"##;
let mut buffer: Vec<u8> = Vec::with_capacity(4096);
buffer.resize(4096, 0);
let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
assert_eq!(consumed, 2);
assert_eq!(size, 36);
assert_eq!(
&buffer[0..size],
&[
36, 0, // length
0, 0, // 0 ids
0, 0, // 0 authors
0, 0, // 0 kinds
255, 255, 255, 255, // max limit
0, 0, 0, 0, // padding
0, 0, 0, 0, 0, 0, 0, 0, // since 1970
255, 255, 255, 255, 255, 255, 255, 255, // until max unixtime
4, 0, 0, 0, // empty tags section
]
);
}
#[test]
fn test_parse_json_filter1() {
let json = br##"{"kinds":[1,30023],"since":1681778790,"authors":["e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"],"until":1704238196,"ids" : [ "7089afc2e77f366bc0fd1662e4048f59f18391c04a35957f21bbd1f3e6a492c4"],"limit":10}"##;
// ,"#e":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7"}"##;
let mut buffer: Vec<u8> = Vec::with_capacity(4096);
buffer.resize(4096, 0);
let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
assert_eq!(consumed, json.len());
assert_eq!(size, 136);
assert_eq!(
&buffer[0..size],
&[
136, 0, // length
1, 0, // 0 ids
2, 0, // 0 authors
2, 0, // 0 kinds
10, 0, 0, 0, // max limit 10
0, 0, 0, 0, // padding
102, 232, 61, 100, 0, 0, 0, 0, // since 1681778790
116, 156, 148, 101, 0, 0, 0, 0, // until 1704238196
// First ID:
112, 137, 175, 194, 231, 127, 54, 107, 192, 253, 22, 98, 228, 4, 143, 89, 241, 131,
145, 192, 74, 53, 149, 127, 33, 187, 209, 243, 230, 164, 146, 196,
// First author:
226, 204, 247, 207, 32, 64, 63, 63, 42, 74, 85, 179, 40, 240, 222, 59, 227, 133, 88,
167, 213, 243, 54, 50, 253, 170, 239, 199, 38, 193, 200, 235,
// Second author:
44, 134, 171, 204, 152, 247, 253, 138, 103, 80, 170, 184, 223, 108, 24, 99, 144, 63,
16, 114, 6, 204, 45, 114, 232, 175, 235, 108, 56, 53, 122, 237, // Kinds,
1, 0, // 1
71, 117, // 30023
4, 0, 0, 0, // empty tags section
]
);
}
#[test]
fn test_parse_json_filter2() {
let json = br##"{"kinds":[1,30023],"since":1681778790,"authors":["e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"],"until":1704238196,"ids" : [ "7089afc2e77f366bc0fd1662e4048f59f18391c04a35957f21bbd1f3e6a492c4"],"limit":10, "#e":["a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7"]}"##;
let mut buffer: Vec<u8> = Vec::with_capacity(4096);
buffer.resize(4096, 0);
let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
assert_eq!(consumed, json.len());
assert_eq!(size, 209);
assert_eq!(
&buffer[0..size],
&[
209, 0, // length
1, 0, // 0 ids
2, 0, // 0 authors
2, 0, // 0 kinds
10, 0, 0, 0, // max limit 10
0, 0, 0, 0, // padding
102, 232, 61, 100, 0, 0, 0, 0, // since 1681778790
116, 156, 148, 101, 0, 0, 0, 0, // until 1704238196
// First ID:
112, 137, 175, 194, 231, 127, 54, 107, 192, 253, 22, 98, 228, 4, 143, 89, 241, 131,
145, 192, 74, 53, 149, 127, 33, 187, 209, 243, 230, 164, 146, 196,
// First author:
226, 204, 247, 207, 32, 64, 63, 63, 42, 74, 85, 179, 40, 240, 222, 59, 227, 133, 88,
167, 213, 243, 54, 50, 253, 170, 239, 199, 38, 193, 200, 235,
// Second author:
44, 134, 171, 204, 152, 247, 253, 138, 103, 80, 170, 184, 223, 108, 24, 99, 144, 63,
16, 114, 6, 204, 45, 114, 232, 175, 235, 108, 56, 53, 122, 237, // Kinds,
1, 0, // 1
71, 117, // 30023
// Tag section:
77, 0, // tags section length is 77
1, 0, // just one tag
6, 0, // offset of 0th tag is 6
// First tag:
2, 0, // 2 fields
// Field 1:
1, 0, // 1 byte long
101, // 'e'
// Field 2:
64, 0, // 64 bytes long
97, 57, 54, 54, 51, 48, 53, 53, 49, 54, 52, 97, 98, 56, 98, 51, 48, 100, 57, 53,
50, 52, 54, 53, 54, 51, 55, 48, 99, 52, 98, 102, 57, 51, 51, 57, 51, 98, 98, 48,
53, 49, 98, 55, 101, 100, 102, 52, 53, 53, 54, 102, 52, 48, 99, 53, 50, 57, 56,
100, 99, 48, 99, 55
]
);
}
#[test]
fn test_filter_parse_and_check() {
let json = br##"{"kinds":[1,5,9,30023],"since":1681778790,"authors":["e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"], "#e":["a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"],"#p":["2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"]}"##;
let mut buffer: Vec<u8> = Vec::with_capacity(4096);
buffer.resize(4096, 0);
let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
assert_eq!(consumed, json.len());
assert_eq!(size, 452);
let filter = Filter::delineate(&buffer).unwrap();
assert_eq!(filter.len(), 452);
assert_eq!(filter.num_ids(), 0);
assert_eq!(filter.num_authors(), 2);
let mut author_iter = filter.authors();
assert_eq!(
author_iter.next(),
Some(
Pubkey::read_hex(
b"e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
)
.unwrap()
)
);
assert_eq!(
author_iter.next(),
Some(
Pubkey::read_hex(
b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"
)
.unwrap()
)
);
assert_eq!(author_iter.next(), None);
assert_eq!(filter.num_kinds(), 4);
let mut kind_iter = filter.kinds();
assert_eq!(kind_iter.next(), Some(Kind(1)));
assert_eq!(kind_iter.next(), Some(Kind(5)));
assert_eq!(kind_iter.next(), Some(Kind(9)));
assert_eq!(kind_iter.next(), Some(Kind(30023)));
assert_eq!(kind_iter.next(), None);
assert_eq!(filter.limit(), u32::MAX);
assert_eq!(filter.since(), Time(1681778790));
assert_eq!(filter.until(), Time::max());
let tags: Tags = filter.tags().unwrap();
let mut tag_iter: TagsIter = tags.iter();
let mut tag1_iter: TagsStringIter = tag_iter.next().unwrap();
assert_eq!(tag1_iter.next(), Some(b"e".as_slice()));
assert_eq!(
tag1_iter.next(),
Some(b"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7".as_slice())
);
assert_eq!(
tag1_iter.next(),
Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
);
assert_eq!(tag1_iter.next(), None);
let mut tag2_iter = tag_iter.next().unwrap();
assert_eq!(tag2_iter.next(), Some(b"p".as_slice()));
assert_eq!(
tag2_iter.next(),
Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
);
assert_eq!(
tag2_iter.next(),
Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
);
assert_eq!(
tag2_iter.next(),
Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
);
assert_eq!(tag2_iter.next(), None);
assert!(tag_iter.next().is_none());
}
}

537
src/types/filter/mod.rs Normal file
View File

@ -0,0 +1,537 @@
use super::{Event, Id, Kind, Pubkey, Tags, Time};
use crate::error::{ChorusError, Error};
use std::fmt;
mod json_filter;
use json_filter::parse_json_filter;
/*
* 0 [2 bytes] length of entire structure
* 2 [2 bytes] num_ids
* 4 [2 bytes] num_authors
* 6 [2 bytes] num_kinds
* 8 [4 bytes] limit u32. Set to u32::max if limit was not set.
* 12 [4 bytes] PADDING
* 16 [8 bytes] since u64. Set to 0 if since was not set.
* 24 [8 bytes] until u64. Set to u64::max if until was not set.
* 32 [ID] array
* [Pubkey] array starts at 32 + num_ids*32
* [Kind] array starts at 32 + num_ids*32 + num_authors*32
* [Tags] object starts at 32 + num_ids*32 + num_authors*32 * num_kinds*2
*/
const NUM_IDS_OFFSET: usize = 2;
const NUM_AUTHORS_OFFSET: usize = 4;
const NUM_KINDS_OFFSET: usize = 6;
const LIMIT_OFFSET: usize = 8;
const SINCE_OFFSET: usize = 16;
const UNTIL_OFFSET: usize = 24;
const ARRAYS_OFFSET: usize = 32;
const ID_SIZE: usize = 32;
const PUBKEY_SIZE: usize = 32;
const KIND_SIZE: usize = 2;
#[derive(Debug, Clone)]
pub struct Filter<'a>(&'a [u8]);
impl<'a> Filter<'a> {
// Parse json into a Filter. Returns the count of consumed input bytes and output
// bytes the Filter
pub fn from_json(
json: &[u8],
output_buffer: &'a mut [u8],
) -> Result<(usize, usize, Filter<'a>), Error> {
let (incount, outcount) = parse_json_filter(json, output_buffer)?;
Ok((incount, outcount, Filter(&output_buffer[..outcount])))
}
pub fn delineate(input: &'a [u8]) -> Result<Filter<'a>, Error> {
if input.len() < ARRAYS_OFFSET {
return Err(ChorusError::EndOfInput.into());
}
let len = parse_u16!(input, 0) as usize;
if input.len() < len {
return Err(ChorusError::EndOfInput.into());
}
Ok(Filter(&input[0..len]))
}
pub fn copy(&self, output: &mut [u8]) -> Result<(), Error> {
if output.len() < self.0.len() {
return Err(ChorusError::EndOfInput.into());
}
output[..self.0.len()].copy_from_slice(self.0);
Ok(())
}
#[inline]
pub fn as_bytes(&self) -> &[u8] {
self.0
}
#[inline]
pub fn len(&self) -> usize {
self.0.len()
}
#[inline]
pub fn is_empty(&self) -> bool {
self.0.is_empty()
}
#[inline]
pub fn num_ids(&self) -> usize {
parse_u16!(self.0, NUM_IDS_OFFSET) as usize
}
#[inline]
pub fn ids(&'a self) -> FilterIdIter<'a> {
FilterIdIter {
filter: self,
next: 0,
}
}
#[inline]
pub fn num_authors(&self) -> usize {
parse_u16!(self.0, NUM_AUTHORS_OFFSET) as usize
}
#[inline]
fn start_of_authors(&self) -> usize {
ARRAYS_OFFSET + self.num_ids() * ID_SIZE
}
#[inline]
pub fn authors(&'a self) -> FilterAuthorIter<'a> {
FilterAuthorIter {
filter: self,
start_of_authors: self.start_of_authors(),
next: 0,
}
}
#[inline]
pub fn num_kinds(&self) -> usize {
parse_u16!(self.0, NUM_KINDS_OFFSET) as usize
}
#[inline]
fn start_of_kinds(&self) -> usize {
ARRAYS_OFFSET + self.num_ids() * ID_SIZE + self.num_authors() * PUBKEY_SIZE
}
#[inline]
pub fn kinds(&'a self) -> FilterKindIter<'a> {
FilterKindIter {
filter: self,
start_of_kinds: self.start_of_kinds(),
next: 0,
}
}
#[inline]
fn start_of_tags(&self) -> usize {
ARRAYS_OFFSET
+ self.num_ids() * ID_SIZE
+ self.num_authors() * PUBKEY_SIZE
+ self.num_kinds() * KIND_SIZE
}
#[inline]
pub fn tags(&'a self) -> Result<Tags<'a>, Error> {
Tags::delineate(&self.0[self.start_of_tags()..])
}
#[inline]
pub fn limit(&self) -> u32 {
parse_u32!(self.0, LIMIT_OFFSET)
}
#[inline]
pub fn since(&self) -> Time {
Time(parse_u64!(self.0, SINCE_OFFSET))
}
#[inline]
pub fn until(&self) -> Time {
Time(parse_u64!(self.0, UNTIL_OFFSET))
}
pub fn event_matches(&self, event: &Event) -> Result<bool, Error> {
// ids
if self.num_ids() != 0 && !self.ids().any(|id| id == event.id()) {
return Ok(false);
}
// authors
if self.num_authors() != 0 && !self.authors().any(|pk| pk == event.pubkey()) {
return Ok(false);
}
// kinds
if self.num_kinds() != 0 && !self.kinds().any(|kind| kind == event.kind()) {
return Ok(false);
}
// since
if event.created_at() < self.since() {
return Ok(false);
}
// until
if event.created_at() > self.until() {
return Ok(false);
}
// tags
let filter_tags = self.tags()?;
if !filter_tags.is_empty() {
let event_tags = event.tags()?;
if event_tags.is_empty() {
return Ok(false);
}
let mut i = 0;
while let Some(letter) = filter_tags.get_string(i, 0) {
let mut j = 1;
let mut found = false;
while let Some(value) = filter_tags.get_string(i, j) {
if event_tags.matches(letter, value) {
found = true;
break;
}
j += 1;
}
if !found {
return Ok(false);
}
i += 1;
}
}
Ok(true)
}
pub fn as_json(&self) -> Result<Vec<u8>, Error> {
let mut output: Vec<u8> = Vec::with_capacity(256);
output.push(b'{');
let mut first = true;
if self.num_ids() > 0 {
output.extend(br#""ids":["#);
for (i, id) in self.ids().enumerate() {
if i > 0 {
output.push(b',');
}
output.push(b'"');
let pos = output.len();
output.resize(pos + 64, 0);
id.write_hex(&mut output[pos..])?;
output.push(b'"');
}
output.push(b']');
first = false;
}
if self.num_authors() > 0 {
if !first {
output.push(b',');
}
output.extend(br#""authors":["#);
for (i, pk) in self.authors().enumerate() {
if i > 0 {
output.push(b',');
}
output.push(b'"');
let pos = output.len();
output.resize(pos + 64, 0);
pk.write_hex(&mut output[pos..])?;
output.push(b'"');
}
output.push(b']');
first = false;
}
if self.num_kinds() > 0 {
if !first {
output.push(b',');
}
output.extend(br#""kinds":["#);
for (i, k) in self.kinds().enumerate() {
if i > 0 {
output.push(b',');
}
output.extend(format!("{}", k.0).as_bytes());
}
output.push(b']');
first = false;
}
let tags = self.tags()?;
if !tags.is_empty() {
// Filter 'tags' are not an array of arrays, they are just a convenient
// way to store similar data. They also elide the '#'. So we have to
// iterate here, we cannot use tags.as_json()
for tag in tags.iter() {
if !first {
output.push(b',');
}
for (i, bytes) in tag.enumerate() {
if i == 0 {
output.extend(b"\"#");
output.extend(bytes);
output.extend(b"\":[");
} else {
if i > 1 {
output.push(b',');
}
output.push(b'"');
output.extend(bytes);
output.push(b'"');
}
}
output.push(b']');
first = false;
}
}
if self.limit() != u32::MAX {
if !first {
output.push(b',');
}
output.extend(format!(r#""limit":{}"#, self.limit()).as_bytes());
first = false;
}
if self.since() != Time::min() {
if !first {
output.push(b',');
}
output.extend(format!(r#""since":{}"#, self.since().0).as_bytes());
first = false;
}
if self.until() != Time::max() {
if !first {
output.push(b',');
}
output.extend(format!(r#""until":{}"#, self.until().0).as_bytes());
}
output.push(b'}');
Ok(output)
}
}
impl fmt::Display for Filter<'_> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
if let Ok(bytes) = self.as_json() {
let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
write!(f, "{s}")
} else {
write!(f, "{{Corrupted Event}}")
}
}
}
#[derive(Debug)]
pub struct FilterIdIter<'a> {
filter: &'a Filter<'a>,
next: usize,
}
impl<'a> Iterator for FilterIdIter<'a> {
type Item = Id;
fn next(&mut self) -> Option<Self::Item> {
let num_ids = parse_u16!(self.filter.0, NUM_IDS_OFFSET) as usize;
if self.next >= num_ids {
None
} else {
let offset = ARRAYS_OFFSET + self.next * ID_SIZE;
self.next += 1;
if self.filter.0.len() < offset + ID_SIZE {
None
} else {
Some(Id(self.filter.0[offset..offset + ID_SIZE]
.try_into()
.unwrap()))
}
}
}
}
#[derive(Debug)]
pub struct FilterAuthorIter<'a> {
filter: &'a Filter<'a>,
start_of_authors: usize,
next: usize,
}
impl<'a> Iterator for FilterAuthorIter<'a> {
type Item = Pubkey;
fn next(&mut self) -> Option<Self::Item> {
let num_authors = parse_u16!(self.filter.0, NUM_AUTHORS_OFFSET) as usize;
if self.next >= num_authors {
None
} else {
let offset = self.start_of_authors + self.next * PUBKEY_SIZE;
self.next += 1;
if self.filter.0.len() < offset + PUBKEY_SIZE {
None
} else {
Some(Pubkey(
self.filter.0[offset..offset + PUBKEY_SIZE]
.try_into()
.unwrap(),
))
}
}
}
}
#[derive(Debug)]
pub struct FilterKindIter<'a> {
filter: &'a Filter<'a>,
start_of_kinds: usize,
next: usize,
}
impl<'a> Iterator for FilterKindIter<'a> {
type Item = Kind;
fn next(&mut self) -> Option<Self::Item> {
let num_kinds = parse_u16!(self.filter.0, NUM_KINDS_OFFSET) as usize;
if self.next >= num_kinds {
None
} else {
let offset = self.start_of_kinds + self.next * KIND_SIZE;
self.next += 1;
if self.filter.0.len() < offset + KIND_SIZE {
None
} else {
Some(Kind(parse_u16!(self.filter.0, offset)))
}
}
}
}
#[derive(Debug, Clone)]
pub struct OwnedFilter(pub Vec<u8>);
impl OwnedFilter {
pub fn as_filter(&self) -> Result<Filter<'_>, Error> {
Filter::delineate(&self.0)
}
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn test_filter() {
/*
* {
* "ids": [ "6b43bc2e373b6d9330ff571f3f4e6d897b32d01d65227df3fa41cdf731c63c3a",
* "1f47034c9d6d0539382a86ba31766f00f2b8312ab167c036729422ec9e7085e8"],
* "authors": [ "52b4a076bcbbbdc3a1aefa3735816cf74993b1b8db202b01c883c58be7fad8bd",
* "ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49" ],
* "kinds": [ 1, 5, 30023 ],
* "since": 1702161345,
* "#p": [ "fa984bd7dbb282f07e16e7ae87b26a2a7b9b90b7246a44771f0cf5ae58018f52" ],
* }
*/
// For comparison
let id1 = Id::read_hex(b"6b43bc2e373b6d9330ff571f3f4e6d897b32d01d65227df3fa41cdf731c63c3a")
.unwrap();
let id2 = Id::read_hex(b"1f47034c9d6d0539382a86ba31766f00f2b8312ab167c036729422ec9e7085e8")
.unwrap();
let pk1 =
Pubkey::read_hex(b"52b4a076bcbbbdc3a1aefa3735816cf74993b1b8db202b01c883c58be7fad8bd")
.unwrap();
let pk2 =
Pubkey::read_hex(b"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49")
.unwrap();
let tagged =
Pubkey::read_hex(b"fa984bd7dbb282f07e16e7ae87b26a2a7b9b90b7246a44771f0cf5ae58018f52")
.unwrap();
let data: Vec<u8> = vec![
211, 0, // length of structure
2, 0, // number of IDs
2, 0, // number of authors
3, 0, // number of kinds
255, 255, 255, 255, // limit
0, 0, 0, 0, // padding
0xC1, 0xEB, 0x74, 0x65, 0, 0, 0, 0, // since
255, 255, 255, 255, 255, 255, 255, 255, // until
0x6b, 0x43, 0xbc, 0x2e, 0x37, 0x3b, 0x6d, 0x93, 0x30, 0xff, 0x57, 0x1f, 0x3f, 0x4e,
0x6d, 0x89, 0x7b, 0x32, 0xd0, 0x1d, 0x65, 0x22, 0x7d, 0xf3, 0xfa, 0x41, 0xcd, 0xf7,
0x31, 0xc6, 0x3c, 0x3a, // ID 1
0x1f, 0x47, 0x03, 0x4c, 0x9d, 0x6d, 0x05, 0x39, 0x38, 0x2a, 0x86, 0xba, 0x31, 0x76,
0x6f, 0x00, 0xf2, 0xb8, 0x31, 0x2a, 0xb1, 0x67, 0xc0, 0x36, 0x72, 0x94, 0x22, 0xec,
0x9e, 0x70, 0x85, 0xe8, // ID 2
0x52, 0xb4, 0xa0, 0x76, 0xbc, 0xbb, 0xbd, 0xc3, 0xa1, 0xae, 0xfa, 0x37, 0x35, 0x81,
0x6c, 0xf7, 0x49, 0x93, 0xb1, 0xb8, 0xdb, 0x20, 0x2b, 0x01, 0xc8, 0x83, 0xc5, 0x8b,
0xe7, 0xfa, 0xd8, 0xbd, // Pubkey 1
0xee, 0x11, 0xa5, 0xdf, 0xf4, 0x0c, 0x19, 0xa5, 0x55, 0xf4, 0x1f, 0xe4, 0x2b, 0x48,
0xf0, 0x0e, 0x61, 0x8c, 0x91, 0x22, 0x56, 0x22, 0xae, 0x37, 0xb6, 0xc2, 0xbb, 0x67,
0xb7, 0x6c, 0x4e, 0x49, // Pubkey 2
1, 0, 5, 0, 71, 117, // 3 kinds
// Tags
45, 0, // tags_len
1, 0, // num_tags
6, 0, // first tag offset at 6
2, 0, // 2 fields long
1, 0, // 1st field is 1 byte
112, // "p"
32, 0, // 2nd field is 32 bytes
// 2nd field
0xfa, 0x98, 0x4b, 0xd7, 0xdb, 0xb2, 0x82, 0xf0, 0x7e, 0x16, 0xe7, 0xae, 0x87, 0xb2,
0x6a, 0x2a, 0x7b, 0x9b, 0x90, 0xb7, 0x24, 0x6a, 0x44, 0x77, 0x1f, 0x0c, 0xf5, 0xae,
0x58, 0x01, 0x8f, 0x52,
];
let filter = Filter::delineate(&data).unwrap();
assert_eq!(filter.num_ids(), 2);
let mut ids = filter.ids();
assert_eq!(ids.next().unwrap(), id1);
assert_eq!(ids.next().unwrap(), id2);
assert!(ids.next().is_none());
assert_eq!(filter.num_authors(), 2);
let mut authors = filter.authors();
assert_eq!(authors.next().unwrap(), pk1);
assert_eq!(authors.next().unwrap(), pk2);
assert!(authors.next().is_none());
assert_eq!(filter.num_kinds(), 3);
let mut kinds = filter.kinds();
assert_eq!(kinds.next().unwrap(), Kind(1));
assert_eq!(kinds.next().unwrap(), Kind(5));
assert_eq!(kinds.next().unwrap(), Kind(30023));
assert!(kinds.next().is_none());
assert_eq!(filter.limit(), u32::MAX);
assert_eq!(filter.since(), Time(1702161345));
assert_eq!(filter.until(), Time::max());
let tags = filter.tags().unwrap();
assert_eq!(tags.len(), 1);
let mut iter = tags.iter();
let mut tag = iter.next().unwrap();
assert!(iter.next().is_none());
assert_eq!(tag.next().unwrap(), b"p");
let p_bytes = tag.next().unwrap();
assert!(tag.next().is_none());
let pk = Pubkey(p_bytes.try_into().unwrap());
assert_eq!(pk, tagged);
}
}

47
src/types/id.rs Normal file
View File

@ -0,0 +1,47 @@
use crate::error::Error;
use serde::{Deserialize, Serialize};
use std::fmt;
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct Id(pub [u8; 32]);
impl Id {
pub fn write_hex(&self, output: &mut [u8]) -> Result<(), Error> {
write_hex!(self.0, output, 32)
}
pub fn read_hex(input: &[u8]) -> Result<Id, Error> {
let mut out: [u8; 32] = [0; 32];
read_hex!(input, &mut out, 32)?;
Ok(Id(out))
}
pub fn as_slice(&self) -> &[u8] {
self.0.as_slice()
}
}
impl fmt::Display for Id {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut bytes: [u8; 64] = [0; 64];
self.write_hex(&mut bytes).unwrap();
let hex = unsafe { std::str::from_utf8_unchecked(&bytes) };
write!(f, "{hex}")
}
}
#[cfg(test)]
mod test {
use super::Id;
#[test]
fn test_id_hex_functions() {
let hex = b"1110ee4ff957fa9c55832eaccb4dc1c45bfc6304e1e4e9fa478f53df4b20062d";
let id = Id::read_hex(hex).unwrap();
eprintln!("{:?}", id);
let mut hex2: [u8; 64] = [0; 64];
id.write_hex(&mut hex2).unwrap();
assert_eq!(hex, &hex2);
assert_eq!(format!("{}", id).as_bytes(), hex);
}
}

28
src/types/kind.rs Normal file
View File

@ -0,0 +1,28 @@
use std::fmt;
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct Kind(pub u16);
impl fmt::Display for Kind {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
}
}
impl Kind {
// Some kinds a relay may need to treat differently
pub const SEAL: u16 = 13;
pub const DM_CHAT: u16 = 14;
pub fn is_replaceable(&self) -> bool {
(10000..20000).contains(&self.0) || self.0 == 0 || self.0 == 3
}
pub fn is_ephemeral(&self) -> bool {
(20000..30000).contains(&self.0)
}
pub fn is_parameterized_replaceable(&self) -> bool {
(30000..40000).contains(&self.0)
}
}

25
src/types/mod.rs Normal file
View File

@ -0,0 +1,25 @@
mod event;
pub use event::{Event, OwnedEvent};
mod filter;
pub use filter::{Filter, OwnedFilter};
mod id;
pub use id::Id;
mod kind;
pub use kind::Kind;
pub mod parse;
mod pubkey;
pub use pubkey::Pubkey;
mod sig;
pub use sig::Sig;
mod tags;
pub use tags::{Tags, TagsIter, TagsStringIter};
mod time;
pub use time::Time;

254
src/types/parse/json_escape.rs Normal file
View File

@ -0,0 +1,254 @@
use super::utf8::{encode_utf8, next_code_point};
use crate::error::{ChorusError, Error};
// LITERAL UNESCAPED: 0x20-0x21, 0x23-0x5B, 0x5D-10FFFF
// ESCAPES: \" \\ \/ /b /f /n /r /t
// UTF ESCAPE: \uXXXX or \uXXXX\uXXXX
// This escapes a string with JSON escapes. It takes an output buffer,
// and returns it filled with the escaped string.
// For performance you should try to pass in a buffer that is already
// allocated big enough.
pub fn json_escape(input: &[u8], mut out: Vec<u8>) -> Result<Vec<u8>, Error> {
let mut read_pos: usize = 0;
while let Some((codepoint, size)) = next_code_point(&input[read_pos..])? {
if is_safe_char(codepoint) {
out.extend(&input[read_pos..read_pos + size]);
} else {
match codepoint {
0x08 => out.extend("\\b".as_bytes()),
0x09 => out.extend("\\t".as_bytes()),
0x0A => out.extend("\\n".as_bytes()),
0x0C => out.extend("\\f".as_bytes()),
0x0D => out.extend("\\r".as_bytes()),
0x22 => out.extend("\\\"".as_bytes()),
0x5C => out.extend("\\\\".as_bytes()),
_ => {
if codepoint > 0x20 {
panic!("unnecessary encoding requested");
}
// This violates NIP-01 which doesn't allow characters like 0x00
// even though JSON UTF-8 does.
out.extend(format!("\\u{:04x}", codepoint).as_bytes());
}
}
}
read_pos += size;
}
Ok(out)
}
macro_rules! output_slice {
($slice:expr, $out:expr, $pos:expr) => {
if $out.len() < *$pos + $slice.len() {
Err(Into::<crate::error::Error>::into(
crate::error::ChorusError::BufferTooSmall,
))
} else {
$out[*$pos..*$pos + $slice.len()].copy_from_slice($slice);
*$pos += $slice.len();
Ok(())
}
};
}
macro_rules! output_byte {
($byte:expr, $out:expr, $pos:expr) => {
if $out.len() < *$pos + 1 {
Err(Into::<crate::error::Error>::into(
crate::error::ChorusError::BufferTooSmall,
))
} else {
unsafe { *$out.get_unchecked_mut(*$pos) = $byte };
*$pos += 1;
Ok(())
}
};
}
/// This unescapes a JSON string into the output.
///
/// The input should start on the first character of the string, and may extend
/// to the ending double-quote and even further.
///
/// This will return how much input was consumed and how much output was written
/// in that order (input_len, output_len)
pub fn json_unescape(input: &[u8], out: &mut [u8]) -> Result<(usize, usize), Error> {
const BACKSPACE: u8 = 0x08;
const FORMFEED: u8 = 0x0C;
const LINEFEED: u8 = 0x0A;
const CR: u8 = 0x0D;
const TAB: u8 = 0x09;
const QUOTE: u8 = 0x22;
const BACKSLASH: u8 = 0x5C;
const SLASH: u8 = 0x2F;
// Write position in the output buffer
let mut write_pos: usize = 0;
let mut inescape: bool = false;
let mut uescape: Option<(usize, u32)> = None;
let mut p: usize = 0;
while let Some((codepoint, size)) = next_code_point(&input[p..])? {
if inescape {
inescape = false;
if codepoint > 255 {
return Err(ChorusError::JsonEscape.into());
}
match codepoint as u8 {
QUOTE | BACKSLASH | SLASH => {
output_slice!(&input[p..p + size], out, &mut write_pos)?
}
b'b' => output_byte!(BACKSPACE, out, &mut write_pos)?,
b'f' => output_byte!(FORMFEED, out, &mut write_pos)?,
b'n' => output_byte!(LINEFEED, out, &mut write_pos)?,
b'r' => output_byte!(CR, out, &mut write_pos)?,
b't' => output_byte!(TAB, out, &mut write_pos)?,
// This violates NIP-01 but we accept it
b'u' => uescape = Some((0, 0)),
_ => return Err(ChorusError::JsonEscape.into()), // nothing else is a legal escape
}
} else if let Some((digit, total)) = uescape {
// must be a digit
if !(48..=57).contains(&codepoint) {
return Err(ChorusError::JsonEscape.into());
}
let total = total + ((codepoint - 48) << (4 * (3 - digit)));
if digit >= 3 {
if (0xD800..=0xDFFF).contains(&total) {
return Err(ChorusError::JsonEscapeSurrogate.into());
}
let s = encode_utf8(total, &mut out[write_pos..])?;
write_pos += s;
uescape = None;
} else {
uescape = Some((digit + 1, total));
}
} else if codepoint == 0x5C {
// backslash
inescape = true;
} else if is_safe_char(codepoint) {
output_slice!(&input[p..p + size], out, &mut write_pos)?;
} else if codepoint == 0x22 {
// ending double quote
break;
} else {
return Err(ChorusError::JsonBadStringChar(codepoint).into());
}
p += size;
}
Ok((p, write_pos))
}
#[inline]
fn is_safe_char(c: u32) -> bool {
let safe_ranges = [(0x20..=0x21), (0x23..=0x5B), (0x5D..=0x10FFFF)];
safe_ranges.iter().any(|range| range.contains(&c))
}
#[cfg(test)]
mod test {
use super::{json_escape, json_unescape};
#[test]
fn test_json_escape() {
let buffer = Vec::with_capacity(255);
let input = "hello\t\tworld
!!!";
let mut buffer = json_escape(input.as_bytes(), buffer).unwrap();
assert_eq!(&buffer[0..19], br#"hello\t\tworld\n!!!"#);
buffer.clear();
let input: [u8; 11] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
let mut buffer = json_escape(input.as_slice(), buffer).unwrap();
assert_eq!(
&buffer[0..54],
br#"\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\b\t\n"#
);
buffer.clear();
let input: [u8; 12] = [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22];
let mut buffer = json_escape(input.as_slice(), buffer).unwrap();
assert_eq!(
&buffer[0..64],
br#"\u000b\f\r\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016"#
);
buffer.clear();
let input: [u8; 4] = [32, 33, 34, 35];
let mut buffer = json_escape(input.as_slice(), buffer).unwrap();
assert_eq!(&buffer[0..5], br##" !\"#"##);
buffer.clear();
let input: [u8; 1] = [92];
let buffer = json_escape(input.as_slice(), buffer).unwrap();
assert_eq!(&buffer[0..2], br#"\\"#);
}
#[test]
fn test_json_unescape() {
let is_ok = |s: &[u8], equals: &[u8]| {
let mut buffer: Vec<u8> = Vec::with_capacity(1024);
buffer.resize(1024, 0);
let r = json_unescape(s, &mut buffer);
assert!(r.is_ok());
let (_inlen, outlen) = r.unwrap();
assert_eq!(outlen, equals.len());
assert_eq!(&buffer[0..equals.len()], equals);
};
let is_err = |s: &[u8]| {
let mut buffer: Vec<u8> = Vec::with_capacity(1024);
buffer.resize(1024, 0);
let r = json_unescape(s, &mut buffer);
assert!(r.is_err());
};
// simple string
is_ok(&b"abc".as_slice(), b"abc");
// carraige return
is_ok(&br#"ab\nc"#.as_slice(), b"ab\nc");
// escaping a character that is not allowed
is_err(&br#"ab\zc"#.as_slice());
// escaping quotes is allowed
is_ok(&br#" \"abc\" "#.as_slice(), br#" "abc" "#);
// high character
is_ok(r#"𝄞"#.as_bytes(), "𝄞".as_bytes());
// high character is interpreted as these four bytes
is_ok(r#"𝄞"#.as_bytes(), b"\xF0\x9D\x84\x9E");
// esacaping a character that is not allowed
is_err(r#"\𝄞"#.as_bytes());
// actual unescaped tab is disallowed
is_err("\t".as_bytes());
// unicode escape and more
is_ok(
r#"{\"name\":\"BagMan\",\"about\":\"Father.\nHusband.\nNerd: \u2020.\"}"#.as_bytes(),
"{\"name\":\"BagMan\",\"about\":\"Father.\nHusband.\nNerd: †.\"}".as_bytes(),
);
// bad unicode escape
is_err(r#"\u8f00"#.as_bytes());
// Check output values
let mut buffer: Vec<u8> = Vec::with_capacity(1024);
buffer.resize(1024, 0);
let (inlen, outlen) = json_unescape(br#"the\nclient", "gossip""#, &mut buffer).unwrap();
assert_eq!(inlen, 11);
assert_eq!(outlen, 10);
}
}

468
src/types/parse/json_parse.rs Normal file
View File

@ -0,0 +1,468 @@
use super::json_escape::json_unescape;
use super::put;
use crate::error::{ChorusError, Error};
#[inline]
pub fn eat_whitespace(input: &[u8], inposp: &mut usize) {
while *inposp < input.len() && [0x20, 0x09, 0x0A, 0x0D].contains(&input[*inposp]) {
*inposp += 1;
}
}
#[inline]
pub fn eat_whitespace_and_commas(input: &[u8], inposp: &mut usize) {
while *inposp < input.len() && [0x20, 0x09, 0x0A, 0x0D, b','].contains(&input[*inposp]) {
*inposp += 1;
}
}
#[inline]
pub fn verify_char(input: &[u8], ch: u8, inposp: &mut usize) -> Result<(), Error> {
if *inposp >= input.len() {
Err(ChorusError::JsonBad("Too Short or Missing Fields", *inposp).into())
} else if input[*inposp] == ch {
*inposp += 1;
Ok(())
} else {
Err(ChorusError::JsonBadCharacter(input[*inposp] as char, *inposp, ch as char).into())
}
}
pub fn eat_colon_with_whitespace(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
eat_whitespace(input, inposp);
verify_char(input, b':', inposp)?;
eat_whitespace(input, inposp);
Ok(())
}
pub fn next_object_field(input: &[u8], inposp: &mut usize) -> Result<bool, Error> {
eat_whitespace(input, inposp);
// next comes either comma or end brace
if *inposp >= input.len() {
return Err(ChorusError::JsonBad("Too short", *inposp).into());
}
if input[*inposp] == b'}' {
*inposp += 1;
Ok(true)
} else if input[*inposp] == b',' {
*inposp += 1;
Ok(false)
} else {
Err(ChorusError::JsonBad("Unexpected char", *inposp).into())
}
}
pub fn read_id(input: &[u8], inposp: &mut usize, output: &mut [u8]) -> Result<(), Error> {
if output.len() < 32 {
return Err(ChorusError::BufferTooSmall.into());
}
verify_char(input, b'"', inposp)?;
if *inposp + 64 >= input.len() {
return Err(ChorusError::JsonBad("Too short reading id", *inposp).into());
}
// Read the hex ID and write the binary ID into the output event structure
read_hex!(&input[*inposp..*inposp + 64], &mut output[..32], 32)?;
*inposp += 64;
verify_char(input, b'"', inposp)?;
Ok(())
}
pub fn read_pubkey(input: &[u8], inposp: &mut usize, output: &mut [u8]) -> Result<(), Error> {
if output.len() < 32 {
return Err(ChorusError::BufferTooSmall.into());
}
verify_char(input, b'"', inposp)?;
if *inposp + 64 >= input.len() {
return Err(ChorusError::JsonBad("Too short reading pubkey", *inposp).into());
}
// Read the hex pubkey and write the binary pubkey into the output event structure
read_hex!(&input[*inposp..*inposp + 64], &mut output[..32], 32)?;
*inposp += 64;
verify_char(input, b'"', inposp)?;
Ok(())
}
pub fn read_u64(input: &[u8], inposp: &mut usize) -> Result<u64, Error> {
let mut value: u64 = 0;
let mut any: bool = false;
while *inposp < input.len() && b"0123456789".contains(&input[*inposp]) {
any = true;
value = (value * 10) + (input[*inposp] - 48) as u64;
*inposp += 1;
}
if !any {
return Err(ChorusError::JsonBad(
"Created at must be a positive or zero valued number",
*inposp,
)
.into());
}
Ok(value)
}
pub fn read_kind(input: &[u8], inposp: &mut usize) -> Result<u16, Error> {
let mut value: u32 = 0;
let mut any: bool = false;
while *inposp < input.len() && b"0123456789".contains(&input[*inposp]) {
any = true;
value = (value * 10) + (input[*inposp] - 48) as u32;
*inposp += 1;
}
if !any {
return Err(ChorusError::JsonBad(
"Kind at must be a positive or zero valued number",
*inposp,
)
.into());
}
if value > 65535 {
Err(ChorusError::JsonBad("Kind larger than 65535", *inposp).into())
} else {
Ok(value as u16)
}
}
// HELP written for event only
// From the outer bracket through to the character after the close outer bracket
// returns the size of data written to the output
pub fn read_tags_array(
input: &[u8],
inposp: &mut usize,
output: &mut [u8],
) -> Result<usize, Error> {
verify_char(input, b'[', inposp)?; // outer array open brace
eat_whitespace(input, inposp);
if output.len() < 4 {
return Err(ChorusError::BufferTooSmall.into());
}
// NOTE: we cannot write any tag strings until after we have counted the tags.
// (our tags structure is optimized for reading, not writing)
let num_tags: usize = count_tags(input, *inposp)?;
put(output, 2, (num_tags as u16).to_ne_bytes().as_slice())?;
// Case where we have no tags
if num_tags == 0 {
put(output, 0, 4_u16.to_ne_bytes().as_slice())?;
burn_array(input, inposp)?;
return Ok(4);
}
verify_char(input, b'[', inposp)?; // opening brace of first tag
eat_whitespace(input, inposp);
let mut tag_num = 0;
let mut outpos: usize = 4 + num_tags * 2;
if output.len() < outpos {
return Err(ChorusError::BufferTooSmall.into());
}
loop {
// Write the offset of this tag
let offset_slot = 4 + tag_num * 2;
put(
output,
offset_slot,
(outpos as u16).to_ne_bytes().as_slice(),
)?;
// Read the tag (bumps inpos and outpos)
read_tag(input, inposp, output, &mut outpos)?;
eat_whitespace(input, inposp);
// Check what is next
match input[*inposp] {
b']' => {
*inposp += 1;
if tag_num != num_tags - 1 {
panic!("Tag count mismatch");
}
break;
}
b',' => {
*inposp += 1;
eat_whitespace(input, inposp);
verify_char(input, b'[', inposp)?;
tag_num += 1;
if tag_num >= num_tags {
panic!("Tag count mismatch");
}
eat_whitespace(input, inposp);
}
_ => return Err(ChorusError::JsonBad("Tag array bad character", *inposp).into()),
}
}
// Write length of tags section
put(output, 0, (outpos as u16).to_ne_bytes().as_slice())?;
Ok(outpos)
}
// From the first inner tag bracket, ending after the outer bracket.
// This just counts the tags, it does not write output or modify the inpos
// This does a quicker pass over the content than actual tag parsing does.
pub fn count_tags(input: &[u8], mut inpos: usize) -> Result<usize, Error> {
// First non-whitespace character after the opening brace
match input[inpos] {
b']' => return Ok(0), // no tags
b'[' => (), // expected
_ => return Err(ChorusError::JsonBad("Tag array bad initial character", inpos).into()),
}
let mut count = 1;
inpos += 1;
burn_tag(input, &mut inpos)?;
eat_whitespace(input, &mut inpos);
loop {
match input[inpos] {
b']' => return Ok(count),
b',' => {
inpos += 1;
eat_whitespace(input, &mut inpos);
verify_char(input, b'[', &mut inpos)?;
count += 1;
burn_tag(input, &mut inpos)?;
eat_whitespace(input, &mut inpos);
}
_ => return Err(ChorusError::JsonBad("Tag array bad character", inpos).into()),
}
}
}
pub fn read_tag(
input: &[u8],
inposp: &mut usize,
output: &mut [u8],
outposp: &mut usize,
) -> Result<(), Error> {
verify_char(input, b'"', inposp)?;
let countpos = *outposp;
*outposp += 2;
let mut num_strings: usize = 1;
loop {
// read string
let (inlen, outlen) = json_unescape(&input[*inposp..], &mut output[*outposp + 2..])?;
// write the length before it
put(output, *outposp, (outlen as u16).to_ne_bytes().as_slice())?;
// bump the outposp past it
*outposp += 2 + outlen;
// bump the inpos past the string (and the ending quote which isn't counted in the len)
*inposp += inlen + 1;
eat_whitespace(input, inposp);
match input[*inposp] {
b',' => {
*inposp += 1;
eat_whitespace(input, inposp);
verify_char(input, b'"', inposp)?;
num_strings += 1;
continue;
}
b']' => {
*inposp += 1;
break;
}
_ => return Err(ChorusError::JsonBad("Tag array bad character", *inposp).into()),
}
}
// Write the count of strings at the very start
put(
output,
countpos,
(num_strings as u16).to_ne_bytes().as_slice(),
)?;
Ok(())
}
pub fn read_content(
input: &[u8],
inposp: &mut usize,
output: &mut [u8],
after_tags: usize,
) -> Result<(), Error> {
verify_char(input, b'"', inposp)?;
// Place content 4 bytes beyond tags, to reserve space for content length
let (inlen, outlen) = json_unescape(&input[*inposp..], &mut output[after_tags + 4..])?;
*inposp += inlen + 1; // +1 to pass the end quote
// Write content length
put(output, after_tags, (outlen as u32).to_ne_bytes().as_slice())?;
// Write event size
let event_len = after_tags + 4 + outlen;
put(output, 0, (event_len as u32).to_ne_bytes().as_slice())?;
Ok(())
}
// FIXME this is too event-offset specific
pub fn read_sig(input: &[u8], inposp: &mut usize, output: &mut [u8]) -> Result<(), Error> {
if output.len() < 144 {
return Err(ChorusError::BufferTooSmall.into());
}
verify_char(input, b'"', inposp)?;
if *inposp + 128 >= input.len() {
return Err(ChorusError::JsonBad("Too short reading sig", *inposp).into());
}
// Read the hex sig and write the binary sig into the output event structure
read_hex!(&input[*inposp..*inposp + 128], &mut output[80..144], 64)?;
*inposp += 128;
verify_char(input, b'"', inposp)?;
Ok(())
}
// from the character after the start quote
// ending on the character following the end quote
pub fn burn_string(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
while *inposp < input.len() && input[*inposp] != b'"' {
if input[*inposp] == b'\\' && *inposp + 1 < input.len() {
*inposp += 2;
} else {
*inposp += 1;
}
}
if input[*inposp] == b'"' {
*inposp += 1;
Ok(())
} else {
Err(ChorusError::JsonBad("Unterminated string", *inposp).into())
}
}
// from the character after the open brace
// ending on the character following the close brace
pub fn burn_tag(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
eat_whitespace(input, inposp);
// assuming that every tag must have at least one string
verify_char(input, b'"', inposp)?;
burn_string(input, inposp)?;
eat_whitespace(input, inposp);
while input[*inposp] == b',' {
*inposp += 1;
eat_whitespace(input, inposp);
verify_char(input, b'"', inposp)?;
burn_string(input, inposp)?;
eat_whitespace(input, inposp);
}
verify_char(input, b']', inposp)?;
Ok(())
}
pub fn burn_key_and_value(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
verify_char(input, b'"', inposp)?;
burn_string(input, inposp)?;
eat_colon_with_whitespace(input, inposp)?;
burn_value(input, inposp)?;
Ok(())
}
// from the character after the open brace
// ending on the character following the close brace
pub fn burn_object(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
loop {
eat_whitespace_and_commas(input, inposp);
// Check for the end
if input[*inposp] == b'}' {
*inposp += 1;
return Ok(());
}
burn_key_and_value(input, inposp)?;
}
}
// from the character after the open bracket
// ending on the character following the close bracket
pub fn burn_array(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
loop {
eat_whitespace_and_commas(input, inposp);
// Check for the end
if input[*inposp] == b']' {
*inposp += 1;
return Ok(());
}
burn_value(input, inposp)?;
}
}
pub fn burn_value(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
if *inposp >= input.len() {
return Err(ChorusError::JsonBad("Too short burning an unused JSON value", *inposp).into());
}
match input[*inposp] {
b'"' => {
*inposp += 1;
burn_string(input, inposp)?
}
b'[' => {
*inposp += 1;
burn_array(input, inposp)?
}
b'{' => {
*inposp += 1;
burn_object(input, inposp)?
}
b't' => burn_true(input, inposp)?,
b'f' => burn_false(input, inposp)?,
b'n' => burn_null(input, inposp)?,
b'-' => burn_number(input, inposp)?,
_ => {
if b"123456789".contains(&input[*inposp]) {
burn_number(input, inposp)?
} else {
return Err(ChorusError::JsonBad(
"Too short burning an unused JSON value",
*inposp,
)
.into());
}
}
}
Ok(())
}
pub fn burn_null(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
if *inposp + 4 <= input.len() && &input[*inposp..*inposp + 4] == b"null" {
*inposp += 4;
Ok(())
} else {
Err(ChorusError::JsonBad("Expected null", *inposp).into())
}
}
pub fn burn_true(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
if *inposp + 4 <= input.len() && &input[*inposp..*inposp + 4] == b"true" {
*inposp += 4;
Ok(())
} else {
Err(ChorusError::JsonBad("Expected true", *inposp).into())
}
}
pub fn burn_false(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
if *inposp + 5 <= input.len() && &input[*inposp..*inposp + 5] == b"false" {
*inposp += 5;
Ok(())
} else {
Err(ChorusError::JsonBad("Expected false", *inposp).into())
}
}
pub fn burn_number(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
// For burning, we don't check validity.
while *inposp < input.len() && b".+-0123456789abcdefABCDEF_oOxXn".contains(&input[*inposp]) {
*inposp += 1;
}
Ok(())
}

15
src/types/parse/mod.rs Normal file
View File

@ -0,0 +1,15 @@
pub mod json_escape;
pub mod json_parse;
pub mod utf8;
#[inline]
pub fn put(output: &mut [u8], offset: usize, data: &[u8]) -> Result<(), crate::error::Error> {
if output.len() < offset + data.len() {
Err(crate::error::ChorusError::BufferTooSmall.into())
} else {
output[offset..offset + data.len()].copy_from_slice(data);
Ok(())
}
}

165
src/types/parse/utf8.rs Normal file
View File

@ -0,0 +1,165 @@
use crate::error::{ChorusError, Error};
// Reads the next code point if UTF-8, and returns it along with the number of characters
// that make it up.
pub fn next_code_point(input: &[u8]) -> Result<Option<(u32, usize)>, Error> {
let len = input.len();
if len < 1 {
return Ok(None);
}
// Decode UTF-8
let x = input[0];
if x < 128 {
return Ok(Some((x as u32, 1)));
}
// Multibyte case follows
// Decode from a byte combination out of: [[[x y] z] w]
let init = utf8_first_byte(x, 2);
if len < 2 {
return Err(ChorusError::Utf8Error.into());
}
let y = input[1];
let mut ch = utf8_acc_cont_byte(init, y);
if x >= 0xE0 {
// [[x y z] w] case
// 5th bit in 0xE0 .. 0xEF is always clear, so `init` is still valid
if len < 3 {
return Err(ChorusError::Utf8Error.into());
}
let z = input[2];
let y_z = utf8_acc_cont_byte((y & CONT_MASK) as u32, z);
ch = init << 12 | y_z;
if x >= 0xF0 {
// [x y z w] case
// use only the lower 3 bits of `init`
if len < 4 {
return Err(ChorusError::Utf8Error.into());
}
let w = input[3];
ch = (init & 7) << 18 | utf8_acc_cont_byte(y_z, w);
Ok(Some((ch, 4)))
} else {
Ok(Some((ch, 3)))
}
} else {
Ok(Some((ch, 2)))
}
}
pub fn encode_utf8(code: u32, dst: &mut [u8]) -> Result<usize, Error> {
// UTF-8 ranges and tags for encoding characters
const TAG_CONT: u8 = 0b1000_0000;
const TAG_TWO_B: u8 = 0b1100_0000;
const TAG_THREE_B: u8 = 0b1110_0000;
const TAG_FOUR_B: u8 = 0b1111_0000;
const MAX_ONE_B: u32 = 0x80;
const MAX_TWO_B: u32 = 0x800;
const MAX_THREE_B: u32 = 0x10000;
let len = unsafe {
if code < MAX_ONE_B && !dst.is_empty() {
*dst.get_unchecked_mut(0) = code as u8;
1
} else if code < MAX_TWO_B && dst.len() >= 2 {
*dst.get_unchecked_mut(0) = (code >> 6 & 0x1F) as u8 | TAG_TWO_B;
*dst.get_unchecked_mut(1) = (code & 0x3F) as u8 | TAG_CONT;
2
} else if code < MAX_THREE_B && dst.len() >= 3 {
*dst.get_unchecked_mut(0) = (code >> 12 & 0x0F) as u8 | TAG_THREE_B;
*dst.get_unchecked_mut(1) = (code >> 6 & 0x3F) as u8 | TAG_CONT;
*dst.get_unchecked_mut(2) = (code & 0x3F) as u8 | TAG_CONT;
3
} else if dst.len() >= 4 {
*dst.get_unchecked_mut(0) = (code >> 18 & 0x07) as u8 | TAG_FOUR_B;
*dst.get_unchecked_mut(1) = (code >> 12 & 0x3F) as u8 | TAG_CONT;
*dst.get_unchecked_mut(2) = (code >> 6 & 0x3F) as u8 | TAG_CONT;
*dst.get_unchecked_mut(3) = (code & 0x3F) as u8 | TAG_CONT;
4
} else {
return Err(ChorusError::BufferTooSmall.into());
}
};
Ok(len)
}
/// Returns the initial codepoint accumulator for the first byte.
/// The first byte is special, only want bottom 5 bits for width 2, 4 bits
/// for width 3, and 3 bits for width 4.
#[inline]
const fn utf8_first_byte(byte: u8, width: u32) -> u32 {
(byte & (0x7F >> width)) as u32
}
/// Returns the value of `ch` updated with continuation byte `byte`.
#[inline]
const fn utf8_acc_cont_byte(ch: u32, byte: u8) -> u32 {
(ch << 6) | (byte & CONT_MASK) as u32
}
/// Mask of the value bits of a continuation byte.
const CONT_MASK: u8 = 0b0011_1111;
#[cfg(test)]
mod test {
use super::{encode_utf8, next_code_point};
#[test]
fn test_next_code_point() {
let (codepoint, size) = next_code_point(r#"𝄞"#.as_bytes()).unwrap().unwrap();
assert_eq!(codepoint, 119070);
assert_eq!(size, 4);
let (codepoint, size) = next_code_point(r#"†"#.as_bytes()).unwrap().unwrap();
assert_eq!(codepoint, 0x2020);
assert_eq!(size, 3);
// four codepoints
let s = [
0x61, 0xE0, 0xA4, 0xA8, 0xE0, 0xA4, 0xBF, 0xE4, 0xBA, 0x9C, 0xF0, 0x90, 0x82, 0x83,
];
let (codepoint, size) = next_code_point(s.as_slice()).unwrap().unwrap();
assert_eq!(codepoint, 0x61);
assert_eq!(size, 1);
let mut start = size;
let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
assert_eq!(codepoint, 0x928);
assert_eq!(size, 3);
start += size;
let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
assert_eq!(codepoint, 0x93F);
assert_eq!(size, 3);
start += size;
let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
assert_eq!(codepoint, 0x4E9C);
assert_eq!(size, 3);
start += size;
let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
assert_eq!(codepoint, 0x10083);
assert_eq!(size, 4);
assert_eq!(next_code_point(&s[0..0]).unwrap(), None);
}
#[test]
fn test_encode_utf8() {
let mut buffer: Vec<u8> = vec![0, 0, 0, 0];
assert_eq!(encode_utf8(0x69, &mut buffer).unwrap(), 1);
assert_eq!(buffer[0], 0x69);
assert_eq!(encode_utf8(0xEC, &mut buffer).unwrap(), 2);
assert_eq!(&buffer[0..2], &[0xC3, 0xAC]);
assert_eq!(encode_utf8(0x5450, &mut buffer).unwrap(), 3);
assert_eq!(&buffer[0..3], &[0xE5, 0x91, 0x90]);
assert_eq!(encode_utf8(0x2825F, &mut buffer).unwrap(), 4);
assert_eq!(&buffer[..], &[0xF0, 0xA8, 0x89, 0x9F]);
}
}

47
src/types/pubkey.rs Normal file
View File

@ -0,0 +1,47 @@
use crate::Error;
use serde::{Deserialize, Serialize};
use std::fmt;
#[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq)]
pub struct Pubkey(pub [u8; 32]);
impl Pubkey {
pub fn write_hex(&self, output: &mut [u8]) -> Result<(), Error> {
write_hex!(self.0, output, 32)
}
pub fn read_hex(input: &[u8]) -> Result<Pubkey, Error> {
let mut out: [u8; 32] = [0; 32];
read_hex!(input, &mut out, 32)?;
Ok(Pubkey(out))
}
pub fn as_slice(&self) -> &[u8] {
self.0.as_slice()
}
}
impl fmt::Display for Pubkey {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut bytes: [u8; 64] = [0; 64];
self.write_hex(&mut bytes).unwrap();
let hex = unsafe { std::str::from_utf8_unchecked(&bytes) };
write!(f, "{hex}")
}
}
#[cfg(test)]
mod test {
use super::Pubkey;
#[test]
fn test_pubkey_hex_functions() {
let hex = b"1110ee4ff957fa9c55832eaccb4dc1c45bfc6304e1e4e9fa478f53df4b20062d";
let pubkey = Pubkey::read_hex(hex).unwrap();
eprintln!("{:?}", pubkey);
let mut hex2: [u8; 64] = [0; 64];
pubkey.write_hex(&mut hex2).unwrap();
assert_eq!(hex, &hex2);
assert_eq!(format!("{}", pubkey).as_bytes(), hex);
}
}

46
src/types/sig.rs Normal file
View File

@ -0,0 +1,46 @@
use crate::Error;
use std::fmt;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Sig(pub [u8; 64]);
impl Sig {
pub fn write_hex(&self, output: &mut [u8]) -> Result<(), Error> {
write_hex!(self.0, output, 64)
}
pub fn read_hex(input: &[u8]) -> Result<Sig, Error> {
let mut out: [u8; 64] = [0; 64];
read_hex!(input, &mut out, 64)?;
Ok(Sig(out))
}
pub fn as_slice(&self) -> &[u8] {
self.0.as_slice()
}
}
impl fmt::Display for Sig {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut bytes: [u8; 128] = [0; 128];
self.write_hex(&mut bytes).unwrap();
let hex = unsafe { std::str::from_utf8_unchecked(&bytes) };
write!(f, "{hex}")
}
}
#[cfg(test)]
mod test {
use super::Sig;
#[test]
fn test_sig_hex_functions() {
let hex = b"f4165cd621d387e0f723c3ca7484ca3da9ede00ffc97eb57c3e695384e095dea1a6215e7328b793e878f436f508006f95957c7e6b652e80d4c3c47b9f9110e7d";
let sig = Sig::read_hex(hex).unwrap();
eprintln!("{:?}", sig);
let mut hex2: [u8; 128] = [0; 128];
sig.write_hex(&mut hex2).unwrap();
assert_eq!(hex, &hex2);
assert_eq!(format!("{}", sig).as_bytes(), hex);
}
}

287
src/types/tags.rs Normal file
View File

@ -0,0 +1,287 @@
use crate::error::{ChorusError, Error};
use std::fmt;
/*
* 0 .. 2 u16 Length of the tags section
* 2 .. 4 u16 num_tags
* 4 .. 6 u16 offset of zeroeth tag
* 6 .. 8 u16 offset of first tag
* ...
*
* 4+num_tags*2 .. beginning of actual tag data
*
* Tag data looks like this for each tag:
* count, (len, data), (len, data), ...
*/
/// This stores an array of tags, each tag being an array of byte-strings.
/// It is stored in a single packed linear byte array.
#[derive(Debug, Clone)]
pub struct Tags<'a>(&'a [u8]);
impl<'a> Tags<'a> {
// this marks off the slice of bytes that represent the tags from a potentially longer input
pub fn delineate(input: &'a [u8]) -> Result<Tags<'a>, Error> {
if input.len() < 2 {
return Err(ChorusError::EndOfInput.into());
}
let len = parse_u16!(input, 0) as usize;
if input.len() < len {
return Err(ChorusError::EndOfInput.into());
}
Ok(Tags(&input[0..len]))
}
// This copies
pub fn copy(&self, output: &mut [u8]) -> Result<(), Error> {
if output.len() < self.0.len() {
return Err(ChorusError::BufferTooSmall.into());
}
output[..self.0.len()].copy_from_slice(self.0);
Ok(())
}
#[inline]
pub fn as_bytes(&self) -> &[u8] {
self.0
}
#[inline]
pub fn len(&self) -> usize {
parse_u16!(self.0, 2) as usize
}
#[inline]
pub fn is_empty(&self) -> bool {
self.len() == 0
}
pub fn iter(&'a self) -> TagsIter<'a> {
TagsIter {
tags: self,
next: 0,
}
}
pub fn get_string(&'a self, tag: usize, string: usize) -> Option<&'a [u8]> {
if tag >= self.len() {
return None;
}
let mut offset = parse_u16!(self.0, 4 + tag * 2) as usize;
let count = parse_u16!(self.0, offset) as usize;
offset += 2;
if string >= count {
return None;
}
let end = parse_u16!(self.0, 0) as usize;
// pass the fields we aren't reading
for _ in 0..string {
let len = parse_u16!(self.0, offset) as usize;
offset += 2 + len;
if offset > end {
// safety check
return None;
}
}
let len = parse_u16!(self.0, offset) as usize;
offset += 2;
if offset + len > end {
// safety check
return None;
}
Some(&self.0[offset..offset + len])
}
pub fn get_value(&'a self, key: &[u8]) -> Option<&'a [u8]> {
for tag in 0..self.len() {
if let Some(thing) = self.get_string(tag, 0) {
if thing == key {
return self.get_string(tag, 1);
}
}
}
None
}
pub fn matches(&self, letter: &[u8], value: &[u8]) -> bool {
for mut tag in self.iter() {
if tag.next() == Some(letter) && tag.next() == Some(value) {
return true;
}
}
false
}
pub fn as_json(&self) -> Vec<u8> {
let mut output: Vec<u8> = Vec::with_capacity(256);
let mut first = true;
output.push(b'[');
for tag in self.iter() {
if !first {
output.push(b',');
}
output.push(b'[');
let mut firststring = true;
for bytes in tag {
if !firststring {
output.push(b',');
}
output.push(b'"');
output.extend(bytes);
output.push(b'"');
firststring = false;
}
output.push(b']');
first = false;
}
output.push(b']');
output
}
}
impl fmt::Display for Tags<'_> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let bytes = self.as_json();
let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
write!(f, "{s}")
}
}
#[derive(Debug)]
pub struct TagsIter<'a> {
tags: &'a Tags<'a>,
next: usize,
}
impl<'a> Iterator for TagsIter<'a> {
type Item = TagsStringIter<'a>;
fn next(&mut self) -> Option<Self::Item> {
if self.next >= self.tags.len() {
None
} else {
let offset_slot = 4 + self.next * 2;
let offset = parse_u16!(self.tags.0, offset_slot) as usize;
let count = parse_u16!(self.tags.0, offset) as usize;
self.next += 1;
Some(TagsStringIter {
tags: self.tags,
count,
cur_offset: offset + 2,
next: 0,
})
}
}
}
#[derive(Debug)]
pub struct TagsStringIter<'a> {
tags: &'a Tags<'a>,
count: usize,
cur_offset: usize,
next: usize,
}
impl<'a> Iterator for TagsStringIter<'a> {
type Item = &'a [u8];
fn next(&mut self) -> Option<Self::Item> {
if self.next >= self.count {
None
} else {
// Read len
let len = parse_u16!(self.tags.0, self.cur_offset) as usize;
let s = &self.tags.0[self.cur_offset + 2..self.cur_offset + 2 + len];
self.cur_offset += 2 + len;
self.next += 1;
Some(s)
}
}
}
#[cfg(test)]
mod test {
use super::Tags;
#[test]
fn test_tag() {
/*
* [
* ["Hello world!", "Hello", "world!"],
* ["p", ""ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"],
* ]
*/
let data: Vec<u8> = vec![
110, 0, // tags_len
2, 0, // num_tags
8, 0, // first tag at offset 8
39, 0, // second tag at offset 39
// 8:
3, 0, // three fields long
12, 0, // first field 12 bytes long
72, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33, // "Hello world!"
5, 0, // second field 5 bytes long
72, 101, 108, 108, 111, // "Hello"
6, 0, // third field 6 bytes long
119, 111, 114, 108, 100, 33, // world!
// 39:
2, 0, // two fields long
1, 0, // first field 1 bytes long
112, // "p"
64, 0, // second field 64 bytes long
101, 101, 49, 49, 97, 53, 100, 102, 102, 52, 48, 99, 49, 57, 97, 53, 53, 53, 102, 52,
49, 102, 101, 52, 50, 98, 52, 56, 102, 48, 48, 101, 54, 49, 56, 99, 57, 49, 50, 50, 53,
54, 50, 50, 97, 101, 51, 55, 98, 54, 99, 50, 98, 98, 54, 55, 98, 55, 54, 99, 52, 101,
52, 57,
// "ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"
];
let tags = Tags::delineate(&data).unwrap();
// Test tag access with get_string()
assert_eq!(tags.get_string(0, 0), Some(b"Hello world!".as_slice()));
assert_eq!(tags.get_string(0, 1), Some(b"Hello".as_slice()));
assert_eq!(tags.get_string(0, 2), Some(b"world!".as_slice()));
assert_eq!(tags.get_string(0, 3), None);
assert_eq!(tags.get_string(1, 0), Some(b"p".as_slice()));
assert_eq!(
tags.get_string(1, 1),
Some(b"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49".as_slice())
);
assert_eq!(tags.get_string(1, 2), None);
assert_eq!(tags.get_string(2, 0), None);
// Test tag access with iterators
let mut iter = tags.iter();
let mut tag1 = iter.next().unwrap();
println!("TagsStringIter 1 {:?}", tag1);
assert_eq!(tag1.next(), Some(b"Hello world!".as_slice()));
assert_eq!(tag1.next(), Some(b"Hello".as_slice()));
assert_eq!(tag1.next(), Some(b"world!".as_slice()));
assert!(tag1.next().is_none());
let mut tag2 = iter.next().unwrap();
assert_eq!(tag2.next(), Some(b"p".as_slice()));
assert_eq!(
tag2.next(),
Some(b"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49".as_slice())
);
assert!(tag2.next().is_none());
let tag3 = iter.next();
assert!(tag3.is_none());
assert_eq!(
format!("{tags}"),
r#"[["Hello world!","Hello","world!"],["p","ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"]]"#
);
}
}

29
src/types/time.rs Normal file
View File

@ -0,0 +1,29 @@
use std::fmt;
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct Time(pub u64);
impl fmt::Display for Time {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
}
}
impl Time {
pub fn as_u64(&self) -> u64 {
self.0
}
pub fn min() -> Time {
Time(0)
}
pub fn max() -> Time {
Time(u64::MAX)
}
pub fn now() -> Time {
// Safety: unwrap() can only panic if the system time is before UNIX_EPOCH
Time(std::time::UNIX_EPOCH.elapsed().unwrap().as_secs())
}
}

108
src/web.rs Normal file
View File

@ -0,0 +1,108 @@
use crate::config::Config;
use crate::error::Error;
use crate::globals::GLOBALS;
use hyper::{Body, Request, Response, StatusCode};
use std::net::SocketAddr;
pub async fn serve_http(peer: SocketAddr, request: Request<Body>) -> Result<Response<Body>, Error> {
log::debug!("{}: HTTP request for {}", peer, request.uri());
let response = Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "*")
.header("Access-Control-Allow-Methods", "*")
.status(StatusCode::OK)
.body("This is a nostr relay. Please use a nostr client to connect.".into())?;
Ok(response)
}
pub async fn serve_nip11(peer: SocketAddr) -> Result<Response<Body>, Error> {
log::debug!("{}: sent NIP-11", peer);
let rid = {
let config = GLOBALS.config.read().await;
GLOBALS.rid.get_or_init(|| build_rid(&config))
};
let response = Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "*")
.header("Access-Control-Allow-Methods", "*")
.header("Content-Type", "application/nostr+json")
.status(StatusCode::OK)
.body(rid.clone().into())?;
Ok(response)
}
fn build_rid(config: &Config) -> String {
let mut rid: String = String::with_capacity(255);
const SUPPORTED_NIPS: [u8; 7] = [
1, // nostr
4, // DMs
9, // Event Deletion
11, // relay information document
42, // AUTH
59, // GiftWrap
65, // Relay List Metadata
];
const _UNSUPPORTED_NIPS: [u8; 7] = [
26, // Delegated Event Signing
28, // Public Chat
40, // Expiration Timestamp
45, // Counting results
50, // SEARCH
94, // File Metadata
96, // HTTP File Storage Integration
];
const _INAPPLICABLE_NIPS: [u8; 43] = [
2, 3, 5, 6, 7, 8, 10, 13, 14, 15, 18, 19, 21, 23, 24, 25, 27, 30, 31, 32, 36, 38, 39, 44,
46, 47, 48, 49, 51, 52, 53, 56, 57, 58, 72, 75, 78, 84, 89, 90, 92, 98, 99,
];
let s = SUPPORTED_NIPS
.iter()
.map(|i| format!("{}", i))
.collect::<Vec<String>>()
.join(",");
rid.push_str(&format!("{{\"supported_nips\":[{}],", s));
let software = env!("CARGO_PKG_NAME");
rid.push_str("\"software\":\"");
rid.push_str(software);
rid.push('\"');
let version = env!("CARGO_PKG_VERSION");
rid.push(',');
rid.push_str("\"version\":\"");
rid.push_str(version);
rid.push('\"');
if let Some(name) = &config.name {
rid.push(',');
rid.push_str("\"name\":\"");
rid.push_str(name);
rid.push('\"');
}
if let Some(description) = &config.description {
rid.push(',');
rid.push_str("\"description\":\"");
rid.push_str(description);
rid.push('\"');
}
if let Some(contact) = &config.contact {
rid.push(',');
rid.push_str("\"contact\":\"");
rid.push_str(contact);
rid.push('\"');
}
if let Some(pubkey) = &config.public_key {
let mut pkh: [u8; 64] = [0; 64];
pubkey.write_hex(&mut pkh).unwrap();
rid.push(',');
rid.push_str("\"pubkey\":\"");
rid.push_str(unsafe { std::str::from_utf8_unchecked(pkh.as_slice()) });
rid.push('\"');
}
rid.push('}');
rid
}

145
src/web/blossom/auth.rs
View File

@ -1,145 +0,0 @@
use crate::error::{ChorusError, Error};
use base64::prelude::*;
use http::header::AUTHORIZATION;
use hyper::body::Incoming;
use hyper::Request;
use pocket_types::Event;
fn s_err(s: &str) -> Result<AuthData, Error> {
Err(ChorusError::BlossomAuthFailure(s.to_owned()).into())
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum AuthVerb {
Upload,
List,
Delete,
Mirror,
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct AuthData {
/// If a verb was included, this is it
pub verb: Option<AuthVerb>,
/// If an 'x' tag was included, this is the hash
pub hash: Option<[u8; 32]>,
}
pub fn verify_auth(request: &Request<Incoming>) -> Result<AuthData, Error> {
// Force every other error into a BlossomAuthFailure error
match verify_auth_inner(request) {
Ok(ad) => Ok(ad),
Err(e) => match e.inner {
ChorusError::BlossomAuthFailure(_) => Err(e),
_ => Err(ChorusError::BlossomAuthFailure(format!("{e}")).into()),
},
}
}
fn verify_auth_inner(request: &Request<Incoming>) -> Result<AuthData, Error> {
// Must have AUTHORIZATION header
let authz = match request.headers().get(AUTHORIZATION) {
Some(h) => h,
None => return s_err("Authorization Required"),
};
// Authorization header must be type "nostr"
if !authz.to_str()?.to_ascii_lowercase().starts_with("nostr ") {
return s_err("You must use the Nostr authorization scheme");
}
let base64 = match authz.to_str()?.get(6..) {
Some(x) => x,
None => return s_err("Missing auth base64 encoded event"),
};
// Authorization header must be base64
let event_bytes = BASE64_STANDARD.decode(base64)?;
// Authorization header base64 must decode to a nostr Event
let mut buffer = vec![0; base64.len()];
let (_size, event) = Event::from_json(&event_bytes, &mut buffer)?;
// Nostr event must be valid
if let Err(e) = event.verify() {
return s_err(&format!("Authorization event is invalid: {}", e));
}
// Nostr event must be signed by a chorus user
if !crate::is_authorized_user(event.pubkey()) {
return s_err("You are not an authorized user");
}
// Event kind must be 24242
if event.kind().as_u16() != 24242 {
return s_err("Authorization event not kind 24242");
}
// Event created_at must be in the past (we give 30 seconds leeway)
use pocket_types::Time;
let now = Time::now();
if event.created_at() > now + 30 {
return s_err("Authorization event too far in the future");
}
let tags = event.tags()?;
// Expiration tag must be in the future
if let Some(v) = tags.get_value(b"expiration") {
let u = parse_u64(v)?;
let expiration = Time::from_u64(u);
if expiration < now {
return s_err("Authorization event has expired");
}
} else {
return s_err("Authorization event missing expiration tag");
}
// We let the caller check the verb and hash since those are specific
// to the endpoint (and the 'x' must be checked later on)
let verb: Option<AuthVerb> = if let Some(t) = tags.get_value(b"t") {
if t == b"upload" {
Some(AuthVerb::Upload)
} else if t == b"list" {
Some(AuthVerb::List)
} else if t == b"delete" {
Some(AuthVerb::Delete)
} else {
None
}
} else {
None
};
let hash: Option<[u8; 32]> = if let Some(v) = tags.get_value(b"x") {
let vec = hex::decode(v)?;
if vec.len() == 32 {
Some(vec.try_into().unwrap())
} else {
return s_err("Authorization event x tag is of the wrong length");
}
} else {
None
};
Ok(AuthData { verb, hash })
}
// FIXME, expose these from pocket-types
fn parse_u64(input: &[u8]) -> Result<u64, Error> {
let mut pos = 0;
let mut value: u64 = 0;
let mut any: bool = false;
while pos < input.len() && b"0123456789".contains(&input[pos]) {
any = true;
value = (value * 10) + (input[pos] - 48) as u64;
pos += 1;
}
if !any {
Err(ChorusError::General("Auth event expiration is not a number".to_string()).into())
} else {
Ok(value)
}
}

360
src/web/blossom/mod.rs
View File

@ -1,360 +0,0 @@
use crate::error::{ChorusError, Error};
use crate::filestore::HashOutput;
use crate::globals::GLOBALS;
use http::header::{
ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS, ACCESS_CONTROL_ALLOW_ORIGIN,
ACCESS_CONTROL_REQUEST_HEADERS, ACCESS_CONTROL_REQUEST_METHOD, ALLOW, CONTENT_LENGTH,
CONTENT_TYPE, ETAG, IF_MATCH, IF_MODIFIED_SINCE, IF_NONE_MATCH, ORIGIN, WWW_AUTHENTICATE,
};
use http::{Method, StatusCode};
//ACCEPT, AUTHORIZATION, DATE, ETAG, ORIGIN
use http_body_util::combinators::BoxBody;
use http_body_util::{BodyExt, Empty, Full};
use hyper::body::{Bytes, Incoming};
use hyper::{Request, Response};
use serde::{Deserialize, Serialize};
mod auth;
use auth::{verify_auth, AuthVerb};
pub async fn handle(request: Request<Incoming>) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
match route(request).await {
Ok(response) => Ok(response),
Err(e) => match e.inner {
ChorusError::SignalNotBlossom => Err(e),
_ => error_response(e),
},
}
}
pub async fn route(request: Request<Incoming>) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
let p = request.uri().path();
#[allow(clippy::int_plus_one)]
if p.starts_with("/")
&& p.len() >= 1 + 64
&& p.chars().skip(1).take(64).all(|c| c.is_ascii_hexdigit())
{
handle_hash(request).await
} else if p == "/upload" {
handle_upload(request).await
} else if p.starts_with("/list/")
&& p.len() >= 6 + 64
&& p.chars().skip(6).take(64).all(|c| c.is_ascii_hexdigit())
{
handle_list(request).await
} else if p == "/mirror" {
handle_mirror(request).await
} else {
Err(ChorusError::SignalNotBlossom.into())
}
}
fn error_response(e: Error) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
use std::io::ErrorKind;
let mut response = Response::builder().header(ACCESS_CONTROL_ALLOW_ORIGIN, "*");
let (status, reason) = match e.inner {
ChorusError::BlossomAuthFailure(m) => {
response = response.header(WWW_AUTHENTICATE, "Nostr");
(StatusCode::UNAUTHORIZED, m)
}
ChorusError::FromHex(_) => (StatusCode::BAD_REQUEST, format!("{e}")),
ChorusError::Io(ref ioerror) => match ioerror.kind() {
ErrorKind::NotFound => (StatusCode::NOT_FOUND, "Not Found".to_owned()),
_ => (StatusCode::INTERNAL_SERVER_ERROR, format!("{e}")),
},
_ => (StatusCode::INTERNAL_SERVER_ERROR, format!("{e}")),
};
Ok(response
.header("X-Reason", reason)
.status(status)
.body(Empty::new().map_err(|e| e.into()).boxed())?)
}
fn options_response(
request: Request<Incoming>,
methods: &str,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
if request
.headers()
.contains_key(ACCESS_CONTROL_REQUEST_HEADERS)
|| request
.headers()
.contains_key(ACCESS_CONTROL_REQUEST_METHOD)
|| request.headers().contains_key(ORIGIN)
{
// CORS OPTIONS response
Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(ACCESS_CONTROL_ALLOW_HEADERS, "Authorization, *")
.header(ACCESS_CONTROL_ALLOW_METHODS, methods)
.header(CONTENT_LENGTH, "0")
.status(StatusCode::OK)
.body(Empty::new().map_err(|e| e.into()).boxed())?)
} else {
// Normal OPTIONS response
Ok(Response::builder()
.header(ALLOW, methods)
.status(StatusCode::NO_CONTENT)
.body(Empty::new().map_err(|e| e.into()).boxed())?)
}
}
pub async fn handle_hash(
request: Request<Incoming>,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
if matches!(request.method(), &Method::OPTIONS) {
return options_response(request, "OPTIONS, HEAD, GET, DELETE");
}
// HEAD, GET, DELETE
let p = request.uri().path();
let hashstr: String = p.chars().skip(1).take(64).collect();
let hash = match HashOutput::from_hex(&hashstr) {
Ok(h) => h,
Err(e) => return error_response(e),
};
let metadata = GLOBALS.filestore.get().unwrap().metadata(hash).await?;
match *request.method() {
Method::HEAD | Method::GET => {
// Honor If-Match (fail if they didn't specify an etag matching the hash)
if let Some(etags) = request.headers().get(IF_MATCH) {
let mut onematch: bool = false;
for part in etags.to_str()?.split(',') {
if part[1..part.len() - 1] == format!("{}", hash) {
onematch = true;
}
}
if !onematch {
return Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(ETAG, format!("\"{}\"", hash))
.status(StatusCode::PRECONDITION_FAILED)
.body(Empty::new().map_err(|e| e.into()).boxed())?);
}
}
// Honor If-None-Match (send NOT_MODIFIED if they specified an etag matching the hash)
// Honor If-Modified-Since (always send NOT_MODIFIED)
let mut send_not_modified: bool = false;
if let Some(etags) = request.headers().get(IF_NONE_MATCH) {
for part in etags.to_str()?.split(',') {
if part[1..part.len() - 1] == format!("{}", hash) {
send_not_modified = true;
}
}
}
if request.headers().get(IF_MODIFIED_SINCE).is_some() {
send_not_modified = true;
}
if send_not_modified {
return Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(ETAG, format!("\"{}\"", hash))
.status(StatusCode::NOT_MODIFIED)
.body(Empty::new().map_err(|e| e.into()).boxed())?);
}
// Normal reasponse (HEAD or GET)
let response = Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, format!("{}", metadata.len()))
.header(ETAG, format!("\"{}\"", hash))
.status(StatusCode::OK);
if matches!(*request.method(), Method::GET) {
let body = GLOBALS.filestore.get().unwrap().retrieve(hash).await?;
Ok(response.body(body)?)
} else {
Ok(response.body(Empty::new().map_err(|e| e.into()).boxed())?)
}
}
Method::DELETE => {
let auth_data = verify_auth(&request)?;
if auth_data.verb != Some(AuthVerb::Delete) {
return Err(ChorusError::BlossomAuthFailure(
"Delete was not authorized".to_string(),
)
.into());
}
GLOBALS.filestore.get().unwrap().delete(hash).await?;
Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::OK)
.body(Empty::new().map_err(|e| e.into()).boxed())?)
}
_ => Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::METHOD_NOT_ALLOWED)
.body(Empty::new().map_err(|e| e.into()).boxed())?),
}
}
pub async fn handle_upload(
request: Request<Incoming>,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
if matches!(request.method(), &Method::OPTIONS) {
return options_response(request, "OPTIONS, HEAD, PUT");
}
let auth_data = verify_auth(&request)?;
if auth_data.verb != Some(AuthVerb::Upload) {
return Err(
ChorusError::BlossomAuthFailure("Upload was not authorized".to_string()).into(),
);
}
match *request.method() {
Method::HEAD => Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::NOT_IMPLEMENTED)
.body(Empty::new().map_err(|e| e.into()).boxed())?),
Method::PUT => {
let expected_hash = auth_data.hash.map(HashOutput::from_bytes);
if expected_hash.is_none() {
return Err(ChorusError::BlossomAuthFailure(
"Put requires an expected hash value x tag in the authorization event"
.to_string(),
)
.into());
}
let uri = request.uri().to_owned();
let maybe_content_type = match request.headers().get(http::header::CONTENT_TYPE) {
Some(s) => match s.to_str() {
Ok(s) => Some(s.to_owned()),
Err(_) => None,
},
None => None,
};
let (size, hash, maybe_sniffed_mime_string) = GLOBALS
.filestore
.get()
.unwrap()
.store(
request.into_body().map_err(|e| e.into()).boxed(),
expected_hash,
)
.await?;
let extension = {
let mut mime_string: String = "".to_owned();
if let Some(ms) = maybe_content_type {
mime_string = ms.to_owned();
} else if let Some(ms) = maybe_sniffed_mime_string {
mime_string = ms.to_owned();
}
mime2ext::mime2ext(&mime_string).unwrap_or("blob")
};
let uri = {
let mut parts = GLOBALS.config.read().uri_parts(uri, true)?;
parts.path_and_query = Some(http::uri::PathAndQuery::from_maybe_shared(format!(
"/{}.{}",
hash, extension
))?);
http::Uri::from_parts(parts)?
};
let blob_descriptor = BlobDescriptor {
url: format!("{}", uri),
sha256: format!("{}", hash),
size,
uploaded: pocket_types::Time::now().as_u64(),
};
let descriptor_json_string = serde_json::to_string(&blob_descriptor)?;
let body_bytes = descriptor_json_string.into_bytes();
let len = body_bytes.len();
let body = Full::new(Bytes::from(body_bytes))
.map_err(|e| e.into())
.boxed();
Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, format!("{}", len))
.header(CONTENT_TYPE, "application/json")
.status(StatusCode::OK)
.body(body)?)
}
_ => Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::METHOD_NOT_ALLOWED)
.body(Empty::new().map_err(|e| e.into()).boxed())?),
}
}
pub async fn handle_list(
request: Request<Incoming>,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
if matches!(request.method(), &Method::OPTIONS) {
return options_response(request, "OPTIONS, GET");
}
let auth_data = verify_auth(&request)?;
if auth_data.verb != Some(AuthVerb::List) {
return Err(ChorusError::BlossomAuthFailure("List was not authorized".to_string()).into());
}
match *request.method() {
Method::GET => Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::NOT_IMPLEMENTED)
.body(Empty::new().map_err(|e| e.into()).boxed())?),
_ => Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::METHOD_NOT_ALLOWED)
.body(Empty::new().map_err(|e| e.into()).boxed())?),
}
}
pub async fn handle_mirror(
request: Request<Incoming>,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
if matches!(request.method(), &Method::OPTIONS) {
return options_response(request, "OPTIONS, PUT");
}
let auth_data = verify_auth(&request)?;
if auth_data.verb != Some(AuthVerb::Mirror) {
return Err(
ChorusError::BlossomAuthFailure("Mirror was not authorized".to_string()).into(),
);
}
match *request.method() {
Method::PUT => Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::NOT_IMPLEMENTED)
.body(Empty::new().map_err(|e| e.into()).boxed())?),
_ => Ok(Response::builder()
.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
.header(CONTENT_LENGTH, "0")
.status(StatusCode::METHOD_NOT_ALLOWED)
.body(Empty::new().map_err(|e| e.into()).boxed())?),
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BlobDescriptor {
pub url: String,
pub sha256: String,
pub size: u64,
// type: String
pub uploaded: u64,
}

116
src/web/management/auth.rs
View File

@ -1,116 +0,0 @@
use crate::error::{ChorusError, Error};
use crate::globals::GLOBALS;
use base64::prelude::*;
use http::header::AUTHORIZATION;
use http_body_util::BodyExt;
use hyper::body::Incoming;
use hyper::Request;
use pocket_types::{Event, Pubkey};
use secp256k1::hashes::{sha256, Hash};
use serde_json::Value;
fn s_err(s: &str) -> Result<(Pubkey, Value), Error> {
Err(ChorusError::ManagementAuthFailure(s.to_owned()).into())
}
pub async fn check_auth(request: Request<Incoming>) -> Result<(Pubkey, Value), Error> {
// Must be POST
if request.method() != hyper::Method::POST {
return s_err("Management RPC only supports POST method");
}
// Must have AUTHORIZATION header
let authz = match request.headers().get(AUTHORIZATION) {
Some(h) => h,
None => return s_err("Authorization header not of type nostr"),
};
// Authorization header must be type "nostr"
let value = String::from_utf8(authz.as_bytes().to_owned())?;
let mut parts = value.split(' ');
match parts.next() {
Some(s) => {
if s.to_lowercase() != "nostr" {
return s_err("Authorization header not of type nostr");
}
}
None => return s_err("Authorization header missing"),
}
// Authorization header second part
let base64event = match parts.next() {
Some(s) => s,
None => return s_err("Authorization header incomplete"),
};
// Authorization header must be base64
let event_bytes = BASE64_STANDARD.decode(base64event)?;
// Authorization header base64 must decode to a nostr Event
let mut buffer = vec![0; base64event.len()];
let (_size, event) = Event::from_json(&event_bytes, &mut buffer)?;
// Nostr event must be valid
if let Err(e) = event.verify() {
return s_err(&format!("Authorization event is invalid: {}", e));
}
// Nostr event must be signed by a moderator
if !crate::is_moderator(event.pubkey()) {
return s_err("Authorization failed as user is not a moderator");
}
// Event kind must be 27235
if event.kind().as_u16() != 27235 {
return s_err("Authorization event not kind 27235");
}
// Event created_at must be within 60 seconds of now
use pocket_types::Time;
let now = Time::now().as_u64();
if event.created_at().as_u64() > now + 60 {
return s_err("Authorization event too far in the future");
}
if event.created_at().as_u64() < now - 60 {
return s_err("Authorization event too far in the past");
}
let tags = event.tags()?;
// Tag 'u' must be the current URL
if let Some(u) = tags.get_value(b"u") {
let auth_url = String::from_utf8(u.to_owned())?;
let actual_url = {
let uri_parts = GLOBALS
.config
.read()
.uri_parts(request.uri().to_owned(), true)?;
let uri = http::Uri::from_parts(uri_parts)?;
format!("{}", uri)
};
if actual_url != auth_url {
return s_err(&format!(
"Authorization event URL {} does not match requqest URL {}",
auth_url, actual_url
));
}
} else {
return s_err("Authorization event URL missing");
}
let body = request.collect().await?.to_bytes();
let hash = sha256::Hash::hash(&body);
let hashref = <sha256::Hash as AsRef<[u8]>>::as_ref(&hash);
let hashrefhex = hex::encode(hashref);
if let Some(payload) = tags.get_value(b"payload") {
if hashrefhex.as_bytes() != payload {
return s_err("Authorization failed: body hash mismatch");
}
} else {
return s_err("Authorization event payload missing");
}
Ok((event.pubkey(), serde_json::from_slice(&body)?))
}

615
src/web/management/mod.rs
View File

@ -1,615 +0,0 @@
use crate::error::{ChorusError, Error};
use crate::globals::GLOBALS;
use crate::ip::HashedPeer;
use http_body_util::combinators::BoxBody;
use http_body_util::{BodyExt, Full};
use hyper::body::{Bytes, Incoming};
use hyper::{Request, Response, StatusCode};
use pocket_db::ScreenResult;
use pocket_types::{Event, Filter, Id, Kind, Pubkey};
use serde::Serialize;
use serde_json::{json, Map, Value};
mod auth;
#[derive(Serialize)]
struct EventResult {
id: String,
#[serde(skip_serializing_if = "Option::is_none")]
reason: Option<String>,
}
#[derive(Serialize)]
struct FullEventResult {
event: String,
#[serde(skip_serializing_if = "Option::is_none")]
reason: Option<String>,
}
#[derive(Serialize)]
struct PubkeyResult {
pubkey: String,
#[serde(skip_serializing_if = "Option::is_none")]
reason: Option<String>,
}
fn respond(
json: serde_json::Value,
status: StatusCode,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
let s: String = serde_json::to_string(&json)?;
let response = Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header("Access-Control-Allow-Methods", "*")
.header("Content-Type", "application/nostr+json+rpc")
.status(status)
.body(
Full::new(s.into_bytes().into())
.map_err(|e| e.into())
.boxed(),
)?;
Ok(response)
}
pub async fn handle(
_peer: HashedPeer,
request: Request<Incoming>,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
let (pubkey, command) = match auth::check_auth(request).await {
Ok((pk, v)) => (pk, v),
Err(e) => {
let result = json!({
"result": {},
"error": format!("{}", e)
});
return respond(result, StatusCode::UNAUTHORIZED);
}
};
match handle_inner(pubkey, command) {
Ok(Some(value)) => respond(value, StatusCode::OK),
Ok(None) => {
let result = json!({
"result": {},
});
respond(result, StatusCode::OK)
}
Err(e) => {
let (result, status) = match e.inner {
ChorusError::BadRequest(s) => (
json!({
"result": {},
"error": format!("{}", s)
}),
StatusCode::BAD_REQUEST,
),
ChorusError::NotImplemented => (
json!({
"result": {},
"error": "not_implemented"
}),
StatusCode::NOT_IMPLEMENTED,
),
_ => (
json!({
"result": {},
"error": format!("{}", e)
}),
StatusCode::INTERNAL_SERVER_ERROR,
),
};
respond(result, status)
}
}
}
pub fn handle_inner(pubkey: Pubkey, command: Value) -> Result<Option<Value>, Error> {
let obj = match command.as_object() {
Some(o) => o,
None => return Err(ChorusError::BadRequest("Command was not a JSON object").into()),
};
let method = match obj.get("method") {
Some(m) => match m.as_str() {
Some(s) => s.to_owned(),
None => return Err(ChorusError::BadRequest("Method not a string").into()),
},
None => return Err(ChorusError::BadRequest("Method missing").into()),
};
match &*method {
"supportedmethods" => Ok(Some(json!({
"result": [
"supportedmethods",
"listeventsneedingmoderation",
"allowevent",
"banevent",
"clearevent",
"removeevent",
"allowpubkey",
"banpubkey",
"clearpubkey",
"listallowedevents",
"listbannedevents",
"fetchbannedevents",
"listallowedpubkeys",
"listbannedpubkeys",
"stats",
"numconnections",
"uptime",
"listadmins",
"listmoderators",
"grantmoderator",
"revokemoderator",
"listusers",
"grantuser",
"revokeuser",
"listrole",
"grantrole",
"revokerole",
]
}))),
"listeventsneedingmoderation" => {
// FIXME this scans the entire database, maybe we need to some process
// that does this in epochs and saves the result.
let allowed_kinds = [
Kind::from(4), // Encrypted Direct Message
Kind::from(1059), // Giftwrap
Kind::from(10002), // Relay list
Kind::from(10050), // DM Relay list
Kind::from(0), // Metadata
Kind::from(3), // Following list
Kind::from(7), // Reaction
];
let mut buffer: [u8; 128] = [0; 128];
let filter = {
let (_incount, _outcount, filter) = Filter::from_json(b"{}", &mut buffer)?;
filter
};
let screen = |e: &Event| -> ScreenResult {
if allowed_kinds.contains(&e.kind())
|| e.kind().is_ephemeral()
|| crate::is_authorized_user(e.pubkey())
{
ScreenResult::Mismatch
} else {
ScreenResult::Match
}
};
let mut need_moderation: Vec<EventResult> = Vec::new();
let (mut events, _redacted) = GLOBALS
.store
.get()
.unwrap()
.find_events(filter, true, 0, 0, screen)?;
for event in events.drain(..) {
// Skip if pubkey marked (either banned or approved)
if matches!(crate::get_pubkey_approval(event.pubkey()), Ok(Some(_))) {
continue;
}
// Skip if event marked (either banned or approved)
if matches!(crate::get_event_approval(event.id()), Ok(Some(_))) {
continue;
}
need_moderation.push(EventResult {
id: event.id().as_hex_string(),
reason: Some("unmoderated".to_string()),
});
}
Ok(Some(json!({
"result": need_moderation,
})))
}
"allowevent" => {
let id = get_id_param(obj)?;
crate::mark_event_approval(id, true)?;
Ok(None)
}
"banevent" => {
let id = get_id_param(obj)?;
crate::mark_event_approval(id, false)?;
Ok(None)
}
"clearevent" => {
let id = get_id_param(obj)?;
crate::clear_event_approval(id)?;
Ok(None)
}
"removeevent" => {
let id = get_id_param(obj)?;
GLOBALS.store.get().unwrap().remove_event(id)?;
Ok(None)
}
"allowpubkey" => {
let pk = get_pubkey_param(obj)?;
crate::mark_pubkey_approval(pk, true)?;
Ok(None)
}
"banpubkey" => {
let pk = get_pubkey_param(obj)?;
crate::mark_pubkey_approval(pk, false)?;
Ok(None)
}
"clearpubkey" => {
let pk = get_pubkey_param(obj)?;
crate::clear_pubkey_approval(pk)?;
Ok(None)
}
"listallowedevents" => {
let approvals = crate::dump_event_approvals()?;
let ids: Vec<EventResult> = approvals
.iter()
.filter_map(|(id, appr)| {
if *appr {
Some(EventResult {
id: id.as_hex_string(),
reason: None,
})
} else {
None
}
})
.collect();
Ok(Some(json!({
"result": ids
})))
}
"listbannedevents" => {
let approvals = crate::dump_event_approvals()?;
let ids: Vec<EventResult> = approvals
.iter()
.filter_map(|(id, appr)| {
if *appr {
None
} else {
Some(EventResult {
id: id.as_hex_string(),
reason: None,
})
}
})
.collect();
Ok(Some(json!({
"result": ids
})))
}
"fetchbannedevents" => {
let approvals = crate::dump_event_approvals()?;
let mut results: Vec<FullEventResult> = Vec::new();
for (id, appr) in approvals.iter() {
if !*appr {
if let Some(event) = GLOBALS.store.get().unwrap().get_event_by_id(*id)? {
results.push(FullEventResult {
event: format!("{event}"),
reason: None,
});
}
}
}
Ok(Some(json!({
"result": results
})))
}
"listallowedpubkeys" => {
let approvals = crate::dump_pubkey_approvals()?;
let pubkeys: Vec<PubkeyResult> = approvals
.iter()
.filter_map(|(pk, appr)| {
if *appr {
Some(PubkeyResult {
pubkey: pk.as_hex_string(),
reason: None,
})
} else {
None
}
})
.collect();
Ok(Some(json!({
"result": pubkeys
})))
}
"listbannedpubkeys" => {
let approvals = crate::dump_pubkey_approvals()?;
let pubkeys: Vec<PubkeyResult> = approvals
.iter()
.filter_map(|(pk, appr)| {
if *appr {
None
} else {
Some(PubkeyResult {
pubkey: pk.as_hex_string(),
reason: None,
})
}
})
.collect();
Ok(Some(json!({
"result": pubkeys
})))
}
"stats" => {
let store_stats = GLOBALS.store.get().unwrap().stats()?;
Ok(Some(json!({
"result": {
"uptime": GLOBALS.start_time.elapsed().as_secs(),
"num_connections": &GLOBALS.num_connections,
"bytes_received": &GLOBALS.bytes_inbound,
"bytes_sent": &GLOBALS.bytes_outbound,
"event_bytes": store_stats.event_bytes,
"num_events": store_stats.index_stats.i_index_entries,
"index_disk_usage": store_stats.index_stats.disk_usage,
"index_memory_usage": store_stats.index_stats.memory_usage,
}
})))
}
"numconnections" => {
let num = &GLOBALS.num_connections;
Ok(Some(json!({
"result": num,
})))
}
"uptime" => {
let uptime_in_secs = GLOBALS.start_time.elapsed().as_secs();
Ok(Some(json!({
"result": uptime_in_secs,
})))
}
"listadmins" => {
let keys = GLOBALS.config.read().admin_hex_keys.clone();
Ok(Some(json!({
"result": keys
})))
}
"listmoderators" => {
let moderators: Vec<String> = crate::dump_authorized_users()?
.iter()
.filter_map(|(pk, moderator)| {
if *moderator {
Some(pk.as_hex_string())
} else {
None
}
})
.collect();
Ok(Some(json!({
"result": moderators
})))
}
"grantmoderator" => {
if !crate::is_admin(pubkey) {
Ok(Some(json!({
"result": {},
"error": "Unauthorized: Only admins can grant moderator status"
})))
} else {
let pk = get_pubkey_param(obj)?;
crate::add_authorized_user(pk, true)?;
Ok(None)
}
}
"revokemoderator" => {
if !crate::is_admin(pubkey) {
Ok(Some(json!({
"result": {},
"error": "Unauthorized: Only admins can revoke moderator status"
})))
} else {
let pk = get_pubkey_param(obj)?;
// Do not do this if they aren't already an authorized user
if !crate::is_authorized_user(pk) {
Ok(None)
} else {
crate::add_authorized_user(pk, false)?;
Ok(None)
}
}
}
"listusers" => {
let users: Vec<String> = crate::dump_authorized_users()?
.iter()
.map(|(pk, _moderator)| pk.as_hex_string())
.collect();
Ok(Some(json!({
"result": users
})))
}
"grantuser" => {
if !crate::is_admin(pubkey) {
Ok(Some(json!({
"result": {},
"error": "Unauthorized: Only admins can grant user status"
})))
} else {
let pk = get_pubkey_param(obj)?;
crate::add_authorized_user(pk, false)?;
Ok(None)
}
}
"revokeuser" => {
if !crate::is_admin(pubkey) {
Ok(Some(json!({
"result": {},
"error": "Unauthorized: Only admins can revoke user status"
})))
} else {
let pk = get_pubkey_param(obj)?;
crate::rm_authorized_user(pk)?;
Ok(None)
}
}
"listrole" => {
if !crate::is_admin(pubkey) {
Ok(Some(json!({
"result": {},
"error": "Unauthorized: Only admins can revoke user status"
})))
} else {
let role = get_string_param(obj)?;
match &*role {
"admin" => {
let keys = GLOBALS.config.read().admin_hex_keys.clone();
Ok(Some(json!({
"result": keys
})))
}
"user" => {
let users: Vec<String> = crate::dump_authorized_users()?
.iter()
.map(|(pk, _moderator)| pk.as_hex_string())
.collect();
Ok(Some(json!({
"result": users
})))
}
"moderator" => {
let moderators: Vec<String> = crate::dump_authorized_users()?
.iter()
.filter_map(|(pk, moderator)| {
if *moderator {
Some(pk.as_hex_string())
} else {
None
}
})
.collect();
Ok(Some(json!({
"result": moderators
})))
}
_ => Ok(Some(json!({
"result": {},
"error": "Unknown role."
}))),
}
}
}
"grantrole" => {
if !crate::is_admin(pubkey) {
Ok(Some(json!({
"result": {},
"error": "Unauthorized: Only admins can grant user status"
})))
} else {
let role = get_string_param(obj)?;
match &*role {
"admin" => Ok(Some(json!({
"result": {},
"error": "The administrators cannot be managed via this interface."
}))),
"user" => {
let pk = get_pubkey_param(obj)?;
crate::add_authorized_user(pk, false)?;
Ok(None)
}
"moderator" => {
let pk = get_pubkey_param(obj)?;
crate::add_authorized_user(pk, true)?;
Ok(None)
}
_ => Ok(Some(json!({
"result": {},
"error": "Unknown role."
}))),
}
}
}
"revokerole" => {
if !crate::is_admin(pubkey) {
Ok(Some(json!({
"result": {},
"error": "Unauthorized: Only admins can grant user status"
})))
} else {
let role = get_string_param(obj)?;
match &*role {
"admin" => Ok(Some(json!({
"result": {},
"error": "The administrators cannot be managed via this interface."
}))),
"user" => {
let pk = get_pubkey_param(obj)?;
crate::rm_authorized_user(pk)?;
Ok(None)
}
"moderator" => {
let pk = get_pubkey_param(obj)?;
// Do not do this if they aren't already an authorized user
if !crate::is_authorized_user(pk) {
Ok(None)
} else {
crate::add_authorized_user(pk, false)?;
Ok(None)
}
}
_ => Ok(Some(json!({
"result": {},
"error": "Unknown role."
}))),
}
}
}
_ => Err(ChorusError::NotImplemented.into()),
}
}
fn get_pubkey_param(obj: &Map<String, Value>) -> Result<Pubkey, Error> {
let pubkey_text = obj
.get("params")
.ok_or(ChorusError::BadRequest("Params field missing").into_err())?
.as_array()
.ok_or(ChorusError::BadRequest("Params not an array").into_err())?
.first()
.ok_or(ChorusError::BadRequest("Missing pubkey parameter").into_err())?
.as_str()
.ok_or(ChorusError::BadRequest("Pubkey parameter is wrong type").into_err())?;
Pubkey::read_hex(pubkey_text.as_bytes())
.map_err(|_| ChorusError::BadRequest("Pubkey could not be parsed").into_err())
}
fn get_id_param(obj: &Map<String, Value>) -> Result<Id, Error> {
let id_text = obj
.get("params")
.ok_or(ChorusError::BadRequest("Params field missing").into_err())?
.as_array()
.ok_or(ChorusError::BadRequest("Params not an array").into_err())?
.first()
.ok_or(ChorusError::BadRequest("Missing ID parameter").into_err())?
.as_str()
.ok_or(ChorusError::BadRequest("ID parameter is wrong type").into_err())?;
Id::read_hex(id_text.as_bytes())
.map_err(|_| ChorusError::BadRequest("ID could not be parsed").into_err())
}
fn get_string_param(obj: &Map<String, Value>) -> Result<String, Error> {
Ok(obj
.get("params")
.ok_or(ChorusError::BadRequest("Params field missing").into_err())?
.as_array()
.ok_or(ChorusError::BadRequest("Params not an array").into_err())?
.first()
.ok_or(ChorusError::BadRequest("Missing parameter").into_err())?
.as_str()
.ok_or(ChorusError::BadRequest("Parameter is not a string as expected").into_err())?
.to_owned())
}

133
src/web/mod.rs
View File

@ -1,133 +0,0 @@
mod blossom;
mod management;
mod nip11;
use crate::error::{ChorusError, Error};
use crate::globals::GLOBALS;
use crate::ip::HashedPeer;
use http::Method;
use http_body_util::combinators::BoxBody;
use http_body_util::{BodyExt, Empty, Full};
use hyper::body::{Bytes, Incoming};
use hyper::{Request, Response, StatusCode};
pub async fn serve_http(
peer: HashedPeer,
request: Request<Incoming>,
) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
let method = request.method().clone();
// Handle server-wide OPTIONS requests
let p = request.uri().path();
if p == "*" && request.method() == Method::OPTIONS {
let response = Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header(
"Access-Control-Allow-Methods",
"OPTIONS, GET, HEAD, PUT, DELETE",
)
.header("Allow", "OPTIONS, GET, HEAD, PUT, DELETE")
.status(StatusCode::OK)
.body(Empty::new().map_err(|e| e.into()).boxed())?;
return Ok(response);
}
// Check if it is a NIP-11 request
if let Some(accept) = request.headers().get("Accept") {
if let Ok(s) = accept.to_str() {
if s == "application/nostr+json" {
return nip11::serve_nip11(peer).await;
}
}
}
// Check if it is a NIP-86 Relay Management request
if let Some(content_type) = request.headers().get("Content-Type") {
if let Ok(s) = content_type.to_str() {
if s == "application/nostr+json+rpc" {
return management::handle(peer, request).await;
}
}
}
let uri = request.uri().to_owned();
if p == "/privacy-policy" {
let config = &*GLOBALS.config.read();
if let Some(pp) = &config.privacy_policy {
let response = Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header("Access-Control-Allow-Methods", "*")
.header("Allow", "OPTIONS, GET, HEAD")
.header("Content-Type", "text/plain")
.status(StatusCode::OK)
.body(Full::new(pp.clone().into()).map_err(|e| e.into()).boxed())?;
return Ok(response);
}
}
if p == "/terms-of-service" {
let config = &*GLOBALS.config.read();
if let Some(tos) = &config.terms_of_service {
let response = Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header("Access-Control-Allow-Methods", "*")
.header("Allow", "OPTIONS, GET, HEAD")
.header("Content-Type", "text/plain")
.status(StatusCode::OK)
.body(Full::new(tos.clone().into()).map_err(|e| e.into()).boxed())?;
return Ok(response);
}
}
// Try blossom if enabled
if GLOBALS.config.read().blossom_directory.is_some() {
match blossom::handle(request).await {
Ok(response) => return Ok(response),
Err(e) => {
if !matches!(e.inner, ChorusError::SignalNotBlossom) {
return Err(e);
}
}
}
}
log::debug!(target: "Client", "{}: HTTP request for {}", peer, uri);
let response = match method {
Method::OPTIONS => Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header("Access-Control-Allow-Methods", "*")
.header("Allow", "OPTIONS, GET, HEAD, PUT, DELETE")
.status(StatusCode::NO_CONTENT)
.body(Empty::new().map_err(|e| e.into()).boxed())?,
Method::HEAD => Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header("Access-Control-Allow-Methods", "*")
.status(StatusCode::OK)
.body(Empty::new().map_err(|e| e.into()).boxed())?,
Method::GET => Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header("Access-Control-Allow-Methods", "*")
.status(StatusCode::OK)
.body(
Full::new("This is a nostr relay. Please use a nostr client to connect.".into())
.map_err(|e| e.into())
.boxed(),
)?,
_ => Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "Authorization, *")
.header("Access-Control-Allow-Methods", "*")
.status(StatusCode::METHOD_NOT_ALLOWED)
.body(Empty::new().map_err(|e| e.into()).boxed())?,
};
Ok(response)
}

174
src/web/nip11.rs
View File

@ -1,174 +0,0 @@
use crate::config::Config;
use crate::error::Error;
use crate::globals::GLOBALS;
use crate::ip::HashedPeer;
use http_body_util::combinators::BoxBody;
use http_body_util::{BodyExt, Full};
use hyper::body::Bytes;
use hyper::http::uri::Uri;
use hyper::{Response, StatusCode};
pub async fn serve_nip11(peer: HashedPeer) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
log::debug!(target: "Client", "{}: sent NIP-11", peer);
let rid = {
let config = &*GLOBALS.config.read();
GLOBALS.rid.get_or_init(|| build_rid(config))
};
let response = Response::builder()
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Headers", "*")
.header("Access-Control-Allow-Methods", "*")
.header("Content-Type", "application/nostr+json")
.status(StatusCode::OK)
.body(Full::new(rid.clone().into()).map_err(|e| e.into()).boxed())?;
Ok(response)
}
fn build_rid(config: &Config) -> String {
let mut rid: String = String::with_capacity(255);
const SUPPORTED_NIPS: [u8; 9] = [
1, // nostr
4, // DMs
9, // Event Deletion
11, // relay information document
40, // Expiration Timestamp
42, // AUTH
45, // Counting results
59, // GiftWrap
65, // Relay List Metadata
];
const _UNSUPPORTED_NIPS: [u8; 5] = [
26, // Delegated Event Signing
29, // Relay-based Groups
50, // SEARCH
94, // File Metadata
96, // HTTP File Storage Integration
];
const _INAPPLICABLE_NIPS: [u8; 45] = [
2, 3, 5, 6, 7, 8, 10, 13, 14, 15, 18, 19, 21, 23, 24, 25, 27, 28, 30, 31, 32, 34, 36, 38,
39, 44, 46, 47, 48, 49, 51, 52, 53, 56, 57, 58, 72, 75, 78, 84, 89, 90, 92, 98, 99,
];
let s = SUPPORTED_NIPS
.iter()
.map(|i| format!("{}", i))
.collect::<Vec<String>>()
.join(",");
rid.push_str(&format!("{{\"supported_nips\":[{}],", s));
let software = env!("CARGO_PKG_NAME");
rid.push_str("\"software\":\"");
rid.push_str(software);
rid.push('\"');
let version = env!("CARGO_PKG_VERSION");
rid.push(',');
rid.push_str("\"version\":\"");
rid.push_str(version);
rid.push('\"');
if let Some(name) = &config.name {
rid.push(',');
rid.push_str("\"name\":\"");
rid.push_str(name);
rid.push('\"');
}
if let Some(description) = &config.description {
rid.push(',');
rid.push_str("\"description\":\"");
rid.push_str(description);
rid.push('\"');
}
if let Some(banner_url) = &config.banner_url {
rid.push(',');
rid.push_str("\"banner\":\"");
rid.push_str(banner_url);
rid.push('\"');
}
if let Some(icon_url) = &config.icon_url {
rid.push(',');
rid.push_str("\"icon\":\"");
rid.push_str(icon_url);
rid.push('\"');
}
if let Some(pubkey) = &config.contact_public_key {
let mut pkh: [u8; 64] = [0; 64];
pubkey.write_hex(&mut pkh).unwrap();
rid.push(',');
rid.push_str("\"pubkey\":\"");
rid.push_str(unsafe { std::str::from_utf8_unchecked(pkh.as_slice()) });
rid.push('\"');
}
if let Some(contact) = &config.contact {
rid.push(',');
rid.push_str("\"contact\":\"");
rid.push_str(contact);
rid.push('\"');
}
if config.privacy_policy.is_some() {
rid.push(',');
rid.push_str("\"privacy_policy\":\"");
let url = match config.uri_parts(
Uri::from_static("https://authority-will-be-replaced/privacy-policy"),
true,
) {
Ok(parts) => match Uri::from_parts(parts) {
Ok(uri) => format!("{}", uri),
Err(_) => "".to_owned(),
},
Err(_) => "".to_owned(),
};
rid.push_str(&url);
rid.push('\"');
}
if config.terms_of_service.is_some() {
rid.push(',');
rid.push_str("\"terms_of_service\":\"");
let url = match config.uri_parts(
Uri::from_static("https://authority-will-be-replaced/terms-of-service"),
true,
) {
Ok(parts) => match Uri::from_parts(parts) {
Ok(uri) => format!("{}", uri),
Err(_) => "".to_owned(),
},
Err(_) => "".to_owned(),
};
rid.push_str(&url);
rid.push('\"');
}
// Limitation
rid.push(',');
rid.push_str("\"limitation\":{");
{
rid.push_str("\"payment_required\":false,\"auth_required\":false,\"restricted_writes\":true,\"max_message_length\":1048576");
rid.push_str(&format!(
",\"max_subscriptions\":{}",
config.max_subscriptions
));
}
rid.push('}');
// Retention
rid.push(',');
rid.push_str("\"retention\":[{\"time\": null}]");
// Services
rid.push(',');
rid.push_str("\"services\":{");
rid.push_str("\"public\":[\"ephemeral\",\"directory\"]");
rid.push(',');
rid.push_str("\"private\":[\"outbox\",\"inbox\"]");
rid.push(',');
rid.push_str("\"paid\":[]");
rid.push(',');
rid.push_str("\"unavailable\":[\"search\"]");
rid.push('}');
rid.push('}');
rid
}

3
test_blossom/.gitignore vendored
View File

@ -1,3 +0,0 @@
event.map
lmdb/
downloaded

BIN
test_blossom/Example.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 2.3 KiB

7
test_blossom/README.md
View File

@ -1,7 +0,0 @@
# Testing Blossom functionality of Chorus
In one shell run `run.sh` to run a local chorus.
Then in another shell run `test.sh` to run tests against that running instance.
When done, break out of the server and run `clean.sh`

2
test_blossom/blossom/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

3
test_blossom/clean.sh
View File

@ -1,3 +0,0 @@
#!/bin/bash
rm -rf ./event.map ./lmdb/ ./blossom/* ./downloaded

34
test_blossom/config.toml
View File

@ -1,34 +0,0 @@
# See contrib/chorus.toml for a documented config file
data_directory = "."
ip_address = "127.0.0.1"
port = 8089
hostname = "localhost"
chorus_is_behind_a_proxy = false
use_tls = false
certchain_pem_path = "tls/fullchain.pem"
key_pem_path = "tls/privkey.pem"
name = "Chorus Sample"
description = "A sample run of the Chorus relay"
# icon_url =
open_relay = false
admin_hex_keys = [
"12bb541d03bfc3cab0f4a8e4db28947f60faae6fca4e315eb27f809c6eff9a0b"
]
verify_events = true
allow_scraping = false
allow_scrape_if_limited_to = 100
allow_scrape_if_max_seconds = 7200
max_subscriptions = 128
serve_ephemeral = true
serve_relay_lists = true
server_log_level = "Debug"
library_log_level = "Debug"
client_log_level = "Debug"
enable_ip_blocking = true
minimum_ban_seconds = 1
timeout_seconds = 60
max_connections_per_ip = 5
throttling_bytes_per_second = 131072
throttling_burst = 4194304
blossom_directory = "./blossom"

26
test_blossom/create_auth.sh
View File

@ -1,26 +0,0 @@
#!/bin/bash
PUBKEY=12bb541d03bfc3cab0f4a8e4db28947f60faae6fca4e315eb27f809c6eff9a0b
PRIVKEY=b4a98d96270b6cd30c80e4fd594461d2b22d8dbcfbcd1f7b11bf0ef2b028a56b
AUTH_EXPIRATION=1900000000
VERB=$1
HASH=$2
if [ x$VERB = x ] ; then
echo "USAGE: create_auth.sh VERB HASH"
exit 1
fi
if [ x$HASH = x ] ; then
echo "USAGE: create_auth.sh VERB HASH"
exit 1
fi
PRE_EVENT='{"pubkey": "'$PUBKEY'", "kind": 24242, "created_at": 0, "tags": [["expiration","'$AUTH_EXPIRATION'"], ["t","'$VERB'"], ["x","'$HASH'"]], "content":""}'
EVENT=$(echo "$PRE_EVENT" | nak event --sec $PRIVKEY)
EVENT_BASE64=$(echo $EVENT | base64 -w 0)
echo "Authorization: Nostr $EVENT_BASE64"

6
test_blossom/run.sh
View File

@ -1,6 +0,0 @@
#!/bin/bash
pushd ..
cargo build --release || exit 1
popd
../target/release/chorus ./config.toml

84
test_blossom/test.sh
View File

@ -1,84 +0,0 @@
#!/bin/bash
if ! command -v curl 2>&1 >/dev/null
then
echo "curl command is required."
exit 1
fi
if ! command -v jq 2>&1 >/dev/null
then
echo "jq command is required."
exit 1
fi
if ! command -v nak 2>&1 >/dev/null
then
echo "nak command is required. https://github.com/fiatjaf/nak"
exit 1
fi
# ADD ADMIN AS A USER ------------
echo "Adding user..."
PUBKEY=12bb541d03bfc3cab0f4a8e4db28947f60faae6fca4e315eb27f809c6eff9a0b
../target/release/chorus_cmd ./config.toml add_user $PUBKEY 0
# UPLOAD TEST ------------
FILE="./Example.png"
HASH=$(sha256sum $FILE | awk '{print $1}')
# Generate nostr auth
AUTH=$(./create_auth.sh upload $HASH)
# Upload (note we clobber the content type)
DESCRIPTOR=$(curl -vfs --data-binary @"$FILE" -H "Content-Type: " -X PUT --header "$AUTH" http://127.0.0.1:8089/upload)
if [ $? -ne 0 ] ; then
echo "FAILED: Curl (uploading) exited with a non-zero status"
exit 1
fi
echo "PASS: FILE UPLOADED"
## FIXME check for 4xx and 5xx error codes
# Extract the sha256 and compare it
DHASH=$(echo "$DESCRIPTOR" | jq -r .sha256)
if [ $? -ne 0 ] ; then
echo "FAILED: jq failed extracting sha256 from descriptor"
exit 1
fi
if [ "$HASH" != "$DHASH" ] ; then
echo "returned descriptor 'sha256' does not match the hash"
exit 1
fi
echo "PASS: DESCRIPTOR HASH MATCHES"
# Extract the URL for download
URL=$(echo "$DESCRIPTOR" | jq -r .url)
if [ $? -ne 0 ] ; then
echo "FAILED: jq failed extracting url from descriptor"
exit 1
fi
echo "Descriptor URL = $URL"
# DOWNLOAD TEST -----------
curl -s "$URL" > downloaded
if [ $? -ne 0 ] ; then
echo "FAILED: Curl (downloading) exited with a non-zero status"
exit 1
fi
echo "PASS: FILE DOWNLOADED"
# Compare the files
if cmp -s "$FILE" downloaded; then
echo "PASS: THE DOWNLOADED FILE MATCHES THE UPLOADED FILE"
else
echo "FAIL: THE DOWNLOADED FILE DOES NOT MATCH THE UPLOADED FILE"
fi
echo "end."
exit 0

3
test_with_relay_tester/.gitignore vendored
View File

@ -1,3 +0,0 @@
data
relay-tester

12
test_with_relay_tester/README.md
View File

@ -1,12 +0,0 @@
# Testing Chorus with relay-tester
First, git clone https://github.com/mikedilger/relay-tester and build that project
(cargo build --release).
Then copy that target/release/relay-tester binary into this directory.
Then run from two different shells, in this order:
shell1: ./test_chorus.sh
shell2: ./run_relay_tester.sh

12
test_with_relay_tester/run_relay_tester.sh
View File

@ -1,12 +0,0 @@
#!/bin/bash
if [ ! -x relay-tester ] ; then
echo "You must build https://github.com/mikedilge/relay-tester and copy the "
echo "resultant target/release/relay-tester binary into this directory."
exit 1
fi
./relay-tester \
ws://localhost:8080/ \
nsec16xfd467kyd3xpu9x5u4933u00v73xrl0jyq9rk5ktd9t2j38k20qtwxuj3 \
nsec1l50yuf6uxm2l5qxm87fkm56z3m7g88jnfy5s6az5wscxpu5l2yqq6qwk88

10
test_with_relay_tester/test_chorus.sh
View File

@ -1,10 +0,0 @@
#!/bin/bash
echo "Use ws://localhost:8080/ as the relay url"
cargo build --release
rm -rf ./data/
../target/release/chorus_init ./test_chorus.toml
../target/release/chorus_cmd ./test_chorus.toml add_user de16d3ed2d5ceb91d33e39dbe30585164e0c19f3f2e2a5b121def086b447a2e5 0
../target/release/chorus_cmd ./test_chorus.toml add_user 35d6bbcf17fc31a9c4f7a2f68aa40ad32c8f9de1ae77505dc5eb3722d8b2987d 0
../target/release/chorus ./test_chorus.toml

40
test_with_relay_tester/test_chorus.toml
View File

@ -1,40 +0,0 @@
# See contrib/chorus.toml for a documented config file
data_directory = "./data"
ip_address = "127.0.0.1"
port = 8080
hostname = "localhost"
chorus_is_behind_a_proxy = false
use_tls = false
certchain_pem_path = "tls/fullchain.pem"
key_pem_path = "tls/privkey.pem"
name = "Chorus Sample"
description = "A sample run of the Chorus relay"
#icon_url =
open_relay = false
admin_hex_keys = [
"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49",
# npub1mctd8mfdtn4er5e788d7xpv9ze8qcx0n7t32tvfpmmcgddz85tjsuyxe7z
"de16d3ed2d5ceb91d33e39dbe30585164e0c19f3f2e2a5b121def086b447a2e5",
# npub1xhtthnchlsc6n38h5tmg4fq26vkgl80p4em4qhw9avmj9k9jnp7sm78ql6
"35d6bbcf17fc31a9c4f7a2f68aa40ad32c8f9de1ae77505dc5eb3722d8b2987d"
]
verify_events = true
allow_scraping = false
allow_scrape_if_limited_to = 100
allow_scrape_if_max_seconds = 7200
max_subscriptions = 128
serve_ephemeral = true
serve_relay_lists = true
server_log_level = "Info"
library_log_level = "Info"
client_log_level = "Debug"
enable_ip_blocking = false
minimum_ban_seconds = 1
timeout_seconds = 60
max_connections_per_ip = 5
throttling_bytes_per_second = 131072
throttling_burst = 4194304
enable_negentropy = true