2.0.2

* Use docker-compose
 * Make it work with podman too
 * Run as root inside the container, no need for chorus user

CHANGELOG.md

@@ -0,0 +1,182 @@
+# v2.0.0
+
+- IMPORTANT: You need to manually [Migrate](docs/MIGRATION.md) your users and moderators.
+- Management commands have been added:  listadmins, listmoderators, grantmoderator, revokemoderator,
+  listusers, grantuser, revokeuser, stats, listeventsneedingmoderation, clearevent, clearpubkey,
+  listrole, grantrole, revokerole, fetchbannedevents
+- fix: subscriptions will be CLOSED: auth-required if any matching event requires auth. Chorus
+  will serve the redacted results first, but will not EOSE.
+- fix: subscriptions will be CLOSED when completed, without EOSE, if the filter has any ids set.
+
+# v1.7.2
+
+- Support for NIP-62 (PR #1256) Right to Vanish
+- Fix NIP-86 management request header determination
+- Management adds: numconnections, uptime
+- chorus_cmd: fetch_by_id
+
+# v1.7.1
+
+- Config setting allow_scrape_if_negentropy=true
+- CLOSED no longer sent after CLOSE
+
+# v1.7.0
+
+- Negentropy support (you must set enable_negentropy=true)
+- Blossom extensions are better
+
+# v1.6.1
+
+- base_url added (you should set this if you are behind a proxy like nginx)
+- NIP-45 COUNT support
+- NIP-45 PR #1561 HyperLogLog count support
+
+# v1.6.0
+
+- Support for running as a blossom server
+- Websocket close frames provided on close
+- Better pre-flight checks
+- server-wide OPTIONS requests now handled
+
+# v1.5.3
+
+- Performance improvements (streaming body)
+- Approval not required for kinds 0, 3, and reactions (as well as prior list)
+- FIX honor enable_ip_blocking = false
+- NEW `chorus_cmd` with `delete_by_id` and `delete_by_pubkey`
+- optional `icon_url` for NIP-11
+
+# version 1.5.2 (2024-07-31, d8689540)
+
+- NEW CONFIG: `throttling_bytes_per_second` how many bytes are allowed per second
+- NEW CONFIG: `throttling_burst` how many bytes can be accepted per connection in a burst
+- Don't wait more than 5 seconds for connections to close when shutting down
+
+# version 1.5.1 (2024-07-14, c9c71311)
+
+- FIX: large non-utf8 messages were attempted to be logged causing a panic
+- FIX: parse errors were not being punished so a nasty client could do a DoS of chorus with
+  simple junk.
+
+# version 1.5.0 (2024-07-13, 870e470d)
+
+- BREAKING: If you run chorus behind a proxy like nginx, you MUST set the new `chorus_is_behind_a_proxy`
+  config variable to true, and your proxy MUST set the `X-Real-IP` header.  If the header is missing,
+  connections will not be served. If you fail to set the `chorus_is_behind_a_proxy` setting, the proxy
+  IP address will be used directly, generally causing all connections to quickly become banned due to
+  the bad behavior of just one client, or due to too many connections from a single IP.
+- NEW CONFIG: `chorus_is_behind_a_proxy` (please set to true or false)
+- NEW CONFIG: `max_connections_per_ip` (defaults to 5)
+- NEW CONFIG: `moderator_hex_keys` (see next bullet point)
+- A rudimentary Management API is now available using https://github.com/nostr-protocol/nips/pull/1325
+  To use management front ends against chorus, you must add hex pubkeys to `moderator_hex_keys`.
+- Errors about DM kinds are much less common now, as we don't explicitly error unless they specify some set
+  of kinds (we implicitly filter out the DMs still)
+- Accurate count of bytes sent/received (SSL header data is now counted)
+- Kind 10050 dm relay list events are now treated the same way as kind 10002 relay list events.
+- Error message detail (e.g. source code line numbers) is now no longer sent to clients.
+- Some mild errors are now swallowed.
+- Updates of many dependencies, some updates were very large jumps and may change network/http behavior.
+- Receipt of a deleted EVENT now returns OK false (was OK true)
+
+# v1.4.0 (2024-05-07, 25058ef4)
+
+- Origin header logged
+- CLOSED: auth-required sent if DMs requested and not authenticated yet
+- config: minimum_ban_seconds, timeout_seconds, enable_ip_blocking
+- default for allow_scrape_if_max_seconds raised from 3600 to 7200
+- default for max_subscriptions raised from 32 to 128
+- timeouts no longer affect ban seconds
+- Internal: switched to pocket for the backend storage
+- creates lmdb subdirectory if missing
+- several bugfixes: filter parsing, empty tags, event ordering, chorus_compress was fixed
+
+# v1.3.0 (2024-04-10, 7883d754)
+
+- `chorus_moderate` tool to approve events or pubkeys
+- `chorus_dump_approvals` tool to view prior approvals
+- `chorus_compress` to backup and rebuild your data, compressing as it goes.
+- NIP-40 event expiration support
+- extended NIP-11 document, with updates to supported NIPs
+- Traffic bytes counted and logged on exit (or HUP)
+- Logging improvements
+- Data migrations. Less space consumed.
+- performance improved via 8-byte alignment of events
+
+# v1.2.0 (2024-03-22, a701e148)
+
+- Config option to run as an open relay
+- Config variable for scraper behavior
+- More efficient scraping
+- Reload config on HUP without dropping connections
+- IP addresses no longer logged. They are hashed and tracked by their hash.
+
+# v1.1.1 (2024-03-02, fda607a6)
+
+- FIX some LIMIT related bugs
+- FIX some indexing bugs that cuased some events that should have been returned to not be.
+- Allow scrape up to limit=100, or time range of 1 hour
+
+# v1.1.0 (2024-02-21, ec315d98)
+
+- Update docs
+
+# v1.0.7 (2024-02-21, 4ebe2545)
+
+- BREAKING: Config changed to TOML format
+- Logging improvements
+- Error handling improvements
+- Ban time improvements
+- Documentation updates
+- Restructured into a `bin` and a `lib`
+- Tool to dump all events as JSON to stdout
+
+# v1.0.6 (2024-02-21, 50995a75)
+
+- FIX: REQ errors now properly CLOSED
+- Logging improvements
+- Persistent IP reputation data for adjusting ban times
+- Config settings for `serve_ephmeral` and `serve_relay_lists`
+- Uses 'X-Real-Ip' header if behind a proxy
+
+# v1.0.5 (2024-02-20, 5e7c1c38)
+
+- FIX: significant performance problem addressed
+- Logging improvements
+- EVENTs submitted that are known to be deleted are now responded to with 'blocked:'
+- Longer filters now accepted
+
+# v1.0.4 (2024-02-20, 9c7aa299)
+
+- FIX: after EOSE, outbound events were not being screened properly
+- FIX: shutdown issue
+- NIP-11 updated
+- Idle connections with no REQs now timed out at 5 seconds
+
+# v1.0.3 (2024-02-20, 015d847f)
+
+- FIX: limits were not working properly
+- Update IP banning logic and timing
+- Improve logging
+- Allow more REQ filters (with low limits)
+- Better performance when filters are more open
+
+# v1.0.2 (2024-02-19, b85cd929)
+
+- FIX: deadlock (nested transaction issue)
+- Logging changes
+- Documentation updated
+
+# v1.0.1 (2024-02-19, 9d65d773)
+
+- FIX: AUTH was failing due to a bad time difference comparison
+- AUTH failures now include detail as to why
+- documentation updated
+
+# v1.0.0 (2024-02-19, 7ed36b95)
+
+- Initial release.
+- Works as a personal relay
+- Supported NIPs:  1, 4, 9, 11, 42, 59, 65
+- Bans IPs to prevent abuse
+- Limits to 32 subscriptions by default, configurable

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "chorus"
-version = "1.0.1"
+version = "2.0.2"
 description = "A personal relay for nostr"
 authors = ["Mike Dilger <mike@mikedilger.com>"]
 license = "MIT"
@@ -8,21 +8,37 @@ repository = "https://github.com/mikedilger/chorus"
 edition = "2021"
 
 [dependencies]
-env_logger = "0.10"
+base64 = "0.22"
+bitcoin_hashes = "0.19"
+dashmap = "6"
+env_logger = "0.11"
 futures = "0.3"
-heed = { git = "https://github.com/meilisearch/heed", rev = "64fd6fec293c0dee94855b8267557ce03e7ce5d8" }
+hex = "0.4"
-hyper = { version = "0.14", features = [ "http1", "server", "runtime", "stream" ] }
+http = "1.3"
-hyper-tungstenite = "0.11"
+http-body-util = "0.1"
-lazy_static = "1.4"
+hyper = { version = "1.6", features = [ "http1", "server" ] }
+hyper-tungstenite = "0.17"
+hyper-util = "0.1"
+lazy_static = "1.5"
 log = "0.4"
-mmap-append = { git = "https://github.com/mikedilger/mmap-append", rev = "0d20e193e7f13a442865a4a40a5da9a120e87411" }
+mime-sniffer = "0.1"
-ron = "0.8"
+mime2ext = "0.1"
-rustls-pemfile = "1.0"
+negentropy = "0.5"
-secp256k1 = { version = "0.28", features = [ "hashes", "global-context", "rand-std" ] }
+pocket-types = { git = "https://github.com/mikedilger/pocket", ref = "329334f20948c796c6016b673b92551ac4855ad7" }
+pocket-db = { git = "https://github.com/mikedilger/pocket", ref = "329334f20948c796c6016b673b92551ac4855ad7" }
+parking_lot = "0.12"
+rustls-pki-types = "1.11"
+rustls-pemfile = "2.2"
+secp256k1 = { version = "0.31", features = [ "hashes", "global-context" ] }
 serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+speedy = "0.8"
 textnonce = "1"
 tokio = { version = "1", features = [ "full" ] }
-tokio-rustls = "0.24"
+tokio-rustls = "0.26"
+tokio-util = { version = "0.7", features = [ "io" ] }
+tokio-stream = "0.1"
+toml = "0.8"
 url = "2.5"
 
 [dev-dependencies]

README.md

@@ -4,7 +4,9 @@ Chorus is a nostr relay.
 
 It is pretty fast: [docs/PERFORMANCE.md](docs/PERFORMANCE.md).
 
-It can work as your personal relay: [docs/PERSONAL_RELAY.md](docs/PERSONAL_RELAY.md).
+It can work as your personal relay (or as an open relay): [docs/PERSONAL_RELAY.md](docs/PERSONAL_RELAY.md)
+
+It can serve as a blossom server.
 
 To deploy chorus, read [docs/DEPLOYING.md](docs/DEPLOYING.md)
 
@@ -12,8 +14,50 @@ To configure chorus read [docs/CONFIG.md](docs/CONFIG.md)
 
 To undertand the relay's behavior, read [docs/BEHAVIOR.md](docs/BEHAVIOR.md)
 
+To understand command line tools, read [docs/TOOLS.md](docs/TOOLS.md)
+
+To understand the management API, read [docs/MANAGEMENT.md](docs/MANAGEMENT.md)
+
 Chorus does not have any provisions for charging users.
 
-## Status
+Chorus does not have any provisions for synchronizing events with other relays outside of the nostr protocol.
 
-Chorus is now released.
+## Alternatives
+
+### strfry
+
+[strfry](https://github.com/hoytech/strfry) is a more mature relay that offers additional features including:
+
+- Synchronizing events with other relays efficiently (negentropy)
+- Zero-downtime restarts
+- Websocket permessage-deflate
+- Plugins for event sifting
+
+However:
+
+- Chorus is probably faster (more efficient)
+- Chorus has personal relay rules by default
+- Chorus has extensive IP banning and rate limiting to protect your relay from abuse
+- Chorus supports NIP-42 (AUTH), NIP-59 (GiftWrap), NIP-65 (Relay Lists) and PR 1030 and PR 1325.
+- Chorus has a moderation cmd line tool and a moderation API (PR 1325)
+- Chorus can act as a blossom server
+
+### nostream
+
+[nostream](https://github.com/Cameri/nostream)
+
+### nostr-rs-relay
+
+[nostr-rs-relay](https://git.sr.ht/~gheartsfield/nostr-rs-relay)
+
+### khatru
+
+[khatru](https://github.com/fiatjaf/khatru)
+
+## Git branches
+
+Use the branch `latest`.
+
+Do not run off of the `master` branch. Master is updated with breaking changes that are
+not only unstable, but which may require you to update your configuration. I will not
+announce upgrade instructions until release.

contrib/chorus-direct.service

@@ -12,7 +12,8 @@ AmbientCapabilities=CAP_NET_BIND_SERVICE
 ExecStartPre=+cp -H /etc/letsencrypt/live/chorus.example.com/fullchain.pem /opt/chorus/etc/tls/
 ExecStartPre=+cp -H /etc/letsencrypt/live/chorus.example.com/privkey.pem /opt/chorus/etc/tls/
 ExecStartPre=+chown chorus /opt/chorus/etc/tls/fullchain.pem /opt/chorus/etc/tls/privkey.pem
-ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.ron
+ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.toml
+ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=5
 

contrib/chorus-proxied.service

@@ -8,7 +8,8 @@ Environment="RUST_BACKTRACE=1"
 Environment="RUST_LOG=info"
 WorkingDirectory=/opt/chorus
 User=chorus
-ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.ron
+ExecStart=/opt/chorus/sbin/chorus /opt/chorus/etc/chorus.toml
+ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=5
 

contrib/chorus.nginx.conf

@@ -23,7 +23,7 @@ server {
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "Upgrade";
         proxy_set_header Host $host;
-	    proxy_set_header X-Forwarded-For $remote_addr;
+	    proxy_set_header X-Real-Ip $remote_addr;
         proxy_read_timeout 1d;
         proxy_send_timeout 1d;
     }

contrib/chorus.ron

@@ -1,16 +0,0 @@
-FriendlyConfig(
-    data_directory: "/opt/chorus/var/chorus",
-    ip_address: "127.0.0.1",
-    port: 443,
-    use_tls: true,
-    certchain_pem_path: "/opt/chorus/etc/tls/fullchain.pem",
-    key_pem_path: "/opt/chorus/etc/tls/privkey.pem",
-    name: Some("Chorus Default"),
-    description: Some("A default config of the Chorus relay"),
-    contact: None,
-    public_key_hex: None,
-    user_hex_keys: [
-    ],
-    verify_events: true,
-    allow_scraping: false,
-)

contrib/chorus.toml

@@ -0,0 +1,354 @@
+# This is a config file for the Chorus nostr relay
+# Refer to https://github.com/mikedilger/chorus
+
+# This is the directory where chorus stores data.
+#
+# Default is "/tmp".
+#
+# If deployed according to docs/DEPLOYING.md, is "/opt/chorus/var/chorus".
+#
+data_directory = "/opt/chorus/var/chorus"
+
+
+# This is the IP address that chorus listens on. If deployed directly on the
+# Internet, this should be an Internet globally accessible IP address.
+# If proxied or if testing locally, this can be a localhost address.
+#
+# Default is "127.0.0.1".
+#
+ip_address = "127.0.0.1"
+
+
+# This is the port that chorus listens on. If deployed directly on the Internet,
+# this should probably be 443 which is the expected default port for the
+# "wss://" protocol.
+#
+# Default is 443.
+#
+port = 443
+
+
+# This is the DNS hostname of your relay.
+# This is used for verifying AUTH events, which specify your relay host name.
+#
+hostname = "localhost"
+
+
+# If chorus is behind a proxy like nginx, set this to true. In this case chorus will look for and
+# trust the `X-Real-Ip` HTTP request header to get the real IP of the client. This header MUST exist
+# or the connection will not be served.
+#
+# Default is false.
+#
+chorus_is_behind_a_proxy = false
+
+
+# If chorus is behing a proxy, it can't compute it's Internet-visible URL. So set it here.
+# This is used currently by web management and blossom and maybe more in the future.
+#
+# Default is not set.
+#
+# base_url =
+
+# If true, chorus will handle TLS, running over HTTPS. If false, chorus run over HTTP.
+#
+# If you are proxying via nginx, normally you will set this to false and allow nginx to handle TLS.
+#
+use_tls = true
+
+
+# This is the path to your TLS certificate chain file.
+#
+# If use_tls is false, this value is irrelevant.
+#
+# Default is "./tls/fullchain.pem".
+#
+# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
+# "/opt/chorus/etc/tls/fullchain.pem" and the systemd service copies letsencrypt TLS
+# certificates into this position on start.
+#
+certchain_pem_path = "/opt/chorus/etc/tls/fullchain.pem"
+
+
+# This is the path to yoru TLS private key file.
+#
+# If use_tls is false, this value is irrelevant.
+#
+# Default is "./tls/privkey.pem".
+#
+# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
+# "/opt/chorus/etc/tls/privkey.pem" and the systemd service copies letsencrypt TLS
+# certificates into this position on start.
+#
+key_pem_path = "/opt/chorus/etc/tls/privkey.pem"
+
+
+# This is a name for your relay, displayed in the NIP-11 response.
+#
+# Default is "Chorus Default"
+#
+# name = "Chorus Default"
+
+
+# This is a description for your relay, displayed in the NIP-11 response.
+#
+# Default is "A default config of the Chorus relay"
+#
+# description = "A default config of the Chorus relay"
+
+
+# This is a banner URL pointing to an image representing your relay, displayed in the NIP-11
+# response.
+#
+# Default is not set
+#
+# banner_url =
+
+
+# This is an icon URL pointing to an image representing your relay, displayed in the NIP-11
+# response.
+#
+# Default is not set
+#
+# icon_url =
+
+
+# This is an optional privacy policy as a blob of text (not a URL, not HTML).
+#
+# Default is not set
+#
+# privacy_policy = ""
+
+
+# This is an optional terms of service document as a blob of text (not a URL, not HTML).
+#
+# Default is not set
+#
+# terms_of_service = ""
+
+
+# This is an optional contact for your relay, displayed in the NIP-11 response.
+#
+# Default is not set
+#
+# contact =
+
+
+# This is an optional public key (hex format) for your relay, displayed in the NIP-11 response.
+#
+# Default is not set
+#
+# contact_public_key_hex =
+
+
+# If open_relay is true, the relay behaves as an open public relay.
+#
+# Default is false.
+#
+# open_relay = false
+
+
+# These are the public keys (hex format) of your relay's administrators. This does NOT
+# automatically make them a relay user, but it will eventually allow them to add/remove users
+# and moderators.
+#
+# Default is []
+#
+admin_hex_keys = []
+
+
+# This is a boolean indicating whether or not chorus verifies incoming events.
+#
+# This setting only skips verification of events that are submitted by AUTHed and
+# authorized users. Chorus always verifies incoming AUTH events, and any event that
+# is not submitted by an AUTHed and authorized relay user.
+#
+# Default is true.
+#
+verify_events = true
+
+
+# This is a boolean indicating whether or not scraping is allowed. Scraping is any filter
+# where all of the following are true:
+#
+# - `ids` is missing or empty
+# - `authors` is missing or empty
+# - There are no `#X` tag filters
+#
+# Filter that fail to match these conditions will be rejected if allow_scraping is false.
+#
+# If allow_scraping is true, be aware that filters that don't match any of these conditions
+# (and are not bound by since/until/limit) have no indexes to speed up their query, so they
+# scan through every single event on the relay.
+#
+# See also `allow_scrape_if_limited_to` and `allow_scrape_if_max_seconds`.
+#
+# Default is false.
+#
+allow_scraping = false
+
+
+# This is a u32 count of events indicating a filter `limit` value under which a scrape is
+# allowed, irrespective of the `allow_scraping` setting. Such scrapes are not expensive due
+# to the limit.  The limit must be specified in the filter; it does not do the scrape and
+# then count to see if it was under the limit.
+#
+# See `allow_scraping` to learn the definition of a scrape.
+#
+# The default is 100.
+#
+allow_scrape_if_limited_to = 100
+
+
+# This is a u64 number of seconds indicating a filter time range under which a scrape is
+# allowed, irrespective of the `allow_scraping` setting. Such scrapes are rarely expensive
+# due to the short time period.
+#
+# See `allow_scraping` to learn the definition of a scrape.
+#
+# The default is 7200.
+#
+allow_scrape_if_max_seconds = 7200
+
+
+# Within negentropy synchronization, allow scraping
+#
+# The default is true
+#
+allow_scrape_if_negentropy = true
+
+
+# This is an integer indicating the maximum number of subscriptions a connection can have open
+# at a given time.
+#
+# If you set this too low, clients will be incentivised to resubmit updated subscriptions which
+# will pull down the same events over again, instead of submitting a new subscription that only
+# gets the additional events that the client wants. It may seem intuitive that setting this to
+# a low value like 10 will decrease server load, but it will probably increase server load.
+#
+# It is strongly recommended to not go below 16.
+#
+# Default is 128.
+#
+max_subscriptions = 128
+
+
+# Whether or not to accept and serve all ephemeral events to everybody.
+#
+# Default is true.
+#
+serve_ephemeral = true
+
+# Whether or not to accept and serve kind 10002 Relay List Metadata (NIP-65) events to everybody.
+#
+# Default is true.
+#
+serve_relay_lists = true
+
+# How verbose to log issues with the main server code.
+#
+# Possible values are: Trace, Debug, Info, Warn, Error
+#
+# Default is Info
+#
+server_log_level = "Info"
+
+
+# How verbose to log library issues and other general issues
+#
+# Possible values are: Trace, Debug, Info, Warn, Error
+#
+# Default is Info
+#
+library_log_level = "Info"
+
+
+# How verbose to log issues with client requests
+#
+# Possible values are: Trace, Debug, Info, Warn, Error
+#
+# Default is Info
+#
+client_log_level = "Info"
+
+
+# Whether to block incoming connections based on recent prior behavior
+#
+# Chorus normally blocks IP addresses for a short period preventing quick reconnections,
+# and for a longer period if the previous connection ended in some error condition.
+#
+# By setting this variable to false, it will allow all connections, incluing poorly behaving
+# clients that reconnect over and over in a tight loop.
+#
+# Default is true
+#
+enable_ip_blocking = true
+
+
+# Number of seconds to ban an IP address after disconnection.
+#
+# Enforcing this minimum ban prevents clients from immediately reconnecting which can cause tight loops.
+# Only relevant if enable_ip_blocking is true.
+#
+# Default is 1
+#
+minimum_ban_seconds = 1
+
+
+# Number of seconds beyond which chorus times out a client that has no open subscriptions.
+#
+# Default is 60
+#
+timeout_seconds = 60
+
+
+# Maximum number of websocket connections per IP address
+#
+# Default is 5
+#
+max_connections_per_ip = 5
+
+
+# The maximum rate (excluding bursts) of data that will be transmitted over a websocket connection
+# (both directions, per connection). Beyond this rate (in a sustained way) the connection will be
+# closed.
+#
+# Default is 1024576 bytes per second.
+#
+throttling_bytes_per_second = 1024576
+
+
+# The allowable bursts of data beyond the normal rate.
+#
+# We keep a count of how many bytes they are allowed, and that count starts at this number.
+# As bytes are consumed the count goes down, but we refund throttling_bytes_per_second every
+# second. If that bucket doesn't have enough, the burst won't be allowed and the connection
+# will be closed.
+#
+# Default is 16777216 bytes.
+#
+throttling_burst = 16777216
+
+
+# Blossom server directory
+#
+# Set to a filesystem directory where you want chorus to store files.
+#
+# Blossom allows clients to upload files making them available for the public to download.
+# Our implementation makes all files publicly readable, but only chorus users can upload
+# or delete.  See https://github.com/hzrd149/blossom
+#
+# Default is not set
+#
+# blossom_directory =
+
+
+# Whether to allow negentropy syncing or not
+#
+# If this is enabled, keep in mind that some negentropy syncs are scrapes and if you want to
+# allow those, you should also enable allow_scraping. But this will require scans of the entire
+# database since scrapes have no indexes.
+#
+# Default is false
+#
+enable_negentropy = false

docker/.env

@@ -0,0 +1 @@
+PODMAN_USERNS=keep-id

docker/Dockerfile

@@ -0,0 +1,35 @@
+FROM rust:alpine as builder
+RUN apk add --no-cache git
+WORKDIR /root
+RUN git clone https://github.com/mikedilger/chorus
+WORKDIR /root/chorus
+RUN git checkout latest
+RUN cargo build --release
+RUN strip ./target/release/* || true
+
+FROM alpine:latest
+
+# Install system packages
+# RUN apk add --no-cache curl gcc musl-dev openssl-dev pkgconfig git make cmake
+
+# Setup chorus user and directories
+RUN mkdir -p /opt/chorus/etc /opt/chorus/src/chorus /opt/chorus/var /opt/chorus/sbin /opt/chorus/lib && \
+    mkdir -p /opt/chorus/var/chorus /opt/chorus/var/www && \
+    mkdir -p /opt/chorus/lib/systemd/system
+
+COPY --from=builder /root/chorus/target/release/chorus /opt/chorus/sbin/chorus
+COPY --from=builder /root/chorus/target/release/chorus_cmd /opt/chorus/sbin/chorus_cmd
+COPY --from=builder /root/chorus/target/release/chorus_compress /opt/chorus/sbin/chorus_compress
+COPY --from=builder /root/chorus/target/release/chorus_dump /opt/chorus/sbin/chorus_dump
+COPY --from=builder /root/chorus/target/release/chorus_dump_approvals /opt/chorus/sbin/chorus_dump_approvals
+COPY --from=builder /root/chorus/target/release/chorus_moderate /opt/chorus/sbin/chorus_moderate
+
+RUN chmod 0700 /opt/chorus/sbin/*
+
+VOLUME /opt/chorus/etc
+VOLUME /opt/chorus/var
+
+WORKDIR /opt/chorus
+ENV RUST_BACKTRACE=1
+ENV RUST_LOG=info
+ENTRYPOINT ["/opt/chorus/sbin/chorus", "/opt/chorus/etc/chorus.toml"]

docker/README.md

@@ -0,0 +1,6 @@
+# Docker install of Chorus
+
+You need to `cp chorus.toml.in chorus.toml` and then edit this file
+to set up your configuration
+
+Then you can build the container: `docker build .`

docker/docker-compose.yml

@@ -0,0 +1,12 @@
+services:
+  chorus:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: chorus
+    restart: unless-stopped
+    volumes:
+      - ./etc:/opt/chorus/etc
+      - ./var:/opt/chorus/var
+    ports:
+      - 443:1444

docker/etc/.gitignore

@@ -0,0 +1 @@
+/chorus.toml

docker/etc/chorus.toml.in

@@ -0,0 +1,354 @@
+# This is a config file for the Chorus nostr relay
+# Refer to https://github.com/mikedilger/chorus
+
+# This is the directory where chorus stores data.
+#
+# Default is "/tmp".
+#
+# If deployed according to docs/DEPLOYING.md, is "/opt/chorus/var/chorus".
+#
+data_directory = "/opt/chorus/var/chorus"
+
+
+# This is the IP address that chorus listens on. If deployed directly on the
+# Internet, this should be an Internet globally accessible IP address.
+# If proxied or if testing locally, this can be a localhost address.
+#
+# Default is "127.0.0.1".
+#
+ip_address = "0.0.0.0"
+
+
+# This is the port that chorus listens on. If deployed directly on the Internet,
+# this should probably be 443 which is the expected default port for the
+# "wss://" protocol.
+#
+# Default is 443.
+#
+port = 443
+
+
+# This is the DNS hostname of your relay.
+# This is used for verifying AUTH events, which specify your relay host name.
+#
+hostname = "localhost"
+
+
+# If chorus is behind a proxy like nginx, set this to true. In this case chorus will look for and
+# trust the `X-Real-Ip` HTTP request header to get the real IP of the client. This header MUST exist
+# or the connection will not be served.
+#
+# Default is false.
+#
+chorus_is_behind_a_proxy = true
+
+
+# If chorus is behing a proxy, it can't compute it's Internet-visible URL. So set it here.
+# This is used currently by web management and blossom and maybe more in the future.
+#
+# Default is not set.
+#
+# base_url =
+
+# If true, chorus will handle TLS, running over HTTPS. If false, chorus run over HTTP.
+#
+# If you are proxying via nginx, normally you will set this to false and allow nginx to handle TLS.
+#
+use_tls = false
+
+
+# This is the path to your TLS certificate chain file.
+#
+# If use_tls is false, this value is irrelevant.
+#
+# Default is "./tls/fullchain.pem".
+#
+# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
+# "/opt/chorus/etc/tls/fullchain.pem" and the systemd service copies letsencrypt TLS
+# certificates into this position on start.
+#
+certchain_pem_path = "/use/an/nginx/proxy"
+
+
+# This is the path to yoru TLS private key file.
+#
+# If use_tls is false, this value is irrelevant.
+#
+# Default is "./tls/privkey.pem".
+#
+# If deployed according to docs/DEPLOYING.md using the direct method, this is set to
+# "/opt/chorus/etc/tls/privkey.pem" and the systemd service copies letsencrypt TLS
+# certificates into this position on start.
+#
+key_pem_path = "/use/an/nginx/proxy"
+
+
+# This is a name for your relay, displayed in the NIP-11 response.
+#
+# Default is "Chorus Default"
+#
+# name = "Chorus Default"
+
+
+# This is a description for your relay, displayed in the NIP-11 response.
+#
+# Default is "A default config of the Chorus relay"
+#
+# description = "A default config of the Chorus relay"
+
+
+# This is a banner URL pointing to an image representing your relay, displayed in the NIP-11
+# response.
+#
+# Default is not set
+#
+# banner_url =
+
+
+# This is an icon URL pointing to an image representing your relay, displayed in the NIP-11
+# response.
+#
+# Default is not set
+#
+# icon_url =
+
+
+# This is an optional privacy policy as a blob of text (not a URL, not HTML).
+#
+# Default is not set
+#
+# privacy_policy = ""
+
+
+# This is an optional terms of service document as a blob of text (not a URL, not HTML).
+#
+# Default is not set
+#
+# terms_of_service = ""
+
+
+# This is an optional contact for your relay, displayed in the NIP-11 response.
+#
+# Default is not set
+#
+# contact =
+
+
+# This is an optional public key (hex format) for your relay, displayed in the NIP-11 response.
+#
+# Default is not set
+#
+# contact_public_key_hex =
+
+
+# If open_relay is true, the relay behaves as an open public relay.
+#
+# Default is false.
+#
+# open_relay = false
+
+
+# These are the public keys (hex format) of your relay's administrators. This does NOT
+# automatically make them a relay user, but it will eventually allow them to add/remove users
+# and moderators.
+#
+# Default is []
+#
+admin_hex_keys = []
+
+
+# This is a boolean indicating whether or not chorus verifies incoming events.
+#
+# This setting only skips verification of events that are submitted by AUTHed and
+# authorized users. Chorus always verifies incoming AUTH events, and any event that
+# is not submitted by an AUTHed and authorized relay user.
+#
+# Default is true.
+#
+verify_events = true
+
+
+# This is a boolean indicating whether or not scraping is allowed. Scraping is any filter
+# where all of the following are true:
+#
+# - `ids` is missing or empty
+# - `authors` is missing or empty
+# - There are no `#X` tag filters
+#
+# Filter that fail to match these conditions will be rejected if allow_scraping is false.
+#
+# If allow_scraping is true, be aware that filters that don't match any of these conditions
+# (and are not bound by since/until/limit) have no indexes to speed up their query, so they
+# scan through every single event on the relay.
+#
+# See also `allow_scrape_if_limited_to` and `allow_scrape_if_max_seconds`.
+#
+# Default is false.
+#
+allow_scraping = false
+
+
+# This is a u32 count of events indicating a filter `limit` value under which a scrape is
+# allowed, irrespective of the `allow_scraping` setting. Such scrapes are not expensive due
+# to the limit.  The limit must be specified in the filter; it does not do the scrape and
+# then count to see if it was under the limit.
+#
+# See `allow_scraping` to learn the definition of a scrape.
+#
+# The default is 100.
+#
+allow_scrape_if_limited_to = 500
+
+
+# This is a u64 number of seconds indicating a filter time range under which a scrape is
+# allowed, irrespective of the `allow_scraping` setting. Such scrapes are rarely expensive
+# due to the short time period.
+#
+# See `allow_scraping` to learn the definition of a scrape.
+#
+# The default is 7200.
+#
+allow_scrape_if_max_seconds = 14400
+
+
+# Within negentropy synchronization, allow scraping
+#
+# The default is true
+#
+allow_scrape_if_negentropy = true
+
+
+# This is an integer indicating the maximum number of subscriptions a connection can have open
+# at a given time.
+#
+# If you set this too low, clients will be incentivised to resubmit updated subscriptions which
+# will pull down the same events over again, instead of submitting a new subscription that only
+# gets the additional events that the client wants. It may seem intuitive that setting this to
+# a low value like 10 will decrease server load, but it will probably increase server load.
+#
+# It is strongly recommended to not go below 16.
+#
+# Default is 128.
+#
+max_subscriptions = 128
+
+
+# Whether or not to accept and serve all ephemeral events to everybody.
+#
+# Default is true.
+#
+serve_ephemeral = true
+
+# Whether or not to accept and serve kind 10002 Relay List Metadata (NIP-65) events to everybody.
+#
+# Default is true.
+#
+serve_relay_lists = true
+
+# How verbose to log issues with the main server code.
+#
+# Possible values are: Trace, Debug, Info, Warn, Error
+#
+# Default is Info
+#
+server_log_level = "Info"
+
+
+# How verbose to log library issues and other general issues
+#
+# Possible values are: Trace, Debug, Info, Warn, Error
+#
+# Default is Info
+#
+library_log_level = "Info"
+
+
+# How verbose to log issues with client requests
+#
+# Possible values are: Trace, Debug, Info, Warn, Error
+#
+# Default is Info
+#
+client_log_level = "Info"
+
+
+# Whether to block incoming connections based on recent prior behavior
+#
+# Chorus normally blocks IP addresses for a short period preventing quick reconnections,
+# and for a longer period if the previous connection ended in some error condition.
+#
+# By setting this variable to false, it will allow all connections, incluing poorly behaving
+# clients that reconnect over and over in a tight loop.
+#
+# Default is true
+#
+enable_ip_blocking = true
+
+
+# Number of seconds to ban an IP address after disconnection.
+#
+# Enforcing this minimum ban prevents clients from immediately reconnecting which can cause tight loops.
+# Only relevant if enable_ip_blocking is true.
+#
+# Default is 1
+#
+minimum_ban_seconds = 1
+
+
+# Number of seconds beyond which chorus times out a client that has no open subscriptions.
+#
+# Default is 60
+#
+timeout_seconds = 60
+
+
+# Maximum number of websocket connections per IP address
+#
+# Default is 5
+#
+max_connections_per_ip = 5
+
+
+# The maximum rate (excluding bursts) of data that will be transmitted over a websocket connection
+# (both directions, per connection). Beyond this rate (in a sustained way) the connection will be
+# closed.
+#
+# Default is 1024576 bytes per second.
+#
+throttling_bytes_per_second = 1024576
+
+
+# The allowable bursts of data beyond the normal rate.
+#
+# We keep a count of how many bytes they are allowed, and that count starts at this number.
+# As bytes are consumed the count goes down, but we refund throttling_bytes_per_second every
+# second. If that bucket doesn't have enough, the burst won't be allowed and the connection
+# will be closed.
+#
+# Default is 16777216 bytes.
+#
+throttling_burst = 16777216
+
+
+# Blossom server directory
+#
+# Set to a filesystem directory where you want chorus to store files.
+#
+# Blossom allows clients to upload files making them available for the public to download.
+# Our implementation makes all files publicly readable, but only chorus users can upload
+# or delete.  See https://github.com/hzrd149/blossom
+#
+# Default is not set
+#
+# blossom_directory =
+
+
+# Whether to allow negentropy syncing or not
+#
+# If this is enabled, keep in mind that some negentropy syncs are scrapes and if you want to
+# allow those, you should also enable allow_scraping. But this will require scans of the entire
+# database since scrapes have no indexes.
+#
+# Default is false
+#
+enable_negentropy = false

docker/var/.gitignore

@@ -0,0 +1 @@
+

docs/BEHAVIOR.md

@@ -4,7 +4,11 @@
 
 If `verify_events` is set in the configuration, chorus rejects invalid events in all cases.
 
-Chorus accepts all events submitted by authorized users.
+If an event has a '-' tag, it must be submitted by an AUTHed user that matches the pubkey of the event, else it is rejected.
+
+If `open_relay` is true, all other events are accepted. If false, the remaining rules apply.
+
+Chorus accepts all events submitted by AUTHed authorized users.
 
 Chorus accepts relay list metadata (kind 10002) from anybody.
 
@@ -18,33 +22,41 @@ If you wish to change these rules, change the source code at `nostr.rs:screen_in
 
 ## REQ Read permissions and behavior
 
+Chorus does not serve any DM (kind 4) or GiftWrap (kind 1059) unless the connection is AUTHed and the user matches either a tagged person or the author of the event.
+
+If `open_relay` is true, all other events are served. If false, the remaining rules apply.
+
 Chorus serves all relay list metadata (kind 10002) events queried.
 
 Chorus serves all ephemeral events.
 
-Chorus does not serve any DM (kind 4) or GiftWrap (kind 1059) unless the connection is AUTHed and the user matches either a tagged person or the author of the event.
+Chorus serves all events to AUTHed authorized users.
-
-Chorus serves all events to AUTHed and authorized users.
 
 Chorus serves all events which were authored by an authorized user.
 
+Filters which are broad are considered scrapers and are not serviced. Scraping is any filter where all of the following are true:
+
+- `ids` is missing or empty
+- `authors` is missing or empty
+- There are no `#X` tag filters
+
 If you wish to change these rules, change the source code at `nostr.rs:screen_outgoing_event()`
 
 ## Abuse, Banning, Throttling, and the like
 
-WebSocket frames and messages are limited to 1 MB.
+WebSocket frames and messages are limited to 1 MB (slightly less for messages due to some overhead).
 
 Each connection has a memory buffer used for JSON deserialization that is no larger
 than the WebSocket message. No more than one such buffer exists per connection, and memory
 allocation is generally tightly controlled.
 
-Every connection is IP banned for 4 seconds after disconnection, whether or not the connection
+Every connection is IP banned for 2 seconds after disconnection, whether or not the connection
 was well behaved. We don't think clients should ever reconnect immediately. If chorus is
 run directly (not behind an nginx proxy), this IP banning is more efficient because it happens
 prior to SSL setup.
 
-A maximum of 32 subscriptions are allowed by default, although this is configurable with the
+A maximum of 32 subscriptions are allowed by default (per connection), although this is
-`max_subscriptions` configuration setting.
+configurable with the `max_subscriptions` configuration setting.
 
 ## NIP Support
 
@@ -73,8 +85,6 @@ as well as remembering these (id,pubkey) pairs to reject such events subsequentl
 
 Chorus fully complies with NIP-11.
 
-Chorus returns a brief result without much detail.
-
 ### NIP-26 Delegated Event Signing
 
 Chorus does not support NIP-26.
@@ -124,4 +134,3 @@ Chorus does not support NIP-94.
 ### NIP-96 HTTP File Storage Integration
 
 Chorus does not support NIP-96.
-

docs/CONFIG.md

@@ -2,7 +2,7 @@
 
 The chorus binary requires one command line parameter which specifies the config file path.
 
-The config file must be in RON format. See the [RON documentation](https://docs.rs/ron/latest/ron/)
+The config file must be in TOML format. See the [TOML documentation](https://github.com/toml-lang/toml)
 
 ## Configuration Variables
 
@@ -10,7 +10,7 @@ The config file must be in RON format. See the [RON documentation](https://docs.
 
 This is the directory where chorus stores data.
 
-Default is "/tmp".
+Default is "/opt/chorus/var/chorus".
 
 If deployed according to [docs/DEPLOYING.md](docs/DEPLOYING.md), is "/opt/chorus/var/chorus".
 
@@ -34,6 +34,23 @@ Default is 443.
 This is the DNS hostname of your relay. This is used for verifying AUTH events, which specify
 your relay host name.
 
+Default is localhost
+
+### chorus_is_behind_a_proxy
+
+If chorus is behind a proxy like nginx, set this to true. In this case chorus will look for and
+trust the `X-Real-Ip` HTTP request header to get the real IP of the client. This header MUST exist
+or the connection will not be served.
+
+Default is false.
+
+### base_url
+
+If chorus is behing a proxy, it can't compute it's Internet-visible URL. So set it here.
+This is used currently by web management and blossom and maybe more in the future.
+
+Default is not set.
+
 ### use_tls
 
 If true, chorus will handle TLS, running over HTTPS.  If false, chorus run over HTTP.
@@ -41,57 +58,81 @@ If true, chorus will handle TLS, running over HTTPS.  If false, chorus run over
 If you are proxying via nginx, normally you will set this to false and allow nginx to handle
 TLS.
 
+Default is true
+
 ### certchain_pem_path
 
 This is the path to your TLS certificate chain file.
 
 If `use_tls` is false, this value is irrelevant.
 
-Default is "./tls/fullchain.pem".
+Default is "/opt/chorus/etc/tls/fullchain.pem"
 
 If deployed according to [docs/DEPLOYING.md](docs/DEPLOYING.md) using the direct method,
-this is set to "/opt/chorus/etc/tls/fullchain.pem" and the systemd service copies letsencrypt
+systemd service copies letsencrypt TLS certificates into this position on start.
-TLS certificates into this position on start.
 
 ### key_pem_path
 
-This is the path to yoru TLS private key file.
+This is the path to your TLS private key file.
 
 If `use_tls` is false, this value is irrelevant.
 
-Default is "./tls/privkey.pem".
+Default is "/opt/chorus/etc/tls/privkey.pem"
 
 If deployed according to [docs/DEPLOYING.md](docs/DEPLOYING.md) using the direct method,
-this is set to "/opt/chorus/etc/tls/privkey.pem" and the systemd service copies letsencrypt
+systemd service copies letsencrypt TLS certificates into this position on start.
-TLS certificates into this position on start.
 
 ### name
 
 This is an optional name for your relay, displayed in the NIP-11 response.
 
-Default is None.
+Default is "Chorus Default"
 
 ### description
 
 This is an optional description for your relay, displayed in the NIP-11 response.
 
-Default is None.
+Default is "A default config of the Chorus relay".
+
+### banner_url
+
+This is an optional URL for an graphical banner representing your relay, displayed in the NIP-11 response.
+
+### icon_url
+
+This is an optional URL for an graphical icon representing your relay, displayed in the NIP-11 response.
+
+### privacy_policy
+
+This is an optional privacy policy as a blob of text (not a URL, not HTML).
+
+### terms_of_service
+
+This is an optional terms of service document as a blob of text (not a URL, not HTML).
 
 ### contact
 
-This is an optional contact for your relay, displayed in the NIP-11 response.
+This is an optional administrative contact for your relay, displayed in the NIP-11 response.
 
 Default is None.
 
-### public_key_hex
+### contact_public_key_hex
 
-This is an optional public key (hex format) for your relay, displayed in the NIP-11 response.
+This is an optional public key (hex format) for your relay's administrative contact, displayed in the NIP-11 response.
+
+Deprecated "public_key_hex" also works.
 
 Default is None.
 
-### user_hex_keys
+### open_relay
 
-These are the public keys (hex format) of your relay's authorized users. See [BEHAVIOR.md](BEHAVIOR.md) to understand how chorus uses these.
+If open_relay true, the relay behaves as an open public relay.
+
+Default is false.
+
+### admin_hex_keys
+
+These are the public keys (hex format) of your relay's administrators. This does NOT automatically make them a relay user, but it will eventually allow them to add/remove users and moderators.
 
 Default is `[]`
 
@@ -105,27 +146,157 @@ Default is true.
 
 ### allow_scraping
 
-This is a boolean indicating whether or not scraping is allowed. Scraping is any filter that does not match one of the following conditions:
+This is a boolean indicating whether or not scraping is allowed. Scraping is any filter where all of the following are true:
 
-- A non-empty `id` list is set
+- `ids` is missing or empty
-- A non-empty `authors` list is set and a non-empty `kinds` list is set
+- `authors` is missing or empty
-- A non-empty `authors` list is set and at least one tag is set.
+- There are no `#X` tag filters
-- A non-empty `kinds` list is set and at least one tag is set.
 
 Filter that fail to match these conditions will be rejected if `allow_scraping` is false.
 
-If `allow_scraping` is true, be aware that filters that don't match any of these conditions have no indexes to speed up their query, so they scan through every single event on the relay.
+If `allow_scraping` is true, be aware that filters that don't match any of these conditions (and are not bound by since/until/limit) have no indexes to speed up their query, so they scan through every single event on the relay.
 
-The purpose of this setting is as a temporary setting that allows you to dump every single event on your relay.
+See also `allow_scrape_if_limited_to` and # `allow_scrape_if_max_seconds`.
 
 Default is false.
 
+### allow_scrape_if_limited_to
+
+This is a u32 count of events indicating a filter `limit` value under which a scrape is allowed, irrespective of the `allow_scraping` setting. Such scrapes are not expensive due to the limit. The limit must be specified in the filter; it does not do the scrape and then count to see if it was under the limit.
+
+See `allow_scraping` to learn the definition of a scrape.
+
+The default is 100.
+
+### allow_scrape_if_max_seconds
+
+This is a u64 number of seconds indicating a filter time range under which a scrape is allowed, irrespective of the `allow_scraping` setting. Such scrapes are rarely expensive due to the short time period.
+
+See `allow_scraping` to learn the definition of a scrape.
+
+The default is 7200.
+
+### allow_scrape_if_negentropy
+
+Within negentropy synchronization, allow scraping.
+
+The default is true
+
 ### max_subscriptions
 
-This is a usize indicating the maximum number of subscriptions a connection can have open at a given time.
+This is an integer indicating the maximum number of subscriptions a connection can have open at a given time.
 
 If you set this too low, clients will be incentivised to resubmit updated subscriptions which will pull down the same events over again, instead of submitting a new subscription that only gets the additional events that the client wants. It may seem intuitive that setting this to a low value like 10 will decrease server load, but it will probably increase server load.
 
 It is strongly recommended to not go below 16.
 
-Default is 32.
+Default is 128.
+
+### serve_ephemeral
+
+Whether or not to accept and serve all ephemeral events to everybody.
+
+Default is true.
+
+### serve_relay_lists
+
+Whether or not to accept and serve kind 10002 Relay List Metadata (NIP-65) events to everybody.
+
+Default is true.
+
+### server_log_level
+
+How verbose to log issues with the main server code.
+
+Possible values are: Trace, Debug, Info, Warn, Error
+
+Default is Info
+
+### library_log_level
+
+How verbose to log library issues and other general issues
+
+Possible values are: Trace, Debug, Info, Warn, Error
+
+Default is Info
+
+### client_log_level
+
+How verbose to log issues with client requests
+
+Possible values are: Trace, Debug, Info, Warn, Error
+
+Default is Info
+
+### enable_ip_blocking
+
+Whether to block incoming connections based on recent prior behavior
+
+Chorus normally blocks IP addresses for a short period preventing quick reconnections, and for a longer period if the previous connection ended in some error condition.
+
+By setting this variable to false, it will allow all connections, incluing poorly behaving clients that reconnect over and over in a tight loop.
+
+Default is true
+
+### minimum_ban_seconds
+
+Number of seconds to ban an IP address after disconnection.
+
+Enforcing this minimum ban prevents clients from immediately reconnecting which can cause tight loops.
+
+Only relevant if enable_ip_blocking is true.
+
+Default is 1
+
+### timeout_seconds
+
+Number of seconds beyond which chorus times out a client that has no open subscriptions.
+
+Default is 60
+
+### max_connections_per_ip
+
+Maximum number of websocket connections per IP address
+
+Default is 5
+
+### throttling_bytes_per_second
+
+The maximum rate (excluding bursts) of data that will be transmitted over a websocket connection
+(both directions, per connection). Beyond this rate (in a sustained way) the connection will be
+closed.
+
+Default is 1024576 bytes per second.
+
+### throttling_burst
+
+The allowable bursts of data beyond the normal rate.
+
+We keep a count of how many bytes they are allowed, and that count starts at this number.
+As bytes are consumed the count goes down, but we refund throttling_bytes_per_second every
+second. If that bucket doesn't have enough, the burst won't be allowed and the connection
+will be closed.
+
+Default is 16777216 bytes.
+
+### blossom_directory
+
+Blossom server directory
+
+Set to a filesystem directory where you want chorus to store files.
+
+Blossom allows clients to upload files making them available for the public to download.
+Our implementation makes all files publicly readable, but only chorus users can upload
+or delete.  See https://github.com/hzrd149/blossom
+
+Default is None
+
+### enable_negentropy
+
+Whether to allow negentropy syncing or not
+
+If this is enabled, keep in mind that some negentropy syncs are scrapes and if you want to
+allow those, you should also enable allow_scraping. But this will require scans of the entire
+database since scrapes have no indexes.
+
+Default is false

docs/DEPLOYING.md

@@ -8,6 +8,15 @@ Generally these are servers in data centres, but you might be able to make a por
 to the Internet on a home machine if your ISP doesn't use CGNAT and you know how to
 configure your firewall/router for this. We leave this up to you.
 
+## Operating System
+
+Chorus was developed on and for linux. It might work on other operating systems, but I
+won't spend any time getting it working on non-linux systems.
+
+In particular, I wrote [mmap-append](https://github.com/mikedilger/mmap-append) to use
+linux specific mmap functionality. Ashkan Kiani came later and generalized the code but
+I haven't tried it elsewhere and so I can't vouch for it elsewhere.
+
 ## Deploying the files
 
 You'll want to create a `chorus` user. Here is an example for debian based systems:
@@ -34,6 +43,7 @@ sudo chown $(id -u) /opt/chorus/src
 cd /opt/chorus/src
 git clone https://github.com/mikedilger/chorus
 cd chorus
+git checkout latest
 ```
 
 Check if you have `rustc` and `cargo` installed. If so, you can skip this part.
@@ -56,16 +66,16 @@ Now let's continue by building chorus:
 cargo build --release
 ```
 
-Ok now let's install that:
+Ok now let's install that (along with the utility binaries):
 
 ```bash
-sudo install --mode=0700 --owner=chorus ./target/release/chorus /opt/chorus/sbin/chorus
+sudo install --mode=0700 --owner=chorus ./target/release/{chorus,chorus_compress,chorus_dump,chorus_dump_approvals,chorus_moderate,chorus_cmd} /opt/chorus/sbin/
 ```
 
 Now let's create our config file
 
 ```bash
-sudo -u chorus cp /opt/chorus/src/chorus/contrib/chorus.ron /opt/chorus/etc/
+sudo -u chorus cp /opt/chorus/src/chorus/contrib/chorus.toml /opt/chorus/etc/
 ```
 
 Go ahead and edit that file to your liking. In particular:
@@ -73,8 +83,9 @@ Go ahead and edit that file to your liking. In particular:
 - Change the `ip_address` to your internet-accessible IP address (if you are running directly)
   or to 127.0.0.1 with a local port like 8080 (if you are proxying behind nginx)
 - Change the port if necessary
-- Change the name, description, and contact (e.g. your email address) as desired
+- Change the name, description, banner_url, icon_url, privacy_policy, terms_of_service and
-- Set your public_key_hex (it is an option, so use `Some()`)
+  contact (e.g. your email address) as desired
+- Set your contact_public_key_hex (it is an option, so use `Some()`)
 - Set hex keys of users for which this relay will act as a personal relay
 
 
@@ -156,6 +167,10 @@ Restart nginx
 sudo systemctl restart nginx.service
 ```
 
+## Adding users and moderators
+
+See [MANAGEMENT](MANAGEMENT.md) for how to add users and moderators.
+
 ## Monitoring the service
 
 You can watch the logs with a command like this
@@ -164,3 +179,23 @@ You can watch the logs with a command like this
 sudo journalctl -f -u chorus.service
 ```
 
+## Updating
+
+````bash
+cd /opt/chorus/src/chorus
+git pull
+cargo build --release
+sudo install --mode=0700 --owner=chorus ./target/release/{chorus,chorus_compress,chorus_dump,chorus_dump_approvals,chorus_moderate,chorus_cmd} /opt/chorus/sbin/
+sudo systemctl restart chorus.service
+````
+
+## Uninstalling
+
+```bash
+sudo systemctl disable chorus.service
+sudo rm -f /etc/nginx/sites-available/chorus.nginx.conf
+sudo rm -f /etc/nginx/sites-enabled/chorus.nginx.conf
+sudo systemctl restart nginx.service
+sudo rm -rf /opt/chorus
+sudo userdel chorus
+```

docs/MANAGEMENT.md

@@ -0,0 +1,42 @@
+# Chorus Management
+
+This page is about using the online Management API or command line [TOOLS.md](TOOLS.md)
+to manage users and moderate events.
+
+## Managing users
+
+To list users:  `chorus_cmd <configtoml> dump_users`
+
+To add a user:  `chorus_cmd <configtoml> add_user <pubkeyhex> 0`
+
+To add a moderator:  `chorus_cmd <configtoml> add_user <pubkeyhex> 1`
+
+To remove moderator flag, just add the user again with 0.
+
+To remove a user: `chorus_cmd <configtoml> rm_user <pubkeyhex>`
+
+## Managing events
+
+To cycle through all events needing moderation, and address each one, use `chorus_moderate <configtoml>`
+
+To remove an event by id: `chorus_cmd <configtoml> delete_by_id <idhex>`
+
+To remove multiple events by pubkey: `chorus_cmd <configtoml> delete_by_pubkey <pubkeyhex>`
+
+## Relay Management NIP
+
+The Relay Management API is in flux currently. It is still a pull request on the NIPs repo: [PR 1325](https://github.com/nostr-protocol/nips/pull/1325).
+
+This document may go out of date as things are changing rapidly.
+
+## The status of a pubkey (user)
+
+Users can be in one of four moderation states: Authorized, Approved, Banned, and Default.
+
+**Authorized**: These are the users you have added as authorized (whether or not they are a moderator). These user's events are always accepted and these users can read everything (except other user's DMs for example).
+
+**Approved**: These are users who are not authorized, but which via moderation have been approved. Approved user's posts are publically available to anybody to read. Because they are not authorized, they can only post replies to authorized users.
+
+**Banned**: These are users who cannot make any posts at all to the relay.
+
+**Default**: All pubkeys not explicitly put into any of the other three categories default to this category. Because they are not authorized, they can only post replies to authorized users. Because they are not approved, these replies are only visible to authorized users and are not publicly visible (unless and until a moderator approves the specific post).

docs/MIGRATION.md

@@ -0,0 +1,25 @@
+# Migration
+
+## From 1.0 to 2.0
+
+1) Add to your config file `admin_hex_keys` to include the nostr hex keys of administrators.
+   These will (eventually) be allowed to manage users and moderators remotely via the management
+   interface.  Note that being an admin does NOT automatically grant user or moderator rights,
+   it ONLY grants the right to administer users.
+
+2) Users and moderators are now dynamically configured in the database. Use `chorus_cmd` to
+   manage them from the command line:
+
+* Adding a user:  `chorus_cmd <chorus.toml> add_user <pubkey> 0`
+* Adding a moderator: `chorus_cmd <chorus.toml> add_user <pubkey> 1`
+* Removing a user or moderator: `chorus_cmd <chorus.toml> rm_user <pubkey>`
+* Listing users and moderators: `chorus_cmd <chorus.toml> dump_users`
+
+3) Remove the following from your config file as these are no longer used:
+
+* `user_hex_keys` - users are now dynamically configured and configuration is in the database.
+* `moderator_hex_keys` - moderators are now dynamically configured and configuration is in the database.
+
+4) Configuration setting `public_key_kex` has been renamed `contact_public_key_hex` and is
+   used only for the NIP-11 data.
+

docs/PERSONAL_RELAY.md

@@ -5,14 +5,14 @@ users, perhaps just yourself.
 
 It hosts your own events and makes them available to the public.
 
-It accepts events that tag your users, but doesn't allow the public to read them back. Only
+It accepts events that tag your users, but doesn't allow the public to read them back until
-your users (after being authenticated) can see them.
+they have passed moderation and until then, only your users (after being authenticated) can see
+them.
 
 One nice thing about a personal relay is that you are in control, and you have a record
 of your events in your possession.
 
-Another nice thing is that you don't have to moderate content, since the only content
+Another nice thing is that you don't have to moderate content if you don't care to.
-that is served to the public is content that your trusted users created.
 
 ## The Dark Side
 
@@ -23,3 +23,7 @@ they will need to setup SSL connections to far too many relays.
 It may be better for others if people were to use a smaller number of larger relay services.
 
 But who am I to say? You are the sovereign. Here is your personal relay.
+
+## Open Relay
+
+You can run chorus as an open public relay. Just set `open_relay` to true in the config.

docs/TOOLS.md

@@ -0,0 +1,35 @@
+# Tools
+
+Chorus comes with four binaries other than the main `chorus` binary.
+
+## chorus_dump
+
+Usage: **chorus_dump** *<path_to_config_file\>*
+
+This dumps every event to STDOUT in JSON format.
+
+## chorus_compress
+
+Usage: **chorus_compress** *<path_to_config_file\>*
+
+This compresses the data by rewriting it entirely.
+
+This leaves the old data under `.bak` extensions (`event.map.bak` and `lmdb.bak`) If these arready exist, the compress command will fail. You are responsible for deleting or saving this files.
+
+## chorus_dump_approvals
+
+Usage: **chorus_dump_approvals** *<path_to_config_file\>*
+
+This shows the public keys you have approved.
+
+## chorus_moderate
+
+Usage: **chorus_moderate** *<path_to_config_file\>*
+
+This is an interactive tool to moderate events which have been accepted, but which are not authored by a chorus user.  If you chose to approve them, they will become available to the public.
+
+## chorus_cmd
+
+Usage: **chorus_cmd** *<path_to_config_file\>* *<command\>* *[args...]*
+
+Commands available:   delete_by_id (specify the ID in hex),  delete_by_pubkey (specify the pubkey in hex)

run_sample.sh

@@ -1,6 +1,4 @@
 #!/bin/bash
 
-export RUST_LOG='info,chorus=debug'
-
 cargo build --release && \
-    ./target/release/chorus ./sample/sample.config.ron
+    ./target/release/chorus ./sample/sample.config.toml

sample/.gitignore

@@ -1,2 +1,4 @@
 lmdb/
+lmdb.bak/
 event.map
+event.map.bak

sample/sample.config.ron

@@ -1,22 +0,0 @@
-FriendlyConfig(
-    data_directory: "./sample",
-    ip_address: "127.0.0.1",
-    port: 8080,
-    hostname: "localhost",
-    use_tls: false,
-    certchain_pem_path: "tls/fullchain.pem",
-    key_pem_path: "tls/privkey.pem",
-    name: Some("Chorus Sample"),
-    description: Some("A sample run of the Chorus relay"),
-    contact: None,
-    public_key_hex: None,
-    user_hex_keys: [
-      "ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"
-    ],
-    verify_events: true,
-
-    // This is a bad idea in production, but useful for testing or for dumping
-    // your entire relay
-    allow_scraping: true,
-    max_subscriptions: 32,
-)

sample/sample.config.toml

@@ -0,0 +1,35 @@
+# See contrib/chorus.toml for a documented config file
+
+data_directory = "./sample"
+ip_address = "127.0.0.1"
+port = 8081
+hostname = "localhost"
+chorus_is_behind_a_proxy = false
+use_tls = false
+certchain_pem_path = "tls/fullchain.pem"
+key_pem_path = "tls/privkey.pem"
+name = "Chorus Sample"
+description = "A sample run of the Chorus relay"
+privacy_policy = "This relay guarantees nothing. Privacy is your concern, not ours."
+terms_of_service = "This relay guarantees nothing. Use at your own risk."
+# icon_url =
+open_relay = false
+admin_hex_keys = [
+  "ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"
+]
+verify_events = true
+allow_scraping = false
+allow_scrape_if_limited_to = 100
+allow_scrape_if_max_seconds = 7200
+max_subscriptions = 128
+serve_ephemeral = true
+serve_relay_lists = true
+server_log_level = "Info"
+library_log_level = "Info"
+client_log_level = "Warn"
+enable_ip_blocking = true
+minimum_ban_seconds = 1
+timeout_seconds = 60
+max_connections_per_ip = 5
+throttling_bytes_per_second = 131072
+throttling_burst = 4194304

src/bin/chorus.rs

@@ -0,0 +1,192 @@
+use chorus::config::{Config, FriendlyConfig};
+use chorus::counting_stream::CountingStream;
+use chorus::error::Error;
+use chorus::globals::GLOBALS;
+use chorus::ip::HashedPeer;
+use std::env;
+use std::fs::OpenOptions;
+use std::io::Read;
+use std::sync::atomic::Ordering;
+use std::time::Duration;
+use tokio::net::TcpListener;
+use tokio::signal::unix::{signal, SignalKind};
+
+#[tokio::main]
+async fn main() -> Result<(), Error> {
+    // Get args (config path)
+    let mut args = env::args();
+    if args.len() <= 1 {
+        panic!("USAGE: chorus <config_path>");
+    }
+    let _ = args.next(); // ignore program name
+    let config_path = args.next().unwrap();
+
+    let config = chorus::load_config(&config_path)?;
+
+    chorus::setup_logging(&config);
+
+    // Log host name
+    log::info!(target: "Server", "HOSTNAME = {}", config.hostname);
+
+    chorus::setup_store(&config)?;
+
+    if let Some(ref blossom_directory) = config.blossom_directory {
+        let filestore = chorus::filestore::FileStore::new(blossom_directory).await?;
+        let _ = GLOBALS.filestore.set(filestore);
+    }
+
+    // TLS setup
+    let maybe_tls_acceptor = if config.use_tls {
+        log::info!(target: "Server", "Using TLS");
+        Some(chorus::tls::tls_acceptor(&config)?)
+    } else {
+        log::info!(target: "Server", "Not using TLS");
+        None
+    };
+
+    // Bind listener to port
+    let listener = TcpListener::bind((&*config.ip_address, config.port)).await?;
+    log::info!(target: "Server", "Running on {}:{}", config.ip_address, config.port);
+
+    // Store config into GLOBALS
+    *GLOBALS.config.write() = config;
+
+    let mut interrupt_signal = signal(SignalKind::interrupt())?;
+    let mut quit_signal = signal(SignalKind::quit())?;
+    let mut terminate_signal = signal(SignalKind::terminate())?;
+    let mut hup_signal = signal(SignalKind::hangup())?;
+
+    loop {
+        tokio::select! {
+            // Exits gracefully upon exit-type signals
+            v = interrupt_signal.recv() => if v.is_some() {
+                log::info!(target: "Server", "SIGINT");
+                break;
+            },
+            v = quit_signal.recv() => if v.is_some() {
+                log::info!(target: "Server", "SIGQUIT");
+                break;
+            },
+            v = terminate_signal.recv() => if v.is_some() {
+                log::info!(target: "Server", "SIGTERM");
+                break;
+            },
+
+            // Reload config on HUP
+            v = hup_signal.recv() => if v.is_some() {
+                log::info!(target: "Server", "SIGHUP: Reloading configuration");
+
+                // Reload the config file
+                let mut file = OpenOptions::new().read(true).open(config_path.clone())?;
+                let mut contents = String::new();
+                file.read_to_string(&mut contents)?;
+                let friendly_config: FriendlyConfig = toml::from_str(&contents)?;
+                let config: Config = friendly_config.into_config()?;
+
+                *GLOBALS.config.write() = config;
+
+                chorus::print_stats();
+            },
+
+            // Accepts network connections and spawn a task to serve each one
+            v = listener.accept() => {
+                let (tcp_stream, hashed_peer) = {
+                    let (tcp_stream, peer_addr) = v?;
+                    let hashed_peer = HashedPeer::new(peer_addr);
+                    (tcp_stream, hashed_peer)
+                };
+
+                // Possibly IP block early
+                if ! GLOBALS.config.read().chorus_is_behind_a_proxy
+                    && GLOBALS.config.read().enable_ip_blocking
+                {
+                    let ip_data = chorus::get_ip_data(hashed_peer.ip())?;
+                    if ip_data.is_banned() {
+                        log::debug!(target: "Client",
+                                    "{}: Blocking reconnection until {}",
+                                    hashed_peer.ip(),
+                                    ip_data.ban_until);
+                        // note: no need to shutdown() which only drops the write half.
+                        // the whole thing gets dropped when we continue.
+                        continue;
+                    }
+                }
+
+                let counting_stream = CountingStream(tcp_stream);
+
+                let maybe_tls_acceptor_clone = maybe_tls_acceptor.clone();
+                tokio::spawn(async move {
+                    match maybe_tls_acceptor_clone {
+                        Some(tls_acceptor) => {
+                            match tls_acceptor.accept(counting_stream).await {
+                                Ok(stream) => {
+                                    let io = hyper_util::rt::TokioIo::new(stream);
+                                    chorus::serve(io, hashed_peer).await;
+                                },
+                                Err(e) => {
+                                    log::error!(
+                                        target: "Client",
+                                        "{}: TLS accept: {}", hashed_peer, e
+                                    );
+                                }
+                            }
+                        },
+                        None => {
+                            let io = hyper_util::rt::TokioIo::new(counting_stream);
+                            chorus::serve(io, hashed_peer).await;
+                        }
+                    };
+                });
+            }
+        };
+    }
+
+    // Pre-sync in case something below hangs up
+    let _ = GLOBALS.store.get().unwrap().sync();
+
+    // Set the shutting down signal
+    let _ = GLOBALS.shutting_down.send(true);
+
+    // Wait for active websockets to shutdown gracefully
+    let mut num_connections = GLOBALS.num_connections.load(Ordering::Relaxed);
+    if num_connections != 0 {
+        log::info!(target: "Server", "Waiting for {num_connections} websockets to shutdown...");
+
+        // We will check if all clients have shutdown every 50ms
+        let interval = tokio::time::interval(Duration::from_millis(50));
+        tokio::pin!(interval);
+
+        let mut ms = 0;
+
+        while num_connections != 0 {
+            // If we get another shutdown signal, stop waiting for websockets
+            tokio::select! {
+                v = interrupt_signal.recv() => if v.is_some() {
+                    break;
+                },
+                v = quit_signal.recv() => if v.is_some() {
+                    break;
+                },
+                v = terminate_signal.recv() => if v.is_some() {
+                    break;
+                },
+                _instant = interval.tick() => {
+                    ms += 50;
+                    if ms > 5_000 {
+                        log::info!(target: "Server", "Some connections were hung.");
+                        break;
+                    }
+                    num_connections = GLOBALS.num_connections.load(Ordering::Relaxed);
+                    continue;
+                }
+            }
+        }
+    }
+
+    chorus::print_stats();
+
+    log::info!(target: "Server", "Syncing and shutting down.");
+    let _ = GLOBALS.store.get().unwrap().sync();
+
+    Ok(())
+}

src/bin/chorus_cmd.rs

@@ -0,0 +1,107 @@
+use chorus::error::{ChorusError, Error};
+use chorus::globals::GLOBALS;
+use pocket_db::ScreenResult;
+use pocket_types::{Filter, Id, Pubkey, Tags};
+use std::env;
+
+const USAGE: &str = "usage: chorus_cmd <config_path> <command> [args...]";
+
+fn main() -> Result<(), Error> {
+    // Get args (config path)
+    let mut args = env::args();
+    if args.len() <= 1 {
+        panic!("USAGE: chorus_cmd <config_path> <command> [args...]");
+    }
+    let _ = args.next(); // ignore program name
+
+    // Load config
+    let config_path = args
+        .next()
+        .ok_or::<Error>(ChorusError::General(USAGE.to_owned()).into())?;
+    let mut config = chorus::load_config(config_path)?;
+    // Force allow of scraping (this program is a scraper)
+    config.allow_scraping = true;
+
+    chorus::setup_logging(&config);
+    chorus::setup_store(&config)?;
+
+    // Handle command
+    let command = args
+        .next()
+        .ok_or::<Error>(ChorusError::General(USAGE.to_owned()).into())?;
+    match &*command {
+        "delete_by_id" => {
+            let idstr = args
+                .next()
+                .ok_or::<Error>(ChorusError::General("ID argument missing".to_owned()).into())?;
+            let id: Id = Id::read_hex(idstr.as_bytes())?;
+            GLOBALS.store.get().unwrap().remove_event(id)?;
+            println!("Done.");
+        }
+        "delete_by_pubkey" => {
+            let pubstr = args.next().ok_or::<Error>(
+                ChorusError::General("Pubkey argument missing".to_owned()).into(),
+            )?;
+            let pk: Pubkey = Pubkey::read_hex(pubstr.as_bytes())?;
+
+            let mut tags_buffer: [u8; 128] = [0; 128];
+            let (_, tags) = Tags::from_json(b"[]", &mut tags_buffer)?;
+            let mut filter_buffer: [u8; 128] = [0; 128];
+            let filter =
+                Filter::from_parts(&[], &[pk], &[], tags, None, None, None, &mut filter_buffer)?;
+            let (events, _redacted) =
+                GLOBALS
+                    .store
+                    .get()
+                    .unwrap()
+                    .find_events(filter, true, 0, 0, |_| ScreenResult::Match)?;
+            for event in events.iter() {
+                GLOBALS.store.get().unwrap().remove_event(event.id())?;
+            }
+            println!("Done.");
+        }
+        "fetch_by_id" => {
+            let idstr = args
+                .next()
+                .ok_or::<Error>(ChorusError::General("ID argument missing".to_owned()).into())?;
+            let id: Id = Id::read_hex(idstr.as_bytes())?;
+            if let Some(event) = GLOBALS.store.get().unwrap().get_event_by_id(id)? {
+                println!("{event}");
+            } else {
+                println!("Not found.");
+            }
+        }
+        "dump_users" => {
+            let users = chorus::dump_authorized_users()?;
+            for (pubkey, moderator) in users.iter() {
+                println!("{} {}", pubkey, if *moderator { "moderator" } else { "" });
+            }
+        }
+        "add_user" => {
+            let pubstr = args.next().ok_or::<Error>(
+                ChorusError::General("Pubkey argument missing".to_owned()).into(),
+            )?;
+            let pk: Pubkey = Pubkey::read_hex(pubstr.as_bytes())?;
+
+            let moderator = args.next().ok_or::<Error>(
+                ChorusError::General("Moderator argument missing".to_owned()).into(),
+            )?;
+            let moderator: bool = moderator == "1";
+
+            chorus::add_authorized_user(pk, moderator)?;
+        }
+        "rm_user" => {
+            let pubstr = args.next().ok_or::<Error>(
+                ChorusError::General("Pubkey argument missing".to_owned()).into(),
+            )?;
+            let pk: Pubkey = Pubkey::read_hex(pubstr.as_bytes())?;
+
+            chorus::rm_authorized_user(pk)?;
+        }
+        _ => {
+            return Err(ChorusError::General("Unknown command.".to_owned()).into());
+        }
+    }
+
+    Ok(())
+}

src/bin/chorus_compress.rs

@@ -0,0 +1,33 @@
+use chorus::error::Error;
+use std::env;
+use std::io::BufRead;
+fn main() -> Result<(), Error> {
+    // Get args (config path)
+    let mut args = env::args();
+    if args.len() <= 1 {
+        panic!("USAGE: chorus_compress <config_path>");
+    }
+    let _ = args.next(); // ignore program name
+    let config_path = args.next().unwrap();
+
+    let config = chorus::load_config(config_path)?;
+
+    chorus::setup_logging(&config);
+
+    println!("Chorus must NOT be running when you do this.");
+    println!("Proceed? (break out with ^C, or press <ENTER> to proceed)");
+    let stdin = std::io::stdin();
+    let _ = stdin.lock().lines().next().unwrap().unwrap();
+
+    let store = chorus::setup_store_and_return(&config)?;
+
+    let pre_stats = store.stats()?;
+    println!("{:?}", pre_stats);
+
+    let new_store = unsafe { store.rebuild()? };
+
+    let post_stats = new_store.stats()?;
+    println!("{:?}", post_stats);
+
+    Ok(())
+}

src/bin/chorus_dump.rs

@@ -0,0 +1,43 @@
+use chorus::error::Error;
+use chorus::globals::GLOBALS;
+use pocket_db::ScreenResult;
+use pocket_types::{Event, Filter};
+use std::env;
+
+fn main() -> Result<(), Error> {
+    // Get args (config path)
+    let mut args = env::args();
+    if args.len() <= 1 {
+        panic!("USAGE: chorus_dump <chorus_config_path>");
+    }
+    let _ = args.next(); // ignore program name
+    let config_path = args.next().unwrap();
+
+    let mut config = chorus::load_config(config_path)?;
+
+    // Force allow of scraping (this program is a scraper)
+    config.allow_scraping = true;
+
+    chorus::setup_logging(&config);
+    chorus::setup_store(&config)?;
+
+    let mut buffer: [u8; 128] = [0; 128];
+    let (_incount, _outcount, filter) = Filter::from_json(b"{}", &mut buffer)?;
+    let screen = |_: &Event| -> ScreenResult { ScreenResult::Match };
+
+    let (mut events, _redacted) = GLOBALS.store.get().unwrap().find_events(
+        filter,
+        config.allow_scraping,
+        config.allow_scrape_if_limited_to,
+        config.allow_scrape_if_max_seconds,
+        screen,
+    )?;
+
+    for event in events.drain(..) {
+        let bytes = event.as_json()?;
+        let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
+        println!("{s}");
+    }
+
+    Ok(())
+}

src/bin/chorus_dump_approvals.rs

@@ -0,0 +1,30 @@
+use chorus::error::Error;
+use std::env;
+
+fn main() -> Result<(), Error> {
+    // Get args (config path)
+    let mut args = env::args();
+    if args.len() <= 1 {
+        panic!("USAGE: chorus_moderate <config_path>");
+    }
+    let _ = args.next(); // ignore program name
+    let config_path = args.next().unwrap();
+
+    let mut config = chorus::load_config(config_path)?;
+
+    // Force allow of scraping (this program is a scraper)
+    config.allow_scraping = true;
+
+    chorus::setup_logging(&config);
+    chorus::setup_store(&config)?;
+
+    for (id, approved) in chorus::dump_event_approvals()? {
+        println!("ID {} = {}", id, approved);
+    }
+
+    for (pubkey, approved) in chorus::dump_pubkey_approvals()? {
+        println!("PUBKEY {} = {}", pubkey, approved);
+    }
+
+    Ok(())
+}

src/bin/chorus_init.rs

@@ -0,0 +1,27 @@
+use chorus::error::Error;
+use chorus::globals::GLOBALS;
+use std::env;
+
+#[tokio::main]
+async fn main() -> Result<(), Error> {
+    // Get args (config path)
+    let mut args = env::args();
+    if args.len() <= 1 {
+        panic!("USAGE: chorus <config_path>");
+    }
+    let _ = args.next(); // ignore program name
+    let config_path = args.next().unwrap();
+
+    let config = chorus::load_config(&config_path)?;
+
+    chorus::setup_logging(&config);
+
+    // Log host name
+    log::info!(target: "Server", "HOSTNAME = {}", config.hostname);
+
+    chorus::setup_store(&config)?;
+
+    let _ = GLOBALS.store.get().unwrap().sync();
+
+    Ok(())
+}

src/bin/chorus_moderate.rs

@@ -0,0 +1,140 @@
+use chorus::error::Error;
+use chorus::globals::GLOBALS;
+use pocket_db::ScreenResult;
+use pocket_types::{Event, Filter, Kind};
+use std::env;
+use std::io::Write;
+
+fn main() -> Result<(), Error> {
+    // Get args (config path)
+    let mut args = env::args();
+    if args.len() <= 1 {
+        panic!("USAGE: chorus_moderate <config_path>");
+    }
+    let _ = args.next(); // ignore program name
+    let config_path = args.next().unwrap();
+
+    let mut config = chorus::load_config(config_path)?;
+    // Force allow of scraping (this program is a scraper)
+    config.allow_scraping = true;
+
+    chorus::setup_logging(&config);
+
+    // These kinds don't need approval:
+    let allowed_kinds = [
+        Kind::from(4),     // Encrypted Direct Message
+        Kind::from(1059),  // Giftwrap
+        Kind::from(10002), // Relay list
+        Kind::from(10050), // DM Relay list
+        Kind::from(0),     // Metadata
+        Kind::from(3),     // Following list
+        Kind::from(7),     // Reaction
+    ];
+
+    chorus::setup_store(&config)?;
+
+    let mut buffer: [u8; 128] = [0; 128];
+    let (_incount, _outcount, filter) = Filter::from_json(b"{}", &mut buffer)?;
+    let screen = |_: &Event| -> ScreenResult { ScreenResult::Match };
+
+    let (mut events, _redacted) = GLOBALS.store.get().unwrap().find_events(
+        filter,
+        config.allow_scraping,
+        config.allow_scrape_if_limited_to,
+        config.allow_scrape_if_max_seconds,
+        screen,
+    )?;
+
+    let mut stdout = std::io::stdout();
+    let stdin = std::io::stdin();
+    let mut input = String::new();
+
+    'eventloop: for event in events.drain(..) {
+        if allowed_kinds.contains(&event.kind()) {
+            continue;
+        }
+
+        // Skip ephemeral events which don't need approval
+        if event.kind().is_ephemeral() {
+            continue;
+        }
+
+        // Skip if the author is authorized user
+        if chorus::is_authorized_user(event.pubkey()) {
+            continue;
+        }
+
+        //let bytes = event.as_json()?;
+        //let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
+        //println!("{s}");
+
+        // Skip if event marked approved
+        if matches!(chorus::get_event_approval(event.id()), Ok(Some(true))) {
+            continue;
+        }
+
+        // Skip if pubkey marked approved
+        if matches!(chorus::get_pubkey_approval(event.pubkey()), Ok(Some(true))) {
+            continue;
+        }
+
+        // Delete if pubkey marked banned
+        if matches!(chorus::get_pubkey_approval(event.pubkey()), Ok(Some(false))) {
+            GLOBALS.store.get().unwrap().remove_event(event.id())?;
+            continue;
+        }
+
+        println!("---------------------------------------------------------------");
+        println!("kind={} id={}", event.kind(), event.id());
+        println!("{}", String::from_utf8_lossy(event.content()));
+        println!("---------------------------------------------------------------");
+
+        println!("   Pubkey:  (p) approve, (P) ban and delete");
+        println!("   Id:      (i) approve, (I) ban and delete");
+        println!("   other:   (s) skip, (q) quit");
+
+        loop {
+            print!(">> ");
+            let _ = stdout.flush();
+            input.clear();
+            stdin.read_line(&mut input)?;
+            if input.is_empty() {
+                continue;
+            }
+            match input.bytes().next().unwrap() {
+                b'p' => {
+                    chorus::mark_pubkey_approval(event.pubkey(), true)?;
+                    println!("User approved.");
+                    break;
+                }
+                b'P' => {
+                    chorus::mark_pubkey_approval(event.pubkey(), false)?;
+                    GLOBALS.store.get().unwrap().remove_event(event.id())?;
+                    println!("User banned.");
+                    break;
+                }
+                b'i' => {
+                    chorus::mark_event_approval(event.id(), true)?;
+                    println!("Event approved.");
+                    break;
+                }
+                b'I' => {
+                    chorus::mark_event_approval(event.id(), false)?;
+                    GLOBALS.store.get().unwrap().remove_event(event.id())?;
+                    println!("Event banned.");
+                    break;
+                }
+                b's' => {
+                    println!("Skipped.");
+                    break;
+                }
+                b'q' => break 'eventloop,
+                _ => {
+                    println!("?");
+                }
+            }
+        }
+    }
+
+    Ok(())
+}

src/config.rs

@@ -1,45 +1,95 @@
 use crate::error::Error;
-use crate::types::Pubkey;
+use hyper::http::uri::{Authority, Scheme, Uri};
+use pocket_types::Pubkey;
 use serde::{Deserialize, Serialize};
+use std::str::FromStr;
 use url::Host;
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(default)]
 pub struct FriendlyConfig {
     pub data_directory: String,
     pub ip_address: String,
     pub port: u16,
     pub hostname: String,
+    pub chorus_is_behind_a_proxy: bool,
+    pub base_url: Option<String>,
     pub use_tls: bool,
     pub certchain_pem_path: String,
     pub key_pem_path: String,
     pub name: Option<String>,
     pub description: Option<String>,
+    pub banner_url: Option<String>,
+    pub icon_url: Option<String>,
+    pub privacy_policy: Option<String>,
+    pub terms_of_service: Option<String>,
     pub contact: Option<String>,
-    pub public_key_hex: Option<String>,
+    #[serde(alias = "public_key_hex")]
-    pub user_hex_keys: Vec<String>,
+    pub contact_public_key_hex: Option<String>,
+    pub open_relay: bool,
+    pub admin_hex_keys: Vec<String>,
     pub verify_events: bool,
     pub allow_scraping: bool,
+    pub allow_scrape_if_limited_to: u32,
+    pub allow_scrape_if_max_seconds: u64,
+    pub allow_scrape_if_negentropy: bool,
     pub max_subscriptions: usize,
+    pub serve_ephemeral: bool,
+    pub serve_relay_lists: bool,
+    pub server_log_level: String,
+    pub library_log_level: String,
+    pub client_log_level: String,
+    pub enable_ip_blocking: bool,
+    pub minimum_ban_seconds: u64,
+    pub timeout_seconds: u64,
+    pub max_connections_per_ip: usize,
+    pub throttling_bytes_per_second: usize,
+    pub throttling_burst: usize,
+    pub blossom_directory: Option<String>,
+    pub enable_negentropy: bool,
 }
 
 impl Default for FriendlyConfig {
     fn default() -> FriendlyConfig {
         FriendlyConfig {
-            data_directory: "/tmp".to_string(),
+            data_directory: "/opt/chorus/var/chorus".to_string(),
             ip_address: "127.0.0.1".to_string(),
-            port: 80,
+            port: 443,
             hostname: "localhost".to_string(),
-            use_tls: false,
+            chorus_is_behind_a_proxy: false,
-            certchain_pem_path: "./tls/fullchain.pem".to_string(),
+            base_url: None,
-            key_pem_path: "./tls/privkey.pem".to_string(),
+            use_tls: true,
-            name: None,
+            certchain_pem_path: "/opt/chorus/etc/tls/fullchain.pem".to_string(),
-            description: None,
+            key_pem_path: "/opt/chorus/etc/tls/privkey.pem".to_string(),
+            name: Some("Chorus Default".to_string()),
+            description: Some("A default config of the Chorus relay".to_string()),
+            banner_url: None,
+            icon_url: None,
+            privacy_policy: None,
+            terms_of_service: None,
             contact: None,
-            public_key_hex: None,
+            contact_public_key_hex: None,
-            user_hex_keys: vec![],
+            open_relay: false,
+            admin_hex_keys: vec![],
             verify_events: true,
             allow_scraping: false,
-            max_subscriptions: 32,
+            allow_scrape_if_limited_to: 100,
+            allow_scrape_if_max_seconds: 7200,
+            allow_scrape_if_negentropy: true,
+            max_subscriptions: 128,
+            serve_ephemeral: true,
+            serve_relay_lists: true,
+            server_log_level: "Info".to_string(),
+            library_log_level: "Info".to_string(),
+            client_log_level: "Info".to_string(),
+            enable_ip_blocking: true,
+            minimum_ban_seconds: 1,
+            timeout_seconds: 60,
+            max_connections_per_ip: 5,
+            throttling_bytes_per_second: 1024 * 1024,
+            throttling_burst: 1024 * 1024 * 16,
+            blossom_directory: None,
+            enable_negentropy: false,
         }
     }
 }
@@ -51,48 +101,101 @@ impl FriendlyConfig {
             ip_address,
             port,
             hostname,
+            chorus_is_behind_a_proxy,
+            base_url,
             use_tls,
             certchain_pem_path,
             key_pem_path,
             name,
             description,
+            banner_url,
+            icon_url,
+            privacy_policy,
+            terms_of_service,
             contact,
-            public_key_hex,
+            contact_public_key_hex,
-            user_hex_keys,
+            open_relay,
+            admin_hex_keys,
             verify_events,
             allow_scraping,
+            allow_scrape_if_limited_to,
+            allow_scrape_if_max_seconds,
+            allow_scrape_if_negentropy,
             max_subscriptions,
+            serve_ephemeral,
+            serve_relay_lists,
+            server_log_level,
+            library_log_level,
+            client_log_level,
+            enable_ip_blocking,
+            minimum_ban_seconds,
+            timeout_seconds,
+            max_connections_per_ip,
+            throttling_bytes_per_second,
+            throttling_burst,
+            blossom_directory,
+            enable_negentropy,
         } = self;
 
-        let mut public_key: Option<Pubkey> = None;
+        let mut contact_public_key: Option<Pubkey> = None;
-        if let Some(pkh) = public_key_hex {
+        if let Some(pkh) = contact_public_key_hex {
-            public_key = Some(Pubkey::read_hex(pkh.as_bytes())?);
+            contact_public_key = Some(Pubkey::read_hex(pkh.as_bytes())?);
         };
 
-        let mut user_keys: Vec<Pubkey> = Vec::with_capacity(user_hex_keys.len());
+        let mut admin_keys: Vec<Pubkey> = Vec::with_capacity(admin_hex_keys.len());
-        for pkh in user_hex_keys.iter() {
+        for pkh in admin_hex_keys.iter() {
-            user_keys.push(Pubkey::read_hex(pkh.as_bytes())?);
+            admin_keys.push(Pubkey::read_hex(pkh.as_bytes())?);
         }
 
         let hostname = Host::parse(&hostname)?;
 
+        let server_log_level =
+            log::LevelFilter::from_str(&server_log_level).unwrap_or(log::LevelFilter::Info);
+        let library_log_level =
+            log::LevelFilter::from_str(&library_log_level).unwrap_or(log::LevelFilter::Info);
+        let client_log_level =
+            log::LevelFilter::from_str(&client_log_level).unwrap_or(log::LevelFilter::Info);
+
         Ok(Config {
             data_directory,
             ip_address,
             port,
             hostname,
+            chorus_is_behind_a_proxy,
+            base_url,
             use_tls,
             certchain_pem_path,
             key_pem_path,
             name,
             description,
+            banner_url,
+            icon_url,
+            privacy_policy,
+            terms_of_service,
             contact,
-            public_key,
+            contact_public_key,
-            user_keys,
+            open_relay,
-            user_hex_keys,
+            admin_keys,
+            admin_hex_keys,
             verify_events,
             allow_scraping,
+            allow_scrape_if_limited_to,
+            allow_scrape_if_max_seconds,
+            allow_scrape_if_negentropy,
             max_subscriptions,
+            serve_ephemeral,
+            serve_relay_lists,
+            server_log_level,
+            library_log_level,
+            client_log_level,
+            enable_ip_blocking,
+            minimum_ban_seconds,
+            timeout_seconds,
+            max_connections_per_ip,
+            throttling_bytes_per_second,
+            throttling_burst,
+            blossom_directory,
+            enable_negentropy,
         })
     }
 }
@@ -103,16 +206,76 @@ pub struct Config {
     pub ip_address: String,
     pub port: u16,
     pub hostname: Host,
+    pub chorus_is_behind_a_proxy: bool,
+    pub base_url: Option<String>,
     pub use_tls: bool,
     pub certchain_pem_path: String,
     pub key_pem_path: String,
     pub name: Option<String>,
     pub description: Option<String>,
+    pub banner_url: Option<String>,
+    pub icon_url: Option<String>,
+    pub privacy_policy: Option<String>,
+    pub terms_of_service: Option<String>,
     pub contact: Option<String>,
-    pub public_key: Option<Pubkey>,
+    pub contact_public_key: Option<Pubkey>,
-    pub user_keys: Vec<Pubkey>,
+    pub open_relay: bool,
-    pub user_hex_keys: Vec<String>,
+    pub admin_keys: Vec<Pubkey>,
+    pub admin_hex_keys: Vec<String>,
     pub verify_events: bool,
     pub allow_scraping: bool,
+    pub allow_scrape_if_limited_to: u32,
+    pub allow_scrape_if_max_seconds: u64,
+    pub allow_scrape_if_negentropy: bool,
     pub max_subscriptions: usize,
+    pub serve_ephemeral: bool,
+    pub serve_relay_lists: bool,
+    pub server_log_level: log::LevelFilter,
+    pub library_log_level: log::LevelFilter,
+    pub client_log_level: log::LevelFilter,
+    pub enable_ip_blocking: bool,
+    pub minimum_ban_seconds: u64,
+    pub timeout_seconds: u64,
+    pub max_connections_per_ip: usize,
+    pub throttling_bytes_per_second: usize,
+    pub throttling_burst: usize,
+    pub blossom_directory: Option<String>,
+    pub enable_negentropy: bool,
+}
+
+impl Default for Config {
+    fn default() -> Config {
+        let friendly = FriendlyConfig::default();
+
+        // We know the default config passes into_config without error:
+        friendly.into_config().unwrap()
+    }
+}
+
+impl Config {
+    /// Get the URI for our server matching the inner Uri, overridden with either
+    /// our base_url parts or our hostname/port.
+    pub fn uri_parts(&self, inner: Uri, http: bool) -> Result<http::uri::Parts, Error> {
+        let mut uri_parts = inner.into_parts();
+
+        if let Some(s) = &self.base_url {
+            let base_uri = s.parse::<Uri>()?;
+            let base_uri_parts = base_uri.into_parts();
+            uri_parts.scheme = base_uri_parts.scheme;
+            uri_parts.authority = base_uri_parts.authority;
+        } else {
+            let scheme = match (self.use_tls, http) {
+                (false, false) => Scheme::from_str("ws").unwrap(),
+                (true, false) => Scheme::from_str("wss").unwrap(),
+                (false, true) => Scheme::HTTP,
+                (true, true) => Scheme::HTTPS,
+            };
+            uri_parts.scheme = Some(scheme);
+
+            let authority = Authority::from_str(&format!("{}:{}", self.hostname, self.port))?;
+            uri_parts.authority = Some(authority);
+        }
+
+        Ok(uri_parts)
+    }
 }

src/counting_stream.rs

@@ -0,0 +1,53 @@
+use crate::globals::GLOBALS;
+use std::pin::Pin;
+use std::sync::atomic::Ordering;
+use std::task::{Context, Poll};
+use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
+
+#[derive(Debug)]
+pub struct CountingStream<S>(pub S);
+
+impl<S: AsyncRead + Unpin> AsyncRead for CountingStream<S> {
+    fn poll_read(
+        self: Pin<&mut Self>,
+        cx: &mut Context<'_>,
+        buf: &mut ReadBuf<'_>,
+    ) -> Poll<std::io::Result<()>> {
+        // Count bytes for statistics
+        let pre = buf.filled().len();
+        let result = Pin::new(&mut self.get_mut().0).poll_read(cx, buf);
+        let post = buf.filled().len();
+        let count = post - pre;
+        if count > 0 {
+            let _ = GLOBALS
+                .bytes_inbound
+                .fetch_add(count as u64, Ordering::SeqCst);
+        }
+        result
+    }
+}
+
+impl<S: AsyncWrite + Unpin> AsyncWrite for CountingStream<S> {
+    fn poll_write(
+        self: Pin<&mut Self>,
+        cx: &mut Context<'_>,
+        buf: &[u8],
+    ) -> Poll<Result<usize, std::io::Error>> {
+        // Count bytes for statistics
+        let _ = GLOBALS
+            .bytes_outbound
+            .fetch_add(buf.len() as u64, Ordering::SeqCst);
+        Pin::new(&mut self.get_mut().0).poll_write(cx, buf)
+    }
+
+    fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), std::io::Error>> {
+        Pin::new(&mut self.get_mut().0).poll_flush(cx)
+    }
+
+    fn poll_shutdown(
+        self: Pin<&mut Self>,
+        cx: &mut Context<'_>,
+    ) -> Poll<Result<(), std::io::Error>> {
+        Pin::new(&mut self.get_mut().0).poll_shutdown(cx)
+    }
+}

src/error.rs

@@ -1,3 +1,4 @@
+use std::convert::Infallible;
 use std::error::Error as StdError;
 use std::panic::Location;
 
@@ -22,93 +23,150 @@ impl std::fmt::Display for Error {
 /// Errors that can occur in the chorus crate
 #[derive(Debug)]
 pub enum ChorusError {
-    // Auth failure
+    // Nostr AUTH failure
     AuthFailure(String),
 
     // Auth required
     AuthRequired,
 
-    // Bad event id
+    // Bad request
-    BadEventId,
+    BadRequest(&'static str),
 
-    // Bad hex input
+    // Bad X-Real-Ip header
-    BadHexInput,
+    BadRealIpHeader(String),
 
-    // Output buffer too small
+    // Bad X-Real-Ip header characters
-    BufferTooSmall,
+    BadRealIpHeaderCharacters,
+
+    // Event is banned
+    BannedEvent,
+
+    // User is banned
+    BannedUser,
+
+    // Base64 Decode Error
+    Base64Decode(base64::DecodeError), // 24b
+
+    // Blocked IP
+    BlockedIp,
+
+    // Blossom Authorization failure
+    BlossomAuthFailure(String),
 
     // Channel Recv
-    ChannelRecv(tokio::sync::broadcast::error::RecvError),
+    ChannelRecv(tokio::sync::broadcast::error::RecvError), // 24b
 
     // Channel Send
-    ChannelSend(tokio::sync::broadcast::error::SendError<usize>),
+    ChannelSend(tokio::sync::broadcast::error::SendError<u64>), // 16b
 
     // Config
-    Config(ron::error::SpannedError),
+    Config(Box<toml::de::Error>), // 16b
 
     // Crypto
-    Crypto(secp256k1::Error),
+    Crypto(secp256k1::Error), // 16b
 
-    // Deleted event
+    // Closing on error(s)
-    Deleted,
+    ErrorClose,
-
-    // Duplicate event
-    Duplicate,
-
-    // End of Input
-    EndOfInput,
 
     // Event is Invalid
     EventIsInvalid(String),
 
+    // From hex
+    FromHex(hex::FromHexError), // 24b
+
+    // From UTF8
+    FromUtf8(Box<std::string::FromUtf8Error>),
+
+    // General
+    General(String),
+
     // Http
-    Http(hyper::http::Error),
+    Http(Box<hyper::http::Error>),
 
     // Hyper
     Hyper(hyper::Error),
 
+    // Infallible
+    Infallible,
+
+    // Invalid URI
+    InvalidUri(hyper::http::uri::InvalidUri),
+
+    // Invalid URI Parts
+    InvalidUriParts(hyper::http::uri::InvalidUriParts),
+
     // I/O
     Io(std::io::Error),
 
-    // JSON Bad (general)
+    // Management Authorization failure
-    JsonBad(&'static str, usize),
+    ManagementAuthFailure(String),
 
-    // JSON Bad Character
+    // Missing Table
-    JsonBadCharacter(char, usize, char),
+    MissingTable(&'static str),
 
-    // JSON Bad Event
+    // Negentropy error
-    JsonBadEvent(&'static str, usize),
+    Negentropy(negentropy::Error),
 
-    // JSON Bad Filter
+    // Non-ASCII HTTP header value
-    JsonBadFilter(&'static str, usize),
+    NonAsciiHttpHeaderValue(http::header::ToStrError),
-
-    // JSON Bad String Character
-    JsonBadStringChar(u32),
-
-    // JSON Escape
-    JsonEscape,
-
-    // JSON Escape Surrogate
-    JsonEscapeSurrogate,
-
-    // LMDB
-    Lmdb(heed::Error),
 
     // No private key
     NoPrivateKey,
 
+    // Not Implemented
+    NotImplemented,
+
+    // No such subscription
+    NoSuchSubscription,
+
+    // Protected Event
+    ProtectedEvent,
+
+    // Pocket Db Error
+    PocketDb(pocket_db::Error),
+
+    // Pocket Db Heed Error
+    PocketDbHeed(Box<pocket_db::heed::Error>),
+
+    // Pocket Types Error
+    PocketType(pocket_types::Error),
+
+    // Rate limit exceeded
+    RateLimitExceeded,
+
+    // X-Real-Ip header is missing
+    RealIpHeaderMissing,
+
     // Restricted
     Restricted,
 
     // Rustls
-    Rustls(tokio_rustls::rustls::Error),
+    Rustls(Box<tokio_rustls::rustls::Error>),
-
-    // Tungstenite
-    Tungstenite(hyper_tungstenite::tungstenite::error::Error),
 
     // Filter is underspecified
     Scraper,
 
+    // Serde JSON
+    SerdeJson(serde_json::Error),
+
+    // Shutting Down
+    ShuttingDown,
+
+    // Signal - Not Blossom Request
+    SignalNotBlossom,
+
+    // Speedy
+    Speedy(speedy::Error),
+
+    // Timed Out
+    TimedOut,
+
+    // Too many subscriptions
+    TooManySubscriptions,
+
+    // Tungstenite
+    Tungstenite(Box<hyper_tungstenite::tungstenite::error::Error>),
+
     // URL Parse
     UrlParse(url::ParseError),
 
@@ -127,45 +185,56 @@ impl std::fmt::Display for ChorusError {
         match self {
             ChorusError::AuthFailure(s) => write!(f, "AUTH failure: {s}"),
             ChorusError::AuthRequired => write!(f, "AUTH required"),
-            ChorusError::BadEventId => write!(f, "Bad event id, does not match hash"),
+            ChorusError::BadRequest(s) => write!(f, "Bad Request: {s}"),
-            ChorusError::BadHexInput => write!(f, "Bad hex input"),
+            ChorusError::BadRealIpHeader(s) => write!(f, "Bad X-Real-Ip header: {s}"),
-            ChorusError::BufferTooSmall => write!(f, "Output buffer too small"),
+            ChorusError::BadRealIpHeaderCharacters => {
+                write!(f, "Bad X-Real-Ip header (non utf-8 characters)")
+            }
+            ChorusError::BannedEvent => write!(f, "Event is banned"),
+            ChorusError::BannedUser => write!(f, "User is banned"),
+            ChorusError::Base64Decode(e) => write!(f, "{e}"),
+            ChorusError::BlockedIp => write!(f, "IP is temporarily blocked"),
+            ChorusError::BlossomAuthFailure(s) => write!(f, "Authorization failure: {s}"),
             ChorusError::ChannelRecv(e) => write!(f, "{e}"),
             ChorusError::ChannelSend(e) => write!(f, "{e}"),
             ChorusError::Config(e) => write!(f, "{e}"),
             ChorusError::Crypto(e) => write!(f, "{e}"),
-            ChorusError::Deleted => write!(f, "Event was previously deleted"),
+            ChorusError::ErrorClose => write!(f, "Closing due to error(s)"),
-            ChorusError::Duplicate => write!(f, "Duplicate event"),
-            ChorusError::EndOfInput => write!(f, "End of input"),
             ChorusError::EventIsInvalid(s) => write!(f, "Event is invalid: {s}"),
+            ChorusError::FromHex(e) => write!(f, "{e}"),
+            ChorusError::FromUtf8(e) => write!(f, "{e}"),
+            ChorusError::General(s) => write!(f, "{s}"),
             ChorusError::Http(e) => write!(f, "{e}"),
             ChorusError::Hyper(e) => write!(f, "{e}"),
+            ChorusError::Infallible => panic!("INFALLIBLE"),
+            ChorusError::InvalidUri(e) => write!(f, "{e}"),
+            ChorusError::InvalidUriParts(e) => write!(f, "{e}"),
             ChorusError::Io(e) => write!(f, "{e}"),
-            ChorusError::JsonBad(err, pos) => write!(f, "JSON bad: {err} at position {pos}"),
+            ChorusError::ManagementAuthFailure(s) => write!(f, "Authorization failure: {s}"),
-            ChorusError::JsonBadCharacter(c, pos, ec) => write!(
+            ChorusError::MissingTable(t) => write!(f, "Missing table: {t}"),
-                f,
+            ChorusError::Negentropy(e) => write!(f, "Negentropy: {e}"),
-                "JSON bad character: {c} at position {pos}, {ec} was expected"
+            ChorusError::NonAsciiHttpHeaderValue(e) => {
-            ),
+                write!(f, "Non ASCII HTTP header value: {e}")
-            ChorusError::JsonBadEvent(err, pos) => {
-                write!(f, "JSON bad event: {err} at position {pos}")
             }
-            ChorusError::JsonBadFilter(err, pos) => {
-                write!(f, "JSON bad filter: {err} at position {pos}")
-            }
-            ChorusError::JsonBadStringChar(ch) => {
-                write!(f, "JSON string bad character: codepoint {ch}")
-            }
-            ChorusError::JsonEscape => write!(f, "JSON string escape error"),
-            ChorusError::JsonEscapeSurrogate => write!(
-                f,
-                "JSON string escape surrogate (ancient style) is not supported"
-            ),
-            ChorusError::Lmdb(e) => write!(f, "{e}"),
             ChorusError::NoPrivateKey => write!(f, "Private Key Not Found"),
+            ChorusError::NotImplemented => write!(f, "Not implemented"),
+            ChorusError::NoSuchSubscription => write!(f, "No such subscription"),
+            ChorusError::PocketDb(e) => write!(f, "{e}"),
+            ChorusError::PocketDbHeed(e) => write!(f, "{e}"),
+            ChorusError::PocketType(e) => write!(f, "{e}"),
+            ChorusError::RateLimitExceeded => write!(f, "Rate limit exceeded"),
+            ChorusError::ProtectedEvent => write!(f, "Protected event"),
+            ChorusError::RealIpHeaderMissing => write!(f, "X-Real-Ip header is missing"),
             ChorusError::Restricted => write!(f, "Restricted"),
             ChorusError::Rustls(e) => write!(f, "{e}"),
-            ChorusError::Tungstenite(e) => write!(f, "{e}"),
             ChorusError::Scraper => write!(f, "Filter is underspecified. Scrapers are not allowed"),
+            ChorusError::SerdeJson(e) => write!(f, "{e}"),
+            ChorusError::ShuttingDown => write!(f, "Shutting down"),
+            ChorusError::SignalNotBlossom => write!(f, "internal-signal-not-blossom"),
+            ChorusError::Speedy(e) => write!(f, "{e}"),
+            ChorusError::TimedOut => write!(f, "Timed out"),
+            ChorusError::TooManySubscriptions => write!(f, "Too many subscriptions"),
+            ChorusError::Tungstenite(e) => write!(f, "{e}"),
             ChorusError::UrlParse(e) => write!(f, "{e}"),
             ChorusError::Utf8(e) => write!(f, "{e}"),
             ChorusError::Utf8Error => write!(f, "UTF-8 error"),
@@ -177,15 +246,25 @@ impl std::fmt::Display for ChorusError {
 impl StdError for ChorusError {
     fn source(&self) -> Option<&(dyn StdError + 'static)> {
         match self {
+            ChorusError::Base64Decode(e) => Some(e),
             ChorusError::ChannelRecv(e) => Some(e),
             ChorusError::ChannelSend(e) => Some(e),
             ChorusError::Config(e) => Some(e),
             ChorusError::Crypto(e) => Some(e),
+            ChorusError::FromHex(e) => Some(e),
+            ChorusError::FromUtf8(e) => Some(e),
             ChorusError::Http(e) => Some(e),
             ChorusError::Hyper(e) => Some(e),
+            ChorusError::InvalidUri(e) => Some(e),
+            ChorusError::InvalidUriParts(e) => Some(e),
             ChorusError::Io(e) => Some(e),
-            ChorusError::Lmdb(e) => Some(e),
+            ChorusError::NonAsciiHttpHeaderValue(e) => Some(e),
+            ChorusError::PocketDb(e) => Some(e),
+            ChorusError::PocketDbHeed(e) => Some(e),
+            ChorusError::PocketType(e) => Some(e),
             ChorusError::Rustls(e) => Some(e),
+            ChorusError::SerdeJson(e) => Some(e),
+            ChorusError::Speedy(e) => Some(e),
             ChorusError::Tungstenite(e) => Some(e),
             ChorusError::UrlParse(e) => Some(e),
             ChorusError::Utf8(e) => Some(e),
@@ -195,6 +274,73 @@ impl StdError for ChorusError {
     }
 }
 
+impl ChorusError {
+    #[track_caller]
+    pub fn into_err(self) -> Error {
+        Error {
+            inner: self,
+            location: std::panic::Location::caller(),
+        }
+    }
+
+    pub fn punishment(&self) -> f32 {
+        match self {
+            ChorusError::AuthFailure(_) => 0.25,
+            ChorusError::AuthRequired => 0.0,
+            ChorusError::BadRequest(_) => 0.1,
+            ChorusError::BadRealIpHeader(_) => 0.0,
+            ChorusError::BadRealIpHeaderCharacters => 0.0,
+            ChorusError::BannedEvent => 0.1,
+            ChorusError::BannedUser => 0.2,
+            ChorusError::Base64Decode(_) => 0.0,
+            ChorusError::BlockedIp => 0.0,
+            ChorusError::BlossomAuthFailure(_) => 0.0,
+            ChorusError::ChannelRecv(_) => 0.0,
+            ChorusError::ChannelSend(_) => 0.0,
+            ChorusError::Config(_) => 0.0,
+            ChorusError::Crypto(_) => 0.1,
+            ChorusError::ErrorClose => 1.0,
+            ChorusError::EventIsInvalid(_) => 0.2,
+            ChorusError::FromHex(_) => 0.2,
+            ChorusError::FromUtf8(_) => 0.2,
+            ChorusError::General(_) => 0.0,
+            ChorusError::Http(_) => 0.0,
+            ChorusError::Hyper(_) => 0.0,
+            ChorusError::Infallible => panic!("INFALLIBLE"),
+            ChorusError::InvalidUri(_) => 0.0,
+            ChorusError::InvalidUriParts(_) => 0.0,
+            ChorusError::Io(_) => 0.0,
+            ChorusError::ManagementAuthFailure(_) => 0.0,
+            ChorusError::MissingTable(_) => 0.0,
+            ChorusError::Negentropy(_) => 0.1,
+            ChorusError::NonAsciiHttpHeaderValue(_) => 0.2,
+            ChorusError::NoPrivateKey => 0.0,
+            ChorusError::NotImplemented => 0.0,
+            ChorusError::NoSuchSubscription => 0.05,
+            ChorusError::PocketDb(_) => 0.0,
+            ChorusError::PocketDbHeed(_) => 0.0,
+            ChorusError::PocketType(_) => 0.25,
+            ChorusError::RateLimitExceeded => 1.0,
+            ChorusError::ProtectedEvent => 0.35,
+            ChorusError::RealIpHeaderMissing => 0.0,
+            ChorusError::Restricted => 0.1,
+            ChorusError::Rustls(_) => 0.0,
+            ChorusError::Scraper => 0.4,
+            ChorusError::SerdeJson(_) => 0.0,
+            ChorusError::ShuttingDown => 0.0,
+            ChorusError::SignalNotBlossom => 0.0,
+            ChorusError::Speedy(_) => 0.0,
+            ChorusError::TimedOut => 0.1,
+            ChorusError::TooManySubscriptions => 0.1,
+            ChorusError::Tungstenite(_) => 0.0,
+            ChorusError::UrlParse(_) => 0.1,
+            ChorusError::Utf8(_) => 0.1,
+            ChorusError::Utf8Error => 0.1,
+            ChorusError::WebsocketProtocol(_) => 0.1,
+        }
+    }
+}
+
 // Note: we impl Into because our typical pattern is ChorusError::Variant.into()
 //       when we tried implementing From, the location was deep in rust code's
 //       blanket into implementation, which wasn't the line number we wanted.
@@ -222,9 +368,9 @@ impl From<tokio::sync::broadcast::error::RecvError> for Error {
     }
 }
 
-impl From<tokio::sync::broadcast::error::SendError<usize>> for Error {
+impl From<tokio::sync::broadcast::error::SendError<u64>> for Error {
     #[track_caller]
-    fn from(err: tokio::sync::broadcast::error::SendError<usize>) -> Self {
+    fn from(err: tokio::sync::broadcast::error::SendError<u64>) -> Self {
         Error {
             inner: ChorusError::ChannelSend(err),
             location: std::panic::Location::caller(),
@@ -232,11 +378,11 @@ impl From<tokio::sync::broadcast::error::SendError<usize>> for Error {
     }
 }
 
-impl From<ron::error::SpannedError> for Error {
+impl From<toml::de::Error> for Error {
     #[track_caller]
-    fn from(err: ron::error::SpannedError) -> Self {
+    fn from(err: toml::de::Error) -> Self {
         Error {
-            inner: ChorusError::Config(err),
+            inner: ChorusError::Config(Box::new(err)),
             location: std::panic::Location::caller(),
         }
     }
@@ -256,7 +402,7 @@ impl From<hyper::http::Error> for Error {
     #[track_caller]
     fn from(err: hyper::http::Error) -> Self {
         Error {
-            inner: ChorusError::Http(err),
+            inner: ChorusError::Http(Box::new(err)),
             location: std::panic::Location::caller(),
         }
     }
@@ -272,6 +418,26 @@ impl From<hyper::Error> for Error {
     }
 }
 
+impl From<hyper::http::uri::InvalidUri> for Error {
+    #[track_caller]
+    fn from(err: hyper::http::uri::InvalidUri) -> Self {
+        Error {
+            inner: ChorusError::InvalidUri(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<hyper::http::uri::InvalidUriParts> for Error {
+    #[track_caller]
+    fn from(err: hyper::http::uri::InvalidUriParts) -> Self {
+        Error {
+            inner: ChorusError::InvalidUriParts(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
 impl From<std::io::Error> for Error {
     #[track_caller]
     fn from(err: std::io::Error) -> Self {
@@ -282,11 +448,41 @@ impl From<std::io::Error> for Error {
     }
 }
 
-impl From<heed::Error> for Error {
+impl From<http::header::ToStrError> for Error {
     #[track_caller]
-    fn from(err: heed::Error) -> Self {
+    fn from(err: http::header::ToStrError) -> Self {
         Error {
-            inner: ChorusError::Lmdb(err),
+            inner: ChorusError::NonAsciiHttpHeaderValue(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<pocket_db::Error> for Error {
+    #[track_caller]
+    fn from(err: pocket_db::Error) -> Self {
+        Error {
+            inner: ChorusError::PocketDb(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<pocket_db::heed::Error> for Error {
+    #[track_caller]
+    fn from(err: pocket_db::heed::Error) -> Self {
+        Error {
+            inner: ChorusError::PocketDbHeed(Box::new(err)),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<pocket_types::Error> for Error {
+    #[track_caller]
+    fn from(err: pocket_types::Error) -> Self {
+        Error {
+            inner: ChorusError::PocketType(err),
             location: std::panic::Location::caller(),
         }
     }
@@ -296,7 +492,7 @@ impl From<tokio_rustls::rustls::Error> for Error {
     #[track_caller]
     fn from(err: tokio_rustls::rustls::Error) -> Self {
         Error {
-            inner: ChorusError::Rustls(err),
+            inner: ChorusError::Rustls(Box::new(err)),
             location: std::panic::Location::caller(),
         }
     }
@@ -306,7 +502,7 @@ impl From<hyper_tungstenite::tungstenite::error::Error> for Error {
     #[track_caller]
     fn from(err: hyper_tungstenite::tungstenite::error::Error) -> Self {
         Error {
-            inner: ChorusError::Tungstenite(err),
+            inner: ChorusError::Tungstenite(Box::new(err)),
             location: std::panic::Location::caller(),
         }
     }
@@ -341,3 +537,75 @@ impl From<url::ParseError> for Error {
         }
     }
 }
+
+impl From<speedy::Error> for Error {
+    #[track_caller]
+    fn from(err: speedy::Error) -> Self {
+        Error {
+            inner: ChorusError::Speedy(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<serde_json::Error> for Error {
+    #[track_caller]
+    fn from(err: serde_json::Error) -> Self {
+        Error {
+            inner: ChorusError::SerdeJson(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<hex::FromHexError> for Error {
+    #[track_caller]
+    fn from(err: hex::FromHexError) -> Self {
+        Error {
+            inner: ChorusError::FromHex(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<std::string::FromUtf8Error> for Error {
+    #[track_caller]
+    fn from(err: std::string::FromUtf8Error) -> Self {
+        Error {
+            inner: ChorusError::FromUtf8(Box::new(err)),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<base64::DecodeError> for Error {
+    #[track_caller]
+    fn from(err: base64::DecodeError) -> Self {
+        Error {
+            inner: ChorusError::Base64Decode(err),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<negentropy::Error> for Error {
+    #[track_caller]
+    fn from(e: negentropy::Error) -> Error {
+        Error {
+            inner: ChorusError::Negentropy(e),
+            location: std::panic::Location::caller(),
+        }
+    }
+}
+
+impl From<Infallible> for Error {
+    fn from(_: Infallible) -> Self {
+        panic!("INFALLIBLE")
+    }
+}
+
+impl From<Error> for std::io::Error {
+    fn from(e: Error) -> std::io::Error {
+        std::io::Error::other(e)
+    }
+}

src/filestore/hash_output.rs

@@ -0,0 +1,65 @@
+use crate::{ChorusError, Error};
+use std::fmt;
+use std::path::{Path, PathBuf};
+
+/// A simple type for a SHA-256 hash output of 32 bytes
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub struct HashOutput([u8; 32]);
+
+impl HashOutput {
+    pub fn from_engine(engine: bitcoin_hashes::sha256::HashEngine) -> HashOutput {
+        use bitcoin_hashes::sha256;
+        let hashvalue = sha256::Hash::from_engine(engine);
+        HashOutput(hashvalue.as_byte_array()[0..32].try_into().unwrap())
+    }
+
+    pub fn from_hex(input: &str) -> Result<HashOutput, Error> {
+        let bytes = hex::decode(input)?;
+        if bytes.len() == 32 {
+            Ok(HashOutput(bytes.try_into().unwrap()))
+        } else {
+            Err(
+                ChorusError::General("HashOutput::from_hex() got wrong length string".to_string())
+                    .into(),
+            )
+        }
+    }
+
+    pub fn from_bytes(bytes: [u8; 32]) -> HashOutput {
+        HashOutput(bytes)
+    }
+
+    pub fn to_pathbuf<P: AsRef<Path>>(&self, base: P) -> PathBuf {
+        let s = hex::encode(self.0);
+        let mut output: PathBuf = PathBuf::new();
+        output.push(base);
+        output.push(&s[0..=1]);
+        output.push(&s[2..=3]);
+        output.push(&s[4..]);
+        output
+    }
+}
+
+impl fmt::Display for HashOutput {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "{}", hex::encode(self.0))
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    fn test_hash_output_to_pathbuf() {
+        let hash = HashOutput::from_hex(
+            "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        )
+        .unwrap();
+
+        assert_eq!(
+            &format!("{}", hash.to_pathbuf("/tmp").display()),
+            "/tmp/e3/b0/c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+        );
+    }
+}

src/filestore/mod.rs

@@ -0,0 +1,173 @@
+use crate::error::{ChorusError, Error};
+use futures::TryStreamExt;
+use http_body_util::combinators::BoxBody;
+use http_body_util::{BodyDataStream, BodyExt, StreamBody};
+use hyper::body::{Bytes, Frame};
+use std::fs::Metadata;
+use std::path::{Path, PathBuf};
+use tokio::fs;
+use tokio::fs::File;
+use tokio_util::io::{InspectReader, ReaderStream, StreamReader};
+
+mod hash_output;
+pub use hash_output::HashOutput;
+
+pub struct FileStore {
+    pub base: PathBuf,
+    pub temp: PathBuf,
+}
+
+impl FileStore {
+    pub async fn new<P: AsRef<Path>>(base: P) -> Result<FileStore, Error> {
+        let base = base.as_ref().to_owned();
+
+        let temp = {
+            let mut temp = base.clone();
+            temp.push("temp");
+            temp
+        };
+
+        if !fs::try_exists(&temp).await? {
+            fs::create_dir_all(&temp).await?;
+        }
+
+        Ok(FileStore { base, temp })
+    }
+
+    fn tmpfile(&self) -> PathBuf {
+        let mut tf = self.temp.clone();
+        let nonce = textnonce::TextNonce::sized_urlsafe(32).unwrap();
+        tf.push(&nonce.0);
+        tf
+    }
+
+    /// Store a file in storage, streamed from a hyper BoxBody
+    ///
+    /// Returns it's HashOutput by which it can be later retrieved or deleted.
+    pub async fn store(
+        &self,
+        data: BoxBody<Bytes, Error>,
+        expected_hash: Option<HashOutput>,
+    ) -> Result<(u64, HashOutput, Option<String>), Error> {
+        use bitcoin_hashes::sha256;
+        use std::io::Write; // for hash_engine.write_all()
+
+        // We will download into a temporary file (as we don't know the hash yet)
+        let temppathbuf = self.tmpfile();
+        let mut tempfile = File::options()
+            .create(true)
+            .truncate(true)
+            .write(true)
+            .open(&temppathbuf)
+            .await?;
+
+        // Convert the BoxBody into a Data Stream
+        let body_stream = BodyDataStream::new(data);
+
+        // Convert the Data Stream into something that is AsyncRead (over bytes)
+        let stream_reader = StreamReader::new(body_stream);
+
+        // Wrap this in something that lets us inspect the content so we can
+        // hash it as it comes in, as well as count the size
+        let mut size: u64 = 0;
+        let mut hash_engine = sha256::HashEngine::default();
+        let mut inspect_reader = InspectReader::new(stream_reader, |bytes: &[u8]| {
+            size += bytes.len() as u64;
+            hash_engine.write_all(bytes).unwrap(); // I don't think hashing will fail
+        });
+
+        // Copy the data into the tempfile (hashing and counting as we go)
+        let count = tokio::io::copy(&mut inspect_reader, &mut tempfile).await?;
+        drop(tempfile);
+
+        // Verify our code was correct
+        if count != size {
+            return Err(ChorusError::General("INTERNAL COUNT MISMATCH".to_string()).into());
+        }
+
+        // Finish the hash
+        let hash = HashOutput::from_engine(hash_engine);
+
+        // Verify the expected hash matches
+        if let Some(expected) = expected_hash {
+            if hash != expected {
+                // Remove the temporary file
+                fs::remove_file(&temppathbuf).await?;
+
+                // And complain
+                return Err(ChorusError::BlossomAuthFailure(
+                    "File hash does not match authorized hash".to_string(),
+                )
+                .into());
+            }
+        }
+
+        // Sniff the mime-type
+        let maybe_mime_string = {
+            use mime_sniffer::MimeTypeSniffer;
+            use tokio::io::AsyncReadExt;
+            let mut readtempfile = File::open(&temppathbuf).await?;
+            let mut buffer: Vec<u8> = vec![0; 128];
+            let _ = readtempfile.read(&mut buffer).await?;
+            buffer.sniff_mime_type().map(|s| s.to_string())
+        };
+
+        // Compute the proper path
+        let pathbuf = hash.to_pathbuf(&self.base);
+
+        // If it already exists, trust the existing copy
+        if fs::try_exists(&pathbuf).await? {
+            // Just clean up
+            fs::remove_file(&temppathbuf).await?;
+
+            return Ok((size, hash, maybe_mime_string));
+        }
+
+        // Make the parent directory
+        fs::create_dir_all(pathbuf.parent().unwrap()).await?;
+
+        // Move the file
+        fs::rename(&temppathbuf, &pathbuf).await?;
+
+        Ok((size, hash, maybe_mime_string))
+    }
+
+    /// Retrieve a file from storage by its HashOutput, streamed to a hyper BoxBoxy
+    pub async fn retrieve(&self, hash: HashOutput) -> Result<BoxBody<Bytes, Error>, Error> {
+        // Compute the path
+        let pathbuf = hash.to_pathbuf(&self.base);
+
+        // Open the file
+        let file = File::open(&pathbuf).await?;
+
+        // Convert the AsyncRead file into a Stream
+        let reader_stream = ReaderStream::new(file);
+
+        // Convert the Stream into a Body
+        let stream_body = StreamBody::new(reader_stream.map_ok(Frame::data));
+
+        // Box the body, mapping the error
+        let boxed_body = BodyExt::map_err(stream_body, |e| e.into()).boxed();
+
+        Ok(boxed_body)
+    }
+
+    /// Check if a file exists and provide it's metadata (including .len())
+    pub async fn metadata(&self, hash: HashOutput) -> Result<Metadata, Error> {
+        // Compute the path
+        let pathbuf = hash.to_pathbuf(&self.base);
+
+        Ok(tokio::fs::metadata(&pathbuf).await?)
+    }
+
+    /// Delete a file from storage by its HashOutput
+    pub async fn delete(&self, hash: HashOutput) -> Result<(), Error> {
+        // Compute the path
+        let pathbuf = hash.to_pathbuf(&self.base);
+
+        // Delete the file
+        tokio::fs::remove_file(&pathbuf).await?;
+
+        Ok(())
+    }
+}

src/globals.rs

@@ -1,51 +1,63 @@
-use crate::config::{Config, FriendlyConfig};
+use crate::config::Config;
-use crate::store::Store;
+use crate::filestore::FileStore;
-use crate::types::Time;
+use crate::ip::HashedIp;
-use hyper::server::conn::Http;
+use dashmap::DashMap;
+use hyper::server::conn::http1;
+use hyper_util::rt::tokio::TokioTimer;
 use lazy_static::lazy_static;
-use std::collections::HashMap;
+use parking_lot::RwLock;
-use std::net::IpAddr;
+use pocket_db::Store;
-use std::sync::atomic::AtomicUsize;
+use std::sync::atomic::{AtomicU64, AtomicUsize};
 use std::sync::OnceLock;
+use std::time::{Duration, Instant};
 use tokio::sync::broadcast::Sender as BroadcastSender;
 use tokio::sync::watch::Sender as WatchSender;
-use tokio::sync::RwLock;
 
 pub struct Globals {
+    pub start_time: Instant,
+    pub bytes_inbound: AtomicU64,
+    pub bytes_outbound: AtomicU64,
     pub config: RwLock<Config>,
     pub store: OnceLock<Store>,
-    pub http_server: Http,
+    pub filestore: OnceLock<FileStore>,
+    pub http1builder: http1::Builder,
     pub rid: OnceLock<String>,
 
     /// This is a broadcast channel where new incoming events are advertised by their offset.
     /// Every handler needs to listen to it and check if the incoming event matches any
     /// subscribed fitlers for their client, and if so, send the event to their client under
     /// that subscription.
-    pub new_events: BroadcastSender<usize>,
+    pub new_events: BroadcastSender<u64>,
 
-    pub num_clients: AtomicUsize,
+    pub num_connections: AtomicUsize,
+    pub num_connections_per_ip: DashMap<HashedIp, usize>,
     pub shutting_down: WatchSender<bool>,
-    pub banlist: RwLock<HashMap<IpAddr, Time>>,
 }
 
 lazy_static! {
     pub static ref GLOBALS: Globals = {
-        let mut http_server = hyper::server::conn::Http::new();
-        http_server.http1_only(true);
-        http_server.http1_keep_alive(true);
-
         let (new_events, _) = tokio::sync::broadcast::channel(512);
         let (shutting_down, _) = tokio::sync::watch::channel(false);
 
+        let mut http1builder = http1::Builder::new();
+        http1builder.ignore_invalid_headers(true);
+        http1builder.keep_alive(true);
+        http1builder.timer(TokioTimer::new());
+        http1builder.header_read_timeout(Duration::from_secs(5));
+
         Globals {
-            config: RwLock::new(FriendlyConfig::default().into_config().unwrap()),
+            start_time: Instant::now(),
+            bytes_inbound: AtomicU64::new(0),
+            bytes_outbound: AtomicU64::new(0),
+            config: RwLock::new(Default::default()),
             store: OnceLock::new(),
-            http_server,
+            filestore: OnceLock::new(),
+            http1builder,
             rid: OnceLock::new(),
             new_events,
-            num_clients: AtomicUsize::new(0),
+            num_connections: AtomicUsize::new(0),
+            num_connections_per_ip: DashMap::new(),
             shutting_down,
-            banlist: RwLock::new(HashMap::new()),
         }
     };
 }

src/ip.rs

@@ -0,0 +1,172 @@
+use pocket_types::Time;
+use speedy::{Readable, Writable};
+use std::net::{IpAddr, SocketAddr};
+
+#[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
+pub struct HashedIp(pub [u8; 20], bool);
+
+impl std::fmt::Display for HashedIp {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        unsafe { write!(f, "{}", std::str::from_utf8_unchecked(self.0.as_slice())) }
+    }
+}
+
+impl HashedIp {
+    pub fn new(ip_addr: IpAddr) -> HashedIp {
+        use base64::prelude::*;
+        use secp256k1::hashes::{sha256, Hash};
+        let bytes = ip_addr.write_to_vec().unwrap();
+        let hashvalue: sha256::Hash = Hash::hash(&bytes);
+        let tag = BASE64_STANDARD.encode(&hashvalue.as_byte_array()[0..16]);
+        HashedIp(
+            tag.as_bytes()[..20].try_into().unwrap(),
+            ip_addr.is_loopback(),
+        )
+    }
+
+    pub fn from_bytes(bytes: &[u8]) -> HashedIp {
+        HashedIp(bytes[0..20].try_into().unwrap(), false)
+    }
+
+    pub fn is_loopback(&self) -> bool {
+        self.1
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct HashedPeer(pub HashedIp, pub u16);
+
+impl std::fmt::Display for HashedPeer {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}:{}", self.0, self.1)
+    }
+}
+
+impl HashedPeer {
+    pub fn new(peer_addr: SocketAddr) -> HashedPeer {
+        let hashed_ip = HashedIp::new(peer_addr.ip());
+        HashedPeer(hashed_ip, peer_addr.port())
+    }
+
+    pub fn from_parts(hashed_ip: HashedIp, port: u16) -> HashedPeer {
+        HashedPeer(hashed_ip, port)
+    }
+
+    pub fn ip(&self) -> HashedIp {
+        self.0
+    }
+
+    pub fn port(&self) -> u16 {
+        self.1
+    }
+}
+
+// Single-session exit condition
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum SessionExit {
+    // No problems
+    Ok,
+
+    // Session exited with an error
+    ErrorExit,
+
+    // Session exited because of too many nostr command errors
+    // (based on a per-session command punishment value that we don't track long term)
+    TooManyErrors,
+
+    // Session timed out without an active subscription
+    Timeout,
+}
+
+// Long term reputation of an IP address
+// The values are running totals, updating with (9/10) of old and (1/10) of new.
+//
+// Used to determine ban time multiplier from short-term violations
+#[derive(Debug, Clone, Default, Readable, Writable)]
+pub struct IpReputation {
+    pub good: f32,
+    pub errored: f32,
+    pub too_many_errors: f32,
+    pub timed_out: f32,
+}
+
+impl IpReputation {
+    pub fn update(&mut self, session_exit: SessionExit) {
+        // Lessen the running totals
+        self.good *= 9.0 / 10.0;
+        self.errored *= 9.0 / 10.0;
+        self.too_many_errors *= 9.0 / 10.0;
+        self.timed_out *= 9.0 / 10.0;
+
+        match session_exit {
+            SessionExit::Ok => self.good += 1.0,
+            SessionExit::ErrorExit => self.errored += 1.0,
+            SessionExit::TooManyErrors => self.too_many_errors += 1.0,
+            SessionExit::Timeout => self.timed_out += 1.0,
+        };
+    }
+
+    pub fn ban_multiplier(&self) -> f32 {
+        let good_endings = 1.0 + self.good + (self.errored / 2.0);
+
+        let bad_endings = 1.0 + self.timed_out + self.too_many_errors + (self.errored / 2.0);
+
+        bad_endings / good_endings
+    }
+}
+
+// Memory-only short-term record of IP handling
+#[derive(Debug, Clone, Default, Readable, Writable)]
+pub struct IpData {
+    pub ban_until: u64,
+    pub reputation: IpReputation,
+}
+
+impl IpData {
+    pub fn update_on_session_close(
+        &mut self,
+        session_exit: SessionExit,
+        minimum_ban_seconds: u64,
+    ) -> u64 {
+        // Update reputation
+        self.reputation.update(session_exit);
+
+        // Compute ban_until
+        let mut until = Time::now();
+        let seconds = self.ban_seconds(session_exit, minimum_ban_seconds);
+        until = until + seconds;
+
+        self.ban_until = self.ban_until.max(until.as_u64());
+
+        seconds
+    }
+
+    pub fn is_banned(&self) -> bool {
+        Time::from_u64(self.ban_until) > Time::now()
+    }
+
+    fn ban_seconds(&self, session_exit: SessionExit, minimum_ban_seconds: u64) -> u64 {
+        let multiplier = self.reputation.ban_multiplier();
+
+        match session_exit {
+            SessionExit::Ok => minimum_ban_seconds,
+            SessionExit::Timeout => minimum_ban_seconds,
+            SessionExit::ErrorExit => minimum_ban_seconds + (2.0 * multiplier) as u64,
+            SessionExit::TooManyErrors => minimum_ban_seconds + (5.0 * multiplier) as u64,
+        }
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    fn test_hashed_ip() {
+        let ipaddr: std::net::IpAddr = "127.0.0.1".parse().unwrap();
+        println!("HashedIP={}", HashedIp::new(ipaddr));
+
+        let socketaddr = std::net::SocketAddr::new(ipaddr, 80);
+        println!("HashedPEER={}", HashedPeer::new(socketaddr));
+    }
+}

src/lib.rs

@@ -0,0 +1,942 @@
+pub mod config;
+pub mod counting_stream;
+pub mod error;
+pub mod filestore;
+pub mod globals;
+pub mod ip;
+mod neg_storage;
+pub mod nostr;
+pub mod reply;
+pub mod tls;
+pub mod web;
+
+use crate::config::{Config, FriendlyConfig};
+use crate::error::{ChorusError, Error};
+use crate::globals::GLOBALS;
+use crate::ip::{HashedIp, HashedPeer, IpData, SessionExit};
+use crate::reply::NostrReply;
+use futures::{sink::SinkExt, stream::StreamExt};
+use http_body_util::combinators::BoxBody;
+use http_body_util::{BodyExt, Empty};
+use hyper::body::{Bytes, Incoming};
+use hyper::service::Service;
+use hyper::upgrade::Upgraded;
+use hyper::StatusCode;
+use hyper::{Request, Response};
+use hyper_tungstenite::tungstenite;
+use hyper_tungstenite::{HyperWebsocket, WebSocketStream};
+use hyper_util::rt::TokioIo;
+use neg_storage::NegentropyStorageVector;
+use pocket_db::{ScreenResult, Store};
+use pocket_types::{Id, OwnedFilter, Pubkey};
+use speedy::{Readable, Writable};
+use std::collections::HashMap;
+use std::error::Error as StdError;
+use std::fs::OpenOptions;
+use std::future::Future;
+use std::io::Read;
+use std::net::IpAddr;
+use std::path::Path;
+use std::pin::Pin;
+use std::sync::atomic::Ordering;
+use std::time::Duration;
+use textnonce::TextNonce;
+use tokio::io::{AsyncRead, AsyncWrite};
+use tokio::time::Instant;
+use tungstenite::protocol::frame::Utf8Bytes;
+use tungstenite::protocol::WebSocketConfig;
+use tungstenite::Message;
+
+/// Serve a single network connection
+pub async fn serve<T>(stream: TokioIo<T>, peer: HashedPeer)
+where
+    T: AsyncRead + AsyncWrite + Unpin + Send + 'static,
+{
+    // Serve the network stream with our http server and our ChorusService
+    let service = ChorusService { peer };
+
+    let http1builder = GLOBALS.http1builder.clone();
+    let connection = http1builder
+        .serve_connection(stream, service)
+        .with_upgrades();
+
+    // If our service exits with an error, log the error
+    if let Err(he) = connection.await {
+        if let Some(src) = he.source() {
+            if &*format!("{}", src) == "Transport endpoint is not connected (os error 107)" {
+                // do nothing
+            } else {
+                // Print in detail
+                log::error!(target: "Client", "{}: {:?}", peer, src);
+            }
+        } else {
+            // Print in less detail
+            let e: Error = he.into();
+            log::error!(target: "Client", "{}: {}", peer, e);
+        }
+    }
+}
+
+// This is our per-connection HTTP service
+struct ChorusService {
+    peer: HashedPeer,
+}
+
+impl Service<Request<Incoming>> for ChorusService {
+    type Response = Response<BoxBody<Bytes, Self::Error>>;
+    type Error = Error;
+    type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
+
+    // This is called for each HTTP request made by the client
+    // NOTE: it is not called for each websocket message once upgraded.
+    fn call(&self, req: Request<Incoming>) -> Self::Future {
+        let mut hashed_peer = self.peer;
+
+        let failvalue =
+            |c: ChorusError| -> Self::Future { Box::pin(futures::future::ready(Err(c.into()))) };
+
+        if GLOBALS.config.read().chorus_is_behind_a_proxy {
+            // If chorus is behind a proxy that sets an "X-Real-Ip" header, we use
+            // that ip address instead (otherwise their log file will just give the proxy IP
+            // for every peer)
+            //
+            // This header must be found and be valid for us to proceed
+            if let Some(rip) = req.headers().get("x-real-ip") {
+                if let Ok(ripstr) = rip.to_str() {
+                    if let Ok(ipaddr) = ripstr.parse::<IpAddr>() {
+                        let hashed_ip = HashedIp::new(ipaddr);
+                        hashed_peer = HashedPeer::from_parts(hashed_ip, hashed_peer.port());
+                    } else {
+                        return failvalue(ChorusError::BadRealIpHeader(ripstr.to_owned()));
+                    }
+                } else {
+                    return failvalue(ChorusError::BadRealIpHeaderCharacters);
+                }
+            } else {
+                return failvalue(ChorusError::RealIpHeaderMissing);
+            }
+
+            // Possibly IP block late (if behind a proxy)
+            if GLOBALS.config.read().enable_ip_blocking {
+                if let Ok(ip_data) = crate::get_ip_data(hashed_peer.ip()) {
+                    if ip_data.is_banned() {
+                        log::debug!(target: "Client",
+                                    "{}: Blocking reconnection until {}",
+                                    hashed_peer.ip(),
+                                    ip_data.ban_until);
+                        return failvalue(ChorusError::BlockedIp);
+                    }
+                }
+            }
+        }
+
+        Box::pin(async move { handle_http_request(hashed_peer, req).await })
+    }
+}
+
+async fn handle_http_request(
+    peer: HashedPeer,
+    mut request: Request<Incoming>,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    let ua = match request.headers().get("user-agent") {
+        Some(ua) => ua.to_str().unwrap_or("NON-UTF8-HEADER").to_owned(),
+        None => "(no user-agent)".to_owned(),
+    };
+
+    let origin = match request.headers().get("origin") {
+        Some(o) => o.to_str().unwrap_or("NON-UTF8-HEADER").to_owned(),
+        None => "(no origin)".to_owned(),
+    };
+
+    let max_conn = GLOBALS.config.read().max_connections_per_ip;
+    if let Some(cur) = GLOBALS.num_connections_per_ip.get(&peer.ip()) {
+        if *cur.value() >= max_conn {
+            return Ok(Response::builder()
+                .status(StatusCode::TOO_MANY_REQUESTS)
+                .body(Empty::new().map_err(|e| e.into()).boxed())?);
+        }
+    }
+
+    if hyper_tungstenite::is_upgrade_request(&request) {
+        // If the client asks for a Sec-Websocket-Protocol that we don't understand,
+        // Respond with 501 Not Implemented
+        let maybe_protocol: Option<String> = match request.headers().get("sec-websocket-protocol") {
+            None => None,
+            Some(hv) => hv.to_str().ok().map(|s| s.to_owned()),
+        };
+        if let Some(ref protocol) = maybe_protocol {
+            let mut we_can_do_nostr: bool = false;
+            for option in protocol.split(',') {
+                if option.trim() == "nostr" {
+                    we_can_do_nostr = true;
+                    break;
+                }
+            }
+            if !we_can_do_nostr {
+                return Ok(Response::builder()
+                    .status(StatusCode::NOT_IMPLEMENTED)
+                    .body(Empty::new().map_err(|e| e.into()).boxed())?);
+            }
+        }
+
+        let mut web_socket_config = WebSocketConfig::default();
+        web_socket_config.max_write_buffer_size = 1024 * 1024; // 1 MB
+        web_socket_config.max_message_size = Some(1024 * 1024); // 1 MB
+        web_socket_config.max_frame_size = Some(1024 * 1024); // 1 MB
+
+        let (mut response, websocket) =
+            hyper_tungstenite::upgrade(&mut request, Some(web_socket_config))?;
+
+        // If the client asked for Sec-Websocket-Protocol, then we already checked it must
+        // have asked for 'nostr', so send that as a response header
+        if maybe_protocol.is_some() {
+            response.headers_mut().insert(
+                http::header::SEC_WEBSOCKET_PROTOCOL,
+                http::header::HeaderValue::from_static("nostr"),
+            );
+        }
+
+        // Start the websocket thread
+        tokio::spawn(async move { websocket_thread(peer, websocket, origin, ua).await });
+
+        Ok(response.map(|body| body.map_err(|e| e.into()).boxed()))
+    } else {
+        web::serve_http(peer, request).await
+    }
+}
+
+async fn websocket_thread(peer: HashedPeer, websocket: HyperWebsocket, origin: String, ua: String) {
+    // Await the websocket upgrade process
+    match websocket.await {
+        Ok(websocket) => {
+            // Build a websocket service
+            let mut ws_service = WebSocketService {
+                peer,
+                subscriptions: HashMap::new(),
+                neg_subscriptions: HashMap::new(),
+                // We start with a 1-page buffer, and grow it if needed.
+                buffer: vec![0; 4096],
+                websocket,
+                last_message: Instant::now(),
+                burst_tokens: GLOBALS.config.read().throttling_burst,
+                challenge: TextNonce::new().into_string(),
+                authed_as: Vec::new(),
+                error_punishment: 0.0,
+                replied: false,
+                negentropy_sub: None,
+            };
+
+            // Increment connection count
+            let old_num_websockets = GLOBALS.num_connections.fetch_add(1, Ordering::SeqCst);
+
+            // Increment per-ip connection count
+            GLOBALS
+                .num_connections_per_ip
+                .entry(peer.ip())
+                .and_modify(|count| *count += 1)
+                .or_insert(1);
+
+            // we cheat somewhat and log these websocket open and close messages
+            // as server messages
+            log::info!(
+                target: "Server",
+                "{}: TOTAL={}, New Connection: {}, {}",
+                peer,
+                old_num_websockets + 1,
+                origin,
+                ua,
+            );
+
+            // Everybody gets a ban on disconnect to prevent rapid reconnection
+            let mut session_exit: SessionExit = SessionExit::Ok;
+            let mut msg = "Closed";
+
+            // Handle the websocket
+            if let Err(e) = ws_service.handle_websocket_stream().await {
+                match e.inner {
+                    ChorusError::Tungstenite(ref t) => {
+                        match *t.as_ref() {
+                            tungstenite::error::Error::Protocol(
+                                tungstenite::error::ProtocolError::ResetWithoutClosingHandshake,
+                            ) => {
+                                // So they disconnected ungracefully.
+                                // No big deal, still SessionExit::Ok
+                                msg = "Reset";
+                            }
+                            tungstenite::error::Error::Io(ref ioerror) => {
+                                match ioerror.kind() {
+                                    std::io::ErrorKind::ConnectionReset
+                                    | std::io::ErrorKind::ConnectionAborted
+                                    | std::io::ErrorKind::UnexpectedEof => {
+                                        // no biggie.
+                                        msg = "Reset";
+                                    }
+                                    _ => {
+                                        log::error!(target: "Client", "{}: {}", peer, e);
+                                        session_exit = SessionExit::ErrorExit;
+                                        msg = "Error Exited";
+                                    }
+                                }
+                            }
+                            _ => {
+                                log::error!(target: "Client", "{}: {}", peer, e);
+                                session_exit = SessionExit::ErrorExit;
+                                msg = "Error Exited";
+                            }
+                        }
+                    }
+                    ChorusError::ErrorClose => {
+                        session_exit = SessionExit::TooManyErrors;
+                        msg = "Errored Out";
+                    }
+                    ChorusError::RateLimitExceeded => {
+                        session_exit = SessionExit::TooManyErrors; // close enough for now.
+                        msg = "Rate Limit Exceeded";
+                    }
+                    ChorusError::TimedOut => {
+                        session_exit = SessionExit::Timeout;
+                        msg = "Timed Out (with no subscriptions)";
+                    }
+                    ChorusError::Io(_) => {
+                        // Usually "Connection reset by peer" but any I/O error
+                        // isn't a big deal.
+                        msg = "Reset";
+                    }
+                    _ => {
+                        log::error!(target: "Client", "{}: {}", peer, e);
+                        session_exit = SessionExit::ErrorExit;
+                        msg = "Error Exited";
+                    }
+                }
+            }
+
+            // Decrement connection count
+            let old_num_websockets = GLOBALS.num_connections.fetch_sub(1, Ordering::SeqCst);
+
+            // Decrement per-ip connection count
+            match GLOBALS.num_connections_per_ip.get_mut(&peer.ip()) {
+                Some(mut refmut) => {
+                    if *refmut.value_mut() > 0 {
+                        *refmut.value_mut() -= 1;
+                    } else {
+                        unreachable!("The connection should be in the map")
+                    }
+                }
+                None => unreachable!("The connection count should be greater than zero"),
+            };
+
+            // Update ip data (including ban time)
+            let minimum_ban_seconds = GLOBALS.config.read().minimum_ban_seconds;
+            let ban_seconds = if GLOBALS.config.read().enable_ip_blocking {
+                let mut ban_seconds = 0;
+                if let Ok(mut ip_data) = get_ip_data(peer.ip()) {
+                    ban_seconds =
+                        ip_data.update_on_session_close(session_exit, minimum_ban_seconds);
+                    let _ = update_ip_data(peer.ip(), &ip_data);
+                }
+                ban_seconds
+            } else {
+                minimum_ban_seconds
+            };
+
+            // we cheat somewhat and log these websocket open and close messages
+            // as server messages
+            log::info!(
+                target: "Server",
+                "{}: TOTAL={}, {}, ban={}s",
+                peer,
+                old_num_websockets - 1,
+                msg,
+                ban_seconds
+            );
+        }
+        Err(e) => {
+            log::error!(target: "Client", "{}: {}", peer, e);
+        }
+    }
+}
+
+struct WebSocketService {
+    pub peer: HashedPeer,
+    pub subscriptions: HashMap<String, Vec<OwnedFilter>>,
+    pub neg_subscriptions: HashMap<String, NegentropyStorageVector>,
+    pub buffer: Vec<u8>,
+    pub websocket: WebSocketStream<TokioIo<Upgraded>>,
+    pub last_message: Instant,
+    pub burst_tokens: usize,
+    pub challenge: String,
+    pub authed_as: Vec<Pubkey>,
+    pub error_punishment: f32,
+    pub replied: bool,
+    pub negentropy_sub: Option<String>,
+}
+
+impl WebSocketService {
+    async fn send(&mut self, m: Message) -> Result<(), Error> {
+        log::trace!(target: "Client", "{}: {}", self.peer, m);
+
+        // Throttling: we consume burst tokens, but we do not throttle on output
+        if m.len() > self.burst_tokens {
+            log::info!(target: "Client", "{}: Rate limited exceeded", self.peer);
+            let reply = NostrReply::Notice("Rate limit exceeded.".into());
+            self.websocket.send(Message::text(reply.as_json()?)).await?;
+            let error = ChorusError::RateLimitExceeded;
+            self.error_punishment += error.punishment();
+            return Err(error.into());
+        } else {
+            self.burst_tokens -= m.len();
+        }
+
+        self.replied = true;
+        Ok(self.websocket.send(m).await?)
+    }
+
+    async fn wsclose(&mut self, error: Error) -> Result<(), Error> {
+        use tungstenite::protocol::frame::coding::CloseCode;
+        use tungstenite::protocol::frame::CloseFrame;
+
+        let (code, reason) = match &error.inner {
+            ChorusError::TimedOut => (CloseCode::Policy, Utf8Bytes::from_static("timed out")),
+            ChorusError::ShuttingDown => (CloseCode::Restart, Utf8Bytes::from_static("restarting")),
+            ChorusError::BannedUser | ChorusError::BlockedIp => {
+                (CloseCode::Policy, Utf8Bytes::from_static("banned"))
+            }
+            e => (CloseCode::Error, format!("{}", e).into()),
+        };
+
+        let close_frame = CloseFrame { code, reason };
+
+        // NOTE: This is the same as sending Message::Close(..)
+        self.websocket.close(Some(close_frame)).await?;
+
+        // Drive to completion
+        while (self.websocket.next().await).is_some() {}
+
+        Err(error)
+    }
+
+    async fn handle_websocket_stream(&mut self) -> Result<(), Error> {
+        // Subscribe to the shutting down channel
+        let mut shutting_down = GLOBALS.shutting_down.subscribe();
+
+        // Subscribe to the new_events broadcast channel
+        let mut new_events = GLOBALS.new_events.subscribe();
+
+        // Offer AUTH to clients right off the bat
+        let reply = NostrReply::Auth(self.challenge.clone());
+        self.send(Message::text(reply.as_json()?)).await?;
+
+        let mut last_message_at = Instant::now();
+
+        let timeout_seconds = GLOBALS.config.read().timeout_seconds;
+
+        let mut interval = tokio::time::interval(Duration::from_secs(1));
+        let _ = interval.tick().await; // consume the first tick
+        tokio::pin!(interval);
+
+        loop {
+            tokio::select! {
+                instant = interval.tick() => {
+                    // Drop them if they have no subscriptions
+                    if self.subscriptions.is_empty() && self.neg_subscriptions.is_empty() {
+                        // And they are idle for timeout_seconds with no subscriptions
+                        if last_message_at + Duration::from_secs(timeout_seconds) < instant {
+                            self.wsclose(ChorusError::TimedOut.into()).await?;
+                        }
+                    }
+                }
+                message_option = self.websocket.next() => {
+                    last_message_at = Instant::now();
+                    match message_option {
+                        Some(message) => {
+                            let message = message?;
+                            if let Err(e) = self.handle_websocket_message(message).await {
+                                self.wsclose(e).await?;
+                            }
+                        },
+                        None => break, // the websocket is closed
+                    }
+                },
+                offset_result = new_events.recv() => {
+                    let offset = offset_result?;
+                    self.handle_new_event(offset).await?;
+                },
+                _r = shutting_down.changed() => {
+                    self.wsclose(ChorusError::ShuttingDown.into()).await?;
+                },
+            }
+        }
+
+        Ok(())
+    }
+
+    // If the event matches a subscription they have open, send them the event
+    async fn handle_new_event(&mut self, new_event_offset: u64) -> Result<(), Error> {
+        if self.subscriptions.is_empty() {
+            return Ok(());
+        }
+
+        let event = GLOBALS
+            .store
+            .get()
+            .unwrap()
+            .get_event_by_offset(new_event_offset)?;
+
+        let event_flags = nostr::event_flags(event, self.authed_as.as_slice());
+        let authorized_user = self.authed_as.iter().any(|u| is_authorized_user(*u));
+
+        'subs: for (subid, filters) in self.subscriptions.iter() {
+            for filter in filters.iter() {
+                if filter.event_matches(event)? {
+                    let screen_result =
+                        nostr::screen_outgoing_event(event, &event_flags, authorized_user);
+                    if screen_result == ScreenResult::Redacted {
+                        // TBD:  Update subscription so the final close can
+                        //       let them know there were redactions from
+                        //       the post-EOSE data
+                    } else if screen_result == ScreenResult::Match {
+                        let message = NostrReply::Event(subid, event);
+                        // note, this is not currently counted in throttling
+                        self.websocket
+                            .send(Message::text(message.as_json()?))
+                            .await?;
+                        continue 'subs;
+                    }
+                }
+            }
+        }
+
+        Ok(())
+    }
+
+    async fn handle_websocket_message(&mut self, message: Message) -> Result<(), Error> {
+        // Throttling
+        {
+            let (throttling_burst, throttling_bytes_per_second) = {
+                let config = GLOBALS.config.read();
+                (config.throttling_burst, config.throttling_bytes_per_second)
+            };
+
+            // Get (and update) timing
+            let elapsed = self.last_message.elapsed();
+            self.last_message = Instant::now();
+
+            // Grant new tokens
+            let new_tokens = throttling_bytes_per_second * elapsed.as_millis() as usize / 1_000;
+            self.burst_tokens += new_tokens;
+
+            // Cap tokens to a maximum
+            if self.burst_tokens > throttling_burst {
+                self.burst_tokens = throttling_burst;
+            }
+
+            // Consume tokens, possibly closing the connection if there are not enough
+            if message.len() > self.burst_tokens {
+                log::info!(target: "Client", "{}: Rate limited exceeded", self.peer);
+                let reply = NostrReply::Notice("Rate limit exceeded.".into());
+                self.websocket.send(Message::text(reply.as_json()?)).await?;
+                let error = ChorusError::RateLimitExceeded;
+                self.error_punishment += error.punishment();
+                return Err(error.into());
+            } else {
+                self.burst_tokens -= message.len();
+            }
+        }
+
+        match message {
+            Message::Text(msg) => {
+                log::trace!(target: "Client", "{}: <= {}", self.peer, msg);
+                self.replied = false;
+                // This is defined in nostr.rs
+                if let Err(e) = self.handle_nostr_message(&msg).await {
+                    self.error_punishment += e.inner.punishment();
+                    log::error!(target: "Client", "{}: {e}", self.peer);
+                    if !matches!(e.inner, ChorusError::AuthRequired) {
+                        if msg.len() < 2048 {
+                            log::warn!(target: "Client", "{}:   msg was {}", self.peer, msg);
+                        } else {
+                            log::warn!(target: "Client", "{}:   msg > 2048 (not shown)", self.peer);
+                        }
+                    }
+                    if !self.replied {
+                        if let Some(subid) = &self.negentropy_sub {
+                            let reply = NostrReply::NegErr(subid, format!("error: {e}"));
+                            self.send(Message::text(reply.as_json()?)).await?;
+                        } else {
+                            let reply = NostrReply::Notice(format!("error: {}", e.inner));
+                            self.send(Message::text(reply.as_json()?)).await?;
+                        }
+                    }
+                    if self.error_punishment >= 1.0 {
+                        let reply = NostrReply::Notice("Closing due to error(s)".into());
+                        self.send(Message::text(reply.as_json()?)).await?;
+                        return Err(ChorusError::ErrorClose.into());
+                    }
+                }
+            }
+            Message::Binary(msg) => {
+                let reply = NostrReply::Notice(
+                    "binary messages are not processed by this relay".to_owned(),
+                );
+                self.send(Message::text(reply.as_json()?)).await?;
+                log::info!(target: "Client",
+                    "{}: Received unhandled binary message: {:02X?}",
+                    self.peer,
+                    msg
+                );
+            }
+            Message::Ping(msg) => {
+                // No need to send a reply: tungstenite takes care of this for you.
+                log::debug!(target: "Client", "{}: Received ping message: {:02X?}", self.peer, msg);
+            }
+            Message::Pong(msg) => {
+                log::debug!(target: "Client", "{}: Received pong message: {:02X?}", self.peer, msg);
+            }
+            Message::Close(msg) => {
+                // No need to send a reply: tungstenite takes care of this for you.
+                if let Some(msg) = &msg {
+                    log::debug!(target: "Client",
+                        "{}: Received websocket close message with code {} and message: {}",
+                        self.peer,
+                        msg.code,
+                        msg.reason
+                    );
+                } else {
+                    log::debug!(
+                        target: "Client",
+                        "{}: Received websocket close message",
+                        self.peer,
+                    );
+                }
+            }
+            Message::Frame(_msg) => {
+                unreachable!();
+            }
+        }
+
+        Ok(())
+    }
+}
+
+/// Print statistics
+pub fn print_stats() {
+    let mut runtime: u64 = GLOBALS.start_time.elapsed().as_secs();
+    if runtime < 1 {
+        runtime = 1;
+    }
+    log::info!(
+        target: "Server",
+        "Runtime: {} seconds", runtime
+    );
+    log::info!(
+        target: "Server",
+        "Inbound: {} bytes ({} B/s)",
+        GLOBALS.bytes_inbound.load(Ordering::Relaxed),
+        (GLOBALS.bytes_inbound.load(Ordering::Relaxed) as f32) / (runtime as f32)
+    );
+    log::info!(
+        target: "Server",
+        "Outbound: {} bytes ({} B/s)",
+        GLOBALS.bytes_outbound.load(Ordering::Relaxed),
+        (GLOBALS.bytes_outbound.load(Ordering::Relaxed) as f32) / (runtime as f32)
+    );
+    if let Ok(status) = GLOBALS.store.get().unwrap().stats() {
+        log::info!(
+            target: "Server",
+            "Store: {} event bytes in {} events, {} bytes for the indexes",
+            status.event_bytes,
+            status.index_stats.i_index_entries,
+            status.index_stats.disk_usage
+        );
+    }
+}
+
+/// Load config file
+pub fn load_config<P: AsRef<Path>>(config_path: P) -> Result<Config, Error> {
+    // Read config file
+    let mut file = OpenOptions::new().read(true).open(config_path)?;
+    let mut contents = String::new();
+    file.read_to_string(&mut contents)?;
+    let friendly_config: FriendlyConfig = toml::from_str(&contents)?;
+    let config: Config = friendly_config.into_config()?;
+    Ok(config)
+}
+
+/// Setup logging
+pub fn setup_logging(config: &Config) {
+    env_logger::Builder::new()
+        .filter_level(config.library_log_level)
+        .filter(Some("Server"), config.server_log_level)
+        .filter(Some("Client"), config.client_log_level)
+        .format_target(true)
+        .format_module_path(false)
+        .format_timestamp_millis()
+        .init();
+
+    log::debug!(target: "Server", "Loaded config file.");
+}
+
+/// Setup storage
+pub fn setup_store(config: &Config) -> Result<(), Error> {
+    let store = setup_store_and_return(config)?;
+    let _ = GLOBALS.store.set(store);
+    Ok(())
+}
+
+/// Setup storage and return it
+pub fn setup_store_and_return(config: &Config) -> Result<Store, Error> {
+    let store = Store::new(
+        &config.data_directory,
+        vec![
+            "approved-events",  // id.as_slice() -> u8(bool)
+            "approved-pubkeys", // pubkey.as_slice() -> u8(bool)
+            "ip_data",          // HashedIp.0 -> IpData
+            "users",            // pubkey.as_slice() -> u8(bool) true if moderator
+        ],
+    )?;
+    Ok(store)
+}
+
+/// Get IpData from storage about this remote HashedIp
+pub fn get_ip_data(ip: HashedIp) -> Result<IpData, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let ip_data = store
+        .extra_table("ip_data")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable("ip_data")))?;
+    let txn = store.read_txn()?;
+    let key = &ip.0;
+    let bytes = match ip_data.get(&txn, key)? {
+        Some(b) => b,
+        None => return Ok(Default::default()),
+    };
+    Ok(IpData::read_from_buffer(bytes)?)
+}
+
+/// Get IpData in storage about this remote HashedIp
+pub fn update_ip_data(ip: HashedIp, data: &IpData) -> Result<(), Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let ip_data = store
+        .extra_table("ip_data")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable("ip_data")))?;
+    let mut txn = store.write_txn()?;
+    let key = &ip.0;
+    let bytes = data.write_to_vec()?;
+    ip_data.put(&mut txn, key, &bytes)?;
+    txn.commit()?;
+    Ok(())
+}
+
+/// Dump all IpData from storage
+pub fn dump_ip_data() -> Result<Vec<(HashedIp, IpData)>, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let ip_data = store
+        .extra_table("ip_data")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable("ip_data")))?;
+    let txn = store.read_txn()?;
+    let mut output: Vec<(HashedIp, IpData)> = Vec::new();
+    for i in ip_data.iter(&txn)? {
+        let (key, val) = i?;
+        let hashedip = HashedIp::from_bytes(key);
+        let data = IpData::read_from_buffer(val)?;
+        output.push((hashedip, data));
+    }
+    Ok(output)
+}
+
+/// Mark an event as approved or not
+pub fn mark_event_approval(id: Id, approval: bool) -> Result<(), Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let approved_events = store
+        .extra_table("approved-events")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-events",
+        )))?;
+    let mut txn = store.write_txn()?;
+    approved_events.put(&mut txn, id.as_slice(), &[approval as u8])?;
+    txn.commit()?;
+    Ok(())
+}
+
+/// Clear an event approval status
+pub fn clear_event_approval(id: Id) -> Result<(), Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let approved_events = store
+        .extra_table("approved-events")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-events",
+        )))?;
+    let mut txn = store.write_txn()?;
+    approved_events.delete(&mut txn, id.as_slice())?;
+    txn.commit()?;
+    Ok(())
+}
+
+/// Fetch an event approval status
+pub fn get_event_approval(id: Id) -> Result<Option<bool>, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let approved_events = store
+        .extra_table("approved-events")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-events",
+        )))?;
+    let txn = store.read_txn()?;
+    Ok(approved_events
+        .get(&txn, id.as_slice())?
+        .map(|u| !u.is_empty() && u[0] != 0))
+}
+
+/// Dump all event approval statuses
+pub fn dump_event_approvals() -> Result<Vec<(Id, bool)>, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let mut output: Vec<(Id, bool)> = Vec::new();
+    let approved_events = store
+        .extra_table("approved-events")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-events",
+        )))?;
+    let txn = store.read_txn()?;
+    for i in approved_events.iter(&txn)? {
+        let (key, val) = i?;
+        let id = Id::from_bytes(key.try_into().unwrap());
+        let approval: bool = !val.is_empty() && val[0] != 0;
+        output.push((id, approval));
+    }
+    Ok(output)
+}
+
+/// Mark a pubkey as approved or not
+pub fn mark_pubkey_approval(pubkey: Pubkey, approval: bool) -> Result<(), Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let approved_pubkeys = store
+        .extra_table("approved-pubkeys")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-pubkeys",
+        )))?;
+    let mut txn = store.write_txn()?;
+    approved_pubkeys.put(&mut txn, pubkey.as_slice(), &[approval as u8])?;
+    txn.commit()?;
+    Ok(())
+}
+
+/// Clear a pubkey approval status
+pub fn clear_pubkey_approval(pubkey: Pubkey) -> Result<(), Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let approved_pubkeys = store
+        .extra_table("approved-pubkeys")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-pubkeys",
+        )))?;
+    let mut txn = store.write_txn()?;
+    approved_pubkeys.delete(&mut txn, pubkey.as_slice())?;
+    txn.commit()?;
+    Ok(())
+}
+
+/// Fetch a pubkey approval status
+pub fn get_pubkey_approval(pubkey: Pubkey) -> Result<Option<bool>, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let approved_pubkeys = store
+        .extra_table("approved-pubkeys")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-pubkeys",
+        )))?;
+    let txn = store.read_txn()?;
+    Ok(approved_pubkeys
+        .get(&txn, pubkey.as_slice())?
+        .map(|u| !u.is_empty() && u[0] != 0))
+}
+
+/// Dump all pubkey approval statuses
+pub fn dump_pubkey_approvals() -> Result<Vec<(Pubkey, bool)>, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let mut output: Vec<(Pubkey, bool)> = Vec::new();
+    let approved_pubkeys = store
+        .extra_table("approved-pubkeys")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable(
+            "approved-pubkeys",
+        )))?;
+    let txn = store.read_txn()?;
+    for i in approved_pubkeys.iter(&txn)? {
+        let (key, val) = i?;
+        let pubkey = Pubkey::from_bytes(key.try_into().unwrap());
+        let approval: bool = !val.is_empty() && val[0] != 0;
+        output.push((pubkey, approval));
+    }
+    Ok(output)
+}
+
+/// Add authorized user (or change moderator flag)
+pub fn add_authorized_user(pubkey: Pubkey, moderator: bool) -> Result<(), Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let users = store
+        .extra_table("users")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
+    let mut txn = store.write_txn()?;
+    users.put(&mut txn, pubkey.as_slice(), &[moderator as u8])?;
+    txn.commit()?;
+    Ok(())
+}
+
+/// Remove authorized user
+pub fn rm_authorized_user(pubkey: Pubkey) -> Result<(), Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let users = store
+        .extra_table("users")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
+    let mut txn = store.write_txn()?;
+    users.delete(&mut txn, pubkey.as_slice())?;
+    txn.commit()?;
+    Ok(())
+}
+
+/// Get authorized user
+pub fn get_authorized_user(pubkey: Pubkey) -> Result<Option<bool>, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let users = store
+        .extra_table("users")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
+    let txn = store.read_txn()?;
+    Ok(users
+        .get(&txn, pubkey.as_slice())?
+        .map(|u| !u.is_empty() && u[0] != 0))
+}
+
+/// Dump all authorized users
+pub fn dump_authorized_users() -> Result<Vec<(Pubkey, bool)>, Error> {
+    let store = GLOBALS.store.get().unwrap();
+    let mut output: Vec<(Pubkey, bool)> = Vec::new();
+    let users = store
+        .extra_table("users")
+        .ok_or(Into::<Error>::into(ChorusError::MissingTable("users")))?;
+    let txn = store.read_txn()?;
+    for i in users.iter(&txn)? {
+        let (key, val) = i?;
+        let pubkey = Pubkey::from_bytes(key.try_into().unwrap());
+        let moderator: bool = !val.is_empty() && val[0] != 0;
+        output.push((pubkey, moderator));
+    }
+    Ok(output)
+}
+
+/// Is the pubkey an authorized user?
+pub fn is_authorized_user(pubkey: Pubkey) -> bool {
+    match get_authorized_user(pubkey) {
+        Err(_) => false,
+        Ok(None) => false,
+        Ok(Some(_)) => true,
+    }
+}
+
+/// Is the pubkey a moderator?
+pub fn is_moderator(pubkey: Pubkey) -> bool {
+    match get_authorized_user(pubkey) {
+        Err(_) => false,
+        Ok(None) => false,
+        Ok(Some(moderator)) => moderator,
+    }
+}
+
+/// Is the pubkey an admin?
+pub fn is_admin(pubkey: Pubkey) -> bool {
+    GLOBALS.config.read().admin_keys.contains(&pubkey)
+}

src/macros.rs

@@ -1,75 +0,0 @@
-static HEX_CHARS: &[u8; 16] = b"0123456789abcdef";
-#[allow(clippy::zero_prefixed_literal)]
-static HEX_INVERSE: [u8; 128] = {
-    const __: u8 = 255;
-    [
-        //  1  2  3   4  5  6  7   8  9  A  B   C  D  E  F
-        __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 0
-        __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 1
-        __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 2
-        00, 01, 02, 03, 04, 05, 06, 07, 08, 09, __, __, __, __, __, __, // 3
-        __, 10, 11, 12, 13, 14, 15, __, __, __, __, __, __, __, __, __, // 4
-        __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 5
-        __, 10, 11, 12, 13, 14, 15, __, __, __, __, __, __, __, __, __, // 6
-        __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 7
-    ]
-};
-
-macro_rules! write_hex {
-    ($input:expr, $output:expr, $bytelen:expr) => {{
-        assert_eq!($input.len(), $bytelen);
-        if $output.len() != $bytelen * 2 {
-            Err(crate::error::ChorusError::BufferTooSmall.into())
-        } else {
-            for (i, byte) in $input.iter().enumerate() {
-                $output[i * 2] = crate::HEX_CHARS[((byte & 0xF0) >> 4) as usize];
-                $output[i * 2 + 1] = crate::HEX_CHARS[(byte & 0x0F) as usize];
-            }
-            Ok(())
-        }
-    }};
-}
-
-macro_rules! read_hex {
-    ($input:expr, $output:expr, $bytelen:expr) => {{
-        assert_eq!($output.len(), $bytelen);
-        if $input.len() != $bytelen * 2 {
-            Err(Into::<crate::error::Error>::into(crate::error::ChorusError::EndOfInput))
-        } else {
-            let mut i = 0;
-            loop {
-                let high = crate::HEX_INVERSE[$input[i * 2] as usize];
-                if high == 255 {
-                    break Err(crate::error::ChorusError::BadHexInput.into());
-                }
-                let low = crate::HEX_INVERSE[$input[i * 2 + 1] as usize];
-                if low == 255 {
-                    break Err(crate::error::ChorusError::BadHexInput.into());
-                }
-                $output[i] = high * 16 + low;
-                i += 1;
-                if i == $bytelen {
-                    break Ok(());
-                }
-            }
-        }
-    }};
-}
-
-macro_rules! parse_u16 {
-    ($input:expr, $start:expr) => {
-        u16::from_ne_bytes($input[$start .. $start+2].try_into().unwrap())
-    }
-}
-
-macro_rules! parse_u32 {
-    ($input:expr, $start:expr) => {
-        u32::from_ne_bytes($input[$start .. $start+4].try_into().unwrap())
-    }
-}
-
-macro_rules! parse_u64 {
-    ($input:expr, $start:expr) => {
-        u64::from_ne_bytes($input[$start .. $start+8].try_into().unwrap())
-    }
-}

src/main.rs

@@ -1,441 +0,0 @@
-include!("macros.rs");
-
-pub mod config;
-pub mod error;
-pub mod globals;
-pub mod nostr;
-pub mod reply;
-pub mod store;
-pub mod tls;
-pub mod types;
-pub mod web;
-
-use crate::config::{Config, FriendlyConfig};
-use crate::error::{ChorusError, Error};
-use crate::globals::GLOBALS;
-use crate::reply::NostrReply;
-use crate::store::Store;
-use crate::tls::MaybeTlsStream;
-use crate::types::{OwnedFilter, Pubkey, Time};
-use futures::{sink::SinkExt, stream::StreamExt};
-use hyper::service::Service;
-use hyper::upgrade::Upgraded;
-use hyper::{Body, Request, Response};
-use hyper_tungstenite::{tungstenite, WebSocketStream};
-use std::collections::HashMap;
-use std::env;
-use std::error::Error as StdError;
-use std::fs::OpenOptions;
-use std::future::Future;
-use std::io::Read;
-use std::net::SocketAddr;
-use std::pin::Pin;
-use std::sync::atomic::Ordering;
-use std::task::{Context, Poll};
-use std::time::Duration;
-use textnonce::TextNonce;
-use tokio::net::{TcpListener, TcpStream};
-use tokio::signal::unix::{signal, SignalKind};
-use tungstenite::protocol::WebSocketConfig;
-use tungstenite::Message;
-
-#[tokio::main]
-async fn main() -> Result<(), Error> {
-    env_logger::init();
-
-    // Get args (config path)
-    let mut args = env::args();
-    if args.len() <= 1 {
-        panic!("USAGE: chorus <config_path>");
-    }
-    let _ = args.next(); // ignore program name
-    let config_path = args.next().unwrap();
-
-    // Read config file
-    let mut file = OpenOptions::new().read(true).open(config_path)?;
-    let mut contents = String::new();
-    file.read_to_string(&mut contents)?;
-    let friendly_config: FriendlyConfig = ron::from_str(&contents)?;
-    let config: Config = friendly_config.into_config()?;
-    log::debug!("Loaded config file.");
-
-    // Setup store
-    let store = Store::new(&config.data_directory, config.allow_scraping)?;
-    let _ = GLOBALS.store.set(store);
-
-    // TLS setup
-    let maybe_tls_acceptor = if config.use_tls {
-        log::info!("Using TLS");
-        Some(tls::tls_acceptor(&config)?)
-    } else {
-        log::info!("Not using TLS");
-        None
-    };
-
-    // Bind listener to port
-    let listener = TcpListener::bind((&*config.ip_address, config.port)).await?;
-    log::info!("Running on {}:{}", config.ip_address, config.port);
-
-    // Store config into GLOBALS
-    *GLOBALS.config.write().await = config;
-
-    let mut interrupt_signal = signal(SignalKind::interrupt())?;
-    let mut quit_signal = signal(SignalKind::quit())?;
-    let mut terminate_signal = signal(SignalKind::terminate())?;
-
-    loop {
-        tokio::select! {
-            // Exits gracefully upon exit-type signals
-            v = interrupt_signal.recv() => if v.is_some() {
-                log::info!("SIGINT");
-                break;
-            },
-            v = quit_signal.recv() => if v.is_some() {
-                log::info!("SIGQUIT");
-                break;
-            },
-            v = terminate_signal.recv() => if v.is_some() {
-                log::info!("SIGTERM");
-                break;
-            },
-
-            // Accepts network connections and spawn a task to serve each one
-            v = listener.accept() => {
-                let (tcp_stream, peer_addr) = v?;
-                let ipaddr = peer_addr.ip();
-
-                if let Some(ban_until) = GLOBALS.banlist.read().await.get(&ipaddr) {
-                    let now = Time::now();
-                    if *ban_until > now {
-                        log::debug!("{peer_addr}: Blocking reconnection until {ban_until}");
-                        continue;
-                    }
-                }
-
-                if let Some(tls_acceptor) = &maybe_tls_acceptor {
-                    let tls_acceptor_clone = tls_acceptor.clone();
-                    tokio::spawn(async move {
-                        match tls_acceptor_clone.accept(tcp_stream).await {
-                            Err(e) => log::error!("{}", e),
-                            Ok(tls_stream) => {
-                                if let Err(e) = serve(MaybeTlsStream::Rustls(tls_stream), peer_addr).await {
-                                    log::error!("{}", e);
-                                }
-                            }
-                        }
-                    });
-                } else {
-                    serve(MaybeTlsStream::Plain(tcp_stream), peer_addr).await?;
-                }
-            }
-        };
-    }
-
-    // Pre-sync in case something below hangs up
-    let _ = GLOBALS.store.get().unwrap().sync();
-
-    // Set the shutting down signal
-    let _ = GLOBALS.shutting_down.send(true);
-
-    // Wait for active websockets to shutdown gracefully
-    let mut num_clients = GLOBALS.num_clients.load(Ordering::Relaxed);
-    if num_clients != 0 {
-        log::info!("Waiting for {num_clients} websockets to shutdown...");
-
-        // We will check if all clients have shutdown every 25ms
-        let sleep = tokio::time::sleep(Duration::from_millis(25));
-        tokio::pin!(sleep);
-
-        while num_clients != 0 {
-            // If we get another shutdown signal, stop waiting for websockets
-            tokio::select! {
-                v = interrupt_signal.recv() => if v.is_some() {
-                    break;
-                },
-                v = quit_signal.recv() => if v.is_some() {
-                    break;
-                },
-                v = terminate_signal.recv() => if v.is_some() {
-                    break;
-                },
-                () = &mut sleep => {
-                    num_clients = GLOBALS.num_clients.load(Ordering::Relaxed);
-                    continue;
-                }
-            }
-        }
-    }
-
-    log::info!("Syncing and shutting down.");
-    let _ = GLOBALS.store.get().unwrap().sync();
-
-    Ok(())
-}
-
-// Serve a single network connection
-async fn serve(stream: MaybeTlsStream<TcpStream>, peer_addr: SocketAddr) -> Result<(), Error> {
-    // Serve the network stream with our http server and our HttpService
-    let service = HttpService { peer: peer_addr };
-
-    let connection = GLOBALS
-        .http_server
-        .serve_connection(stream, service)
-        .with_upgrades();
-
-    tokio::spawn(async move {
-        // If our service exits with an error, log the error
-        if let Err(he) = connection.await {
-            if let Some(src) = he.source() {
-                if &*format!("{}", src) == "Transport endpoint is not connected (os error 107)" {
-                    // do nothing
-                } else {
-                    // Print in detail
-                    log::error!("{:?}", src);
-                }
-            } else {
-                // Print in less detail
-                let e: Error = he.into();
-                log::error!("{}", e);
-            }
-        }
-    });
-
-    Ok(())
-}
-
-// This is our per-connection HTTP service
-struct HttpService {
-    peer: SocketAddr,
-}
-
-impl Service<Request<Body>> for HttpService {
-    type Response = Response<Body>;
-    type Error = Error;
-    type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
-
-    fn poll_ready(&mut self, _: &mut Context) -> Poll<Result<(), Self::Error>> {
-        Poll::Ready(Ok(()))
-    }
-
-    // This is called for each HTTP request made by the client
-    // NOTE: it is not called for each websocket message once upgraded.
-    fn call(&mut self, req: Request<Body>) -> Self::Future {
-        let peer = self.peer;
-        Box::pin(async move { handle_http_request(peer, req).await })
-    }
-}
-
-async fn handle_http_request(
-    peer: SocketAddr,
-    mut request: Request<Body>,
-) -> Result<Response<Body>, Error> {
-    if hyper_tungstenite::is_upgrade_request(&request) {
-        let web_socket_config = WebSocketConfig {
-            max_write_buffer_size: 1024 * 1024,  // 1 MB
-            max_message_size: Some(1024 * 1024), // 1 MB
-            max_frame_size: Some(1024 * 1024),   // 1 MB
-            ..Default::default()
-        };
-        let (response, websocket) =
-            hyper_tungstenite::upgrade(&mut request, Some(web_socket_config))?;
-        tokio::spawn(async move {
-            // Await the websocket upgrade process
-            match websocket.await {
-                Ok(websocket) => {
-                    // Build a websocket service
-                    let mut ws_service = WebSocketService {
-                        peer,
-                        subscriptions: HashMap::new(),
-                        // We start with a 1-page buffer, and grow it if needed.
-                        buffer: vec![0; 4096],
-                        websocket,
-                        challenge: TextNonce::new().into_string(),
-                        user: None,
-                    };
-
-                    // Increment count of active websockets
-                    let old_num_websockets = GLOBALS.num_clients.fetch_add(1, Ordering::SeqCst);
-
-                    log::info!(
-                        "{}: websocket started (making {} active websockets)",
-                        peer,
-                        old_num_websockets + 1
-                    );
-
-                    // Handle the websocket
-                    if let Err(e) = ws_service.handle_websocket_stream().await {
-                        if matches!(
-                            e.inner,
-                            ChorusError::Tungstenite(tungstenite::error::Error::Protocol(
-                                tungstenite::error::ProtocolError::ResetWithoutClosingHandshake
-                            ))
-                        ) {
-                            // Swallow the boring error
-                        } else {
-                            log::error!("{}: {}", peer, e);
-                        }
-                    }
-
-                    // Decrement count of active websockets
-                    let old_num_websockets = GLOBALS.num_clients.fetch_sub(1, Ordering::SeqCst);
-
-                    log::info!(
-                        "{}: websocket ended (making {} active websockets)",
-                        peer,
-                        old_num_websockets - 1
-                    );
-
-                    // Everybody gets a 4 second ban on disconnect, to prevent rapid reconnection
-                    let ipaddr = peer.ip();
-                    let mut until = Time::now();
-                    until.0 += 4;
-                    if let Some(current_ban) = GLOBALS.banlist.read().await.get(&ipaddr) {
-                        until.0 = current_ban.0.max(until.0);
-                    }
-                    GLOBALS.banlist.write().await.insert(ipaddr, until);
-                }
-                Err(e) => {
-                    log::error!("{}", e);
-                }
-            }
-        });
-        Ok(response)
-    } else {
-        // check for Accept header of application/nostr+json
-        if let Some(accept) = request.headers().get("Accept") {
-            if let Ok(s) = accept.to_str() {
-                if s == "application/nostr+json" {
-                    return web::serve_nip11(peer).await;
-                }
-            }
-        }
-
-        web::serve_http(peer, request).await
-    }
-}
-
-struct WebSocketService {
-    pub peer: SocketAddr,
-    pub subscriptions: HashMap<String, Vec<OwnedFilter>>,
-    pub buffer: Vec<u8>,
-    pub websocket: WebSocketStream<Upgraded>,
-    pub challenge: String,
-    pub user: Option<Pubkey>,
-}
-
-impl WebSocketService {
-    async fn handle_websocket_stream(&mut self) -> Result<(), Error> {
-        // Subscribe to the shutting down channel
-        let mut shutting_down = GLOBALS.shutting_down.subscribe();
-
-        // Subscribe to the new_events broadcast channel
-        let mut new_events = GLOBALS.new_events.subscribe();
-
-        // Offer AUTH to clients right off the bat
-        let reply = NostrReply::Auth(self.challenge.clone());
-        self.websocket.send(Message::text(reply.as_json())).await?;
-
-        loop {
-            tokio::select! {
-                message_option = self.websocket.next() => {
-                    match message_option {
-                        Some(message) => {
-                            let message = message?;
-                            self.handle_websocket_message(message).await?;
-                        },
-                        None => break, // the websocket is closed
-                    }
-                },
-                offset_result = new_events.recv() => {
-                    let offset = offset_result?;
-                    self.handle_new_event(offset).await?;
-                },
-                _r = shutting_down.changed() => {
-                    // Shutdown the websocket gracefully
-                    self.websocket.send(Message::Close(None)).await?;
-                    break;
-                },
-            }
-        }
-
-        Ok(())
-    }
-
-    // If the event matches a subscription they have open, send them the event
-    async fn handle_new_event(&mut self, new_event_offset: usize) -> Result<(), Error> {
-        if self.subscriptions.is_empty() {
-            return Ok(());
-        }
-
-        if let Some(event) = GLOBALS
-            .store
-            .get()
-            .unwrap()
-            .get_event_by_offset(new_event_offset)?
-        {
-            'subs: for (subid, filters) in self.subscriptions.iter() {
-                for filter in filters.iter() {
-                    if filter.as_filter()?.event_matches(&event)? {
-                        let message = NostrReply::Event(subid, event.clone());
-                        self.websocket
-                            .send(Message::text(message.as_json()))
-                            .await?;
-                        continue 'subs;
-                    }
-                }
-            }
-        }
-
-        Ok(())
-    }
-
-    async fn handle_websocket_message(&mut self, message: Message) -> Result<(), Error> {
-        match message {
-            Message::Text(msg) => {
-                log::debug!("{}: <= {}", self.peer, msg);
-                // This is defined in nostr.rs
-                if let Err(e) = self.handle_nostr_message(msg).await {
-                    log::error!("{e}");
-                    let reply = NostrReply::Notice(format!("error: {}", e));
-                    self.websocket.send(Message::text(reply.as_json())).await?;
-                }
-            }
-            Message::Binary(msg) => {
-                let reply = NostrReply::Notice(
-                    "binary messages are not processed by this relay".to_owned(),
-                );
-                self.websocket.send(Message::text(reply.as_json())).await?;
-                log::info!(
-                    "{}: Received unhandled binary message: {:02X?}",
-                    self.peer,
-                    msg
-                );
-            }
-            Message::Ping(msg) => {
-                // No need to send a reply: tungstenite takes care of this for you.
-                log::debug!("{}: Received ping message: {:02X?}", self.peer, msg);
-            }
-            Message::Pong(msg) => {
-                log::debug!("{}: Received pong message: {:02X?}", self.peer, msg);
-            }
-            Message::Close(msg) => {
-                // No need to send a reply: tungstenite takes care of this for you.
-                if let Some(msg) = &msg {
-                    log::debug!(
-                        "{}: Received close message with code {} and message: {}",
-                        self.peer,
-                        msg.code,
-                        msg.reason
-                    );
-                } else {
-                    log::debug!("{}: Received close message", self.peer);
-                }
-            }
-            Message::Frame(_msg) => {
-                unreachable!();
-            }
-        }
-
-        Ok(())
-    }
-}

src/neg_storage.rs

@@ -0,0 +1,123 @@
+use negentropy::{Bound, Error, Id, Item, NegentropyStorageBase};
+
+// We construct our own NegentropyStorageVector since NegentropyStorageBase
+// is not implemented for a &NegentropyStorageVector upstream
+
+/// Negentropy Storage Vector
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct NegentropyStorageVector {
+    items: Vec<Item>,
+    sealed: bool,
+}
+
+impl NegentropyStorageBase for &NegentropyStorageVector {
+    fn size(&self) -> Result<usize, Error> {
+        self.check_sealed()?;
+        Ok(self.items.len())
+    }
+
+    fn get_item(&self, i: usize) -> Result<Option<Item>, Error> {
+        self.check_sealed()?;
+        Ok(self.items.get(i).copied())
+    }
+
+    fn iterate(
+        &self,
+        begin: usize,
+        end: usize,
+        cb: &mut dyn FnMut(Item, usize) -> Result<bool, Error>,
+    ) -> Result<(), Error> {
+        self.check_sealed()?;
+        self.check_bounds(begin, end)?;
+
+        for i in begin..end {
+            if !cb(self.items[i], i)? {
+                break;
+            }
+        }
+
+        Ok(())
+    }
+
+    fn find_lower_bound(&self, mut first: usize, last: usize, value: &Bound) -> usize {
+        let mut count: usize = last - first;
+
+        while count > 0 {
+            let mut it: usize = first;
+            let step: usize = count / 2;
+            it += step;
+
+            if self.items[it] < value.item {
+                it += 1;
+                first = it;
+                count -= step + 1;
+            } else {
+                count = step;
+            }
+        }
+
+        first
+    }
+}
+
+impl NegentropyStorageVector {
+    /// Create new storage
+    #[inline]
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Create new storage with capacity
+    #[inline]
+    pub fn with_capacity(capacity: usize) -> Self {
+        Self {
+            items: Vec::with_capacity(capacity),
+            sealed: false,
+        }
+    }
+
+    /// Insert item
+    pub fn insert(&mut self, created_at: u64, id: Id) -> Result<(), Error> {
+        if self.sealed {
+            return Err(Error::AlreadySealed);
+        }
+
+        let elem: Item = Item::with_timestamp_and_id(created_at, id);
+        self.items.push(elem);
+
+        Ok(())
+    }
+
+    /// Seal
+    pub fn seal(&mut self) -> Result<(), Error> {
+        if self.sealed {
+            return Err(Error::AlreadySealed);
+        }
+        self.sealed = true;
+
+        self.items.sort();
+        self.items.dedup();
+
+        Ok(())
+    }
+
+    /// Unseal
+    pub fn unseal(&mut self) -> Result<(), Error> {
+        self.sealed = false;
+        Ok(())
+    }
+
+    fn check_sealed(&self) -> Result<(), Error> {
+        if !self.sealed {
+            return Err(Error::NotSealed);
+        }
+        Ok(())
+    }
+
+    fn check_bounds(&self, begin: usize, end: usize) -> Result<(), Error> {
+        if begin > end || end > self.items.len() {
+            return Err(Error::BadRange);
+        }
+        Ok(())
+    }
+}

src/reply.rs

@@ -1,6 +1,8 @@
-use crate::types::{Event, Id};
+use crate::Error;
+use pocket_types::{write_hex, Event, Hll8, Id};
 use std::fmt;
 
+#[derive(Debug, Clone, Copy)]
 pub enum NostrReplyPrefix {
     None,
     AuthRequired,
@@ -29,32 +31,74 @@ impl fmt::Display for NostrReplyPrefix {
     }
 }
 
+#[derive(Debug, Clone)]
+#[allow(clippy::large_enum_variant)]
 pub enum NostrReply<'a> {
     Auth(String),
-    Event(&'a str, Event<'a>),
+    Event(&'a str, &'a Event),
     Ok(Id, bool, NostrReplyPrefix, String),
     Eose(&'a str),
     Closed(&'a str, NostrReplyPrefix, String),
     Notice(String),
+    Count(&'a str, usize, Option<Hll8>),
+    NegErr(&'a str, String),
+    NegMsg(&'a str, Vec<u8>),
 }
 
 impl NostrReply<'_> {
-    pub fn as_json(&self) -> String {
+    pub fn as_json(&self) -> Result<String, Error> {
-        match self {
+        Ok(match self {
-            NostrReply::Auth(challenge) => format!(r#"["AUTH", "{challenge}"]"#),
+            NostrReply::Auth(challenge) => {
-            NostrReply::Event(subid, event) => format!(r#"["EVENT", "{subid}", {}]"#, event),
+                let esc_challenge = escape(challenge)?;
-            NostrReply::Ok(id, ok, prefix, msg) => format!(r#"["OK","{id}",{ok},"{prefix}{msg}"]"#),
+                format!(r#"["AUTH","{esc_challenge}"]"#)
-            NostrReply::Eose(subid) => format!(r#"["EOSE","{subid}"]"#),
+            }
+            NostrReply::Event(subid, event) => {
+                let esc_subid = escape(subid)?;
+                format!(r#"["EVENT","{esc_subid}",{event}]"#)
+            }
+            NostrReply::Ok(id, ok, prefix, msg) => {
+                let esc_msg = escape(msg)?;
+                format!(r#"["OK","{id}",{ok},"{prefix}{esc_msg}"]"#)
+            }
+            NostrReply::Eose(subid) => {
+                let esc_subid = escape(subid)?;
+                format!(r#"["EOSE","{esc_subid}"]"#)
+            }
             NostrReply::Closed(subid, prefix, msg) => {
                 format!(r#"["CLOSED","{subid}","{prefix}{msg}"]"#)
             }
-            NostrReply::Notice(msg) => format!(r#"["NOTICE","{msg}"]"#),
+            NostrReply::Notice(msg) => {
+                let esc_msg = escape(msg)?;
+                format!(r#"["NOTICE","{esc_msg}"]"#)
             }
+            NostrReply::Count(subid, c, opthll) => {
+                let esc_subid = escape(subid)?;
+                if let Some(hll) = opthll {
+                    let hll = hll.to_hex_string();
+                    format!(r#"["COUNT","{esc_subid}",{{"count":{c}, "hll":"{hll}"}}]"#)
+                } else {
+                    format!(r#"["COUNT","{esc_subid}",{{"count":{c}}}]"#)
+                }
+            }
+            NostrReply::NegErr(subid, reason) => {
+                let esc_subid = escape(subid)?;
+                let esc_reason = escape(reason)?;
+                format!(r#"["NEG-ERR","{esc_subid}","{esc_reason}"]"#)
+            }
+            NostrReply::NegMsg(subid, msg) => {
+                let esc_subid = escape(subid)?;
+                // write msg as hex
+                let mut buf: Vec<u8> = vec![0; msg.len() * 2];
+                write_hex!(msg, &mut buf, msg.len()).unwrap();
+                let msg_hex = unsafe { std::str::from_utf8_unchecked(&buf) };
+                format!(r#"["NEG-MSG","{esc_subid}","{}"]"#, msg_hex)
+            }
+        })
     }
 }
 
-impl fmt::Display for NostrReply<'_> {
+fn escape(s: &str) -> Result<String, Error> {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+    let v: Vec<u8> = Vec::with_capacity(256);
-        write!(f, "{}", self.as_json())
+    let e = pocket_types::json::json_escape(s.as_bytes(), v)?;
-    }
+    Ok(unsafe { String::from_utf8_unchecked(e) })
 }

src/store/event_store.rs

@@ -1,183 +0,0 @@
-use crate::error::Error;
-use crate::types::Event;
-use mmap_append::MmapAppend;
-use std::fs::{File, OpenOptions};
-use std::mem;
-use std::path::Path;
-use std::sync::atomic::{AtomicUsize, Ordering};
-
-/// This is the size of the initial event map, and also how large it grows by
-/// when we need to grow it. This should be a multiple of the page size (4096)
-// While debugging, we like to use a small value to increase the frequency of
-// resizing, to help detect if there are problems in the algorithm.
-#[cfg(debug_assertions)]
-const EVENT_MAP_CHUNK: usize = 2048;
-#[cfg(not(debug_assertions))]
-const EVENT_MAP_CHUNK: usize = 4096 * 1024; // grow by 4 megabytes at a time
-
-/// An EventStore is a fast storage facility for events.
-#[derive(Debug)]
-pub struct EventStore {
-    // the Mmap doesn't need us to keep the file, but we keep it for resizing.
-    event_map_file: File,
-    event_map_file_len: AtomicUsize,
-
-    // This is a linear sequence of events in an append-only memory mapped file which
-    // internally remembers the 'end' pointer and internally prevents multiple writers.
-    event_map: MmapAppend,
-}
-
-impl EventStore {
-    /// Create a new `EventStore`. The `event_map_file` is the eventually large file
-    /// that holds all the events.
-    pub fn new<P: AsRef<Path>>(event_map_file: P) -> Result<EventStore, Error> {
-        // Open the event map file, possibly creating if it isn't there
-        let event_map_file = OpenOptions::new()
-            .read(true)
-            .write(true)
-            .create(true)
-            .open(event_map_file)?;
-
-        // Get it's size
-        let metadata = event_map_file.metadata()?;
-        let mut len = metadata.len() as usize;
-
-        // Determine if we just created it
-        // (not long enough for the required end offset)
-        let new = len < mem::size_of::<usize>();
-
-        // If brand new:
-        if new {
-            // grow to initial size
-            len = EVENT_MAP_CHUNK;
-            event_map_file.set_len(EVENT_MAP_CHUNK as u64)?;
-        }
-
-        // Memory map it
-        let event_map = unsafe { MmapAppend::new(&event_map_file, new)? };
-
-        log::info!(
-            "Event Store: new={:?} end={} len={}",
-            new,
-            event_map.get_end(),
-            len
-        );
-
-        Ok(EventStore {
-            event_map_file,
-            event_map_file_len: AtomicUsize::new(len),
-            event_map,
-        })
-    }
-
-    /// Get the number of bytes used in the event map
-    #[inline]
-    pub fn read_event_map_end(&self) -> usize {
-        self.event_map.get_end()
-    }
-
-    /// Get an event by its offset in the map
-    pub fn get_event_by_offset(&self, offset: usize) -> Result<Option<Event>, Error> {
-        // deserialize event
-        let event = Event::delineate(&self.event_map[offset..])?;
-        Ok(Some(event))
-    }
-
-    // This stores an event
-    // It does NOT validate the event.
-    // It does NOT check first if the event is already stored, so it could store a duplicate
-    // It does NOT record the event into any indexes
-    // But it does grow the file if needed and returns the offset where it was stored
-    pub fn store_event(&self, event: &Event) -> Result<usize, Error> {
-        let event_size = event.length();
-
-        loop {
-            let result = self.event_map.append(event_size, |dst| event.macopy(dst));
-
-            match result {
-                Ok(offset) => return Ok(offset),
-                Err(e) => {
-                    if e.kind() == std::io::ErrorKind::Other {
-                        if e.to_string() == "Out of space" {
-                            // Determine the new size
-                            let new_file_len = {
-                                let file_len = self.event_map_file_len.load(Ordering::Relaxed);
-                                file_len + EVENT_MAP_CHUNK
-                            };
-
-                            // Grow the file
-                            self.event_map_file.set_len(new_file_len as u64)?;
-
-                            // Resize the memory map
-                            self.event_map.resize(new_file_len)?;
-
-                            // Save this new length
-                            self.event_map_file_len
-                                .store(new_file_len, Ordering::Relaxed);
-
-                            // Try again
-                            continue;
-                        } else {
-                            return Err(e.into());
-                        }
-                    } else {
-                        return Err(e.into());
-                    }
-                }
-            }
-        }
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use crate::store::EventStore;
-    use crate::types::Event;
-
-    #[test]
-    fn test_event_store() {
-        let tempdir = tempfile::tempdir().unwrap();
-        let path = tempdir.path().join("mmap");
-        let store = EventStore::new(&path).unwrap();
-
-        println!("Event map has {} used bytes", store.read_event_map_end());
-
-        let e1str = br#"{"id":"000000005ccb8402fe9af2ecc72ca1dbbf2dbeb9a0c6353b7f8198a65106f04a","pubkey":"7bdef7be22dd8e59f4600e044aa53a1cf975a9dc7d27df5833bc77db784a5805","created_at":1677930312,"kind":1,"tags":[["p","fe1d10131ca6103715d261f1615a8cd31f6b68a3d1a3272aab8e9e83f787126e"],["e","f76a6f60df029c388e98d4e97c67424cb71e34455302580d3aa48d0be96dfaef","wss://nos.lol","root"],["e","f529676213b123f55117ae9b93ca7dd6c85c872b7c260da7dbf60d71875879f6","wss://brb.io","reply"],["nonce","12912720851603613541","25"]],"content":"https://www.nostr.guru/p/fe1d10131ca6103715d261f1615a8cd31f6b68a3d1a3272aab8e9e83f787126e\nHere I can see only the today contact list.\n\nI tried to query relay.damus.io and nostr.wine but I can get only this last contact list, perhaps they have a time limit; you could ask some relay owner for a backup.\nIf can be useful this is the REQ filter to use:\n{\"kinds\": [3], \"authors\": [\"fe1d10131ca6103715d261f1615a8cd31f6b68a3d1a3272aab8e9e83f787126e\"], \"since\": 1672569529}","sig":"7c72d15cf9b4244cb8c49564d3735d0ac620dca69253956975def12054bc50217307f062f3e9ea879de8bae756c844a9315f03aabbb5f5fd94f9fc9beb76f457"}"#;
-        let mut buffer1: Vec<u8> = Vec::with_capacity(4096);
-        buffer1.resize(4096, 0);
-        let (_insize, event1) = Event::from_json(&e1str[..], &mut buffer1).unwrap();
-        let offset1 = store.store_event(&event1).unwrap();
-
-        let e2str = br#"{"id":"00000000a6fa8ee15b17fcc5bb49f09f85c15cddf5349986db09fddc0a123f7d","pubkey":"7cc328a08ddb2afdf9f9be77beff4c83489ff979721827d628a542f32a247c0e","created_at":1678835260,"kind":1,"tags":[["p","0000000033f569c7069cdec575ca000591a31831ebb68de20ed9fb783e3fc287"],["e","52338357568d06379ad2412a2a2033f23f224754d74148bbfc659b872c9477c4","wss://relay.damus.io/","root"],["nonce","9223372036941030112","32"]],"content":"oh shit, it got released?! no wayyy","sig":"ec9bcdcaa843a2a275857999556136d2b3cc47dac52c899d313dd70291b9661590d4c10b13680bbfa685d67d319798cdfa3ca58f61af5f8acffd0e25bca5ab95"}"#;
-        let mut buffer2: Vec<u8> = Vec::with_capacity(4096);
-        buffer2.resize(4096, 0);
-        let (_insize, event2) = Event::from_json(&e2str[..], &mut buffer2).unwrap();
-        let offset2 = store.store_event(&event2).unwrap();
-
-        let e3str = br#"{"id":"00000000ad0efde5b63e9b24b12a586dc98df372e1fd6f96ac6ad24ea2ed1350","pubkey":"c5fb6ecc876e0458e3eca9918e370cbcd376901c58460512fe537a46e58c38bb","created_at":1681739201,"kind":7,"tags":[["e","193bd20beb8fc13f4218ea106928c3be81ee3b2ad2b1bdbdd2c55efd859a195a","wss://eden.nostr.land/"],["p","3f770d65d3a764a9c5cb503ae123e62ec7598ad035d836e2a810f3877a745b24"],["client","gossip"],["nonce","2305843009213833122","22"]],"content":"+","sig":"f344111c221d2fea5f006865b98b0767b40ed1cc2907d8a325a8dea4b98414d008296ff4f4bd4666d52ec86ffd4739e807c6655ce43de98b473326e30957fcb2"}"#;
-        let mut buffer3: Vec<u8> = Vec::with_capacity(4096);
-        buffer3.resize(4096, 0);
-        let (_insize, event3) = Event::from_json(&e3str[..], &mut buffer3).unwrap();
-        let offset3 = store.store_event(&event3).unwrap();
-
-        println!("Event map has {} used bytes", store.read_event_map_end());
-
-        if let Some(event) = store.get_event_by_offset(offset1).unwrap() {
-            assert_eq!(event, event1);
-        } else {
-            panic!("EVENT 1 IS WRONG");
-        }
-
-        if let Some(event) = store.get_event_by_offset(offset2).unwrap() {
-            assert_eq!(event, event2);
-        } else {
-            panic!("EVENT 2 IS WRONG");
-        }
-
-        if let Some(event) = store.get_event_by_offset(offset3).unwrap() {
-            assert_eq!(event, event3);
-        } else {
-            panic!("EVENT 3 IS WRONG");
-        }
-    }
-}

src/store/mod.rs

@@ -1,641 +0,0 @@
-pub mod event_store;
-pub use event_store::EventStore;
-
-use crate::error::{ChorusError, Error};
-use crate::types::{Event, Filter, Id, Kind, Pubkey, Time};
-use heed::byteorder::BigEndian;
-use heed::types::{OwnedType, UnalignedSlice, Unit, U64};
-use heed::{Database, Env, EnvFlags, EnvOpenOptions, RwTxn};
-use std::fs;
-use std::ops::Bound;
-
-#[derive(Debug)]
-pub struct Store {
-    events: EventStore,
-    env: Env,
-    ids: Database<UnalignedSlice<u8>, OwnedType<usize>>,
-    akci: Database<UnalignedSlice<u8>, OwnedType<usize>>,
-    atci: Database<UnalignedSlice<u8>, OwnedType<usize>>,
-    ktci: Database<UnalignedSlice<u8>, OwnedType<usize>>,
-    deleted_offsets: Database<U64<BigEndian>, Unit>,
-    deleted_events: Database<UnalignedSlice<u8>, Unit>,
-    allow_scraping: bool,
-}
-
-impl Store {
-    /// Setup persistent storage
-    pub fn new(data_directory: &str, allow_scraping: bool) -> Result<Store, Error> {
-        let mut builder = EnvOpenOptions::new();
-        unsafe {
-            builder.flags(EnvFlags::NO_TLS);
-        }
-        builder.max_dbs(32);
-        builder.map_size(1048576 * 1024 * 24); // 24 GB
-
-        let dir = format!("{}/lmdb", data_directory);
-        fs::create_dir_all(&dir)?;
-
-        let env = match builder.open(&dir) {
-            Ok(env) => env,
-            Err(e) => {
-                log::error!("Unable to open LMDB at {}", dir);
-                return Err(e.into());
-            }
-        };
-
-        // Open/Create maps
-        let mut txn = env.write_txn()?;
-        let ids = env
-            .database_options()
-            .types::<UnalignedSlice<u8>, OwnedType<usize>>()
-            .name("ids")
-            .create(&mut txn)?;
-        let akci = env
-            .database_options()
-            .types::<UnalignedSlice<u8>, OwnedType<usize>>()
-            .name("akci")
-            .create(&mut txn)?;
-        let atci = env
-            .database_options()
-            .types::<UnalignedSlice<u8>, OwnedType<usize>>()
-            .name("atci")
-            .create(&mut txn)?;
-        let ktci = env
-            .database_options()
-            .types::<UnalignedSlice<u8>, OwnedType<usize>>()
-            .name("ktci")
-            .create(&mut txn)?;
-        let deleted_offsets = env
-            .database_options()
-            .types::<U64<BigEndian>, Unit>()
-            .name("deleted_offsets")
-            .create(&mut txn)?;
-        let deleted_events = env
-            .database_options()
-            .types::<UnalignedSlice<u8>, Unit>()
-            .name("deleted-events")
-            .create(&mut txn)?;
-        txn.commit()?;
-
-        log::info!("Store is setup");
-
-        let event_map_file = format!("{}/event.map", data_directory);
-
-        Ok(Store {
-            events: EventStore::new(event_map_file)?,
-            env,
-            ids,
-            akci,
-            atci,
-            ktci,
-            deleted_offsets,
-            deleted_events,
-            allow_scraping,
-        })
-    }
-
-    /// Sync the data to disk. This happens periodically, but sometimes it's useful to force
-    /// it.
-    pub fn sync(&self) -> Result<(), Error> {
-        self.env.force_sync()?;
-        Ok(())
-    }
-
-    /// Store an event.
-    ///
-    /// Returns the offset where the event is stored at, which can be used to fetch
-    /// the event via get_event_by_offset().
-    ///
-    /// If the event already exists, you will get a ChorusError::Duplicate
-    ///
-    /// If the event is ephemeral, it will be stored and you will get an offset, but
-    /// it will not be indexed.
-    pub fn store_event(&self, event: &Event) -> Result<usize, Error> {
-        // TBD: should we validate the event?
-
-        let mut txn = self.env.write_txn()?;
-        let offset;
-
-        // Only if it doesn't already exist
-        if self.ids.get(&txn, event.id().0.as_slice())?.is_none() {
-            // Reject event if it was deleted
-            {
-                let deleted_key = Self::key_deleted_events(event.id(), event.pubkey());
-                if self.deleted_events.get(&txn, &deleted_key)?.is_some() {
-                    return Err(ChorusError::Deleted.into());
-                }
-            }
-
-            // Store the event
-            offset = self.events.store_event(event)?;
-
-            if event.kind().is_ephemeral() {
-                // Do not index ephemeral events, not even by id.
-                // But save them in the deleted table
-                let offset_u64 = offset as u64;
-                self.deleted_offsets.put(&mut txn, &offset_u64, &())?;
-            } else {
-                // Index the event
-                self.index(&mut txn, event, offset)?;
-            }
-
-            // If replaceable or parameterized replaceable,
-            // find and delete all but the first one in the group
-            if event.kind().is_replaceable() || event.kind().is_parameterized_replaceable() {
-                self.delete_replaced(&mut txn, event)?;
-            }
-
-            // Handle deletion events
-            if event.kind() == Kind(5) {
-                self.handle_deletion_event(&mut txn, event)?;
-            }
-
-            txn.commit()?;
-        } else {
-            return Err(ChorusError::Duplicate.into());
-        }
-
-        Ok(offset)
-    }
-
-    pub fn handle_deletion_event(&self, txn: &mut RwTxn<'_>, event: &Event) -> Result<(), Error> {
-        for mut tag in event.tags()?.iter() {
-            if let Some(tagname) = tag.next() {
-                if tagname == b"e" {
-                    if let Some(id_hex) = tag.next() {
-                        if let Ok(id) = Id::read_hex(id_hex) {
-                            // Add deletion pair to the event_deleted table
-                            let deleted_key = Self::key_deleted_events(id, event.pubkey());
-                            self.deleted_events.put(txn, &deleted_key, &())?;
-
-                            // Delete pair
-                            if let Some(target) = self.get_event_by_id(id)? {
-                                if target.pubkey() == event.pubkey() {
-                                    self.delete(txn, id)?;
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        }
-
-        Ok(())
-    }
-
-    /// Get an event by its offset.
-    pub fn get_event_by_offset(&self, offset: usize) -> Result<Option<Event>, Error> {
-        self.events.get_event_by_offset(offset)
-    }
-
-    /// Get an event by Id
-    pub fn get_event_by_id(&self, id: Id) -> Result<Option<Event>, Error> {
-        let txn = self.env.read_txn()?;
-        if let Some(offset) = self.ids.get(&txn, id.0.as_slice())? {
-            self.events.get_event_by_offset(offset)
-        } else {
-            Ok(None)
-        }
-    }
-
-    /// Find all events that match the filter
-    pub fn find_events(&self, filter: Filter) -> Result<Vec<Event>, Error> {
-        let mut output: Vec<Event> = Vec::new();
-
-        if filter.num_ids() > 0 {
-            // Fetch by id
-            for id in filter.ids() {
-                if let Some(event) = self.get_event_by_id(id)? {
-                    // and check each against the rest of the filter
-                    if filter.event_matches(&event)? {
-                        output.push(event);
-                    }
-                }
-                // Stop if limited
-                if output.len() >= filter.limit() as usize {
-                    return Ok(output);
-                }
-            }
-        } else if filter.num_authors() > 0 && filter.num_kinds() > 0 {
-            for author in filter.authors() {
-                'kind: for kind in filter.kinds() {
-                    let start_prefix = Self::key_akci(
-                        author,
-                        kind,
-                        filter.until(), // scan goes backwards in time
-                        Id([0; 32]),
-                    );
-                    let end_prefix = Self::key_akci(
-                        author,
-                        kind,
-                        filter.since(), // scan goes backwards in time
-                        Id([255; 32]),
-                    );
-                    let range = (
-                        Bound::Included(&*start_prefix),
-                        Bound::Excluded(&*end_prefix),
-                    );
-                    let txn = self.env.read_txn()?;
-                    let iter = self.akci.range(&txn, &range)?;
-                    for result in iter {
-                        let (_key, offset) = result?;
-                        if let Some(event) = self.events.get_event_by_offset(offset)? {
-                            // check against the rest of the filter
-                            if filter.event_matches(&event)? {
-                                output.push(event);
-                                // If kind is replaceable (and not parameterized)
-                                // then don't take any more events for this author-kind
-                                // pair.
-                                // NOTE that this optimization is difficult to implement
-                                // for other replaceable event situations
-                                if kind.is_replaceable() {
-                                    continue 'kind;
-                                }
-                            }
-                        }
-                        // Stop if limited
-                        if output.len() >= filter.limit() as usize {
-                            return Ok(output);
-                        }
-                    }
-                }
-            }
-        } else if filter.num_authors() > 0 && !filter.tags()?.is_empty() {
-            for author in filter.authors() {
-                let tags = filter.tags()?;
-                for mut tag in tags.iter() {
-                    if let Some(tag0) = tag.next() {
-                        if let Some(tagvalue) = tag.next() {
-                            let start_prefix = Self::key_atci(
-                                author,
-                                tag0[0],
-                                tagvalue,
-                                filter.until(), // scan goes backwards in time
-                                Id([0; 32]),
-                            );
-                            let end_prefix = Self::key_atci(
-                                author,
-                                tag0[0],
-                                tagvalue,
-                                filter.since(), // scan goes backwards in time
-                                Id([255; 32]),
-                            );
-                            let range = (
-                                Bound::Included(&*start_prefix),
-                                Bound::Excluded(&*end_prefix),
-                            );
-                            let txn = self.env.read_txn()?;
-                            let iter = self.akci.range(&txn, &range)?;
-                            for result in iter {
-                                let (_key, offset) = result?;
-                                if let Some(event) = self.events.get_event_by_offset(offset)? {
-                                    // check against the rest of the filter
-                                    if filter.event_matches(&event)? {
-                                        output.push(event);
-                                    }
-                                }
-                                // Stop if limited
-                                if output.len() >= filter.limit() as usize {
-                                    return Ok(output);
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        } else if filter.num_kinds() > 0 && !filter.tags()?.is_empty() {
-            for kind in filter.kinds() {
-                let tags = filter.tags()?;
-                for mut tag in tags.iter() {
-                    if let Some(tag0) = tag.next() {
-                        if let Some(tagvalue) = tag.next() {
-                            let start_prefix = Self::key_ktci(
-                                kind,
-                                tag0[0],
-                                tagvalue,
-                                filter.until(), // scan goes backwards in time
-                                Id([0; 32]),
-                            );
-                            let end_prefix = Self::key_ktci(
-                                kind,
-                                tag0[0],
-                                tagvalue,
-                                filter.since(), // scan goes backwards in time
-                                Id([255; 32]),
-                            );
-                            let range = (
-                                Bound::Included(&*start_prefix),
-                                Bound::Excluded(&*end_prefix),
-                            );
-                            let txn = self.env.read_txn()?;
-                            let iter = self.akci.range(&txn, &range)?;
-                            for result in iter {
-                                let (_key, offset) = result?;
-                                if let Some(event) = self.events.get_event_by_offset(offset)? {
-                                    // check against the rest of the filter
-                                    if filter.event_matches(&event)? {
-                                        output.push(event);
-                                    }
-                                }
-                                // Stop if limited
-                                if output.len() >= filter.limit() as usize {
-                                    return Ok(output);
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        } else if self.allow_scraping {
-            // This is INEFFICIENT as it scans through EVERY EVENT
-            // but the filter is a scraper and we don't have a lot of support
-            // for scrapers.
-            let txn = self.env.read_txn()?;
-            let iter = self.ids.iter(&txn)?;
-            for result in iter {
-                let (_key, offset) = result?;
-                if let Some(event) = self.events.get_event_by_offset(offset)? {
-                    if filter.event_matches(&event)? {
-                        output.push(event);
-                    }
-                }
-            }
-        } else {
-            return Err(ChorusError::Scraper.into());
-        }
-
-        Ok(output)
-    }
-
-    /// Delete an event by id
-    fn delete(&self, txn: &mut RwTxn<'_>, id: Id) -> Result<(), Error> {
-        if let Some(offset) = self.ids.get(txn, id.0.as_slice())? {
-            self.set_offset_as_deleted(offset)?;
-
-            // Also remove from the id index
-            self.ids.delete(txn, id.0.as_slice())?;
-        }
-        Ok(())
-    }
-
-    // Index the event
-    fn index(&self, txn: &mut RwTxn<'_>, event: &Event, offset: usize) -> Result<(), Error> {
-        // Index by id
-        self.ids.put(txn, event.id().0.as_slice(), &offset)?;
-
-        // Index by author and kind (with created_at and id)
-        self.akci.put(
-            txn,
-            &Self::key_akci(event.pubkey(), event.kind(), event.created_at(), event.id()),
-            &offset,
-        )?;
-
-        for mut tsi in event.tags()?.iter() {
-            if let Some(tagname) = tsi.next() {
-                // FIXME make sure it is a letter too
-                if tagname.len() == 1 {
-                    if let Some(tagvalue) = tsi.next() {
-                        // Index by author and tag (with created_at and id)
-                        self.atci.put(
-                            txn,
-                            &Self::key_atci(
-                                event.pubkey(),
-                                tagname[0],
-                                tagvalue,
-                                event.created_at(),
-                                event.id(),
-                            ),
-                            &offset,
-                        )?;
-
-                        // Index by kind and tag (with created_at and id)
-                        self.ktci.put(
-                            txn,
-                            &Self::key_ktci(
-                                event.kind(),
-                                tagname[0],
-                                tagvalue,
-                                event.created_at(),
-                                event.id(),
-                            ),
-                            &offset,
-                        )?;
-                    }
-                }
-            }
-        }
-
-        Ok(())
-    }
-
-    // Remove the event from all indexes (except the 'id' index)
-    fn deindex(&self, txn: &mut RwTxn<'_>, event: &Event) -> Result<(), Error> {
-        for mut tsi in event.tags()?.iter() {
-            if let Some(tagname) = tsi.next() {
-                // FIXME make sure it is a letter too
-                if tagname.len() == 1 {
-                    if let Some(tagvalue) = tsi.next() {
-                        // Index by author and tag (with created_at and id)
-                        self.atci.delete(
-                            txn,
-                            &Self::key_atci(
-                                event.pubkey(),
-                                tagname[0],
-                                tagvalue,
-                                event.created_at(),
-                                event.id(),
-                            ),
-                        )?;
-
-                        // Index by kind and tag (with created_at and id)
-                        self.ktci.delete(
-                            txn,
-                            &Self::key_ktci(
-                                event.kind(),
-                                tagname[0],
-                                tagvalue,
-                                event.created_at(),
-                                event.id(),
-                            ),
-                        )?;
-                    }
-                }
-            }
-        }
-
-        // Index by author and kind (with created_at and id)
-        self.akci.delete(
-            txn,
-            &Self::key_akci(event.pubkey(), event.kind(), event.created_at(), event.id()),
-        )?;
-
-        // We leave it in the id map. If someone wants to load the replaced event by id
-        // they can still do it.
-        // self.ids.delete(&mut txn, event.id().0.as_slice())?;
-
-        Ok(())
-    }
-
-    // Set an event as deleted
-    // This removes it from indexes (except the id index) and adds it to the deleted table
-    fn set_offset_as_deleted(&self, offset: usize) -> Result<(), Error> {
-        let mut txn = self.env.write_txn()?;
-
-        let offset_u64 = offset as u64;
-
-        // Check if it is already deleted
-        if self.deleted_offsets.get(&txn, &offset_u64)?.is_some() {
-            return Ok(());
-        }
-
-        // Add to deleted database in case we need to get at it in the future.
-        self.deleted_offsets.put(&mut txn, &offset_u64, &())?;
-
-        // Get event
-        let event = match self.events.get_event_by_offset(offset)? {
-            Some(event) => event,
-            None => return Ok(()),
-        };
-
-        // Remove from indexes
-        self.deindex(&mut txn, &event)?;
-
-        txn.commit()?;
-
-        Ok(())
-    }
-
-    // If the event is replaceable or parameterized replaceable
-    // this deletes all the events in that group except the most recent one.
-    fn delete_replaced(&self, txn: &mut RwTxn<'_>, event: &Event) -> Result<(), Error> {
-        if event.kind().is_replaceable() {
-            let start_prefix = Self::key_akci(
-                event.pubkey(),
-                event.kind(),
-                Time::max(), // database is ordered in reverse time
-                Id([0; 32]),
-            );
-            let end_prefix =
-                Self::key_akci(event.pubkey(), event.kind(), Time::min(), Id([255; 32]));
-            let range = (
-                Bound::Included(&*start_prefix),
-                Bound::Excluded(&*end_prefix),
-            );
-            let iter = self.akci.range(txn, &range)?;
-            let mut first = true;
-            for result in iter {
-                // Keep the first result
-                if first {
-                    first = false;
-                    continue;
-                }
-
-                let (_key, offset) = result?;
-
-                // Delete the event
-                self.set_offset_as_deleted(offset)?;
-            }
-        } else if event.kind().is_parameterized_replaceable() {
-            let tags = event.tags()?;
-            if let Some(identifier) = tags.get_value(b"d") {
-                let start_prefix =
-                    Self::key_atci(event.pubkey(), b'd', identifier, Time::max(), Id([0; 32]));
-                let end_prefix =
-                    Self::key_atci(event.pubkey(), b'd', identifier, Time::min(), Id([255; 32]));
-                let range = (
-                    Bound::Included(&*start_prefix),
-                    Bound::Excluded(&*end_prefix),
-                );
-                let iter = self.akci.range(txn, &range)?;
-                let mut first = true;
-                for result in iter {
-                    // Keep the first result
-                    if first {
-                        first = false;
-                        continue;
-                    }
-
-                    let (_key, offset) = result?;
-
-                    // Delete the event
-                    self.set_offset_as_deleted(offset)?;
-                }
-            }
-        }
-
-        Ok(())
-    }
-
-    // For looking up event by Author and Kind
-    // author(32) + kind(2) + reversecreatedat(8) + id(32)
-    #[allow(dead_code)]
-    fn key_akci(author: Pubkey, kind: Kind, created_at: Time, id: Id) -> Vec<u8> {
-        let mut key: Vec<u8> = Vec::with_capacity(
-            std::mem::size_of::<Pubkey>()
-                + std::mem::size_of::<Kind>()
-                + std::mem::size_of::<Time>()
-                + std::mem::size_of::<Id>(),
-        );
-        key.extend(author.as_slice());
-        key.extend(kind.0.to_be_bytes());
-        key.extend((u64::MAX - created_at.0).to_be_bytes().as_slice());
-        key.extend(id.as_slice());
-        key
-    }
-
-    // For looking up event by Author and Tag
-    // author(32) + tagletter(1) + fixlentag(182) + reversecreatedat(8) + id(32)
-    #[allow(dead_code)]
-    fn key_atci(author: Pubkey, letter: u8, tag_value: &[u8], created_at: Time, id: Id) -> Vec<u8> {
-        const PADLEN: usize = 182;
-        let mut key: Vec<u8> = Vec::with_capacity(
-            std::mem::size_of::<Pubkey>()
-                + PADLEN
-                + std::mem::size_of::<Time>()
-                + std::mem::size_of::<Id>(),
-        );
-        key.extend(author.as_slice());
-        key.push(letter);
-        if tag_value.len() <= PADLEN {
-            key.extend(tag_value);
-            key.extend(core::iter::repeat(0).take(PADLEN - tag_value.len()));
-        } else {
-            key.extend(&tag_value[..PADLEN]);
-        }
-        key.extend((u64::MAX - created_at.0).to_be_bytes().as_slice());
-        key.extend(id.as_slice());
-        key
-    }
-
-    // For looking up event by Kind and Tag
-    // kind(2) + tagletter(1) + fixlentag(182) + reversecreatedat(8) + id(32)
-    #[allow(dead_code)]
-    fn key_ktci(kind: Kind, letter: u8, tag_value: &[u8], created_at: Time, id: Id) -> Vec<u8> {
-        const PADLEN: usize = 182;
-        let mut key: Vec<u8> = Vec::with_capacity(
-            std::mem::size_of::<Kind>()
-                + PADLEN
-                + std::mem::size_of::<Time>()
-                + std::mem::size_of::<Id>(),
-        );
-        key.extend(kind.0.to_be_bytes());
-        key.push(letter);
-        if tag_value.len() <= PADLEN {
-            key.extend(tag_value);
-            key.extend(core::iter::repeat(0).take(PADLEN - tag_value.len()));
-        } else {
-            key.extend(&tag_value[..PADLEN]);
-        }
-        key.extend((u64::MAX - created_at.0).to_be_bytes().as_slice());
-        key.extend(id.as_slice());
-        key
-    }
-
-    fn key_deleted_events(id: Id, pubkey: Pubkey) -> Vec<u8> {
-        let mut key: Vec<u8> =
-            Vec::with_capacity(std::mem::size_of::<Id>() + std::mem::size_of::<Pubkey>());
-        key.extend(id.as_slice());
-        key.extend(pubkey.as_slice());
-        key
-    }
-}

src/tls.rs

@@ -1,27 +1,26 @@
 use crate::config::Config;
 use crate::error::{ChorusError, Error};
-use rustls::{Certificate, PrivateKey};
+use rustls_pki_types::{CertificateDer, PrivateKeyDer};
 use std::fs::File;
 use std::io::BufReader;
-use std::pin::Pin;
 use std::sync::Arc;
-use std::task::{Context, Poll};
-use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
 use tokio_rustls::{rustls, TlsAcceptor};
 
 pub fn tls_acceptor(config: &Config) -> Result<TlsAcceptor, Error> {
-    let certs: Vec<Certificate> =
+    let cert_file = File::open(&config.certchain_pem_path)?;
-        rustls_pemfile::certs(&mut BufReader::new(File::open(&config.certchain_pem_path)?))?
+    let mut certificates: Vec<CertificateDer<'static>> = Vec::new();
-            .drain(..)
+    for maybe_cert in rustls_pemfile::certs(&mut BufReader::new(cert_file)) {
-            .map(Certificate)
+        let cert = maybe_cert?;
-            .collect();
+        certificates.push(cert);
+    }
 
-    let mut keys: Vec<PrivateKey> =
+    let key_file = File::open(&config.key_pem_path)?;
-        rustls_pemfile::pkcs8_private_keys(&mut BufReader::new(File::open(&config.key_pem_path)?))?
+    let mut keys: Vec<PrivateKeyDer> = Vec::new();
-            .drain(..)
+    for maybe_key in rustls_pemfile::pkcs8_private_keys(&mut BufReader::new(key_file)) {
-            .rev()
+        let key = maybe_key?;
-            .map(PrivateKey)
+        keys.push(PrivateKeyDer::Pkcs8(key));
-            .collect();
+    }
+    keys.reverse();
 
     let key = match keys.pop() {
         Some(k) => k,
@@ -29,62 +28,8 @@ pub fn tls_acceptor(config: &Config) -> Result<TlsAcceptor, Error> {
     };
 
     let tls_config = rustls::ServerConfig::builder()
-        .with_safe_defaults()
         .with_no_client_auth()
-        .with_single_cert(certs, key)?;
+        .with_single_cert(certificates, key)?;
 
     Ok(TlsAcceptor::from(Arc::new(tls_config)))
 }
-
-/// A stream that might be protected with TLS.
-#[allow(clippy::large_enum_variant)] // not great though
-#[derive(Debug)]
-pub enum MaybeTlsStream<S> {
-    /// Unencrypted socket stream.
-    Plain(S),
-    /// Encrypted socket stream using `rustls`.
-    Rustls(tokio_rustls::server::TlsStream<S>),
-}
-
-impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
-    fn poll_read(
-        self: Pin<&mut Self>,
-        cx: &mut Context<'_>,
-        buf: &mut ReadBuf<'_>,
-    ) -> Poll<std::io::Result<()>> {
-        match self.get_mut() {
-            MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_read(cx, buf),
-            MaybeTlsStream::Rustls(s) => Pin::new(s).poll_read(cx, buf),
-        }
-    }
-}
-
-impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
-    fn poll_write(
-        self: Pin<&mut Self>,
-        cx: &mut Context<'_>,
-        buf: &[u8],
-    ) -> Poll<Result<usize, std::io::Error>> {
-        match self.get_mut() {
-            MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_write(cx, buf),
-            MaybeTlsStream::Rustls(s) => Pin::new(s).poll_write(cx, buf),
-        }
-    }
-
-    fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), std::io::Error>> {
-        match self.get_mut() {
-            MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_flush(cx),
-            MaybeTlsStream::Rustls(s) => Pin::new(s).poll_flush(cx),
-        }
-    }
-
-    fn poll_shutdown(
-        self: Pin<&mut Self>,
-        cx: &mut Context<'_>,
-    ) -> Poll<Result<(), std::io::Error>> {
-        match self.get_mut() {
-            MaybeTlsStream::Plain(ref mut s) => Pin::new(s).poll_shutdown(cx),
-            MaybeTlsStream::Rustls(s) => Pin::new(s).poll_shutdown(cx),
-        }
-    }
-}

src/types/event/json_event.rs

@@ -1,318 +0,0 @@
-use crate::error::{ChorusError, Error};
-use crate::types::parse::json_parse::*;
-
-/// Parses a JSON event from the `input` buffer. Places the parsed event into the `output` buffer.
-/// Returns the count of consumed bytes and output bytes
-pub fn parse_json_event(input: &[u8], output: &mut [u8]) -> Result<(usize, usize), Error> {
-    // Minimum-sized JSON event is 204 characters long
-    if input.len() < 204 {
-        return Err(ChorusError::JsonBadEvent("Too Short", 0).into());
-    }
-
-    // NOTE: 152 is the minimum binary event
-    if output.len() < 152 {
-        return Err(ChorusError::BufferTooSmall.into());
-    }
-
-    // This tracks where we are currently looking in the input as we scan forward.
-    // It is short for INput POSition.
-    let mut inpos = 0;
-
-    // If tags comes before content, content can use this to know where to put itself.
-    // This is the length of the tags output section. 0 means it hasn't been written yet.
-    let mut tags_size: usize = 0;
-
-    // If content comes before tags, we cannot write it because we don't know how much
-    // space Tags will take.  So we instead just remember where the content string
-    // begins so we can write it later.
-    let mut content_input_start: usize = 0;
-
-    // Remember which fields we have read using bit flags.
-    // We must get all seven of these fields for an event to be valid.
-    const HAVE_ID: u8 = 0x1 << 0;
-    const HAVE_PUBKEY: u8 = 0x1 << 1;
-    const HAVE_SIG: u8 = 0x1 << 2;
-    const HAVE_CREATED_AT: u8 = 0x1 << 3;
-    const HAVE_KIND: u8 = 0x1 << 4;
-    const HAVE_CONTENT: u8 = 0x1 << 5;
-    const HAVE_TAGS: u8 = 0x1 << 6;
-    let mut complete: u8 = 0;
-
-    eat_whitespace(input, &mut inpos);
-    verify_char(input, b'{', &mut inpos)?;
-    loop {
-        eat_whitespace(input, &mut inpos);
-
-        // Presuming that we must have at least one field, we don't have to look
-        // for the end of the object yet.
-
-        // Move to the start of the field name
-        verify_char(input, b'"', &mut inpos)?;
-
-        // No matter which field is next, we need at least 7 bytes for the smallest
-        // field and value: kind":1
-        // This allows us to skip length tests below that are shorter than inpos+7
-        if inpos + 7 > input.len() {
-            return Err(ChorusError::JsonBadEvent("Too Short or Missing Fields", inpos).into());
-        }
-
-        if &input[inpos..inpos + 3] == b"id\"" {
-            if complete & HAVE_ID == HAVE_ID {
-                return Err(ChorusError::JsonBadEvent("Duplicate id field", inpos).into());
-            }
-            inpos += 3;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            read_id(input, &mut inpos, &mut output[16..48])?;
-            complete |= HAVE_ID;
-        } else if &input[inpos..inpos + 4] == b"sig\"" {
-            if complete & HAVE_SIG == HAVE_SIG {
-                return Err(ChorusError::JsonBadEvent("Duplicate sig field", inpos).into());
-            }
-            inpos += 4;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            read_sig(input, &mut inpos, output)?;
-            complete |= HAVE_SIG;
-        } else if &input[inpos..inpos + 5] == b"kind\"" {
-            if complete & HAVE_KIND == HAVE_KIND {
-                return Err(ChorusError::JsonBadEvent("Duplicate kind field", inpos).into());
-            }
-            inpos += 5;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            let kind = read_kind(input, &mut inpos)?;
-            output[4..6].copy_from_slice(kind.to_ne_bytes().as_slice());
-            complete |= HAVE_KIND;
-        } else if &input[inpos..inpos + 5] == b"tags\"" {
-            if complete & HAVE_TAGS == HAVE_TAGS {
-                return Err(ChorusError::JsonBadEvent("Duplicate tags field", inpos).into());
-            }
-            inpos += 5;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            tags_size = read_tags_array(input, &mut inpos, &mut output[144..])?;
-            complete |= HAVE_TAGS;
-            if content_input_start != 0 {
-                // Content was found earlier than tags.
-                // Now that tags have been read, we should read the content
-                read_content(input, &mut content_input_start, output, 144 + tags_size)?;
-                complete |= HAVE_CONTENT;
-            }
-        } else if &input[inpos..inpos + 7] == b"pubkey\"" {
-            if complete & HAVE_PUBKEY == HAVE_PUBKEY {
-                return Err(ChorusError::JsonBadEvent("Duplicate pubkey field", inpos).into());
-            }
-            inpos += 7;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            read_pubkey(input, &mut inpos, &mut output[48..80])?;
-            complete |= HAVE_PUBKEY;
-        } else if inpos + 8 <= input.len() && &input[inpos..inpos + 8] == b"content\"" {
-            if complete & HAVE_CONTENT == HAVE_CONTENT {
-                return Err(ChorusError::JsonBadEvent("Duplicate pubkey field", inpos).into());
-            }
-            inpos += 8;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            if tags_size == 0 {
-                // Oops, we haven't read the tags yet. That means we don't yet know where
-                // to place the content.  In this case we just remember the offset where
-                // this needs to be done, so we can do this later.
-                content_input_start = inpos;
-                // skip past it so we can read the subsequent fields
-                verify_char(input, b'"', &mut inpos)?;
-                burn_string(input, &mut inpos)?;
-            } else {
-                read_content(input, &mut inpos, output, 144 + tags_size)?;
-                complete |= HAVE_CONTENT;
-            }
-        } else if inpos + 11 <= input.len() && &input[inpos..inpos + 11] == b"created_at\"" {
-            if complete & HAVE_CREATED_AT == HAVE_CREATED_AT {
-                return Err(ChorusError::JsonBadEvent("Duplicate created_at field", inpos).into());
-            }
-            inpos += 11;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            let u = read_u64(input, &mut inpos)?;
-            output[8..16].copy_from_slice(u.to_ne_bytes().as_slice());
-            complete |= HAVE_CREATED_AT;
-        } else {
-            burn_key_and_value(input, &mut inpos)?;
-        }
-
-        // get past the comma, or detect the close brace and exit
-        if next_object_field(input, &mut inpos)? {
-            break;
-        }
-    }
-
-    if complete == 0b0111_1111 {
-        Ok((
-            inpos,
-            u32::from_ne_bytes(output[0..4].try_into().unwrap()) as usize,
-        ))
-    } else {
-        Err(ChorusError::JsonBadEvent("Missing Fields", inpos).into())
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::*;
-
-    #[test]
-    fn test_parse_json_event() {
-        if 256_u16.to_ne_bytes() == [1, 0] {
-            test_parse_json_event_big_endian();
-        } else {
-            test_parse_json_event_little_endian();
-        }
-    }
-
-    fn test_parse_json_event_little_endian() {
-        let json = br#"{"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"kind":1,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","content":"He got snowed in","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]]}"#;
-        let mut buffer: Vec<u8> = Vec::with_capacity(4096);
-        buffer.resize(4096, 0);
-        let (_insize, size) = parse_json_event(&json[..], &mut buffer).unwrap();
-        assert_eq!(size, 372);
-        assert_eq!(
-            &buffer[0..size],
-            &[
-                116, 1, 0, 0, // 372 bytes long
-                1, 0, // kind 1
-                0, 0, // padding
-                102, 232, 61, 100, 0, 0, 0, 0, // created at 1681778790
-                169, 102, 48, 85, 22, 74, 184, 179, 13, 149, 36, 101, 99, 112, 196, 191, 147, 57,
-                59, 176, 81, 183, 237, 244, 85, 111, 64, 197, 41, 141, 192, 199, // id
-                238, 17, 165, 223, 244, 12, 25, 165, 85, 244, 31, 228, 43, 72, 240, 14, 97, 140,
-                145, 34, 86, 34, 174, 55, 182, 194, 187, 103, 183, 108, 78, 73, // pubkey
-                77, 254, 161, 166, 247, 49, 65, 213, 105, 30, 67, 175, 195, 35, 77, 190, 115, 1,
-                109, 176, 251, 32, 124, 242, 71, 224, 18, 124, 194, 89, 30, 230, 180, 190, 91, 70,
-                34, 114, 3, 10, 155, 222, 117, 136, 42, 174, 129, 15, 53, 150, 130, 177, 182, 206,
-                108, 187, 151, 32, 17, 65, 197, 118, 219, 66, // sig
-                // 144:
-                208, 0, // tags section is 208 bytes long
-                3, 0, // there are three tags
-                10, 0, // first tag is at offset 10
-                28, 0, // second tag is at offset 28
-                99, 0, // third tag is at offset 99
-                // 154: (144+10)
-                2, 0, // the first tag has 2 strings
-                6, 0, // the first string is 6 bytes long
-                99, 108, 105, 101, 110, 116, // "client"
-                6, 0, // the second string is 6 bytes long
-                103, 111, 115, 115, 105, 112, // "gossip"
-                // 172: (144+28)
-                2, 0, // the second tag has two strings
-                1, 0,   // the first string is 1 char long
-                112, // "p"
-                64, 0, // the second string is 64 bytes long
-                101, 50, 99, 99, 102, 55, 99, 102, 50, 48, 52, 48, 51, 102, 51, 102, 50, 97, 52,
-                97, 53, 53, 98, 51, 50, 56, 102, 48, 100, 101, 51, 98, 101, 51, 56, 53, 53, 56, 97,
-                55, 100, 53, 102, 51, 51, 54, 51, 50, 102, 100, 97, 97, 101, 102, 99, 55, 50, 54,
-                99, 49, 99, 56, 101,
-                98, // "e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
-                // 243: (144+99)
-                4, 0, // the third tag has 4 strings
-                1, 0,   // the first string is 1 char long
-                101, // "e"
-                64, 0, // the second string is 64 bytes long
-                50, 99, 56, 54, 97, 98, 99, 99, 57, 56, 102, 55, 102, 100, 56, 97, 54, 55, 53, 48,
-                97, 97, 98, 56, 100, 102, 54, 99, 49, 56, 54, 51, 57, 48, 51, 102, 49, 48, 55, 50,
-                48, 54, 99, 99, 50, 100, 55, 50, 101, 56, 97, 102, 101, 98, 54, 99, 51, 56, 51, 53,
-                55, 97, 101,
-                100, // "2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"
-                30, 0, // the third string is 30 bytes long
-                119, 115, 115, 58, 47, 47, 110, 111, 115, 116, 114, 45, 112, 117, 98, 46, 119, 101,
-                108, 108, 111, 114, 100, 101, 114, 46, 110, 101, 116,
-                47, //  "wss://nostr-pub.wellorder.net/"
-                4, 0, // the fourth string is 4 bytes long
-                114, 111, 111, 116, // "root"
-                // 352: (144+208)
-                16, 0, 0, 0, // the content is 16 bytes long
-                72, 101, 32, 103, 111, 116, 32, 115, 110, 111, 119, 101, 100, 32, 105,
-                110, // "He got snowed in"
-
-                     // 372:
-            ]
-        );
-
-        // Same event in a different order
-        let json2 = br#"{"kind":1,"pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]],"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","content":"He got snowed in"}"#;
-        let mut buffer2: Vec<u8> = Vec::with_capacity(4096);
-        buffer2.resize(4096, 0);
-        let (_insize, size) = parse_json_event(&json2[..], &mut buffer2).unwrap();
-        assert_eq!(size, 372);
-        assert_eq!(&buffer[..372], &buffer2[..372]);
-    }
-
-    fn test_parse_json_event_big_endian() {
-        let json = br#"{"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"kind":1,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","content":"He got snowed in","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]]}"#;
-        let mut buffer: Vec<u8> = Vec::with_capacity(4096);
-        buffer.resize(4096, 0);
-        let (_insize, size) = parse_json_event(&json[..], &mut buffer).unwrap();
-        assert_eq!(size, 372);
-        assert_eq!(
-            &buffer[0..size],
-            &[
-                0, 0, 1, 116, // 372 bytes long
-                0, 1, // kind 1
-                0, 0, // padding
-                0, 0, 0, 0, 100, 61, 232, 102, // created at 1681778790
-                169, 102, 48, 85, 22, 74, 184, 179, 13, 149, 36, 101, 99, 112, 196, 191, 147, 57,
-                59, 176, 81, 183, 237, 244, 85, 111, 64, 197, 41, 141, 192, 199, // id
-                238, 17, 165, 223, 244, 12, 25, 165, 85, 244, 31, 228, 43, 72, 240, 14, 97, 140,
-                145, 34, 86, 34, 174, 55, 182, 194, 187, 103, 183, 108, 78, 73, // pubkey
-                77, 254, 161, 166, 247, 49, 65, 213, 105, 30, 67, 175, 195, 35, 77, 190, 115, 1,
-                109, 176, 251, 32, 124, 242, 71, 224, 18, 124, 194, 89, 30, 230, 180, 190, 91, 70,
-                34, 114, 3, 10, 155, 222, 117, 136, 42, 174, 129, 15, 53, 150, 130, 177, 182, 206,
-                108, 187, 151, 32, 17, 65, 197, 118, 219, 66, // sig
-                // 144:
-                0, 208, // tags section is 208 bytes long
-                3, 0, // there are three tags
-                0, 10, // first tag is at offset 10
-                0, 28, // second tag is at offset 28
-                0, 99, // third tag is at offset 99
-                // 154: (144+10)
-                0, 2, // the first tag has 2 strings
-                0, 6, // the first string is 6 bytes long
-                99, 108, 105, 101, 110, 116, // "client"
-                0, 6, // the second string is 6 bytes long
-                103, 111, 115, 115, 105, 112, // "gossip"
-                // 172: (144+28)
-                0, 2, // the second tag has two strings
-                0, 1,   // the first string is 1 char long
-                112, // "p"
-                0, 64, // the second string is 64 bytes long
-                101, 50, 99, 99, 102, 55, 99, 102, 50, 48, 52, 48, 51, 102, 51, 102, 50, 97, 52,
-                97, 53, 53, 98, 51, 50, 56, 102, 48, 100, 101, 51, 98, 101, 51, 56, 53, 53, 56, 97,
-                55, 100, 53, 102, 51, 51, 54, 51, 50, 102, 100, 97, 97, 101, 102, 99, 55, 50, 54,
-                99, 49, 99, 56, 101,
-                98, // "e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
-                // 243: (144+99)
-                0, 4, // the third tag has 4 strings
-                0, 1,   // the first string is 1 char long
-                101, // "e"
-                0, 64, // the second string is 64 bytes long
-                50, 99, 56, 54, 97, 98, 99, 99, 57, 56, 102, 55, 102, 100, 56, 97, 54, 55, 53, 48,
-                97, 97, 98, 56, 100, 102, 54, 99, 49, 56, 54, 51, 57, 48, 51, 102, 49, 48, 55, 50,
-                48, 54, 99, 99, 50, 100, 55, 50, 101, 56, 97, 102, 101, 98, 54, 99, 51, 56, 51, 53,
-                55, 97, 101,
-                100, // "2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"
-                0, 30, // the third string is 30 bytes long
-                119, 115, 115, 58, 47, 47, 110, 111, 115, 116, 114, 45, 112, 117, 98, 46, 119, 101,
-                108, 108, 111, 114, 100, 101, 114, 46, 110, 101, 116,
-                47, //  "wss://nostr-pub.wellorder.net/"
-                0, 4, // the fourth string is 4 bytes long
-                114, 111, 111, 116, // "root"
-                // 352: (144+208)
-                0, 0, 0, 16, // the content is 16 bytes long
-                72, 101, 32, 103, 111, 116, 32, 115, 110, 111, 119, 101, 100, 32, 105,
-                110, // "He got snowed in"
-                     // 372:
-            ]
-        );
-
-        // Same event in a different order
-        let json2 = br#"{"kind":1,"pubkey":"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49","created_at":1681778790,"sig":"4dfea1a6f73141d5691e43afc3234dbe73016db0fb207cf247e0127cc2591ee6b4be5b462272030a9bde75882aae810f359682b1b6ce6cbb97201141c576db42","tags":[["client","gossip"],["p","e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"],["e","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","wss://nostr-pub.wellorder.net/","root"]],"id":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","content":"He got snowed in"}"#;
-        let mut buffer2: Vec<u8> = Vec::with_capacity(4096);
-        buffer2.resize(4096, 0);
-        let (_insize, size) = parse_json_event(&json2[..], &mut buffer2).unwrap();
-        assert_eq!(size, 372);
-        assert_eq!(&buffer[..372], &buffer2[..372]);
-    }
-}

src/types/event/mod.rs

@@ -1,188 +0,0 @@
-use super::{Id, Kind, Pubkey, Sig, Tags, Time};
-use crate::error::{ChorusError, Error};
-use crate::types::parse::json_escape::json_escape;
-use std::fmt;
-
-mod json_event;
-use json_event::parse_json_event;
-
-/*
- * 0 [4 bytes] length of the event structure
- * 4 [2 bytes] kind
- * 6 [2 bytes] PADDING
- * 8 [8 bytes] created_at
- * 16 [32 bytes] id
- * 48 [32 bytes] pubkey
- * 80 [64 bytes] sig
- * 144 [T bytes] Tags
- * 144+T [4 bytes] content length
- * 144+T+4 [C bytes] content
- * 144+T+4+C <--- beginning of region beyond the event
- */
-
-#[derive(Debug, Clone, PartialEq)]
-pub struct Event<'a>(&'a [u8]);
-
-impl<'a> Event<'a> {
-    // Parse json into an Event. Returns the count of consumed input bytes and the Event
-    pub fn from_json(
-        json: &[u8],
-        output_buffer: &'a mut [u8],
-    ) -> Result<(usize, Event<'a>), Error> {
-        let (incount, outcount) = parse_json_event(json, output_buffer)?;
-        Ok((incount, Event(&output_buffer[..outcount])))
-    }
-
-    // this marks off the slice of bytes that represent an event from a potentially longer input
-    pub fn delineate(input: &'a [u8]) -> Result<Event<'a>, Error> {
-        if input.len() < 144 + 4 + 4 {
-            return Err(ChorusError::EndOfInput.into());
-        }
-        let len = parse_u32!(input, 0) as usize;
-        if input.len() < len {
-            return Err(ChorusError::EndOfInput.into());
-        }
-        Ok(Event(&input[0..len]))
-    }
-
-    // This copies
-    pub fn copy(&self, output: &mut [u8]) -> Result<(), Error> {
-        if output.len() < self.0.len() {
-            return Err(ChorusError::BufferTooSmall.into());
-        }
-        output[..self.0.len()].copy_from_slice(self.0);
-        Ok(())
-    }
-
-    // This copies, using the event_store mmap-append api
-    pub fn macopy(&self, output: &mut [u8]) -> Result<usize, std::io::Error> {
-        if output.len() < self.0.len() {
-            return Err(std::io::Error::other(ChorusError::BufferTooSmall));
-        }
-        output[..self.0.len()].copy_from_slice(self.0);
-        Ok(self.0.len())
-    }
-
-    pub fn as_bytes(&self) -> &[u8] {
-        self.0
-    }
-
-    pub fn length(&self) -> usize {
-        self.0.len()
-    }
-
-    pub fn kind(&self) -> Kind {
-        Kind(parse_u16!(self.0, 4))
-    }
-
-    pub fn created_at(&self) -> Time {
-        Time(parse_u64!(self.0, 8))
-    }
-
-    pub fn id(&self) -> Id {
-        Id(self.0[16..16 + 32].try_into().unwrap())
-    }
-
-    pub fn pubkey(&self) -> Pubkey {
-        Pubkey(self.0[48..48 + 32].try_into().unwrap())
-    }
-
-    pub fn sig(&self) -> Sig {
-        Sig(self.0[80..80 + 64].try_into().unwrap())
-    }
-
-    pub fn tags(&'a self) -> Result<Tags<'a>, Error> {
-        Tags::delineate(&self.0[144..])
-    }
-
-    pub fn content(&'a self) -> &'a [u8] {
-        let t = parse_u16!(self.0, 144) as usize;
-        let c = parse_u32!(self.0, 144 + t) as usize;
-        &self.0[144 + t + 4..144 + t + 4 + c]
-    }
-
-    pub fn as_json(&self) -> Result<Vec<u8>, Error> {
-        let mut output: Vec<u8> = Vec::with_capacity(256);
-        output.extend(br#"{"id":""#);
-        let pos = output.len();
-        output.resize(pos + 64, 0);
-        self.id().write_hex(&mut output[pos..]).unwrap();
-        output.extend(br#"","pubkey":""#);
-        let pos = output.len();
-        output.resize(pos + 64, 0);
-        self.pubkey().write_hex(&mut output[pos..]).unwrap();
-        output.extend(br#"","kind":"#);
-        output.extend(format!("{}", self.kind().0).as_bytes());
-        output.extend(br#","created_at":"#);
-        output.extend(format!("{}", self.created_at().0).as_bytes());
-        output.extend(br#","tags":"#);
-        output.extend(self.tags()?.as_json());
-        output.extend(br#","content":""#);
-        // This is okay if it is not accurate. It generally avoids
-        // lots of little mallocs when the capacity is already allocated
-        output.reserve(self.content().len() * 7 / 6);
-        let mut output = json_escape(self.content(), output)?;
-        output.extend(br#"","sig":""#);
-        let pos = output.len();
-        output.resize(pos + 128, 0);
-        self.sig().write_hex(&mut output[pos..]).unwrap();
-        output.extend(br#""}"#);
-        Ok(output)
-    }
-
-    pub fn verify(&self) -> Result<(), Error> {
-        use secp256k1::hashes::{sha256, Hash};
-        use secp256k1::schnorr::Signature;
-        use secp256k1::{Message, XOnlyPublicKey};
-
-        // This is okay if it is not accurate. It generally avoids
-        // lots of little mallocs when the capacity is already allocated
-        let escaped_content = Vec::with_capacity(self.content().len() * 7 / 6);
-        let escaped_content = json_escape(self.content(), escaped_content)?;
-
-        let signable = format!(
-            r#"[0,"{}",{},{},{},"{}"]"#,
-            self.pubkey(),
-            self.created_at(),
-            self.kind(),
-            self.tags()?,
-            unsafe { std::str::from_utf8_unchecked(&escaped_content[..]) },
-        );
-
-        drop(escaped_content);
-
-        let hash = sha256::Hash::hash(signable.as_bytes());
-
-        let hashref = <sha256::Hash as AsRef<[u8]>>::as_ref(&hash);
-        if hashref != self.id().as_slice() {
-            return Err(ChorusError::BadEventId.into());
-        }
-
-        let pubkey = XOnlyPublicKey::from_slice(self.pubkey().as_slice())?;
-        let sig = Signature::from_slice(self.sig().as_slice())?;
-        let message = Message::from_digest_slice(hashref)?;
-        sig.verify(&message, &pubkey)?;
-
-        Ok(())
-    }
-}
-
-impl fmt::Display for Event<'_> {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        if let Ok(bytes) = self.as_json() {
-            let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
-            write!(f, "{s}")
-        } else {
-            write!(f, "{{Corrupted Event}}")
-        }
-    }
-}
-
-#[derive(Debug, Clone)]
-pub struct OwnedEvent(pub Vec<u8>);
-
-impl OwnedEvent {
-    pub fn as_event(&self) -> Result<Event<'_>, Error> {
-        Event::delineate(&self.0)
-    }
-}

src/types/filter/json_filter.rs

@@ -1,540 +0,0 @@
-use crate::error::{ChorusError, Error};
-use crate::types::parse::json_escape::json_unescape;
-use crate::types::parse::json_parse::*;
-use crate::types::parse::put;
-
-/// Parses a JSON filter from the `input` buffer. Places the parsed filter into the `output` buffer.
-/// Returns the count of consumed bytes and output bytes
-pub fn parse_json_filter(input: &[u8], output: &mut [u8]) -> Result<(usize, usize), Error> {
-    if input.len() < 2 {
-        return Err(ChorusError::JsonBadFilter("Too short", 0).into());
-    }
-
-    // This tracks where we are currently looking in the input as we scan forward.
-    // It is short for INput POSition.
-    let mut inpos = 0;
-
-    // Remember which fields we have read using bit flags.
-    // We should only get at maximum one of each
-    const HAVE_IDS: u8 = 0x1 << 0;
-    const HAVE_AUTHORS: u8 = 0x1 << 1;
-    const HAVE_KINDS: u8 = 0x1 << 2;
-    const HAVE_LIMIT: u8 = 0x1 << 3;
-    const HAVE_SINCE: u8 = 0x1 << 4;
-    const HAVE_UNTIL: u8 = 0x1 << 5;
-    let mut found: u8 = 0;
-
-    // Remember which tags we have seen
-    // We track A-Z in the lower 26 bits, and a-z in the next 26 bits
-    let mut found_tags: u64 = 0;
-    let letter_to_tag_bit = |letter: u8| -> Option<u64> {
-        match letter {
-            65..=90 => Some(letter as u64 - 65),
-            97..=122 => Some(letter as u64 - 97 + 26),
-            _ => None,
-        }
-    };
-
-    // Start structure with that of an empty filter
-    put(
-        output,
-        0,
-        &[
-            0, 0, // length (we will fill it in later)
-            0, 0, // 0 ids
-            0, 0, // 0 authors
-            0, 0, // 0 kinds
-            255, 255, 255, 255, // max limit
-            0, 0, 0, 0, // padding
-            0, 0, 0, 0, 0, 0, 0, 0, // since 1970
-            255, 255, 255, 255, 255, 255, 255, 255, // until max unixtime
-        ],
-    )?;
-
-    let mut end: usize = 32;
-
-    // We just store the position of ids, authors, kinds, and tags
-    // and come back to parse them properly again at the end,
-    // since we need to write them in a particular order.
-    let mut start_ids: Option<usize> = None;
-    let mut start_authors: Option<usize> = None;
-    let mut start_kinds: Option<usize> = None;
-    // Allowing up to 32 tag filter fields (plenty!)
-    // (we are not differentiating letters yet, just collecting offsets)
-    // (we make the array to avoid allocation)
-    let mut num_tag_fields = 0;
-    let mut start_tags: [usize; 32] = [usize::MAX; 32];
-
-    eat_whitespace(input, &mut inpos);
-    verify_char(input, b'{', &mut inpos)?;
-    loop {
-        eat_whitespace_and_commas(input, &mut inpos);
-
-        // Check for end
-        if input[inpos] == b'}' {
-            inpos += 1;
-            break;
-        }
-
-        verify_char(input, b'"', &mut inpos)?;
-
-        if inpos + 4 <= input.len() && &input[inpos..inpos + 4] == b"ids\"" {
-            // Check for duplicate
-            if found & HAVE_IDS == HAVE_IDS {
-                return Err(ChorusError::JsonBadFilter("Duplicate id field", inpos).into());
-            }
-            inpos += 4;
-
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            verify_char(input, b'[', &mut inpos)?;
-
-            // Record for later
-            start_ids = Some(inpos);
-
-            // Burn the field
-            while inpos < input.len() && input[inpos] != b']' {
-                inpos += 1;
-            }
-            verify_char(input, b']', &mut inpos)?;
-
-            // Mark as found 'ids'  FIXME this dups `start_ids`
-            found |= HAVE_IDS;
-        } else if inpos + 8 <= input.len() && &input[inpos..inpos + 8] == b"authors\"" {
-            // Check for duplicate
-            if found & HAVE_AUTHORS == HAVE_AUTHORS {
-                return Err(ChorusError::JsonBadFilter("Duplicate authors field", inpos).into());
-            }
-            inpos += 8;
-
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            verify_char(input, b'[', &mut inpos)?;
-
-            // Save the input offset for post-processing
-            start_authors = Some(inpos);
-
-            // Burn the field
-            while inpos < input.len() && input[inpos] != b']' {
-                inpos += 1;
-            }
-            verify_char(input, b']', &mut inpos)?;
-
-            found |= HAVE_AUTHORS;
-        } else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"kinds\"" {
-            // Check for duplicate
-            if found & HAVE_KINDS == HAVE_KINDS {
-                return Err(ChorusError::JsonBadFilter("Duplicate kinds field", inpos).into());
-            }
-            inpos += 6;
-
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            verify_char(input, b'[', &mut inpos)?;
-
-            // Mark this position and bypass this field
-            start_kinds = Some(inpos);
-
-            // Burn the field
-            while inpos < input.len() && input[inpos] != b']' {
-                inpos += 1;
-            }
-            verify_char(input, b']', &mut inpos)?;
-
-            found |= HAVE_KINDS;
-        } else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"since\"" {
-            // Check for duplicate
-            if found & HAVE_SINCE == HAVE_SINCE {
-                return Err(ChorusError::JsonBadFilter("Duplicate since field", inpos).into());
-            }
-            inpos += 6;
-
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            let since = read_u64(input, &mut inpos)?;
-            put(output, 16, since.to_ne_bytes().as_slice())?;
-
-            found |= HAVE_SINCE;
-        } else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"until\"" {
-            // Check for duplicate
-            if found & HAVE_UNTIL == HAVE_UNTIL {
-                return Err(ChorusError::JsonBadFilter("Duplicate until field", inpos).into());
-            }
-            inpos += 6;
-
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            let until = read_u64(input, &mut inpos)?;
-            put(output, 24, until.to_ne_bytes().as_slice())?;
-
-            found |= HAVE_UNTIL;
-        } else if inpos + 6 <= input.len() && &input[inpos..inpos + 6] == b"limit\"" {
-            // Check for duplicate
-            if found & HAVE_LIMIT == HAVE_LIMIT {
-                return Err(ChorusError::JsonBadFilter("Duplicate limit field", inpos).into());
-            }
-            inpos += 6;
-
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            let limit = read_u64(input, &mut inpos)?;
-            let limit: u32 = limit as u32;
-            put(output, 8, limit.to_ne_bytes().as_slice())?;
-
-            found |= HAVE_LIMIT;
-        } else if inpos + 3 <= input.len()
-            && input[inpos] == b'#'
-            && ((input[inpos + 1] >= 65 && input[inpos + 1] <= 90)
-                || (input[inpos + 1] >= 97 && input[inpos + 1] <= 122))
-            && input[inpos + 2] == b'"'
-        {
-            inpos += 1; // pass the hash
-
-            // Mark this position (on the letter itself)
-            start_tags[num_tag_fields] = inpos;
-            num_tag_fields += 1;
-
-            let letter = input[inpos];
-            inpos += 2; // pass the letter and quote
-
-            // Remember we found this tag in the `found_tags` bitfield
-            if let Some(bit) = letter_to_tag_bit(letter) {
-                if found_tags & bit == bit {
-                    return Err(ChorusError::JsonBadFilter("Duplicate tag", inpos).into());
-                }
-                found_tags |= bit;
-            }
-
-            // Burn the rest
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            verify_char(input, b'[', &mut inpos)?;
-            burn_array(input, &mut inpos)?;
-        } else {
-            burn_key_and_value(input, &mut inpos)?;
-        }
-    }
-
-    // Copy ids
-    if let Some(mut inpos) = start_ids {
-        let mut num_ids: u16 = 0;
-        // `inpos` is right after the open bracket of the array
-        loop {
-            eat_whitespace_and_commas(input, &mut inpos);
-            if input[inpos] == b']' {
-                break;
-            }
-            read_id(input, &mut inpos, &mut output[end..])?;
-            num_ids += 1;
-            end += 32;
-        }
-
-        // Write num_ids
-        put(output, 2, num_ids.to_ne_bytes().as_slice())?;
-    }
-
-    // Copy authors
-    if let Some(mut inpos) = start_authors {
-        let mut num_authors: u16 = 0;
-        // `inpos` is right after the open bracket of the array
-        loop {
-            eat_whitespace_and_commas(input, &mut inpos);
-            if input[inpos] == b']' {
-                break;
-            }
-            read_pubkey(input, &mut inpos, &mut output[end..])?;
-            num_authors += 1;
-            end += 32;
-        }
-
-        // write num_authors
-        put(output, 4, num_authors.to_ne_bytes().as_slice())?;
-    }
-
-    // Copy kinds
-    if let Some(mut inpos) = start_kinds {
-        let mut num_kinds: u16 = 0;
-        // `inpos` is right after the open bracket of the array
-        loop {
-            eat_whitespace_and_commas(input, &mut inpos);
-            if input[inpos] == b']' {
-                break;
-            }
-            let u = read_u64(input, &mut inpos)?;
-            if u > 65535 {
-                return Err(
-                    ChorusError::JsonBadFilter("Filter has kind number too large", inpos).into(),
-                );
-            }
-            put(output, end, (u as u16).to_ne_bytes().as_slice())?;
-            num_kinds += 1;
-            end += 2;
-        }
-
-        // write num_kinds
-        put(output, 6, num_kinds.to_ne_bytes().as_slice())?;
-    }
-
-    // Copy tags
-    {
-        let write_tags_start = end;
-        // write number of tags
-        put(
-            output,
-            write_tags_start + 2,
-            (num_tag_fields as u16).to_ne_bytes().as_slice(),
-        )?;
-        // bump end past offset fields
-        end += 4 + 2 * num_tag_fields;
-        // Now pull in each tag
-        #[allow(clippy::needless_range_loop)]
-        for w in 0..num_tag_fields {
-            // Write it's offset
-            put(
-                output,
-                write_tags_start + 4 + (2 * w),
-                ((end - write_tags_start) as u16).to_ne_bytes().as_slice(),
-            )?;
-
-            let mut inpos = start_tags[w];
-            let letter = input[inpos];
-
-            // bump past count output and write letter
-            let countindex = end;
-            end += 2;
-            put(output, end, 1_u16.to_ne_bytes().as_slice())?;
-            if output.len() < end + 2 {
-                return Err(crate::error::ChorusError::BufferTooSmall.into());
-            }
-            output[end + 2] = letter;
-
-            // bump past what we just wrote
-            end += 3;
-
-            // scan further in input
-            inpos += 1; // move off letter
-            verify_char(input, b'"', &mut inpos)?;
-            eat_colon_with_whitespace(input, &mut inpos)?;
-            verify_char(input, b'[', &mut inpos)?;
-
-            let mut count: u16 = 1; // the tag letter itself counts
-            loop {
-                eat_whitespace_and_commas(input, &mut inpos);
-                if input[inpos] == b']' {
-                    break;
-                }
-                verify_char(input, b'"', &mut inpos)?;
-                // copy  data
-                let (inlen, outlen) = json_unescape(&input[inpos..], &mut output[end + 2..])?;
-                // write len
-                put(output, end, (outlen as u16).to_ne_bytes().as_slice())?;
-                end += 2 + outlen;
-                inpos += inlen + 1;
-                count += 1;
-            }
-
-            // write count
-            put(output, countindex, count.to_ne_bytes().as_slice())?;
-        }
-        // write length of tags section
-        put(
-            output,
-            write_tags_start,
-            ((end - write_tags_start) as u16).to_ne_bytes().as_slice(),
-        )?;
-    }
-
-    if end > 65535 {
-        return Err(ChorusError::JsonBadFilter("Filter is too long", end).into());
-    }
-
-    // Write length of filter
-    put(output, 0, (end as u16).to_ne_bytes().as_slice())?;
-
-    Ok((inpos, end))
-}
-
-#[cfg(test)]
-mod test {
-    use super::*;
-    use crate::types::{Filter, Kind, Pubkey, Tags, TagsIter, TagsStringIter, Time};
-
-    #[test]
-    fn test_parse_json_empty_filter() {
-        let json = br##"{}"##;
-        let mut buffer: Vec<u8> = Vec::with_capacity(4096);
-        buffer.resize(4096, 0);
-        let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
-        assert_eq!(consumed, 2);
-        assert_eq!(size, 36);
-        assert_eq!(
-            &buffer[0..size],
-            &[
-                36, 0, // length
-                0, 0, // 0 ids
-                0, 0, // 0 authors
-                0, 0, // 0 kinds
-                255, 255, 255, 255, // max limit
-                0, 0, 0, 0, // padding
-                0, 0, 0, 0, 0, 0, 0, 0, // since 1970
-                255, 255, 255, 255, 255, 255, 255, 255, // until max unixtime
-                4, 0, 0, 0, // empty tags section
-            ]
-        );
-    }
-
-    #[test]
-    fn test_parse_json_filter1() {
-        let json = br##"{"kinds":[1,30023],"since":1681778790,"authors":["e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"],"until":1704238196,"ids" : [ "7089afc2e77f366bc0fd1662e4048f59f18391c04a35957f21bbd1f3e6a492c4"],"limit":10}"##;
-        // ,"#e":"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7"}"##;
-        let mut buffer: Vec<u8> = Vec::with_capacity(4096);
-        buffer.resize(4096, 0);
-        let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
-        assert_eq!(consumed, json.len());
-        assert_eq!(size, 136);
-        assert_eq!(
-            &buffer[0..size],
-            &[
-                136, 0, // length
-                1, 0, // 0 ids
-                2, 0, // 0 authors
-                2, 0, // 0 kinds
-                10, 0, 0, 0, // max limit 10
-                0, 0, 0, 0, // padding
-                102, 232, 61, 100, 0, 0, 0, 0, // since 1681778790
-                116, 156, 148, 101, 0, 0, 0, 0, // until 1704238196
-                // First ID:
-                112, 137, 175, 194, 231, 127, 54, 107, 192, 253, 22, 98, 228, 4, 143, 89, 241, 131,
-                145, 192, 74, 53, 149, 127, 33, 187, 209, 243, 230, 164, 146, 196,
-                // First author:
-                226, 204, 247, 207, 32, 64, 63, 63, 42, 74, 85, 179, 40, 240, 222, 59, 227, 133, 88,
-                167, 213, 243, 54, 50, 253, 170, 239, 199, 38, 193, 200, 235,
-                // Second author:
-                44, 134, 171, 204, 152, 247, 253, 138, 103, 80, 170, 184, 223, 108, 24, 99, 144, 63,
-                16, 114, 6, 204, 45, 114, 232, 175, 235, 108, 56, 53, 122, 237, // Kinds,
-                1, 0, // 1
-                71, 117, // 30023
-                4, 0, 0, 0, // empty tags section
-            ]
-        );
-    }
-
-    #[test]
-    fn test_parse_json_filter2() {
-        let json = br##"{"kinds":[1,30023],"since":1681778790,"authors":["e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"],"until":1704238196,"ids" : [ "7089afc2e77f366bc0fd1662e4048f59f18391c04a35957f21bbd1f3e6a492c4"],"limit":10, "#e":["a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7"]}"##;
-        let mut buffer: Vec<u8> = Vec::with_capacity(4096);
-        buffer.resize(4096, 0);
-        let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
-        assert_eq!(consumed, json.len());
-        assert_eq!(size, 209);
-        assert_eq!(
-            &buffer[0..size],
-            &[
-                209, 0, // length
-                1, 0, // 0 ids
-                2, 0, // 0 authors
-                2, 0, // 0 kinds
-                10, 0, 0, 0, // max limit 10
-                0, 0, 0, 0, // padding
-                102, 232, 61, 100, 0, 0, 0, 0, // since 1681778790
-                116, 156, 148, 101, 0, 0, 0, 0, // until 1704238196
-                // First ID:
-                112, 137, 175, 194, 231, 127, 54, 107, 192, 253, 22, 98, 228, 4, 143, 89, 241, 131,
-                145, 192, 74, 53, 149, 127, 33, 187, 209, 243, 230, 164, 146, 196,
-                // First author:
-                226, 204, 247, 207, 32, 64, 63, 63, 42, 74, 85, 179, 40, 240, 222, 59, 227, 133, 88,
-                167, 213, 243, 54, 50, 253, 170, 239, 199, 38, 193, 200, 235,
-                // Second author:
-                44, 134, 171, 204, 152, 247, 253, 138, 103, 80, 170, 184, 223, 108, 24, 99, 144, 63,
-                16, 114, 6, 204, 45, 114, 232, 175, 235, 108, 56, 53, 122, 237, // Kinds,
-                1, 0, // 1
-                71, 117, // 30023
-                // Tag section:
-                77, 0, // tags section length is 77
-                1, 0, // just one tag
-                6, 0, // offset of 0th tag is 6
-                // First tag:
-                2, 0, // 2 fields
-                // Field 1:
-                1, 0,   // 1 byte long
-                101, // 'e'
-                // Field 2:
-                64, 0, // 64 bytes long
-                97, 57, 54, 54, 51, 48, 53, 53, 49, 54, 52, 97, 98, 56, 98, 51, 48, 100, 57, 53,
-                50, 52, 54, 53, 54, 51, 55, 48, 99, 52, 98, 102, 57, 51, 51, 57, 51, 98, 98, 48,
-                53, 49, 98, 55, 101, 100, 102, 52, 53, 53, 54, 102, 52, 48, 99, 53, 50, 57, 56,
-                100, 99, 48, 99, 55
-            ]
-        );
-    }
-
-    #[test]
-    fn test_filter_parse_and_check() {
-        let json = br##"{"kinds":[1,5,9,30023],"since":1681778790,"authors":["e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"], "#e":["a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"],"#p":["2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed","2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"]}"##;
-        let mut buffer: Vec<u8> = Vec::with_capacity(4096);
-        buffer.resize(4096, 0);
-        let (consumed, size) = parse_json_filter(&json[..], &mut buffer).unwrap();
-        assert_eq!(consumed, json.len());
-        assert_eq!(size, 452);
-        let filter = Filter::delineate(&buffer).unwrap();
-        assert_eq!(filter.len(), 452);
-        assert_eq!(filter.num_ids(), 0);
-
-        assert_eq!(filter.num_authors(), 2);
-        let mut author_iter = filter.authors();
-        assert_eq!(
-            author_iter.next(),
-            Some(
-                Pubkey::read_hex(
-                    b"e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
-                )
-                .unwrap()
-            )
-        );
-        assert_eq!(
-            author_iter.next(),
-            Some(
-                Pubkey::read_hex(
-                    b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed"
-                )
-                .unwrap()
-            )
-        );
-        assert_eq!(author_iter.next(), None);
-
-        assert_eq!(filter.num_kinds(), 4);
-        let mut kind_iter = filter.kinds();
-        assert_eq!(kind_iter.next(), Some(Kind(1)));
-        assert_eq!(kind_iter.next(), Some(Kind(5)));
-        assert_eq!(kind_iter.next(), Some(Kind(9)));
-        assert_eq!(kind_iter.next(), Some(Kind(30023)));
-        assert_eq!(kind_iter.next(), None);
-
-        assert_eq!(filter.limit(), u32::MAX);
-        assert_eq!(filter.since(), Time(1681778790));
-        assert_eq!(filter.until(), Time::max());
-
-        let tags: Tags = filter.tags().unwrap();
-        let mut tag_iter: TagsIter = tags.iter();
-        let mut tag1_iter: TagsStringIter = tag_iter.next().unwrap();
-        assert_eq!(tag1_iter.next(), Some(b"e".as_slice()));
-        assert_eq!(
-            tag1_iter.next(),
-            Some(b"a9663055164ab8b30d9524656370c4bf93393bb051b7edf4556f40c5298dc0c7".as_slice())
-        );
-        assert_eq!(
-            tag1_iter.next(),
-            Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
-        );
-        assert_eq!(tag1_iter.next(), None);
-        let mut tag2_iter = tag_iter.next().unwrap();
-        assert_eq!(tag2_iter.next(), Some(b"p".as_slice()));
-        assert_eq!(
-            tag2_iter.next(),
-            Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
-        );
-        assert_eq!(
-            tag2_iter.next(),
-            Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
-        );
-        assert_eq!(
-            tag2_iter.next(),
-            Some(b"2c86abcc98f7fd8a6750aab8df6c1863903f107206cc2d72e8afeb6c38357aed".as_slice())
-        );
-        assert_eq!(tag2_iter.next(), None);
-        assert!(tag_iter.next().is_none());
-    }
-}

src/types/filter/mod.rs

@@ -1,537 +0,0 @@
-use super::{Event, Id, Kind, Pubkey, Tags, Time};
-use crate::error::{ChorusError, Error};
-use std::fmt;
-
-mod json_filter;
-use json_filter::parse_json_filter;
-
-/*
- *  0 [2 bytes] length of entire structure
- *  2 [2 bytes] num_ids
- *  4 [2 bytes] num_authors
- *  6 [2 bytes] num_kinds
- *  8 [4 bytes] limit				u32.   Set to u32::max if limit was not set.
- *  12 [4 bytes] PADDING
- *  16 [8 bytes] since				u64.   Set to 0 if since was not set.
- *  24 [8 bytes] until				u64.   Set to u64::max if until was not set.
- *  32 [ID] array
- *  [Pubkey] array                  starts at 32 + num_ids*32
- *  [Kind] array                    starts at 32 + num_ids*32 + num_authors*32
- *  [Tags] object                   starts at 32 + num_ids*32 + num_authors*32 * num_kinds*2
- */
-
-const NUM_IDS_OFFSET: usize = 2;
-const NUM_AUTHORS_OFFSET: usize = 4;
-const NUM_KINDS_OFFSET: usize = 6;
-const LIMIT_OFFSET: usize = 8;
-const SINCE_OFFSET: usize = 16;
-const UNTIL_OFFSET: usize = 24;
-const ARRAYS_OFFSET: usize = 32;
-const ID_SIZE: usize = 32;
-const PUBKEY_SIZE: usize = 32;
-const KIND_SIZE: usize = 2;
-
-#[derive(Debug, Clone)]
-pub struct Filter<'a>(&'a [u8]);
-
-impl<'a> Filter<'a> {
-    // Parse json into a Filter. Returns the count of consumed input bytes and output
-    // bytes the Filter
-    pub fn from_json(
-        json: &[u8],
-        output_buffer: &'a mut [u8],
-    ) -> Result<(usize, usize, Filter<'a>), Error> {
-        let (incount, outcount) = parse_json_filter(json, output_buffer)?;
-        Ok((incount, outcount, Filter(&output_buffer[..outcount])))
-    }
-
-    pub fn delineate(input: &'a [u8]) -> Result<Filter<'a>, Error> {
-        if input.len() < ARRAYS_OFFSET {
-            return Err(ChorusError::EndOfInput.into());
-        }
-        let len = parse_u16!(input, 0) as usize;
-        if input.len() < len {
-            return Err(ChorusError::EndOfInput.into());
-        }
-        Ok(Filter(&input[0..len]))
-    }
-
-    pub fn copy(&self, output: &mut [u8]) -> Result<(), Error> {
-        if output.len() < self.0.len() {
-            return Err(ChorusError::EndOfInput.into());
-        }
-        output[..self.0.len()].copy_from_slice(self.0);
-        Ok(())
-    }
-
-    #[inline]
-    pub fn as_bytes(&self) -> &[u8] {
-        self.0
-    }
-
-    #[inline]
-    pub fn len(&self) -> usize {
-        self.0.len()
-    }
-
-    #[inline]
-    pub fn is_empty(&self) -> bool {
-        self.0.is_empty()
-    }
-
-    #[inline]
-    pub fn num_ids(&self) -> usize {
-        parse_u16!(self.0, NUM_IDS_OFFSET) as usize
-    }
-
-    #[inline]
-    pub fn ids(&'a self) -> FilterIdIter<'a> {
-        FilterIdIter {
-            filter: self,
-            next: 0,
-        }
-    }
-
-    #[inline]
-    pub fn num_authors(&self) -> usize {
-        parse_u16!(self.0, NUM_AUTHORS_OFFSET) as usize
-    }
-
-    #[inline]
-    fn start_of_authors(&self) -> usize {
-        ARRAYS_OFFSET + self.num_ids() * ID_SIZE
-    }
-
-    #[inline]
-    pub fn authors(&'a self) -> FilterAuthorIter<'a> {
-        FilterAuthorIter {
-            filter: self,
-            start_of_authors: self.start_of_authors(),
-            next: 0,
-        }
-    }
-
-    #[inline]
-    pub fn num_kinds(&self) -> usize {
-        parse_u16!(self.0, NUM_KINDS_OFFSET) as usize
-    }
-
-    #[inline]
-    fn start_of_kinds(&self) -> usize {
-        ARRAYS_OFFSET + self.num_ids() * ID_SIZE + self.num_authors() * PUBKEY_SIZE
-    }
-
-    #[inline]
-    pub fn kinds(&'a self) -> FilterKindIter<'a> {
-        FilterKindIter {
-            filter: self,
-            start_of_kinds: self.start_of_kinds(),
-            next: 0,
-        }
-    }
-
-    #[inline]
-    fn start_of_tags(&self) -> usize {
-        ARRAYS_OFFSET
-            + self.num_ids() * ID_SIZE
-            + self.num_authors() * PUBKEY_SIZE
-            + self.num_kinds() * KIND_SIZE
-    }
-
-    #[inline]
-    pub fn tags(&'a self) -> Result<Tags<'a>, Error> {
-        Tags::delineate(&self.0[self.start_of_tags()..])
-    }
-
-    #[inline]
-    pub fn limit(&self) -> u32 {
-        parse_u32!(self.0, LIMIT_OFFSET)
-    }
-
-    #[inline]
-    pub fn since(&self) -> Time {
-        Time(parse_u64!(self.0, SINCE_OFFSET))
-    }
-
-    #[inline]
-    pub fn until(&self) -> Time {
-        Time(parse_u64!(self.0, UNTIL_OFFSET))
-    }
-
-    pub fn event_matches(&self, event: &Event) -> Result<bool, Error> {
-        // ids
-        if self.num_ids() != 0 && !self.ids().any(|id| id == event.id()) {
-            return Ok(false);
-        }
-
-        // authors
-        if self.num_authors() != 0 && !self.authors().any(|pk| pk == event.pubkey()) {
-            return Ok(false);
-        }
-
-        // kinds
-        if self.num_kinds() != 0 && !self.kinds().any(|kind| kind == event.kind()) {
-            return Ok(false);
-        }
-
-        // since
-        if event.created_at() < self.since() {
-            return Ok(false);
-        }
-
-        // until
-        if event.created_at() > self.until() {
-            return Ok(false);
-        }
-
-        // tags
-        let filter_tags = self.tags()?;
-        if !filter_tags.is_empty() {
-            let event_tags = event.tags()?;
-            if event_tags.is_empty() {
-                return Ok(false);
-            }
-
-            let mut i = 0;
-            while let Some(letter) = filter_tags.get_string(i, 0) {
-                let mut j = 1;
-                let mut found = false;
-                while let Some(value) = filter_tags.get_string(i, j) {
-                    if event_tags.matches(letter, value) {
-                        found = true;
-                        break;
-                    }
-                    j += 1;
-                }
-                if !found {
-                    return Ok(false);
-                }
-                i += 1;
-            }
-        }
-
-        Ok(true)
-    }
-
-    pub fn as_json(&self) -> Result<Vec<u8>, Error> {
-        let mut output: Vec<u8> = Vec::with_capacity(256);
-        output.push(b'{');
-        let mut first = true;
-
-        if self.num_ids() > 0 {
-            output.extend(br#""ids":["#);
-            for (i, id) in self.ids().enumerate() {
-                if i > 0 {
-                    output.push(b',');
-                }
-                output.push(b'"');
-                let pos = output.len();
-                output.resize(pos + 64, 0);
-                id.write_hex(&mut output[pos..])?;
-                output.push(b'"');
-            }
-            output.push(b']');
-            first = false;
-        }
-
-        if self.num_authors() > 0 {
-            if !first {
-                output.push(b',');
-            }
-            output.extend(br#""authors":["#);
-            for (i, pk) in self.authors().enumerate() {
-                if i > 0 {
-                    output.push(b',');
-                }
-                output.push(b'"');
-                let pos = output.len();
-                output.resize(pos + 64, 0);
-                pk.write_hex(&mut output[pos..])?;
-                output.push(b'"');
-            }
-            output.push(b']');
-            first = false;
-        }
-
-        if self.num_kinds() > 0 {
-            if !first {
-                output.push(b',');
-            }
-            output.extend(br#""kinds":["#);
-            for (i, k) in self.kinds().enumerate() {
-                if i > 0 {
-                    output.push(b',');
-                }
-                output.extend(format!("{}", k.0).as_bytes());
-            }
-            output.push(b']');
-            first = false;
-        }
-
-        let tags = self.tags()?;
-        if !tags.is_empty() {
-            // Filter 'tags' are not an array of arrays, they are just a convenient
-            // way to store similar data. They also elide the '#'. So we have to
-            // iterate here, we cannot use tags.as_json()
-            for tag in tags.iter() {
-                if !first {
-                    output.push(b',');
-                }
-                for (i, bytes) in tag.enumerate() {
-                    if i == 0 {
-                        output.extend(b"\"#");
-                        output.extend(bytes);
-                        output.extend(b"\":[");
-                    } else {
-                        if i > 1 {
-                            output.push(b',');
-                        }
-                        output.push(b'"');
-                        output.extend(bytes);
-                        output.push(b'"');
-                    }
-                }
-                output.push(b']');
-                first = false;
-            }
-        }
-
-        if self.limit() != u32::MAX {
-            if !first {
-                output.push(b',');
-            }
-            output.extend(format!(r#""limit":{}"#, self.limit()).as_bytes());
-            first = false;
-        }
-
-        if self.since() != Time::min() {
-            if !first {
-                output.push(b',');
-            }
-            output.extend(format!(r#""since":{}"#, self.since().0).as_bytes());
-            first = false;
-        }
-
-        if self.until() != Time::max() {
-            if !first {
-                output.push(b',');
-            }
-            output.extend(format!(r#""until":{}"#, self.until().0).as_bytes());
-        }
-
-        output.push(b'}');
-
-        Ok(output)
-    }
-}
-
-impl fmt::Display for Filter<'_> {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        if let Ok(bytes) = self.as_json() {
-            let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
-            write!(f, "{s}")
-        } else {
-            write!(f, "{{Corrupted Event}}")
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct FilterIdIter<'a> {
-    filter: &'a Filter<'a>,
-    next: usize,
-}
-
-impl<'a> Iterator for FilterIdIter<'a> {
-    type Item = Id;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let num_ids = parse_u16!(self.filter.0, NUM_IDS_OFFSET) as usize;
-        if self.next >= num_ids {
-            None
-        } else {
-            let offset = ARRAYS_OFFSET + self.next * ID_SIZE;
-            self.next += 1;
-            if self.filter.0.len() < offset + ID_SIZE {
-                None
-            } else {
-                Some(Id(self.filter.0[offset..offset + ID_SIZE]
-                    .try_into()
-                    .unwrap()))
-            }
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct FilterAuthorIter<'a> {
-    filter: &'a Filter<'a>,
-    start_of_authors: usize,
-    next: usize,
-}
-
-impl<'a> Iterator for FilterAuthorIter<'a> {
-    type Item = Pubkey;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let num_authors = parse_u16!(self.filter.0, NUM_AUTHORS_OFFSET) as usize;
-        if self.next >= num_authors {
-            None
-        } else {
-            let offset = self.start_of_authors + self.next * PUBKEY_SIZE;
-            self.next += 1;
-            if self.filter.0.len() < offset + PUBKEY_SIZE {
-                None
-            } else {
-                Some(Pubkey(
-                    self.filter.0[offset..offset + PUBKEY_SIZE]
-                        .try_into()
-                        .unwrap(),
-                ))
-            }
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct FilterKindIter<'a> {
-    filter: &'a Filter<'a>,
-    start_of_kinds: usize,
-    next: usize,
-}
-
-impl<'a> Iterator for FilterKindIter<'a> {
-    type Item = Kind;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let num_kinds = parse_u16!(self.filter.0, NUM_KINDS_OFFSET) as usize;
-        if self.next >= num_kinds {
-            None
-        } else {
-            let offset = self.start_of_kinds + self.next * KIND_SIZE;
-            self.next += 1;
-            if self.filter.0.len() < offset + KIND_SIZE {
-                None
-            } else {
-                Some(Kind(parse_u16!(self.filter.0, offset)))
-            }
-        }
-    }
-}
-
-#[derive(Debug, Clone)]
-pub struct OwnedFilter(pub Vec<u8>);
-
-impl OwnedFilter {
-    pub fn as_filter(&self) -> Result<Filter<'_>, Error> {
-        Filter::delineate(&self.0)
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::*;
-
-    #[test]
-    fn test_filter() {
-        /*
-         * {
-         *   "ids": [ "6b43bc2e373b6d9330ff571f3f4e6d897b32d01d65227df3fa41cdf731c63c3a",
-         *            "1f47034c9d6d0539382a86ba31766f00f2b8312ab167c036729422ec9e7085e8"],
-         *   "authors": [ "52b4a076bcbbbdc3a1aefa3735816cf74993b1b8db202b01c883c58be7fad8bd",
-         *                "ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49" ],
-         *   "kinds": [ 1, 5, 30023 ],
-         *   "since": 1702161345,
-         *   "#p": [ "fa984bd7dbb282f07e16e7ae87b26a2a7b9b90b7246a44771f0cf5ae58018f52" ],
-         * }
-         */
-
-        // For comparison
-        let id1 = Id::read_hex(b"6b43bc2e373b6d9330ff571f3f4e6d897b32d01d65227df3fa41cdf731c63c3a")
-            .unwrap();
-
-        let id2 = Id::read_hex(b"1f47034c9d6d0539382a86ba31766f00f2b8312ab167c036729422ec9e7085e8")
-            .unwrap();
-        let pk1 =
-            Pubkey::read_hex(b"52b4a076bcbbbdc3a1aefa3735816cf74993b1b8db202b01c883c58be7fad8bd")
-                .unwrap();
-        let pk2 =
-            Pubkey::read_hex(b"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49")
-                .unwrap();
-        let tagged =
-            Pubkey::read_hex(b"fa984bd7dbb282f07e16e7ae87b26a2a7b9b90b7246a44771f0cf5ae58018f52")
-                .unwrap();
-
-        let data: Vec<u8> = vec![
-            211, 0, // length of structure
-            2, 0, // number of IDs
-            2, 0, // number of authors
-            3, 0, // number of kinds
-            255, 255, 255, 255, // limit
-            0, 0, 0, 0, // padding
-            0xC1, 0xEB, 0x74, 0x65, 0, 0, 0, 0, // since
-            255, 255, 255, 255, 255, 255, 255, 255, // until
-            0x6b, 0x43, 0xbc, 0x2e, 0x37, 0x3b, 0x6d, 0x93, 0x30, 0xff, 0x57, 0x1f, 0x3f, 0x4e,
-            0x6d, 0x89, 0x7b, 0x32, 0xd0, 0x1d, 0x65, 0x22, 0x7d, 0xf3, 0xfa, 0x41, 0xcd, 0xf7,
-            0x31, 0xc6, 0x3c, 0x3a, // ID 1
-            0x1f, 0x47, 0x03, 0x4c, 0x9d, 0x6d, 0x05, 0x39, 0x38, 0x2a, 0x86, 0xba, 0x31, 0x76,
-            0x6f, 0x00, 0xf2, 0xb8, 0x31, 0x2a, 0xb1, 0x67, 0xc0, 0x36, 0x72, 0x94, 0x22, 0xec,
-            0x9e, 0x70, 0x85, 0xe8, // ID 2
-            0x52, 0xb4, 0xa0, 0x76, 0xbc, 0xbb, 0xbd, 0xc3, 0xa1, 0xae, 0xfa, 0x37, 0x35, 0x81,
-            0x6c, 0xf7, 0x49, 0x93, 0xb1, 0xb8, 0xdb, 0x20, 0x2b, 0x01, 0xc8, 0x83, 0xc5, 0x8b,
-            0xe7, 0xfa, 0xd8, 0xbd, // Pubkey 1
-            0xee, 0x11, 0xa5, 0xdf, 0xf4, 0x0c, 0x19, 0xa5, 0x55, 0xf4, 0x1f, 0xe4, 0x2b, 0x48,
-            0xf0, 0x0e, 0x61, 0x8c, 0x91, 0x22, 0x56, 0x22, 0xae, 0x37, 0xb6, 0xc2, 0xbb, 0x67,
-            0xb7, 0x6c, 0x4e, 0x49, // Pubkey 2
-            1, 0, 5, 0, 71, 117, // 3 kinds
-            // Tags
-            45, 0, // tags_len
-            1, 0, // num_tags
-            6, 0, // first tag offset at 6
-            2, 0, // 2 fields long
-            1, 0,   // 1st field is 1 byte
-            112, // "p"
-            32, 0, // 2nd field is 32 bytes
-            // 2nd field
-            0xfa, 0x98, 0x4b, 0xd7, 0xdb, 0xb2, 0x82, 0xf0, 0x7e, 0x16, 0xe7, 0xae, 0x87, 0xb2,
-            0x6a, 0x2a, 0x7b, 0x9b, 0x90, 0xb7, 0x24, 0x6a, 0x44, 0x77, 0x1f, 0x0c, 0xf5, 0xae,
-            0x58, 0x01, 0x8f, 0x52,
-        ];
-
-        let filter = Filter::delineate(&data).unwrap();
-
-        assert_eq!(filter.num_ids(), 2);
-        let mut ids = filter.ids();
-        assert_eq!(ids.next().unwrap(), id1);
-        assert_eq!(ids.next().unwrap(), id2);
-        assert!(ids.next().is_none());
-
-        assert_eq!(filter.num_authors(), 2);
-        let mut authors = filter.authors();
-        assert_eq!(authors.next().unwrap(), pk1);
-        assert_eq!(authors.next().unwrap(), pk2);
-        assert!(authors.next().is_none());
-
-        assert_eq!(filter.num_kinds(), 3);
-        let mut kinds = filter.kinds();
-        assert_eq!(kinds.next().unwrap(), Kind(1));
-        assert_eq!(kinds.next().unwrap(), Kind(5));
-        assert_eq!(kinds.next().unwrap(), Kind(30023));
-        assert!(kinds.next().is_none());
-
-        assert_eq!(filter.limit(), u32::MAX);
-        assert_eq!(filter.since(), Time(1702161345));
-        assert_eq!(filter.until(), Time::max());
-
-        let tags = filter.tags().unwrap();
-        assert_eq!(tags.len(), 1);
-        let mut iter = tags.iter();
-        let mut tag = iter.next().unwrap();
-        assert!(iter.next().is_none());
-        assert_eq!(tag.next().unwrap(), b"p");
-        let p_bytes = tag.next().unwrap();
-        assert!(tag.next().is_none());
-        let pk = Pubkey(p_bytes.try_into().unwrap());
-        assert_eq!(pk, tagged);
-    }
-}

src/types/id.rs

@@ -1,47 +0,0 @@
-use crate::error::Error;
-use serde::{Deserialize, Serialize};
-use std::fmt;
-
-#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
-pub struct Id(pub [u8; 32]);
-
-impl Id {
-    pub fn write_hex(&self, output: &mut [u8]) -> Result<(), Error> {
-        write_hex!(self.0, output, 32)
-    }
-
-    pub fn read_hex(input: &[u8]) -> Result<Id, Error> {
-        let mut out: [u8; 32] = [0; 32];
-        read_hex!(input, &mut out, 32)?;
-        Ok(Id(out))
-    }
-
-    pub fn as_slice(&self) -> &[u8] {
-        self.0.as_slice()
-    }
-}
-
-impl fmt::Display for Id {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        let mut bytes: [u8; 64] = [0; 64];
-        self.write_hex(&mut bytes).unwrap();
-        let hex = unsafe { std::str::from_utf8_unchecked(&bytes) };
-        write!(f, "{hex}")
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::Id;
-
-    #[test]
-    fn test_id_hex_functions() {
-        let hex = b"1110ee4ff957fa9c55832eaccb4dc1c45bfc6304e1e4e9fa478f53df4b20062d";
-        let id = Id::read_hex(hex).unwrap();
-        eprintln!("{:?}", id);
-        let mut hex2: [u8; 64] = [0; 64];
-        id.write_hex(&mut hex2).unwrap();
-        assert_eq!(hex, &hex2);
-        assert_eq!(format!("{}", id).as_bytes(), hex);
-    }
-}

src/types/kind.rs

@@ -1,28 +0,0 @@
-use std::fmt;
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
-pub struct Kind(pub u16);
-
-impl fmt::Display for Kind {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "{}", self.0)
-    }
-}
-
-impl Kind {
-    // Some kinds a relay may need to treat differently
-    pub const SEAL: u16 = 13;
-    pub const DM_CHAT: u16 = 14;
-
-    pub fn is_replaceable(&self) -> bool {
-        (10000..20000).contains(&self.0) || self.0 == 0 || self.0 == 3
-    }
-
-    pub fn is_ephemeral(&self) -> bool {
-        (20000..30000).contains(&self.0)
-    }
-
-    pub fn is_parameterized_replaceable(&self) -> bool {
-        (30000..40000).contains(&self.0)
-    }
-}

src/types/mod.rs

@@ -1,25 +0,0 @@
-mod event;
-pub use event::{Event, OwnedEvent};
-
-mod filter;
-pub use filter::{Filter, OwnedFilter};
-
-mod id;
-pub use id::Id;
-
-mod kind;
-pub use kind::Kind;
-
-pub mod parse;
-
-mod pubkey;
-pub use pubkey::Pubkey;
-
-mod sig;
-pub use sig::Sig;
-
-mod tags;
-pub use tags::{Tags, TagsIter, TagsStringIter};
-
-mod time;
-pub use time::Time;

src/types/parse/json_escape.rs

@@ -1,254 +0,0 @@
-use super::utf8::{encode_utf8, next_code_point};
-use crate::error::{ChorusError, Error};
-
-// LITERAL UNESCAPED:   0x20-0x21, 0x23-0x5B, 0x5D-10FFFF
-// ESCAPES:     \"  \\  \/  /b  /f  /n  /r  /t
-// UTF ESCAPE:  \uXXXX or \uXXXX\uXXXX
-
-// This escapes a string with JSON escapes. It takes an output buffer,
-// and returns it filled with the escaped string.
-// For performance you should try to pass in a buffer that is already
-// allocated big enough.
-pub fn json_escape(input: &[u8], mut out: Vec<u8>) -> Result<Vec<u8>, Error> {
-    let mut read_pos: usize = 0;
-    while let Some((codepoint, size)) = next_code_point(&input[read_pos..])? {
-        if is_safe_char(codepoint) {
-            out.extend(&input[read_pos..read_pos + size]);
-        } else {
-            match codepoint {
-                0x08 => out.extend("\\b".as_bytes()),
-                0x09 => out.extend("\\t".as_bytes()),
-                0x0A => out.extend("\\n".as_bytes()),
-                0x0C => out.extend("\\f".as_bytes()),
-                0x0D => out.extend("\\r".as_bytes()),
-                0x22 => out.extend("\\\"".as_bytes()),
-                0x5C => out.extend("\\\\".as_bytes()),
-                _ => {
-                    if codepoint > 0x20 {
-                        panic!("unnecessary encoding requested");
-                    }
-                    // This violates NIP-01 which doesn't allow characters like 0x00
-                    // even though JSON UTF-8 does.
-                    out.extend(format!("\\u{:04x}", codepoint).as_bytes());
-                }
-            }
-        }
-        read_pos += size;
-    }
-
-    Ok(out)
-}
-
-macro_rules! output_slice {
-    ($slice:expr, $out:expr, $pos:expr) => {
-        if $out.len() < *$pos + $slice.len() {
-            Err(Into::<crate::error::Error>::into(
-                crate::error::ChorusError::BufferTooSmall,
-            ))
-        } else {
-            $out[*$pos..*$pos + $slice.len()].copy_from_slice($slice);
-            *$pos += $slice.len();
-            Ok(())
-        }
-    };
-}
-
-macro_rules! output_byte {
-    ($byte:expr, $out:expr, $pos:expr) => {
-        if $out.len() < *$pos + 1 {
-            Err(Into::<crate::error::Error>::into(
-                crate::error::ChorusError::BufferTooSmall,
-            ))
-        } else {
-            unsafe { *$out.get_unchecked_mut(*$pos) = $byte };
-            *$pos += 1;
-            Ok(())
-        }
-    };
-}
-
-/// This unescapes a JSON string into the output.
-///
-/// The input should start on the first character of the string, and may extend
-/// to the ending double-quote and even further.
-///
-/// This will return how much input was consumed and how much output was written
-/// in that order (input_len, output_len)
-pub fn json_unescape(input: &[u8], out: &mut [u8]) -> Result<(usize, usize), Error> {
-    const BACKSPACE: u8 = 0x08;
-    const FORMFEED: u8 = 0x0C;
-    const LINEFEED: u8 = 0x0A;
-    const CR: u8 = 0x0D;
-    const TAB: u8 = 0x09;
-    const QUOTE: u8 = 0x22;
-    const BACKSLASH: u8 = 0x5C;
-    const SLASH: u8 = 0x2F;
-
-    // Write position in the output buffer
-    let mut write_pos: usize = 0;
-
-    let mut inescape: bool = false;
-    let mut uescape: Option<(usize, u32)> = None;
-    let mut p: usize = 0;
-    while let Some((codepoint, size)) = next_code_point(&input[p..])? {
-        if inescape {
-            inescape = false;
-            if codepoint > 255 {
-                return Err(ChorusError::JsonEscape.into());
-            }
-            match codepoint as u8 {
-                QUOTE | BACKSLASH | SLASH => {
-                    output_slice!(&input[p..p + size], out, &mut write_pos)?
-                }
-                b'b' => output_byte!(BACKSPACE, out, &mut write_pos)?,
-                b'f' => output_byte!(FORMFEED, out, &mut write_pos)?,
-                b'n' => output_byte!(LINEFEED, out, &mut write_pos)?,
-                b'r' => output_byte!(CR, out, &mut write_pos)?,
-                b't' => output_byte!(TAB, out, &mut write_pos)?,
-                // This violates NIP-01 but we accept it
-                b'u' => uescape = Some((0, 0)),
-                _ => return Err(ChorusError::JsonEscape.into()), // nothing else is a legal escape
-            }
-        } else if let Some((digit, total)) = uescape {
-            // must be a digit
-            if !(48..=57).contains(&codepoint) {
-                return Err(ChorusError::JsonEscape.into());
-            }
-            let total = total + ((codepoint - 48) << (4 * (3 - digit)));
-            if digit >= 3 {
-                if (0xD800..=0xDFFF).contains(&total) {
-                    return Err(ChorusError::JsonEscapeSurrogate.into());
-                }
-                let s = encode_utf8(total, &mut out[write_pos..])?;
-                write_pos += s;
-                uescape = None;
-            } else {
-                uescape = Some((digit + 1, total));
-            }
-        } else if codepoint == 0x5C {
-            // backslash
-            inescape = true;
-        } else if is_safe_char(codepoint) {
-            output_slice!(&input[p..p + size], out, &mut write_pos)?;
-        } else if codepoint == 0x22 {
-            // ending double quote
-            break;
-        } else {
-            return Err(ChorusError::JsonBadStringChar(codepoint).into());
-        }
-        p += size;
-    }
-
-    Ok((p, write_pos))
-}
-
-#[inline]
-fn is_safe_char(c: u32) -> bool {
-    let safe_ranges = [(0x20..=0x21), (0x23..=0x5B), (0x5D..=0x10FFFF)];
-    safe_ranges.iter().any(|range| range.contains(&c))
-}
-
-#[cfg(test)]
-mod test {
-    use super::{json_escape, json_unescape};
-
-    #[test]
-    fn test_json_escape() {
-        let buffer = Vec::with_capacity(255);
-
-        let input = "hello\t\tworld
-!!!";
-        let mut buffer = json_escape(input.as_bytes(), buffer).unwrap();
-        assert_eq!(&buffer[0..19], br#"hello\t\tworld\n!!!"#);
-
-        buffer.clear();
-
-        let input: [u8; 11] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
-        let mut buffer = json_escape(input.as_slice(), buffer).unwrap();
-        assert_eq!(
-            &buffer[0..54],
-            br#"\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\b\t\n"#
-        );
-
-        buffer.clear();
-
-        let input: [u8; 12] = [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22];
-        let mut buffer = json_escape(input.as_slice(), buffer).unwrap();
-        assert_eq!(
-            &buffer[0..64],
-            br#"\u000b\f\r\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016"#
-        );
-
-        buffer.clear();
-
-        let input: [u8; 4] = [32, 33, 34, 35];
-        let mut buffer = json_escape(input.as_slice(), buffer).unwrap();
-        assert_eq!(&buffer[0..5], br##" !\"#"##);
-
-        buffer.clear();
-
-        let input: [u8; 1] = [92];
-        let buffer = json_escape(input.as_slice(), buffer).unwrap();
-        assert_eq!(&buffer[0..2], br#"\\"#);
-    }
-
-    #[test]
-    fn test_json_unescape() {
-        let is_ok = |s: &[u8], equals: &[u8]| {
-            let mut buffer: Vec<u8> = Vec::with_capacity(1024);
-            buffer.resize(1024, 0);
-            let r = json_unescape(s, &mut buffer);
-            assert!(r.is_ok());
-            let (_inlen, outlen) = r.unwrap();
-            assert_eq!(outlen, equals.len());
-            assert_eq!(&buffer[0..equals.len()], equals);
-        };
-
-        let is_err = |s: &[u8]| {
-            let mut buffer: Vec<u8> = Vec::with_capacity(1024);
-            buffer.resize(1024, 0);
-            let r = json_unescape(s, &mut buffer);
-            assert!(r.is_err());
-        };
-
-        // simple string
-        is_ok(&b"abc".as_slice(), b"abc");
-
-        // carraige return
-        is_ok(&br#"ab\nc"#.as_slice(), b"ab\nc");
-
-        // escaping a character that is not allowed
-        is_err(&br#"ab\zc"#.as_slice());
-
-        // escaping quotes is allowed
-        is_ok(&br#" \"abc\" "#.as_slice(), br#" "abc" "#);
-
-        // high character
-        is_ok(r#"𝄞"#.as_bytes(), "𝄞".as_bytes());
-
-        // high character is interpreted as these four bytes
-        is_ok(r#"𝄞"#.as_bytes(), b"\xF0\x9D\x84\x9E");
-
-        // esacaping a character that is not allowed
-        is_err(r#"\𝄞"#.as_bytes());
-
-        // actual unescaped tab is disallowed
-        is_err("\t".as_bytes());
-
-        // unicode escape and more
-        is_ok(
-            r#"{\"name\":\"BagMan\",\"about\":\"Father.\nHusband.\nNerd: \u2020.\"}"#.as_bytes(),
-            "{\"name\":\"BagMan\",\"about\":\"Father.\nHusband.\nNerd: †.\"}".as_bytes(),
-        );
-
-        // bad unicode escape
-        is_err(r#"\u8f00"#.as_bytes());
-
-        // Check output values
-        let mut buffer: Vec<u8> = Vec::with_capacity(1024);
-        buffer.resize(1024, 0);
-        let (inlen, outlen) = json_unescape(br#"the\nclient", "gossip""#, &mut buffer).unwrap();
-        assert_eq!(inlen, 11);
-        assert_eq!(outlen, 10);
-    }
-}

src/types/parse/json_parse.rs

@@ -1,468 +0,0 @@
-use super::json_escape::json_unescape;
-use super::put;
-use crate::error::{ChorusError, Error};
-
-#[inline]
-pub fn eat_whitespace(input: &[u8], inposp: &mut usize) {
-    while *inposp < input.len() && [0x20, 0x09, 0x0A, 0x0D].contains(&input[*inposp]) {
-        *inposp += 1;
-    }
-}
-
-#[inline]
-pub fn eat_whitespace_and_commas(input: &[u8], inposp: &mut usize) {
-    while *inposp < input.len() && [0x20, 0x09, 0x0A, 0x0D, b','].contains(&input[*inposp]) {
-        *inposp += 1;
-    }
-}
-
-#[inline]
-pub fn verify_char(input: &[u8], ch: u8, inposp: &mut usize) -> Result<(), Error> {
-    if *inposp >= input.len() {
-        Err(ChorusError::JsonBad("Too Short or Missing Fields", *inposp).into())
-    } else if input[*inposp] == ch {
-        *inposp += 1;
-        Ok(())
-    } else {
-        Err(ChorusError::JsonBadCharacter(input[*inposp] as char, *inposp, ch as char).into())
-    }
-}
-
-pub fn eat_colon_with_whitespace(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    eat_whitespace(input, inposp);
-    verify_char(input, b':', inposp)?;
-    eat_whitespace(input, inposp);
-    Ok(())
-}
-
-pub fn next_object_field(input: &[u8], inposp: &mut usize) -> Result<bool, Error> {
-    eat_whitespace(input, inposp);
-    // next comes either comma or end brace
-    if *inposp >= input.len() {
-        return Err(ChorusError::JsonBad("Too short", *inposp).into());
-    }
-    if input[*inposp] == b'}' {
-        *inposp += 1;
-        Ok(true)
-    } else if input[*inposp] == b',' {
-        *inposp += 1;
-        Ok(false)
-    } else {
-        Err(ChorusError::JsonBad("Unexpected char", *inposp).into())
-    }
-}
-
-pub fn read_id(input: &[u8], inposp: &mut usize, output: &mut [u8]) -> Result<(), Error> {
-    if output.len() < 32 {
-        return Err(ChorusError::BufferTooSmall.into());
-    }
-    verify_char(input, b'"', inposp)?;
-    if *inposp + 64 >= input.len() {
-        return Err(ChorusError::JsonBad("Too short reading id", *inposp).into());
-    }
-    // Read the hex ID and write the binary ID into the output event structure
-    read_hex!(&input[*inposp..*inposp + 64], &mut output[..32], 32)?;
-    *inposp += 64;
-    verify_char(input, b'"', inposp)?;
-    Ok(())
-}
-
-pub fn read_pubkey(input: &[u8], inposp: &mut usize, output: &mut [u8]) -> Result<(), Error> {
-    if output.len() < 32 {
-        return Err(ChorusError::BufferTooSmall.into());
-    }
-    verify_char(input, b'"', inposp)?;
-    if *inposp + 64 >= input.len() {
-        return Err(ChorusError::JsonBad("Too short reading pubkey", *inposp).into());
-    }
-    // Read the hex pubkey and write the binary pubkey into the output event structure
-    read_hex!(&input[*inposp..*inposp + 64], &mut output[..32], 32)?;
-    *inposp += 64;
-    verify_char(input, b'"', inposp)?;
-    Ok(())
-}
-
-pub fn read_u64(input: &[u8], inposp: &mut usize) -> Result<u64, Error> {
-    let mut value: u64 = 0;
-    let mut any: bool = false;
-    while *inposp < input.len() && b"0123456789".contains(&input[*inposp]) {
-        any = true;
-        value = (value * 10) + (input[*inposp] - 48) as u64;
-        *inposp += 1;
-    }
-    if !any {
-        return Err(ChorusError::JsonBad(
-            "Created at must be a positive or zero valued number",
-            *inposp,
-        )
-        .into());
-    }
-    Ok(value)
-}
-
-pub fn read_kind(input: &[u8], inposp: &mut usize) -> Result<u16, Error> {
-    let mut value: u32 = 0;
-    let mut any: bool = false;
-    while *inposp < input.len() && b"0123456789".contains(&input[*inposp]) {
-        any = true;
-        value = (value * 10) + (input[*inposp] - 48) as u32;
-        *inposp += 1;
-    }
-    if !any {
-        return Err(ChorusError::JsonBad(
-            "Kind at must be a positive or zero valued number",
-            *inposp,
-        )
-        .into());
-    }
-    if value > 65535 {
-        Err(ChorusError::JsonBad("Kind larger than 65535", *inposp).into())
-    } else {
-        Ok(value as u16)
-    }
-}
-
-// HELP written for event only
-// From the outer bracket through to the character after the close outer bracket
-// returns the size of data written to the output
-pub fn read_tags_array(
-    input: &[u8],
-    inposp: &mut usize,
-    output: &mut [u8],
-) -> Result<usize, Error> {
-    verify_char(input, b'[', inposp)?; // outer array open brace
-    eat_whitespace(input, inposp);
-
-    if output.len() < 4 {
-        return Err(ChorusError::BufferTooSmall.into());
-    }
-
-    // NOTE: we cannot write any tag strings until after we have counted the tags.
-    // (our tags structure is optimized for reading, not writing)
-    let num_tags: usize = count_tags(input, *inposp)?;
-    put(output, 2, (num_tags as u16).to_ne_bytes().as_slice())?;
-
-    // Case where we have no tags
-    if num_tags == 0 {
-        put(output, 0, 4_u16.to_ne_bytes().as_slice())?;
-        burn_array(input, inposp)?;
-        return Ok(4);
-    }
-
-    verify_char(input, b'[', inposp)?; // opening brace of first tag
-    eat_whitespace(input, inposp);
-
-    let mut tag_num = 0;
-    let mut outpos: usize = 4 + num_tags * 2;
-    if output.len() < outpos {
-        return Err(ChorusError::BufferTooSmall.into());
-    }
-
-    loop {
-        // Write the offset of this tag
-        let offset_slot = 4 + tag_num * 2;
-        put(
-            output,
-            offset_slot,
-            (outpos as u16).to_ne_bytes().as_slice(),
-        )?;
-
-        // Read the tag (bumps inpos and outpos)
-        read_tag(input, inposp, output, &mut outpos)?;
-        eat_whitespace(input, inposp);
-
-        // Check what is next
-        match input[*inposp] {
-            b']' => {
-                *inposp += 1;
-                if tag_num != num_tags - 1 {
-                    panic!("Tag count mismatch");
-                }
-                break;
-            }
-            b',' => {
-                *inposp += 1;
-                eat_whitespace(input, inposp);
-                verify_char(input, b'[', inposp)?;
-                tag_num += 1;
-                if tag_num >= num_tags {
-                    panic!("Tag count mismatch");
-                }
-                eat_whitespace(input, inposp);
-            }
-            _ => return Err(ChorusError::JsonBad("Tag array bad character", *inposp).into()),
-        }
-    }
-
-    // Write length of tags section
-    put(output, 0, (outpos as u16).to_ne_bytes().as_slice())?;
-
-    Ok(outpos)
-}
-
-// From the first inner tag bracket, ending after the outer bracket.
-// This just counts the tags, it does not write output or modify the inpos
-// This does a quicker pass over the content than actual tag parsing does.
-pub fn count_tags(input: &[u8], mut inpos: usize) -> Result<usize, Error> {
-    // First non-whitespace character after the opening brace
-    match input[inpos] {
-        b']' => return Ok(0), // no tags
-        b'[' => (),           // expected
-        _ => return Err(ChorusError::JsonBad("Tag array bad initial character", inpos).into()),
-    }
-
-    let mut count = 1;
-    inpos += 1;
-    burn_tag(input, &mut inpos)?;
-    eat_whitespace(input, &mut inpos);
-
-    loop {
-        match input[inpos] {
-            b']' => return Ok(count),
-            b',' => {
-                inpos += 1;
-                eat_whitespace(input, &mut inpos);
-                verify_char(input, b'[', &mut inpos)?;
-                count += 1;
-                burn_tag(input, &mut inpos)?;
-                eat_whitespace(input, &mut inpos);
-            }
-            _ => return Err(ChorusError::JsonBad("Tag array bad character", inpos).into()),
-        }
-    }
-}
-
-pub fn read_tag(
-    input: &[u8],
-    inposp: &mut usize,
-    output: &mut [u8],
-    outposp: &mut usize,
-) -> Result<(), Error> {
-    verify_char(input, b'"', inposp)?;
-
-    let countpos = *outposp;
-    *outposp += 2;
-    let mut num_strings: usize = 1;
-    loop {
-        // read string
-        let (inlen, outlen) = json_unescape(&input[*inposp..], &mut output[*outposp + 2..])?;
-        // write the length before it
-        put(output, *outposp, (outlen as u16).to_ne_bytes().as_slice())?;
-        // bump the outposp past it
-        *outposp += 2 + outlen;
-        // bump the inpos past the string (and the ending quote which isn't counted in the len)
-        *inposp += inlen + 1;
-
-        eat_whitespace(input, inposp);
-        match input[*inposp] {
-            b',' => {
-                *inposp += 1;
-                eat_whitespace(input, inposp);
-                verify_char(input, b'"', inposp)?;
-                num_strings += 1;
-                continue;
-            }
-            b']' => {
-                *inposp += 1;
-                break;
-            }
-            _ => return Err(ChorusError::JsonBad("Tag array bad character", *inposp).into()),
-        }
-    }
-
-    // Write the count of strings at the very start
-    put(
-        output,
-        countpos,
-        (num_strings as u16).to_ne_bytes().as_slice(),
-    )?;
-
-    Ok(())
-}
-
-pub fn read_content(
-    input: &[u8],
-    inposp: &mut usize,
-    output: &mut [u8],
-    after_tags: usize,
-) -> Result<(), Error> {
-    verify_char(input, b'"', inposp)?;
-
-    // Place content 4 bytes beyond tags, to reserve space for content length
-    let (inlen, outlen) = json_unescape(&input[*inposp..], &mut output[after_tags + 4..])?;
-    *inposp += inlen + 1; // +1 to pass the end quote
-
-    // Write content length
-    put(output, after_tags, (outlen as u32).to_ne_bytes().as_slice())?;
-
-    // Write event size
-    let event_len = after_tags + 4 + outlen;
-    put(output, 0, (event_len as u32).to_ne_bytes().as_slice())?;
-
-    Ok(())
-}
-
-// FIXME this is too event-offset specific
-pub fn read_sig(input: &[u8], inposp: &mut usize, output: &mut [u8]) -> Result<(), Error> {
-    if output.len() < 144 {
-        return Err(ChorusError::BufferTooSmall.into());
-    }
-    verify_char(input, b'"', inposp)?;
-    if *inposp + 128 >= input.len() {
-        return Err(ChorusError::JsonBad("Too short reading sig", *inposp).into());
-    }
-    // Read the hex sig and write the binary sig into the output event structure
-    read_hex!(&input[*inposp..*inposp + 128], &mut output[80..144], 64)?;
-    *inposp += 128;
-    verify_char(input, b'"', inposp)?;
-    Ok(())
-}
-
-// from the character after the start quote
-// ending on the character following the end quote
-pub fn burn_string(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    while *inposp < input.len() && input[*inposp] != b'"' {
-        if input[*inposp] == b'\\' && *inposp + 1 < input.len() {
-            *inposp += 2;
-        } else {
-            *inposp += 1;
-        }
-    }
-    if input[*inposp] == b'"' {
-        *inposp += 1;
-        Ok(())
-    } else {
-        Err(ChorusError::JsonBad("Unterminated string", *inposp).into())
-    }
-}
-
-// from the character after the open brace
-// ending on the character following the close brace
-pub fn burn_tag(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    eat_whitespace(input, inposp);
-    // assuming that every tag must have at least one string
-    verify_char(input, b'"', inposp)?;
-    burn_string(input, inposp)?;
-    eat_whitespace(input, inposp);
-    while input[*inposp] == b',' {
-        *inposp += 1;
-        eat_whitespace(input, inposp);
-        verify_char(input, b'"', inposp)?;
-        burn_string(input, inposp)?;
-        eat_whitespace(input, inposp);
-    }
-    verify_char(input, b']', inposp)?;
-    Ok(())
-}
-
-pub fn burn_key_and_value(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    verify_char(input, b'"', inposp)?;
-    burn_string(input, inposp)?;
-    eat_colon_with_whitespace(input, inposp)?;
-    burn_value(input, inposp)?;
-    Ok(())
-}
-
-// from the character after the open brace
-// ending on the character following the close brace
-pub fn burn_object(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    loop {
-        eat_whitespace_and_commas(input, inposp);
-
-        // Check for the end
-        if input[*inposp] == b'}' {
-            *inposp += 1;
-            return Ok(());
-        }
-
-        burn_key_and_value(input, inposp)?;
-    }
-}
-
-// from the character after the open bracket
-// ending on the character following the close bracket
-pub fn burn_array(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    loop {
-        eat_whitespace_and_commas(input, inposp);
-
-        // Check for the end
-        if input[*inposp] == b']' {
-            *inposp += 1;
-            return Ok(());
-        }
-
-        burn_value(input, inposp)?;
-    }
-}
-
-pub fn burn_value(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    if *inposp >= input.len() {
-        return Err(ChorusError::JsonBad("Too short burning an unused JSON value", *inposp).into());
-    }
-    match input[*inposp] {
-        b'"' => {
-            *inposp += 1;
-            burn_string(input, inposp)?
-        }
-        b'[' => {
-            *inposp += 1;
-            burn_array(input, inposp)?
-        }
-        b'{' => {
-            *inposp += 1;
-            burn_object(input, inposp)?
-        }
-        b't' => burn_true(input, inposp)?,
-        b'f' => burn_false(input, inposp)?,
-        b'n' => burn_null(input, inposp)?,
-        b'-' => burn_number(input, inposp)?,
-        _ => {
-            if b"123456789".contains(&input[*inposp]) {
-                burn_number(input, inposp)?
-            } else {
-                return Err(ChorusError::JsonBad(
-                    "Too short burning an unused JSON value",
-                    *inposp,
-                )
-                .into());
-            }
-        }
-    }
-
-    Ok(())
-}
-
-pub fn burn_null(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    if *inposp + 4 <= input.len() && &input[*inposp..*inposp + 4] == b"null" {
-        *inposp += 4;
-        Ok(())
-    } else {
-        Err(ChorusError::JsonBad("Expected null", *inposp).into())
-    }
-}
-
-pub fn burn_true(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    if *inposp + 4 <= input.len() && &input[*inposp..*inposp + 4] == b"true" {
-        *inposp += 4;
-        Ok(())
-    } else {
-        Err(ChorusError::JsonBad("Expected true", *inposp).into())
-    }
-}
-
-pub fn burn_false(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    if *inposp + 5 <= input.len() && &input[*inposp..*inposp + 5] == b"false" {
-        *inposp += 5;
-        Ok(())
-    } else {
-        Err(ChorusError::JsonBad("Expected false", *inposp).into())
-    }
-}
-
-pub fn burn_number(input: &[u8], inposp: &mut usize) -> Result<(), Error> {
-    // For burning, we don't check validity.
-    while *inposp < input.len() && b".+-0123456789abcdefABCDEF_oOxXn".contains(&input[*inposp]) {
-        *inposp += 1;
-    }
-    Ok(())
-}

src/types/parse/mod.rs

@@ -1,15 +0,0 @@
-pub mod json_escape;
-
-pub mod json_parse;
-
-pub mod utf8;
-
-#[inline]
-pub fn put(output: &mut [u8], offset: usize, data: &[u8]) -> Result<(), crate::error::Error> {
-    if output.len() < offset + data.len() {
-        Err(crate::error::ChorusError::BufferTooSmall.into())
-    } else {
-        output[offset..offset + data.len()].copy_from_slice(data);
-        Ok(())
-    }
-}

src/types/parse/utf8.rs

@@ -1,165 +0,0 @@
-use crate::error::{ChorusError, Error};
-
-// Reads the next code point if UTF-8, and returns it along with the number of characters
-// that make it up.
-pub fn next_code_point(input: &[u8]) -> Result<Option<(u32, usize)>, Error> {
-    let len = input.len();
-    if len < 1 {
-        return Ok(None);
-    }
-
-    // Decode UTF-8
-    let x = input[0];
-    if x < 128 {
-        return Ok(Some((x as u32, 1)));
-    }
-
-    // Multibyte case follows
-    // Decode from a byte combination out of: [[[x y] z] w]
-    let init = utf8_first_byte(x, 2);
-    if len < 2 {
-        return Err(ChorusError::Utf8Error.into());
-    }
-    let y = input[1];
-    let mut ch = utf8_acc_cont_byte(init, y);
-    if x >= 0xE0 {
-        // [[x y z] w] case
-        // 5th bit in 0xE0 .. 0xEF is always clear, so `init` is still valid
-        if len < 3 {
-            return Err(ChorusError::Utf8Error.into());
-        }
-        let z = input[2];
-        let y_z = utf8_acc_cont_byte((y & CONT_MASK) as u32, z);
-        ch = init << 12 | y_z;
-        if x >= 0xF0 {
-            // [x y z w] case
-            // use only the lower 3 bits of `init`
-            if len < 4 {
-                return Err(ChorusError::Utf8Error.into());
-            }
-            let w = input[3];
-            ch = (init & 7) << 18 | utf8_acc_cont_byte(y_z, w);
-            Ok(Some((ch, 4)))
-        } else {
-            Ok(Some((ch, 3)))
-        }
-    } else {
-        Ok(Some((ch, 2)))
-    }
-}
-
-pub fn encode_utf8(code: u32, dst: &mut [u8]) -> Result<usize, Error> {
-    // UTF-8 ranges and tags for encoding characters
-    const TAG_CONT: u8 = 0b1000_0000;
-    const TAG_TWO_B: u8 = 0b1100_0000;
-    const TAG_THREE_B: u8 = 0b1110_0000;
-    const TAG_FOUR_B: u8 = 0b1111_0000;
-    const MAX_ONE_B: u32 = 0x80;
-    const MAX_TWO_B: u32 = 0x800;
-    const MAX_THREE_B: u32 = 0x10000;
-
-    let len = unsafe {
-        if code < MAX_ONE_B && !dst.is_empty() {
-            *dst.get_unchecked_mut(0) = code as u8;
-            1
-        } else if code < MAX_TWO_B && dst.len() >= 2 {
-            *dst.get_unchecked_mut(0) = (code >> 6 & 0x1F) as u8 | TAG_TWO_B;
-            *dst.get_unchecked_mut(1) = (code & 0x3F) as u8 | TAG_CONT;
-            2
-        } else if code < MAX_THREE_B && dst.len() >= 3 {
-            *dst.get_unchecked_mut(0) = (code >> 12 & 0x0F) as u8 | TAG_THREE_B;
-            *dst.get_unchecked_mut(1) = (code >> 6 & 0x3F) as u8 | TAG_CONT;
-            *dst.get_unchecked_mut(2) = (code & 0x3F) as u8 | TAG_CONT;
-            3
-        } else if dst.len() >= 4 {
-            *dst.get_unchecked_mut(0) = (code >> 18 & 0x07) as u8 | TAG_FOUR_B;
-            *dst.get_unchecked_mut(1) = (code >> 12 & 0x3F) as u8 | TAG_CONT;
-            *dst.get_unchecked_mut(2) = (code >> 6 & 0x3F) as u8 | TAG_CONT;
-            *dst.get_unchecked_mut(3) = (code & 0x3F) as u8 | TAG_CONT;
-            4
-        } else {
-            return Err(ChorusError::BufferTooSmall.into());
-        }
-    };
-    Ok(len)
-}
-
-/// Returns the initial codepoint accumulator for the first byte.
-/// The first byte is special, only want bottom 5 bits for width 2, 4 bits
-/// for width 3, and 3 bits for width 4.
-#[inline]
-const fn utf8_first_byte(byte: u8, width: u32) -> u32 {
-    (byte & (0x7F >> width)) as u32
-}
-
-/// Returns the value of `ch` updated with continuation byte `byte`.
-#[inline]
-const fn utf8_acc_cont_byte(ch: u32, byte: u8) -> u32 {
-    (ch << 6) | (byte & CONT_MASK) as u32
-}
-
-/// Mask of the value bits of a continuation byte.
-const CONT_MASK: u8 = 0b0011_1111;
-
-#[cfg(test)]
-mod test {
-    use super::{encode_utf8, next_code_point};
-
-    #[test]
-    fn test_next_code_point() {
-        let (codepoint, size) = next_code_point(r#"𝄞"#.as_bytes()).unwrap().unwrap();
-        assert_eq!(codepoint, 119070);
-        assert_eq!(size, 4);
-
-        let (codepoint, size) = next_code_point(r#"†"#.as_bytes()).unwrap().unwrap();
-        assert_eq!(codepoint, 0x2020);
-        assert_eq!(size, 3);
-
-        // four codepoints
-        let s = [
-            0x61, 0xE0, 0xA4, 0xA8, 0xE0, 0xA4, 0xBF, 0xE4, 0xBA, 0x9C, 0xF0, 0x90, 0x82, 0x83,
-        ];
-        let (codepoint, size) = next_code_point(s.as_slice()).unwrap().unwrap();
-        assert_eq!(codepoint, 0x61);
-        assert_eq!(size, 1);
-
-        let mut start = size;
-        let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
-        assert_eq!(codepoint, 0x928);
-        assert_eq!(size, 3);
-
-        start += size;
-        let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
-        assert_eq!(codepoint, 0x93F);
-        assert_eq!(size, 3);
-
-        start += size;
-        let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
-        assert_eq!(codepoint, 0x4E9C);
-        assert_eq!(size, 3);
-
-        start += size;
-        let (codepoint, size) = next_code_point(&s[start..]).unwrap().unwrap();
-        assert_eq!(codepoint, 0x10083);
-        assert_eq!(size, 4);
-
-        assert_eq!(next_code_point(&s[0..0]).unwrap(), None);
-    }
-
-    #[test]
-    fn test_encode_utf8() {
-        let mut buffer: Vec<u8> = vec![0, 0, 0, 0];
-
-        assert_eq!(encode_utf8(0x69, &mut buffer).unwrap(), 1);
-        assert_eq!(buffer[0], 0x69);
-
-        assert_eq!(encode_utf8(0xEC, &mut buffer).unwrap(), 2);
-        assert_eq!(&buffer[0..2], &[0xC3, 0xAC]);
-
-        assert_eq!(encode_utf8(0x5450, &mut buffer).unwrap(), 3);
-        assert_eq!(&buffer[0..3], &[0xE5, 0x91, 0x90]);
-
-        assert_eq!(encode_utf8(0x2825F, &mut buffer).unwrap(), 4);
-        assert_eq!(&buffer[..], &[0xF0, 0xA8, 0x89, 0x9F]);
-    }
-}

src/types/pubkey.rs

@@ -1,47 +0,0 @@
-use crate::Error;
-use serde::{Deserialize, Serialize};
-use std::fmt;
-
-#[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq)]
-pub struct Pubkey(pub [u8; 32]);
-
-impl Pubkey {
-    pub fn write_hex(&self, output: &mut [u8]) -> Result<(), Error> {
-        write_hex!(self.0, output, 32)
-    }
-
-    pub fn read_hex(input: &[u8]) -> Result<Pubkey, Error> {
-        let mut out: [u8; 32] = [0; 32];
-        read_hex!(input, &mut out, 32)?;
-        Ok(Pubkey(out))
-    }
-
-    pub fn as_slice(&self) -> &[u8] {
-        self.0.as_slice()
-    }
-}
-
-impl fmt::Display for Pubkey {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        let mut bytes: [u8; 64] = [0; 64];
-        self.write_hex(&mut bytes).unwrap();
-        let hex = unsafe { std::str::from_utf8_unchecked(&bytes) };
-        write!(f, "{hex}")
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::Pubkey;
-
-    #[test]
-    fn test_pubkey_hex_functions() {
-        let hex = b"1110ee4ff957fa9c55832eaccb4dc1c45bfc6304e1e4e9fa478f53df4b20062d";
-        let pubkey = Pubkey::read_hex(hex).unwrap();
-        eprintln!("{:?}", pubkey);
-        let mut hex2: [u8; 64] = [0; 64];
-        pubkey.write_hex(&mut hex2).unwrap();
-        assert_eq!(hex, &hex2);
-        assert_eq!(format!("{}", pubkey).as_bytes(), hex);
-    }
-}

src/types/sig.rs

@@ -1,46 +0,0 @@
-use crate::Error;
-use std::fmt;
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub struct Sig(pub [u8; 64]);
-
-impl Sig {
-    pub fn write_hex(&self, output: &mut [u8]) -> Result<(), Error> {
-        write_hex!(self.0, output, 64)
-    }
-
-    pub fn read_hex(input: &[u8]) -> Result<Sig, Error> {
-        let mut out: [u8; 64] = [0; 64];
-        read_hex!(input, &mut out, 64)?;
-        Ok(Sig(out))
-    }
-
-    pub fn as_slice(&self) -> &[u8] {
-        self.0.as_slice()
-    }
-}
-
-impl fmt::Display for Sig {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        let mut bytes: [u8; 128] = [0; 128];
-        self.write_hex(&mut bytes).unwrap();
-        let hex = unsafe { std::str::from_utf8_unchecked(&bytes) };
-        write!(f, "{hex}")
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::Sig;
-
-    #[test]
-    fn test_sig_hex_functions() {
-        let hex = b"f4165cd621d387e0f723c3ca7484ca3da9ede00ffc97eb57c3e695384e095dea1a6215e7328b793e878f436f508006f95957c7e6b652e80d4c3c47b9f9110e7d";
-        let sig = Sig::read_hex(hex).unwrap();
-        eprintln!("{:?}", sig);
-        let mut hex2: [u8; 128] = [0; 128];
-        sig.write_hex(&mut hex2).unwrap();
-        assert_eq!(hex, &hex2);
-        assert_eq!(format!("{}", sig).as_bytes(), hex);
-    }
-}

src/types/tags.rs

@@ -1,287 +0,0 @@
-use crate::error::{ChorusError, Error};
-use std::fmt;
-
-/*
- * 0 .. 2    u16      Length of the tags section
- * 2 .. 4    u16      num_tags
- * 4 .. 6    u16      offset of zeroeth tag
- * 6 .. 8    u16      offset of first tag
- * ...
- *
- * 4+num_tags*2 ..    beginning of actual tag data
- *
- *    Tag data looks like this for each tag:
- *    count, (len, data), (len, data), ...
- */
-
-/// This stores an array of tags, each tag being an array of byte-strings.
-/// It is stored in a single packed linear byte array.
-#[derive(Debug, Clone)]
-pub struct Tags<'a>(&'a [u8]);
-
-impl<'a> Tags<'a> {
-    // this marks off the slice of bytes that represent the tags from a potentially longer input
-    pub fn delineate(input: &'a [u8]) -> Result<Tags<'a>, Error> {
-        if input.len() < 2 {
-            return Err(ChorusError::EndOfInput.into());
-        }
-        let len = parse_u16!(input, 0) as usize;
-        if input.len() < len {
-            return Err(ChorusError::EndOfInput.into());
-        }
-        Ok(Tags(&input[0..len]))
-    }
-
-    // This copies
-    pub fn copy(&self, output: &mut [u8]) -> Result<(), Error> {
-        if output.len() < self.0.len() {
-            return Err(ChorusError::BufferTooSmall.into());
-        }
-        output[..self.0.len()].copy_from_slice(self.0);
-        Ok(())
-    }
-
-    #[inline]
-    pub fn as_bytes(&self) -> &[u8] {
-        self.0
-    }
-
-    #[inline]
-    pub fn len(&self) -> usize {
-        parse_u16!(self.0, 2) as usize
-    }
-
-    #[inline]
-    pub fn is_empty(&self) -> bool {
-        self.len() == 0
-    }
-
-    pub fn iter(&'a self) -> TagsIter<'a> {
-        TagsIter {
-            tags: self,
-            next: 0,
-        }
-    }
-
-    pub fn get_string(&'a self, tag: usize, string: usize) -> Option<&'a [u8]> {
-        if tag >= self.len() {
-            return None;
-        }
-
-        let mut offset = parse_u16!(self.0, 4 + tag * 2) as usize;
-        let count = parse_u16!(self.0, offset) as usize;
-        offset += 2;
-        if string >= count {
-            return None;
-        }
-
-        let end = parse_u16!(self.0, 0) as usize;
-
-        // pass the fields we aren't reading
-        for _ in 0..string {
-            let len = parse_u16!(self.0, offset) as usize;
-            offset += 2 + len;
-            if offset > end {
-                // safety check
-                return None;
-            }
-        }
-        let len = parse_u16!(self.0, offset) as usize;
-        offset += 2;
-        if offset + len > end {
-            // safety check
-            return None;
-        }
-
-        Some(&self.0[offset..offset + len])
-    }
-
-    pub fn get_value(&'a self, key: &[u8]) -> Option<&'a [u8]> {
-        for tag in 0..self.len() {
-            if let Some(thing) = self.get_string(tag, 0) {
-                if thing == key {
-                    return self.get_string(tag, 1);
-                }
-            }
-        }
-        None
-    }
-
-    pub fn matches(&self, letter: &[u8], value: &[u8]) -> bool {
-        for mut tag in self.iter() {
-            if tag.next() == Some(letter) && tag.next() == Some(value) {
-                return true;
-            }
-        }
-        false
-    }
-
-    pub fn as_json(&self) -> Vec<u8> {
-        let mut output: Vec<u8> = Vec::with_capacity(256);
-        let mut first = true;
-        output.push(b'[');
-        for tag in self.iter() {
-            if !first {
-                output.push(b',');
-            }
-            output.push(b'[');
-            let mut firststring = true;
-            for bytes in tag {
-                if !firststring {
-                    output.push(b',');
-                }
-                output.push(b'"');
-                output.extend(bytes);
-                output.push(b'"');
-                firststring = false;
-            }
-            output.push(b']');
-            first = false;
-        }
-        output.push(b']');
-        output
-    }
-}
-
-impl fmt::Display for Tags<'_> {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        let bytes = self.as_json();
-        let s = unsafe { std::str::from_utf8_unchecked(&bytes) };
-        write!(f, "{s}")
-    }
-}
-
-#[derive(Debug)]
-pub struct TagsIter<'a> {
-    tags: &'a Tags<'a>,
-    next: usize,
-}
-
-impl<'a> Iterator for TagsIter<'a> {
-    type Item = TagsStringIter<'a>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        if self.next >= self.tags.len() {
-            None
-        } else {
-            let offset_slot = 4 + self.next * 2;
-            let offset = parse_u16!(self.tags.0, offset_slot) as usize;
-            let count = parse_u16!(self.tags.0, offset) as usize;
-            self.next += 1;
-            Some(TagsStringIter {
-                tags: self.tags,
-                count,
-                cur_offset: offset + 2,
-                next: 0,
-            })
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct TagsStringIter<'a> {
-    tags: &'a Tags<'a>,
-    count: usize,
-    cur_offset: usize,
-    next: usize,
-}
-
-impl<'a> Iterator for TagsStringIter<'a> {
-    type Item = &'a [u8];
-
-    fn next(&mut self) -> Option<Self::Item> {
-        if self.next >= self.count {
-            None
-        } else {
-            // Read len
-            let len = parse_u16!(self.tags.0, self.cur_offset) as usize;
-            let s = &self.tags.0[self.cur_offset + 2..self.cur_offset + 2 + len];
-            self.cur_offset += 2 + len;
-            self.next += 1;
-            Some(s)
-        }
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::Tags;
-
-    #[test]
-    fn test_tag() {
-        /*
-         *  [
-         *    ["Hello world!", "Hello", "world!"],
-         *    ["p", ""ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"],
-         *  ]
-         */
-
-        let data: Vec<u8> = vec![
-            110, 0, // tags_len
-            2, 0, // num_tags
-            8, 0, // first tag at offset 8
-            39, 0, // second tag at offset 39
-            // 8:
-            3, 0, // three fields long
-            12, 0, // first field 12 bytes long
-            72, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33, // "Hello world!"
-            5, 0, // second field 5 bytes long
-            72, 101, 108, 108, 111, // "Hello"
-            6, 0, // third field 6 bytes long
-            119, 111, 114, 108, 100, 33, // world!
-            // 39:
-            2, 0, // two fields long
-            1, 0,   // first field 1 bytes long
-            112, // "p"
-            64, 0, // second field 64 bytes long
-            101, 101, 49, 49, 97, 53, 100, 102, 102, 52, 48, 99, 49, 57, 97, 53, 53, 53, 102, 52,
-            49, 102, 101, 52, 50, 98, 52, 56, 102, 48, 48, 101, 54, 49, 56, 99, 57, 49, 50, 50, 53,
-            54, 50, 50, 97, 101, 51, 55, 98, 54, 99, 50, 98, 98, 54, 55, 98, 55, 54, 99, 52, 101,
-            52, 57,
-            // "ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"
-        ];
-
-        let tags = Tags::delineate(&data).unwrap();
-
-        // Test tag access with get_string()
-
-        assert_eq!(tags.get_string(0, 0), Some(b"Hello world!".as_slice()));
-        assert_eq!(tags.get_string(0, 1), Some(b"Hello".as_slice()));
-        assert_eq!(tags.get_string(0, 2), Some(b"world!".as_slice()));
-        assert_eq!(tags.get_string(0, 3), None);
-        assert_eq!(tags.get_string(1, 0), Some(b"p".as_slice()));
-        assert_eq!(
-            tags.get_string(1, 1),
-            Some(b"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49".as_slice())
-        );
-        assert_eq!(tags.get_string(1, 2), None);
-        assert_eq!(tags.get_string(2, 0), None);
-
-        // Test tag access with iterators
-
-        let mut iter = tags.iter();
-
-        let mut tag1 = iter.next().unwrap();
-        println!("TagsStringIter 1 {:?}", tag1);
-        assert_eq!(tag1.next(), Some(b"Hello world!".as_slice()));
-        assert_eq!(tag1.next(), Some(b"Hello".as_slice()));
-        assert_eq!(tag1.next(), Some(b"world!".as_slice()));
-        assert!(tag1.next().is_none());
-
-        let mut tag2 = iter.next().unwrap();
-        assert_eq!(tag2.next(), Some(b"p".as_slice()));
-        assert_eq!(
-            tag2.next(),
-            Some(b"ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49".as_slice())
-        );
-        assert!(tag2.next().is_none());
-
-        let tag3 = iter.next();
-        assert!(tag3.is_none());
-
-        assert_eq!(
-            format!("{tags}"),
-            r#"[["Hello world!","Hello","world!"],["p","ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49"]]"#
-        );
-    }
-}

src/types/time.rs

@@ -1,29 +0,0 @@
-use std::fmt;
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
-pub struct Time(pub u64);
-
-impl fmt::Display for Time {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "{}", self.0)
-    }
-}
-
-impl Time {
-    pub fn as_u64(&self) -> u64 {
-        self.0
-    }
-
-    pub fn min() -> Time {
-        Time(0)
-    }
-
-    pub fn max() -> Time {
-        Time(u64::MAX)
-    }
-
-    pub fn now() -> Time {
-        // Safety: unwrap() can only panic if the system time is before UNIX_EPOCH
-        Time(std::time::UNIX_EPOCH.elapsed().unwrap().as_secs())
-    }
-}

src/web.rs

@@ -1,108 +0,0 @@
-use crate::config::Config;
-use crate::error::Error;
-use crate::globals::GLOBALS;
-use hyper::{Body, Request, Response, StatusCode};
-use std::net::SocketAddr;
-
-pub async fn serve_http(peer: SocketAddr, request: Request<Body>) -> Result<Response<Body>, Error> {
-    log::debug!("{}: HTTP request for {}", peer, request.uri());
-    let response = Response::builder()
-        .header("Access-Control-Allow-Origin", "*")
-        .header("Access-Control-Allow-Headers", "*")
-        .header("Access-Control-Allow-Methods", "*")
-        .status(StatusCode::OK)
-        .body("This is a nostr relay. Please use a nostr client to connect.".into())?;
-    Ok(response)
-}
-
-pub async fn serve_nip11(peer: SocketAddr) -> Result<Response<Body>, Error> {
-    log::debug!("{}: sent NIP-11", peer);
-    let rid = {
-        let config = GLOBALS.config.read().await;
-        GLOBALS.rid.get_or_init(|| build_rid(&config))
-    };
-
-    let response = Response::builder()
-        .header("Access-Control-Allow-Origin", "*")
-        .header("Access-Control-Allow-Headers", "*")
-        .header("Access-Control-Allow-Methods", "*")
-        .header("Content-Type", "application/nostr+json")
-        .status(StatusCode::OK)
-        .body(rid.clone().into())?;
-    Ok(response)
-}
-
-fn build_rid(config: &Config) -> String {
-    let mut rid: String = String::with_capacity(255);
-
-    const SUPPORTED_NIPS: [u8; 7] = [
-        1,  // nostr
-        4,  // DMs
-        9,  // Event Deletion
-        11, // relay information document
-        42, // AUTH
-        59, // GiftWrap
-        65, // Relay List Metadata
-    ];
-    const _UNSUPPORTED_NIPS: [u8; 7] = [
-        26, // Delegated Event Signing
-        28, // Public Chat
-        40, // Expiration Timestamp
-        45, // Counting results
-        50, // SEARCH
-        94, // File Metadata
-        96, // HTTP File Storage Integration
-    ];
-    const _INAPPLICABLE_NIPS: [u8; 43] = [
-        2, 3, 5, 6, 7, 8, 10, 13, 14, 15, 18, 19, 21, 23, 24, 25, 27, 30, 31, 32, 36, 38, 39, 44,
-        46, 47, 48, 49, 51, 52, 53, 56, 57, 58, 72, 75, 78, 84, 89, 90, 92, 98, 99,
-    ];
-
-    let s = SUPPORTED_NIPS
-        .iter()
-        .map(|i| format!("{}", i))
-        .collect::<Vec<String>>()
-        .join(",");
-    rid.push_str(&format!("{{\"supported_nips\":[{}],", s));
-
-    let software = env!("CARGO_PKG_NAME");
-    rid.push_str("\"software\":\"");
-    rid.push_str(software);
-    rid.push('\"');
-
-    let version = env!("CARGO_PKG_VERSION");
-    rid.push(',');
-    rid.push_str("\"version\":\"");
-    rid.push_str(version);
-    rid.push('\"');
-
-    if let Some(name) = &config.name {
-        rid.push(',');
-        rid.push_str("\"name\":\"");
-        rid.push_str(name);
-        rid.push('\"');
-    }
-    if let Some(description) = &config.description {
-        rid.push(',');
-        rid.push_str("\"description\":\"");
-        rid.push_str(description);
-        rid.push('\"');
-    }
-    if let Some(contact) = &config.contact {
-        rid.push(',');
-        rid.push_str("\"contact\":\"");
-        rid.push_str(contact);
-        rid.push('\"');
-    }
-    if let Some(pubkey) = &config.public_key {
-        let mut pkh: [u8; 64] = [0; 64];
-        pubkey.write_hex(&mut pkh).unwrap();
-        rid.push(',');
-        rid.push_str("\"pubkey\":\"");
-        rid.push_str(unsafe { std::str::from_utf8_unchecked(pkh.as_slice()) });
-        rid.push('\"');
-    }
-    rid.push('}');
-
-    rid
-}

src/web/blossom/auth.rs

@@ -0,0 +1,145 @@
+use crate::error::{ChorusError, Error};
+use base64::prelude::*;
+use http::header::AUTHORIZATION;
+use hyper::body::Incoming;
+use hyper::Request;
+use pocket_types::Event;
+
+fn s_err(s: &str) -> Result<AuthData, Error> {
+    Err(ChorusError::BlossomAuthFailure(s.to_owned()).into())
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum AuthVerb {
+    Upload,
+    List,
+    Delete,
+    Mirror,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct AuthData {
+    /// If a verb was included, this is it
+    pub verb: Option<AuthVerb>,
+
+    /// If an 'x' tag was included, this is the hash
+    pub hash: Option<[u8; 32]>,
+}
+
+pub fn verify_auth(request: &Request<Incoming>) -> Result<AuthData, Error> {
+    // Force every other error into a BlossomAuthFailure error
+    match verify_auth_inner(request) {
+        Ok(ad) => Ok(ad),
+        Err(e) => match e.inner {
+            ChorusError::BlossomAuthFailure(_) => Err(e),
+            _ => Err(ChorusError::BlossomAuthFailure(format!("{e}")).into()),
+        },
+    }
+}
+
+fn verify_auth_inner(request: &Request<Incoming>) -> Result<AuthData, Error> {
+    // Must have AUTHORIZATION header
+    let authz = match request.headers().get(AUTHORIZATION) {
+        Some(h) => h,
+        None => return s_err("Authorization Required"),
+    };
+
+    // Authorization header must be type "nostr"
+    if !authz.to_str()?.to_ascii_lowercase().starts_with("nostr ") {
+        return s_err("You must use the Nostr authorization scheme");
+    }
+
+    let base64 = match authz.to_str()?.get(6..) {
+        Some(x) => x,
+        None => return s_err("Missing auth base64 encoded event"),
+    };
+
+    // Authorization header must be base64
+    let event_bytes = BASE64_STANDARD.decode(base64)?;
+
+    // Authorization header base64 must decode to a nostr Event
+    let mut buffer = vec![0; base64.len()];
+    let (_size, event) = Event::from_json(&event_bytes, &mut buffer)?;
+
+    // Nostr event must be valid
+    if let Err(e) = event.verify() {
+        return s_err(&format!("Authorization event is invalid: {}", e));
+    }
+
+    // Nostr event must be signed by a chorus user
+    if !crate::is_authorized_user(event.pubkey()) {
+        return s_err("You are not an authorized user");
+    }
+
+    // Event kind must be 24242
+    if event.kind().as_u16() != 24242 {
+        return s_err("Authorization event not kind 24242");
+    }
+
+    // Event created_at must be in the past (we give 30 seconds leeway)
+    use pocket_types::Time;
+    let now = Time::now();
+    if event.created_at() > now + 30 {
+        return s_err("Authorization event too far in the future");
+    }
+
+    let tags = event.tags()?;
+
+    // Expiration tag must be in the future
+    if let Some(v) = tags.get_value(b"expiration") {
+        let u = parse_u64(v)?;
+        let expiration = Time::from_u64(u);
+        if expiration < now {
+            return s_err("Authorization event has expired");
+        }
+    } else {
+        return s_err("Authorization event missing expiration tag");
+    }
+
+    // We let the caller check the verb and hash since those are specific
+    // to the endpoint (and the 'x' must be checked later on)
+
+    let verb: Option<AuthVerb> = if let Some(t) = tags.get_value(b"t") {
+        if t == b"upload" {
+            Some(AuthVerb::Upload)
+        } else if t == b"list" {
+            Some(AuthVerb::List)
+        } else if t == b"delete" {
+            Some(AuthVerb::Delete)
+        } else {
+            None
+        }
+    } else {
+        None
+    };
+
+    let hash: Option<[u8; 32]> = if let Some(v) = tags.get_value(b"x") {
+        let vec = hex::decode(v)?;
+        if vec.len() == 32 {
+            Some(vec.try_into().unwrap())
+        } else {
+            return s_err("Authorization event x tag is of the wrong length");
+        }
+    } else {
+        None
+    };
+
+    Ok(AuthData { verb, hash })
+}
+
+// FIXME, expose these from pocket-types
+fn parse_u64(input: &[u8]) -> Result<u64, Error> {
+    let mut pos = 0;
+    let mut value: u64 = 0;
+    let mut any: bool = false;
+    while pos < input.len() && b"0123456789".contains(&input[pos]) {
+        any = true;
+        value = (value * 10) + (input[pos] - 48) as u64;
+        pos += 1;
+    }
+    if !any {
+        Err(ChorusError::General("Auth event expiration is not a number".to_string()).into())
+    } else {
+        Ok(value)
+    }
+}

src/web/blossom/mod.rs

@@ -0,0 +1,360 @@
+use crate::error::{ChorusError, Error};
+use crate::filestore::HashOutput;
+use crate::globals::GLOBALS;
+use http::header::{
+    ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS, ACCESS_CONTROL_ALLOW_ORIGIN,
+    ACCESS_CONTROL_REQUEST_HEADERS, ACCESS_CONTROL_REQUEST_METHOD, ALLOW, CONTENT_LENGTH,
+    CONTENT_TYPE, ETAG, IF_MATCH, IF_MODIFIED_SINCE, IF_NONE_MATCH, ORIGIN, WWW_AUTHENTICATE,
+};
+use http::{Method, StatusCode};
+//ACCEPT, AUTHORIZATION, DATE, ETAG, ORIGIN
+use http_body_util::combinators::BoxBody;
+use http_body_util::{BodyExt, Empty, Full};
+use hyper::body::{Bytes, Incoming};
+use hyper::{Request, Response};
+use serde::{Deserialize, Serialize};
+
+mod auth;
+use auth::{verify_auth, AuthVerb};
+
+pub async fn handle(request: Request<Incoming>) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    match route(request).await {
+        Ok(response) => Ok(response),
+        Err(e) => match e.inner {
+            ChorusError::SignalNotBlossom => Err(e),
+            _ => error_response(e),
+        },
+    }
+}
+
+pub async fn route(request: Request<Incoming>) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    let p = request.uri().path();
+    #[allow(clippy::int_plus_one)]
+    if p.starts_with("/")
+        && p.len() >= 1 + 64
+        && p.chars().skip(1).take(64).all(|c| c.is_ascii_hexdigit())
+    {
+        handle_hash(request).await
+    } else if p == "/upload" {
+        handle_upload(request).await
+    } else if p.starts_with("/list/")
+        && p.len() >= 6 + 64
+        && p.chars().skip(6).take(64).all(|c| c.is_ascii_hexdigit())
+    {
+        handle_list(request).await
+    } else if p == "/mirror" {
+        handle_mirror(request).await
+    } else {
+        Err(ChorusError::SignalNotBlossom.into())
+    }
+}
+
+fn error_response(e: Error) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    use std::io::ErrorKind;
+
+    let mut response = Response::builder().header(ACCESS_CONTROL_ALLOW_ORIGIN, "*");
+
+    let (status, reason) = match e.inner {
+        ChorusError::BlossomAuthFailure(m) => {
+            response = response.header(WWW_AUTHENTICATE, "Nostr");
+            (StatusCode::UNAUTHORIZED, m)
+        }
+        ChorusError::FromHex(_) => (StatusCode::BAD_REQUEST, format!("{e}")),
+        ChorusError::Io(ref ioerror) => match ioerror.kind() {
+            ErrorKind::NotFound => (StatusCode::NOT_FOUND, "Not Found".to_owned()),
+            _ => (StatusCode::INTERNAL_SERVER_ERROR, format!("{e}")),
+        },
+        _ => (StatusCode::INTERNAL_SERVER_ERROR, format!("{e}")),
+    };
+
+    Ok(response
+        .header("X-Reason", reason)
+        .status(status)
+        .body(Empty::new().map_err(|e| e.into()).boxed())?)
+}
+
+fn options_response(
+    request: Request<Incoming>,
+    methods: &str,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    if request
+        .headers()
+        .contains_key(ACCESS_CONTROL_REQUEST_HEADERS)
+        || request
+            .headers()
+            .contains_key(ACCESS_CONTROL_REQUEST_METHOD)
+        || request.headers().contains_key(ORIGIN)
+    {
+        // CORS OPTIONS response
+        Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(ACCESS_CONTROL_ALLOW_HEADERS, "Authorization, *")
+            .header(ACCESS_CONTROL_ALLOW_METHODS, methods)
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::OK)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?)
+    } else {
+        // Normal OPTIONS response
+        Ok(Response::builder()
+            .header(ALLOW, methods)
+            .status(StatusCode::NO_CONTENT)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?)
+    }
+}
+
+pub async fn handle_hash(
+    request: Request<Incoming>,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    if matches!(request.method(), &Method::OPTIONS) {
+        return options_response(request, "OPTIONS, HEAD, GET, DELETE");
+    }
+
+    // HEAD, GET, DELETE
+    let p = request.uri().path();
+    let hashstr: String = p.chars().skip(1).take(64).collect();
+    let hash = match HashOutput::from_hex(&hashstr) {
+        Ok(h) => h,
+        Err(e) => return error_response(e),
+    };
+
+    let metadata = GLOBALS.filestore.get().unwrap().metadata(hash).await?;
+
+    match *request.method() {
+        Method::HEAD | Method::GET => {
+            // Honor If-Match (fail if they didn't specify an etag matching the hash)
+            if let Some(etags) = request.headers().get(IF_MATCH) {
+                let mut onematch: bool = false;
+                for part in etags.to_str()?.split(',') {
+                    if part[1..part.len() - 1] == format!("{}", hash) {
+                        onematch = true;
+                    }
+                }
+                if !onematch {
+                    return Ok(Response::builder()
+                        .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+                        .header(ETAG, format!("\"{}\"", hash))
+                        .status(StatusCode::PRECONDITION_FAILED)
+                        .body(Empty::new().map_err(|e| e.into()).boxed())?);
+                }
+            }
+
+            // Honor If-None-Match (send NOT_MODIFIED if they specified an etag matching the hash)
+            // Honor If-Modified-Since (always send NOT_MODIFIED)
+            let mut send_not_modified: bool = false;
+            if let Some(etags) = request.headers().get(IF_NONE_MATCH) {
+                for part in etags.to_str()?.split(',') {
+                    if part[1..part.len() - 1] == format!("{}", hash) {
+                        send_not_modified = true;
+                    }
+                }
+            }
+            if request.headers().get(IF_MODIFIED_SINCE).is_some() {
+                send_not_modified = true;
+            }
+            if send_not_modified {
+                return Ok(Response::builder()
+                    .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+                    .header(ETAG, format!("\"{}\"", hash))
+                    .status(StatusCode::NOT_MODIFIED)
+                    .body(Empty::new().map_err(|e| e.into()).boxed())?);
+            }
+
+            // Normal reasponse (HEAD or GET)
+            let response = Response::builder()
+                .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+                .header(CONTENT_LENGTH, format!("{}", metadata.len()))
+                .header(ETAG, format!("\"{}\"", hash))
+                .status(StatusCode::OK);
+
+            if matches!(*request.method(), Method::GET) {
+                let body = GLOBALS.filestore.get().unwrap().retrieve(hash).await?;
+                Ok(response.body(body)?)
+            } else {
+                Ok(response.body(Empty::new().map_err(|e| e.into()).boxed())?)
+            }
+        }
+        Method::DELETE => {
+            let auth_data = verify_auth(&request)?;
+            if auth_data.verb != Some(AuthVerb::Delete) {
+                return Err(ChorusError::BlossomAuthFailure(
+                    "Delete was not authorized".to_string(),
+                )
+                .into());
+            }
+
+            GLOBALS.filestore.get().unwrap().delete(hash).await?;
+            Ok(Response::builder()
+                .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+                .header(CONTENT_LENGTH, "0")
+                .status(StatusCode::OK)
+                .body(Empty::new().map_err(|e| e.into()).boxed())?)
+        }
+        _ => Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::METHOD_NOT_ALLOWED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?),
+    }
+}
+
+pub async fn handle_upload(
+    request: Request<Incoming>,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    if matches!(request.method(), &Method::OPTIONS) {
+        return options_response(request, "OPTIONS, HEAD, PUT");
+    }
+
+    let auth_data = verify_auth(&request)?;
+    if auth_data.verb != Some(AuthVerb::Upload) {
+        return Err(
+            ChorusError::BlossomAuthFailure("Upload was not authorized".to_string()).into(),
+        );
+    }
+
+    match *request.method() {
+        Method::HEAD => Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::NOT_IMPLEMENTED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?),
+        Method::PUT => {
+            let expected_hash = auth_data.hash.map(HashOutput::from_bytes);
+            if expected_hash.is_none() {
+                return Err(ChorusError::BlossomAuthFailure(
+                    "Put requires an expected hash value x tag in the authorization event"
+                        .to_string(),
+                )
+                .into());
+            }
+
+            let uri = request.uri().to_owned();
+
+            let maybe_content_type = match request.headers().get(http::header::CONTENT_TYPE) {
+                Some(s) => match s.to_str() {
+                    Ok(s) => Some(s.to_owned()),
+                    Err(_) => None,
+                },
+                None => None,
+            };
+
+            let (size, hash, maybe_sniffed_mime_string) = GLOBALS
+                .filestore
+                .get()
+                .unwrap()
+                .store(
+                    request.into_body().map_err(|e| e.into()).boxed(),
+                    expected_hash,
+                )
+                .await?;
+
+            let extension = {
+                let mut mime_string: String = "".to_owned();
+                if let Some(ms) = maybe_content_type {
+                    mime_string = ms.to_owned();
+                } else if let Some(ms) = maybe_sniffed_mime_string {
+                    mime_string = ms.to_owned();
+                }
+
+                mime2ext::mime2ext(&mime_string).unwrap_or("blob")
+            };
+
+            let uri = {
+                let mut parts = GLOBALS.config.read().uri_parts(uri, true)?;
+                parts.path_and_query = Some(http::uri::PathAndQuery::from_maybe_shared(format!(
+                    "/{}.{}",
+                    hash, extension
+                ))?);
+                http::Uri::from_parts(parts)?
+            };
+
+            let blob_descriptor = BlobDescriptor {
+                url: format!("{}", uri),
+                sha256: format!("{}", hash),
+                size,
+                uploaded: pocket_types::Time::now().as_u64(),
+            };
+
+            let descriptor_json_string = serde_json::to_string(&blob_descriptor)?;
+            let body_bytes = descriptor_json_string.into_bytes();
+            let len = body_bytes.len();
+            let body = Full::new(Bytes::from(body_bytes))
+                .map_err(|e| e.into())
+                .boxed();
+
+            Ok(Response::builder()
+                .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+                .header(CONTENT_LENGTH, format!("{}", len))
+                .header(CONTENT_TYPE, "application/json")
+                .status(StatusCode::OK)
+                .body(body)?)
+        }
+        _ => Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::METHOD_NOT_ALLOWED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?),
+    }
+}
+
+pub async fn handle_list(
+    request: Request<Incoming>,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    if matches!(request.method(), &Method::OPTIONS) {
+        return options_response(request, "OPTIONS, GET");
+    }
+
+    let auth_data = verify_auth(&request)?;
+    if auth_data.verb != Some(AuthVerb::List) {
+        return Err(ChorusError::BlossomAuthFailure("List was not authorized".to_string()).into());
+    }
+
+    match *request.method() {
+        Method::GET => Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::NOT_IMPLEMENTED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?),
+        _ => Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::METHOD_NOT_ALLOWED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?),
+    }
+}
+
+pub async fn handle_mirror(
+    request: Request<Incoming>,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    if matches!(request.method(), &Method::OPTIONS) {
+        return options_response(request, "OPTIONS, PUT");
+    }
+
+    let auth_data = verify_auth(&request)?;
+    if auth_data.verb != Some(AuthVerb::Mirror) {
+        return Err(
+            ChorusError::BlossomAuthFailure("Mirror was not authorized".to_string()).into(),
+        );
+    }
+
+    match *request.method() {
+        Method::PUT => Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::NOT_IMPLEMENTED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?),
+        _ => Ok(Response::builder()
+            .header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
+            .header(CONTENT_LENGTH, "0")
+            .status(StatusCode::METHOD_NOT_ALLOWED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?),
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BlobDescriptor {
+    pub url: String,
+    pub sha256: String,
+    pub size: u64,
+    // type: String
+    pub uploaded: u64,
+}

src/web/management/auth.rs

@@ -0,0 +1,116 @@
+use crate::error::{ChorusError, Error};
+use crate::globals::GLOBALS;
+use base64::prelude::*;
+use http::header::AUTHORIZATION;
+use http_body_util::BodyExt;
+use hyper::body::Incoming;
+use hyper::Request;
+use pocket_types::{Event, Pubkey};
+use secp256k1::hashes::{sha256, Hash};
+use serde_json::Value;
+
+fn s_err(s: &str) -> Result<(Pubkey, Value), Error> {
+    Err(ChorusError::ManagementAuthFailure(s.to_owned()).into())
+}
+
+pub async fn check_auth(request: Request<Incoming>) -> Result<(Pubkey, Value), Error> {
+    // Must be POST
+    if request.method() != hyper::Method::POST {
+        return s_err("Management RPC only supports POST method");
+    }
+
+    // Must have AUTHORIZATION header
+    let authz = match request.headers().get(AUTHORIZATION) {
+        Some(h) => h,
+        None => return s_err("Authorization header not of type nostr"),
+    };
+
+    // Authorization header must be type "nostr"
+    let value = String::from_utf8(authz.as_bytes().to_owned())?;
+    let mut parts = value.split(' ');
+    match parts.next() {
+        Some(s) => {
+            if s.to_lowercase() != "nostr" {
+                return s_err("Authorization header not of type nostr");
+            }
+        }
+        None => return s_err("Authorization header missing"),
+    }
+
+    // Authorization header second part
+    let base64event = match parts.next() {
+        Some(s) => s,
+        None => return s_err("Authorization header incomplete"),
+    };
+
+    // Authorization header must be base64
+    let event_bytes = BASE64_STANDARD.decode(base64event)?;
+
+    // Authorization header base64 must decode to a nostr Event
+    let mut buffer = vec![0; base64event.len()];
+    let (_size, event) = Event::from_json(&event_bytes, &mut buffer)?;
+
+    // Nostr event must be valid
+    if let Err(e) = event.verify() {
+        return s_err(&format!("Authorization event is invalid: {}", e));
+    }
+
+    // Nostr event must be signed by a moderator
+    if !crate::is_moderator(event.pubkey()) {
+        return s_err("Authorization failed as user is not a moderator");
+    }
+
+    // Event kind must be 27235
+    if event.kind().as_u16() != 27235 {
+        return s_err("Authorization event not kind 27235");
+    }
+
+    // Event created_at must be within 60 seconds of now
+    use pocket_types::Time;
+    let now = Time::now().as_u64();
+    if event.created_at().as_u64() > now + 60 {
+        return s_err("Authorization event too far in the future");
+    }
+    if event.created_at().as_u64() < now - 60 {
+        return s_err("Authorization event too far in the past");
+    }
+
+    let tags = event.tags()?;
+
+    // Tag 'u' must be the current URL
+    if let Some(u) = tags.get_value(b"u") {
+        let auth_url = String::from_utf8(u.to_owned())?;
+        let actual_url = {
+            let uri_parts = GLOBALS
+                .config
+                .read()
+                .uri_parts(request.uri().to_owned(), true)?;
+            let uri = http::Uri::from_parts(uri_parts)?;
+            format!("{}", uri)
+        };
+
+        if actual_url != auth_url {
+            return s_err(&format!(
+                "Authorization event URL {} does not match requqest URL {}",
+                auth_url, actual_url
+            ));
+        }
+    } else {
+        return s_err("Authorization event URL missing");
+    }
+
+    let body = request.collect().await?.to_bytes();
+    let hash = sha256::Hash::hash(&body);
+    let hashref = <sha256::Hash as AsRef<[u8]>>::as_ref(&hash);
+    let hashrefhex = hex::encode(hashref);
+
+    if let Some(payload) = tags.get_value(b"payload") {
+        if hashrefhex.as_bytes() != payload {
+            return s_err("Authorization failed: body hash mismatch");
+        }
+    } else {
+        return s_err("Authorization event payload missing");
+    }
+
+    Ok((event.pubkey(), serde_json::from_slice(&body)?))
+}

src/web/management/mod.rs

@@ -0,0 +1,615 @@
+use crate::error::{ChorusError, Error};
+use crate::globals::GLOBALS;
+use crate::ip::HashedPeer;
+use http_body_util::combinators::BoxBody;
+use http_body_util::{BodyExt, Full};
+use hyper::body::{Bytes, Incoming};
+use hyper::{Request, Response, StatusCode};
+use pocket_db::ScreenResult;
+use pocket_types::{Event, Filter, Id, Kind, Pubkey};
+use serde::Serialize;
+use serde_json::{json, Map, Value};
+mod auth;
+
+#[derive(Serialize)]
+struct EventResult {
+    id: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    reason: Option<String>,
+}
+
+#[derive(Serialize)]
+struct FullEventResult {
+    event: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    reason: Option<String>,
+}
+
+#[derive(Serialize)]
+struct PubkeyResult {
+    pubkey: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    reason: Option<String>,
+}
+
+fn respond(
+    json: serde_json::Value,
+    status: StatusCode,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    let s: String = serde_json::to_string(&json)?;
+    let response = Response::builder()
+        .header("Access-Control-Allow-Origin", "*")
+        .header("Access-Control-Allow-Headers", "Authorization, *")
+        .header("Access-Control-Allow-Methods", "*")
+        .header("Content-Type", "application/nostr+json+rpc")
+        .status(status)
+        .body(
+            Full::new(s.into_bytes().into())
+                .map_err(|e| e.into())
+                .boxed(),
+        )?;
+    Ok(response)
+}
+
+pub async fn handle(
+    _peer: HashedPeer,
+    request: Request<Incoming>,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    let (pubkey, command) = match auth::check_auth(request).await {
+        Ok((pk, v)) => (pk, v),
+        Err(e) => {
+            let result = json!({
+                "result": {},
+                "error": format!("{}", e)
+            });
+            return respond(result, StatusCode::UNAUTHORIZED);
+        }
+    };
+
+    match handle_inner(pubkey, command) {
+        Ok(Some(value)) => respond(value, StatusCode::OK),
+        Ok(None) => {
+            let result = json!({
+                "result": {},
+            });
+            respond(result, StatusCode::OK)
+        }
+        Err(e) => {
+            let (result, status) = match e.inner {
+                ChorusError::BadRequest(s) => (
+                    json!({
+                        "result": {},
+                        "error": format!("{}", s)
+                    }),
+                    StatusCode::BAD_REQUEST,
+                ),
+                ChorusError::NotImplemented => (
+                    json!({
+                        "result": {},
+                        "error": "not_implemented"
+                    }),
+                    StatusCode::NOT_IMPLEMENTED,
+                ),
+                _ => (
+                    json!({
+                        "result": {},
+                        "error": format!("{}", e)
+                    }),
+                    StatusCode::INTERNAL_SERVER_ERROR,
+                ),
+            };
+            respond(result, status)
+        }
+    }
+}
+
+pub fn handle_inner(pubkey: Pubkey, command: Value) -> Result<Option<Value>, Error> {
+    let obj = match command.as_object() {
+        Some(o) => o,
+        None => return Err(ChorusError::BadRequest("Command was not a JSON object").into()),
+    };
+
+    let method = match obj.get("method") {
+        Some(m) => match m.as_str() {
+            Some(s) => s.to_owned(),
+            None => return Err(ChorusError::BadRequest("Method not a string").into()),
+        },
+        None => return Err(ChorusError::BadRequest("Method missing").into()),
+    };
+
+    match &*method {
+        "supportedmethods" => Ok(Some(json!({
+            "result": [
+                "supportedmethods",
+
+                "listeventsneedingmoderation",
+
+                "allowevent",
+                "banevent",
+                "clearevent",
+                "removeevent",
+
+                "allowpubkey",
+                "banpubkey",
+                "clearpubkey",
+
+                "listallowedevents",
+                "listbannedevents",
+                "fetchbannedevents",
+                "listallowedpubkeys",
+                "listbannedpubkeys",
+
+                "stats",
+                "numconnections",
+                "uptime",
+
+                "listadmins",
+                "listmoderators",
+                "grantmoderator",
+                "revokemoderator",
+                "listusers",
+                "grantuser",
+                "revokeuser",
+                "listrole",
+                "grantrole",
+                "revokerole",
+            ]
+        }))),
+        "listeventsneedingmoderation" => {
+            // FIXME this scans the entire database, maybe we need to some process
+            // that does this in epochs and saves the result.
+
+            let allowed_kinds = [
+                Kind::from(4),     // Encrypted Direct Message
+                Kind::from(1059),  // Giftwrap
+                Kind::from(10002), // Relay list
+                Kind::from(10050), // DM Relay list
+                Kind::from(0),     // Metadata
+                Kind::from(3),     // Following list
+                Kind::from(7),     // Reaction
+            ];
+            let mut buffer: [u8; 128] = [0; 128];
+            let filter = {
+                let (_incount, _outcount, filter) = Filter::from_json(b"{}", &mut buffer)?;
+                filter
+            };
+            let screen = |e: &Event| -> ScreenResult {
+                if allowed_kinds.contains(&e.kind())
+                    || e.kind().is_ephemeral()
+                    || crate::is_authorized_user(e.pubkey())
+                {
+                    ScreenResult::Mismatch
+                } else {
+                    ScreenResult::Match
+                }
+            };
+
+            let mut need_moderation: Vec<EventResult> = Vec::new();
+
+            let (mut events, _redacted) = GLOBALS
+                .store
+                .get()
+                .unwrap()
+                .find_events(filter, true, 0, 0, screen)?;
+
+            for event in events.drain(..) {
+                // Skip if pubkey marked (either banned or approved)
+                if matches!(crate::get_pubkey_approval(event.pubkey()), Ok(Some(_))) {
+                    continue;
+                }
+
+                // Skip if event marked (either banned or approved)
+                if matches!(crate::get_event_approval(event.id()), Ok(Some(_))) {
+                    continue;
+                }
+
+                need_moderation.push(EventResult {
+                    id: event.id().as_hex_string(),
+                    reason: Some("unmoderated".to_string()),
+                });
+            }
+
+            Ok(Some(json!({
+                "result": need_moderation,
+            })))
+        }
+        "allowevent" => {
+            let id = get_id_param(obj)?;
+            crate::mark_event_approval(id, true)?;
+            Ok(None)
+        }
+        "banevent" => {
+            let id = get_id_param(obj)?;
+            crate::mark_event_approval(id, false)?;
+            Ok(None)
+        }
+        "clearevent" => {
+            let id = get_id_param(obj)?;
+            crate::clear_event_approval(id)?;
+            Ok(None)
+        }
+        "removeevent" => {
+            let id = get_id_param(obj)?;
+            GLOBALS.store.get().unwrap().remove_event(id)?;
+            Ok(None)
+        }
+
+        "allowpubkey" => {
+            let pk = get_pubkey_param(obj)?;
+            crate::mark_pubkey_approval(pk, true)?;
+            Ok(None)
+        }
+        "banpubkey" => {
+            let pk = get_pubkey_param(obj)?;
+            crate::mark_pubkey_approval(pk, false)?;
+            Ok(None)
+        }
+        "clearpubkey" => {
+            let pk = get_pubkey_param(obj)?;
+            crate::clear_pubkey_approval(pk)?;
+            Ok(None)
+        }
+
+        "listallowedevents" => {
+            let approvals = crate::dump_event_approvals()?;
+            let ids: Vec<EventResult> = approvals
+                .iter()
+                .filter_map(|(id, appr)| {
+                    if *appr {
+                        Some(EventResult {
+                            id: id.as_hex_string(),
+                            reason: None,
+                        })
+                    } else {
+                        None
+                    }
+                })
+                .collect();
+            Ok(Some(json!({
+                "result": ids
+            })))
+        }
+        "listbannedevents" => {
+            let approvals = crate::dump_event_approvals()?;
+            let ids: Vec<EventResult> = approvals
+                .iter()
+                .filter_map(|(id, appr)| {
+                    if *appr {
+                        None
+                    } else {
+                        Some(EventResult {
+                            id: id.as_hex_string(),
+                            reason: None,
+                        })
+                    }
+                })
+                .collect();
+            Ok(Some(json!({
+                "result": ids
+            })))
+        }
+        "fetchbannedevents" => {
+            let approvals = crate::dump_event_approvals()?;
+            let mut results: Vec<FullEventResult> = Vec::new();
+            for (id, appr) in approvals.iter() {
+                if !*appr {
+                    if let Some(event) = GLOBALS.store.get().unwrap().get_event_by_id(*id)? {
+                        results.push(FullEventResult {
+                            event: format!("{event}"),
+                            reason: None,
+                        });
+                    }
+                }
+            }
+            Ok(Some(json!({
+                "result": results
+            })))
+        }
+        "listallowedpubkeys" => {
+            let approvals = crate::dump_pubkey_approvals()?;
+            let pubkeys: Vec<PubkeyResult> = approvals
+                .iter()
+                .filter_map(|(pk, appr)| {
+                    if *appr {
+                        Some(PubkeyResult {
+                            pubkey: pk.as_hex_string(),
+                            reason: None,
+                        })
+                    } else {
+                        None
+                    }
+                })
+                .collect();
+            Ok(Some(json!({
+                "result": pubkeys
+            })))
+        }
+        "listbannedpubkeys" => {
+            let approvals = crate::dump_pubkey_approvals()?;
+            let pubkeys: Vec<PubkeyResult> = approvals
+                .iter()
+                .filter_map(|(pk, appr)| {
+                    if *appr {
+                        None
+                    } else {
+                        Some(PubkeyResult {
+                            pubkey: pk.as_hex_string(),
+                            reason: None,
+                        })
+                    }
+                })
+                .collect();
+            Ok(Some(json!({
+                "result": pubkeys
+            })))
+        }
+
+        "stats" => {
+            let store_stats = GLOBALS.store.get().unwrap().stats()?;
+            Ok(Some(json!({
+                "result": {
+                    "uptime": GLOBALS.start_time.elapsed().as_secs(),
+                    "num_connections": &GLOBALS.num_connections,
+                    "bytes_received": &GLOBALS.bytes_inbound,
+                    "bytes_sent": &GLOBALS.bytes_outbound,
+                    "event_bytes": store_stats.event_bytes,
+                    "num_events": store_stats.index_stats.i_index_entries,
+                    "index_disk_usage": store_stats.index_stats.disk_usage,
+                    "index_memory_usage": store_stats.index_stats.memory_usage,
+                }
+            })))
+        }
+        "numconnections" => {
+            let num = &GLOBALS.num_connections;
+            Ok(Some(json!({
+                "result": num,
+            })))
+        }
+        "uptime" => {
+            let uptime_in_secs = GLOBALS.start_time.elapsed().as_secs();
+            Ok(Some(json!({
+                "result": uptime_in_secs,
+            })))
+        }
+
+        "listadmins" => {
+            let keys = GLOBALS.config.read().admin_hex_keys.clone();
+            Ok(Some(json!({
+                "result": keys
+            })))
+        }
+        "listmoderators" => {
+            let moderators: Vec<String> = crate::dump_authorized_users()?
+                .iter()
+                .filter_map(|(pk, moderator)| {
+                    if *moderator {
+                        Some(pk.as_hex_string())
+                    } else {
+                        None
+                    }
+                })
+                .collect();
+            Ok(Some(json!({
+                "result": moderators
+            })))
+        }
+        "grantmoderator" => {
+            if !crate::is_admin(pubkey) {
+                Ok(Some(json!({
+                    "result": {},
+                    "error": "Unauthorized: Only admins can grant moderator status"
+                })))
+            } else {
+                let pk = get_pubkey_param(obj)?;
+                crate::add_authorized_user(pk, true)?;
+                Ok(None)
+            }
+        }
+        "revokemoderator" => {
+            if !crate::is_admin(pubkey) {
+                Ok(Some(json!({
+                    "result": {},
+                    "error": "Unauthorized: Only admins can revoke moderator status"
+                })))
+            } else {
+                let pk = get_pubkey_param(obj)?;
+
+                // Do not do this if they aren't already an authorized user
+                if !crate::is_authorized_user(pk) {
+                    Ok(None)
+                } else {
+                    crate::add_authorized_user(pk, false)?;
+                    Ok(None)
+                }
+            }
+        }
+        "listusers" => {
+            let users: Vec<String> = crate::dump_authorized_users()?
+                .iter()
+                .map(|(pk, _moderator)| pk.as_hex_string())
+                .collect();
+            Ok(Some(json!({
+                "result": users
+            })))
+        }
+        "grantuser" => {
+            if !crate::is_admin(pubkey) {
+                Ok(Some(json!({
+                    "result": {},
+                    "error": "Unauthorized: Only admins can grant user status"
+                })))
+            } else {
+                let pk = get_pubkey_param(obj)?;
+                crate::add_authorized_user(pk, false)?;
+                Ok(None)
+            }
+        }
+        "revokeuser" => {
+            if !crate::is_admin(pubkey) {
+                Ok(Some(json!({
+                    "result": {},
+                    "error": "Unauthorized: Only admins can revoke user status"
+                })))
+            } else {
+                let pk = get_pubkey_param(obj)?;
+                crate::rm_authorized_user(pk)?;
+                Ok(None)
+            }
+        }
+        "listrole" => {
+            if !crate::is_admin(pubkey) {
+                Ok(Some(json!({
+                    "result": {},
+                    "error": "Unauthorized: Only admins can revoke user status"
+                })))
+            } else {
+                let role = get_string_param(obj)?;
+                match &*role {
+                    "admin" => {
+                        let keys = GLOBALS.config.read().admin_hex_keys.clone();
+                        Ok(Some(json!({
+                            "result": keys
+                        })))
+                    }
+                    "user" => {
+                        let users: Vec<String> = crate::dump_authorized_users()?
+                            .iter()
+                            .map(|(pk, _moderator)| pk.as_hex_string())
+                            .collect();
+                        Ok(Some(json!({
+                            "result": users
+                        })))
+                    }
+                    "moderator" => {
+                        let moderators: Vec<String> = crate::dump_authorized_users()?
+                            .iter()
+                            .filter_map(|(pk, moderator)| {
+                                if *moderator {
+                                    Some(pk.as_hex_string())
+                                } else {
+                                    None
+                                }
+                            })
+                            .collect();
+                        Ok(Some(json!({
+                            "result": moderators
+                        })))
+                    }
+                    _ => Ok(Some(json!({
+                        "result": {},
+                        "error": "Unknown role."
+                    }))),
+                }
+            }
+        }
+        "grantrole" => {
+            if !crate::is_admin(pubkey) {
+                Ok(Some(json!({
+                    "result": {},
+                    "error": "Unauthorized: Only admins can grant user status"
+                })))
+            } else {
+                let role = get_string_param(obj)?;
+                match &*role {
+                    "admin" => Ok(Some(json!({
+                        "result": {},
+                        "error": "The administrators cannot be managed via this interface."
+                    }))),
+                    "user" => {
+                        let pk = get_pubkey_param(obj)?;
+                        crate::add_authorized_user(pk, false)?;
+                        Ok(None)
+                    }
+                    "moderator" => {
+                        let pk = get_pubkey_param(obj)?;
+                        crate::add_authorized_user(pk, true)?;
+                        Ok(None)
+                    }
+                    _ => Ok(Some(json!({
+                        "result": {},
+                        "error": "Unknown role."
+                    }))),
+                }
+            }
+        }
+        "revokerole" => {
+            if !crate::is_admin(pubkey) {
+                Ok(Some(json!({
+                    "result": {},
+                    "error": "Unauthorized: Only admins can grant user status"
+                })))
+            } else {
+                let role = get_string_param(obj)?;
+                match &*role {
+                    "admin" => Ok(Some(json!({
+                        "result": {},
+                        "error": "The administrators cannot be managed via this interface."
+                    }))),
+                    "user" => {
+                        let pk = get_pubkey_param(obj)?;
+                        crate::rm_authorized_user(pk)?;
+                        Ok(None)
+                    }
+                    "moderator" => {
+                        let pk = get_pubkey_param(obj)?;
+
+                        // Do not do this if they aren't already an authorized user
+                        if !crate::is_authorized_user(pk) {
+                            Ok(None)
+                        } else {
+                            crate::add_authorized_user(pk, false)?;
+                            Ok(None)
+                        }
+                    }
+                    _ => Ok(Some(json!({
+                        "result": {},
+                        "error": "Unknown role."
+                    }))),
+                }
+            }
+        }
+
+        _ => Err(ChorusError::NotImplemented.into()),
+    }
+}
+
+fn get_pubkey_param(obj: &Map<String, Value>) -> Result<Pubkey, Error> {
+    let pubkey_text = obj
+        .get("params")
+        .ok_or(ChorusError::BadRequest("Params field missing").into_err())?
+        .as_array()
+        .ok_or(ChorusError::BadRequest("Params not an array").into_err())?
+        .first()
+        .ok_or(ChorusError::BadRequest("Missing pubkey parameter").into_err())?
+        .as_str()
+        .ok_or(ChorusError::BadRequest("Pubkey parameter is wrong type").into_err())?;
+    Pubkey::read_hex(pubkey_text.as_bytes())
+        .map_err(|_| ChorusError::BadRequest("Pubkey could not be parsed").into_err())
+}
+
+fn get_id_param(obj: &Map<String, Value>) -> Result<Id, Error> {
+    let id_text = obj
+        .get("params")
+        .ok_or(ChorusError::BadRequest("Params field missing").into_err())?
+        .as_array()
+        .ok_or(ChorusError::BadRequest("Params not an array").into_err())?
+        .first()
+        .ok_or(ChorusError::BadRequest("Missing ID parameter").into_err())?
+        .as_str()
+        .ok_or(ChorusError::BadRequest("ID parameter is wrong type").into_err())?;
+    Id::read_hex(id_text.as_bytes())
+        .map_err(|_| ChorusError::BadRequest("ID could not be parsed").into_err())
+}
+
+fn get_string_param(obj: &Map<String, Value>) -> Result<String, Error> {
+    Ok(obj
+        .get("params")
+        .ok_or(ChorusError::BadRequest("Params field missing").into_err())?
+        .as_array()
+        .ok_or(ChorusError::BadRequest("Params not an array").into_err())?
+        .first()
+        .ok_or(ChorusError::BadRequest("Missing parameter").into_err())?
+        .as_str()
+        .ok_or(ChorusError::BadRequest("Parameter is not a string as expected").into_err())?
+        .to_owned())
+}

src/web/mod.rs

@@ -0,0 +1,133 @@
+mod blossom;
+mod management;
+mod nip11;
+
+use crate::error::{ChorusError, Error};
+use crate::globals::GLOBALS;
+use crate::ip::HashedPeer;
+use http::Method;
+use http_body_util::combinators::BoxBody;
+use http_body_util::{BodyExt, Empty, Full};
+use hyper::body::{Bytes, Incoming};
+use hyper::{Request, Response, StatusCode};
+
+pub async fn serve_http(
+    peer: HashedPeer,
+    request: Request<Incoming>,
+) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    let method = request.method().clone();
+
+    // Handle server-wide OPTIONS requests
+    let p = request.uri().path();
+    if p == "*" && request.method() == Method::OPTIONS {
+        let response = Response::builder()
+            .header("Access-Control-Allow-Origin", "*")
+            .header("Access-Control-Allow-Headers", "Authorization, *")
+            .header(
+                "Access-Control-Allow-Methods",
+                "OPTIONS, GET, HEAD, PUT, DELETE",
+            )
+            .header("Allow", "OPTIONS, GET, HEAD, PUT, DELETE")
+            .status(StatusCode::OK)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?;
+        return Ok(response);
+    }
+
+    // Check if it is a NIP-11 request
+    if let Some(accept) = request.headers().get("Accept") {
+        if let Ok(s) = accept.to_str() {
+            if s == "application/nostr+json" {
+                return nip11::serve_nip11(peer).await;
+            }
+        }
+    }
+
+    // Check if it is a NIP-86 Relay Management request
+    if let Some(content_type) = request.headers().get("Content-Type") {
+        if let Ok(s) = content_type.to_str() {
+            if s == "application/nostr+json+rpc" {
+                return management::handle(peer, request).await;
+            }
+        }
+    }
+
+    let uri = request.uri().to_owned();
+
+    if p == "/privacy-policy" {
+        let config = &*GLOBALS.config.read();
+        if let Some(pp) = &config.privacy_policy {
+            let response = Response::builder()
+                .header("Access-Control-Allow-Origin", "*")
+                .header("Access-Control-Allow-Headers", "Authorization, *")
+                .header("Access-Control-Allow-Methods", "*")
+                .header("Allow", "OPTIONS, GET, HEAD")
+                .header("Content-Type", "text/plain")
+                .status(StatusCode::OK)
+                .body(Full::new(pp.clone().into()).map_err(|e| e.into()).boxed())?;
+            return Ok(response);
+        }
+    }
+
+    if p == "/terms-of-service" {
+        let config = &*GLOBALS.config.read();
+        if let Some(tos) = &config.terms_of_service {
+            let response = Response::builder()
+                .header("Access-Control-Allow-Origin", "*")
+                .header("Access-Control-Allow-Headers", "Authorization, *")
+                .header("Access-Control-Allow-Methods", "*")
+                .header("Allow", "OPTIONS, GET, HEAD")
+                .header("Content-Type", "text/plain")
+                .status(StatusCode::OK)
+                .body(Full::new(tos.clone().into()).map_err(|e| e.into()).boxed())?;
+            return Ok(response);
+        }
+    }
+
+    // Try blossom if enabled
+    if GLOBALS.config.read().blossom_directory.is_some() {
+        match blossom::handle(request).await {
+            Ok(response) => return Ok(response),
+            Err(e) => {
+                if !matches!(e.inner, ChorusError::SignalNotBlossom) {
+                    return Err(e);
+                }
+            }
+        }
+    }
+
+    log::debug!(target: "Client", "{}: HTTP request for {}", peer, uri);
+
+    let response = match method {
+        Method::OPTIONS => Response::builder()
+            .header("Access-Control-Allow-Origin", "*")
+            .header("Access-Control-Allow-Headers", "Authorization, *")
+            .header("Access-Control-Allow-Methods", "*")
+            .header("Allow", "OPTIONS, GET, HEAD, PUT, DELETE")
+            .status(StatusCode::NO_CONTENT)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?,
+        Method::HEAD => Response::builder()
+            .header("Access-Control-Allow-Origin", "*")
+            .header("Access-Control-Allow-Headers", "Authorization, *")
+            .header("Access-Control-Allow-Methods", "*")
+            .status(StatusCode::OK)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?,
+        Method::GET => Response::builder()
+            .header("Access-Control-Allow-Origin", "*")
+            .header("Access-Control-Allow-Headers", "Authorization, *")
+            .header("Access-Control-Allow-Methods", "*")
+            .status(StatusCode::OK)
+            .body(
+                Full::new("This is a nostr relay. Please use a nostr client to connect.".into())
+                    .map_err(|e| e.into())
+                    .boxed(),
+            )?,
+        _ => Response::builder()
+            .header("Access-Control-Allow-Origin", "*")
+            .header("Access-Control-Allow-Headers", "Authorization, *")
+            .header("Access-Control-Allow-Methods", "*")
+            .status(StatusCode::METHOD_NOT_ALLOWED)
+            .body(Empty::new().map_err(|e| e.into()).boxed())?,
+    };
+
+    Ok(response)
+}

src/web/nip11.rs

@@ -0,0 +1,174 @@
+use crate::config::Config;
+use crate::error::Error;
+use crate::globals::GLOBALS;
+use crate::ip::HashedPeer;
+use http_body_util::combinators::BoxBody;
+use http_body_util::{BodyExt, Full};
+use hyper::body::Bytes;
+use hyper::http::uri::Uri;
+use hyper::{Response, StatusCode};
+
+pub async fn serve_nip11(peer: HashedPeer) -> Result<Response<BoxBody<Bytes, Error>>, Error> {
+    log::debug!(target: "Client", "{}: sent NIP-11", peer);
+    let rid = {
+        let config = &*GLOBALS.config.read();
+        GLOBALS.rid.get_or_init(|| build_rid(config))
+    };
+
+    let response = Response::builder()
+        .header("Access-Control-Allow-Origin", "*")
+        .header("Access-Control-Allow-Headers", "*")
+        .header("Access-Control-Allow-Methods", "*")
+        .header("Content-Type", "application/nostr+json")
+        .status(StatusCode::OK)
+        .body(Full::new(rid.clone().into()).map_err(|e| e.into()).boxed())?;
+    Ok(response)
+}
+
+fn build_rid(config: &Config) -> String {
+    let mut rid: String = String::with_capacity(255);
+
+    const SUPPORTED_NIPS: [u8; 9] = [
+        1,  // nostr
+        4,  // DMs
+        9,  // Event Deletion
+        11, // relay information document
+        40, // Expiration Timestamp
+        42, // AUTH
+        45, // Counting results
+        59, // GiftWrap
+        65, // Relay List Metadata
+    ];
+    const _UNSUPPORTED_NIPS: [u8; 5] = [
+        26, // Delegated Event Signing
+        29, // Relay-based Groups
+        50, // SEARCH
+        94, // File Metadata
+        96, // HTTP File Storage Integration
+    ];
+    const _INAPPLICABLE_NIPS: [u8; 45] = [
+        2, 3, 5, 6, 7, 8, 10, 13, 14, 15, 18, 19, 21, 23, 24, 25, 27, 28, 30, 31, 32, 34, 36, 38,
+        39, 44, 46, 47, 48, 49, 51, 52, 53, 56, 57, 58, 72, 75, 78, 84, 89, 90, 92, 98, 99,
+    ];
+
+    let s = SUPPORTED_NIPS
+        .iter()
+        .map(|i| format!("{}", i))
+        .collect::<Vec<String>>()
+        .join(",");
+    rid.push_str(&format!("{{\"supported_nips\":[{}],", s));
+
+    let software = env!("CARGO_PKG_NAME");
+    rid.push_str("\"software\":\"");
+    rid.push_str(software);
+    rid.push('\"');
+
+    let version = env!("CARGO_PKG_VERSION");
+    rid.push(',');
+    rid.push_str("\"version\":\"");
+    rid.push_str(version);
+    rid.push('\"');
+
+    if let Some(name) = &config.name {
+        rid.push(',');
+        rid.push_str("\"name\":\"");
+        rid.push_str(name);
+        rid.push('\"');
+    }
+    if let Some(description) = &config.description {
+        rid.push(',');
+        rid.push_str("\"description\":\"");
+        rid.push_str(description);
+        rid.push('\"');
+    }
+    if let Some(banner_url) = &config.banner_url {
+        rid.push(',');
+        rid.push_str("\"banner\":\"");
+        rid.push_str(banner_url);
+        rid.push('\"');
+    }
+    if let Some(icon_url) = &config.icon_url {
+        rid.push(',');
+        rid.push_str("\"icon\":\"");
+        rid.push_str(icon_url);
+        rid.push('\"');
+    }
+    if let Some(pubkey) = &config.contact_public_key {
+        let mut pkh: [u8; 64] = [0; 64];
+        pubkey.write_hex(&mut pkh).unwrap();
+        rid.push(',');
+        rid.push_str("\"pubkey\":\"");
+        rid.push_str(unsafe { std::str::from_utf8_unchecked(pkh.as_slice()) });
+        rid.push('\"');
+    }
+    if let Some(contact) = &config.contact {
+        rid.push(',');
+        rid.push_str("\"contact\":\"");
+        rid.push_str(contact);
+        rid.push('\"');
+    }
+    if config.privacy_policy.is_some() {
+        rid.push(',');
+        rid.push_str("\"privacy_policy\":\"");
+        let url = match config.uri_parts(
+            Uri::from_static("https://authority-will-be-replaced/privacy-policy"),
+            true,
+        ) {
+            Ok(parts) => match Uri::from_parts(parts) {
+                Ok(uri) => format!("{}", uri),
+                Err(_) => "".to_owned(),
+            },
+            Err(_) => "".to_owned(),
+        };
+        rid.push_str(&url);
+        rid.push('\"');
+    }
+    if config.terms_of_service.is_some() {
+        rid.push(',');
+        rid.push_str("\"terms_of_service\":\"");
+        let url = match config.uri_parts(
+            Uri::from_static("https://authority-will-be-replaced/terms-of-service"),
+            true,
+        ) {
+            Ok(parts) => match Uri::from_parts(parts) {
+                Ok(uri) => format!("{}", uri),
+                Err(_) => "".to_owned(),
+            },
+            Err(_) => "".to_owned(),
+        };
+        rid.push_str(&url);
+        rid.push('\"');
+    }
+
+    // Limitation
+    rid.push(',');
+    rid.push_str("\"limitation\":{");
+    {
+        rid.push_str("\"payment_required\":false,\"auth_required\":false,\"restricted_writes\":true,\"max_message_length\":1048576");
+        rid.push_str(&format!(
+            ",\"max_subscriptions\":{}",
+            config.max_subscriptions
+        ));
+    }
+    rid.push('}');
+
+    // Retention
+    rid.push(',');
+    rid.push_str("\"retention\":[{\"time\": null}]");
+
+    // Services
+    rid.push(',');
+    rid.push_str("\"services\":{");
+    rid.push_str("\"public\":[\"ephemeral\",\"directory\"]");
+    rid.push(',');
+    rid.push_str("\"private\":[\"outbox\",\"inbox\"]");
+    rid.push(',');
+    rid.push_str("\"paid\":[]");
+    rid.push(',');
+    rid.push_str("\"unavailable\":[\"search\"]");
+    rid.push('}');
+
+    rid.push('}');
+
+    rid
+}

test_blossom/.gitignore

@@ -0,0 +1,3 @@
+event.map
+lmdb/
+downloaded

test_blossom/README.md

@@ -0,0 +1,7 @@
+# Testing Blossom functionality of Chorus
+
+In one shell run `run.sh` to run a local chorus.
+
+Then in another shell run `test.sh` to run tests against that running instance.
+
+When done, break out of the server and run `clean.sh`

test_blossom/blossom/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

test_blossom/clean.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+rm -rf ./event.map ./lmdb/ ./blossom/* ./downloaded

test_blossom/config.toml

@@ -0,0 +1,34 @@
+# See contrib/chorus.toml for a documented config file
+
+data_directory = "."
+ip_address = "127.0.0.1"
+port = 8089
+hostname = "localhost"
+chorus_is_behind_a_proxy = false
+use_tls = false
+certchain_pem_path = "tls/fullchain.pem"
+key_pem_path = "tls/privkey.pem"
+name = "Chorus Sample"
+description = "A sample run of the Chorus relay"
+# icon_url =
+open_relay = false
+admin_hex_keys = [
+  "12bb541d03bfc3cab0f4a8e4db28947f60faae6fca4e315eb27f809c6eff9a0b"
+]
+verify_events = true
+allow_scraping = false
+allow_scrape_if_limited_to = 100
+allow_scrape_if_max_seconds = 7200
+max_subscriptions = 128
+serve_ephemeral = true
+serve_relay_lists = true
+server_log_level = "Debug"
+library_log_level = "Debug"
+client_log_level = "Debug"
+enable_ip_blocking = true
+minimum_ban_seconds = 1
+timeout_seconds = 60
+max_connections_per_ip = 5
+throttling_bytes_per_second = 131072
+throttling_burst = 4194304
+blossom_directory = "./blossom"

test_blossom/create_auth.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+PUBKEY=12bb541d03bfc3cab0f4a8e4db28947f60faae6fca4e315eb27f809c6eff9a0b
+PRIVKEY=b4a98d96270b6cd30c80e4fd594461d2b22d8dbcfbcd1f7b11bf0ef2b028a56b
+AUTH_EXPIRATION=1900000000
+
+VERB=$1
+HASH=$2
+
+if [ x$VERB = x ] ; then
+    echo "USAGE:  create_auth.sh VERB HASH"
+    exit 1
+fi
+
+if [ x$HASH = x ] ; then
+    echo "USAGE:  create_auth.sh VERB HASH"
+    exit 1
+fi
+
+PRE_EVENT='{"pubkey": "'$PUBKEY'", "kind": 24242, "created_at": 0, "tags": [["expiration","'$AUTH_EXPIRATION'"], ["t","'$VERB'"], ["x","'$HASH'"]], "content":""}'
+
+EVENT=$(echo "$PRE_EVENT" | nak event --sec $PRIVKEY)
+
+EVENT_BASE64=$(echo $EVENT | base64 -w 0)
+
+echo "Authorization: Nostr $EVENT_BASE64"

test_blossom/run.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+pushd ..
+cargo build --release || exit 1
+popd
+../target/release/chorus ./config.toml

test_blossom/test.sh

@@ -0,0 +1,84 @@
+#!/bin/bash
+
+if ! command -v curl 2>&1 >/dev/null
+then
+    echo "curl command is required."
+    exit 1
+fi
+
+if ! command -v jq 2>&1 >/dev/null
+then
+    echo "jq command is required."
+    exit 1
+fi
+
+if ! command -v nak 2>&1 >/dev/null
+then
+    echo "nak command is required. https://github.com/fiatjaf/nak"
+    exit 1
+fi
+
+# ADD ADMIN AS A USER ------------
+
+echo "Adding user..."
+PUBKEY=12bb541d03bfc3cab0f4a8e4db28947f60faae6fca4e315eb27f809c6eff9a0b
+../target/release/chorus_cmd ./config.toml add_user $PUBKEY 0
+
+# UPLOAD TEST ------------
+
+FILE="./Example.png"
+HASH=$(sha256sum $FILE | awk '{print $1}')
+
+# Generate nostr auth
+AUTH=$(./create_auth.sh upload $HASH)
+
+# Upload (note we clobber the content type)
+DESCRIPTOR=$(curl -vfs --data-binary @"$FILE" -H "Content-Type: " -X PUT --header "$AUTH" http://127.0.0.1:8089/upload)
+if [ $? -ne 0 ] ; then
+    echo "FAILED: Curl (uploading) exited with a non-zero status"
+    exit 1
+fi
+echo "PASS:  FILE UPLOADED"
+
+## FIXME check for 4xx and 5xx error codes
+
+
+# Extract the sha256 and compare it
+DHASH=$(echo "$DESCRIPTOR" | jq -r .sha256)
+if [ $? -ne 0 ] ; then
+    echo "FAILED: jq failed extracting sha256 from descriptor"
+    exit 1
+fi
+if [ "$HASH" != "$DHASH" ] ; then
+    echo "returned descriptor 'sha256' does not match the hash"
+    exit 1
+fi
+echo "PASS:  DESCRIPTOR HASH MATCHES"
+
+# Extract the URL for download
+URL=$(echo "$DESCRIPTOR" | jq -r .url)
+if [ $? -ne 0 ] ; then
+    echo "FAILED: jq failed extracting url from descriptor"
+    exit 1
+fi
+
+echo "Descriptor URL = $URL"
+
+# DOWNLOAD TEST -----------
+
+curl -s "$URL" > downloaded
+if [ $? -ne 0 ] ; then
+    echo "FAILED: Curl (downloading) exited with a non-zero status"
+    exit 1
+fi
+echo "PASS:  FILE DOWNLOADED"
+
+# Compare the files
+if cmp -s "$FILE" downloaded; then
+    echo "PASS:  THE DOWNLOADED FILE MATCHES THE UPLOADED FILE"
+else
+    echo "FAIL:  THE DOWNLOADED FILE DOES NOT MATCH THE UPLOADED FILE"
+fi
+
+echo "end."
+exit 0

test_with_relay_tester/.gitignore

@@ -0,0 +1,3 @@
+data
+relay-tester
+

test_with_relay_tester/README.md

@@ -0,0 +1,12 @@
+# Testing Chorus with relay-tester
+
+First, git clone https://github.com/mikedilger/relay-tester and build that project
+(cargo build --release).
+
+Then copy that target/release/relay-tester binary into this directory.
+
+Then run from two different shells, in this order:
+
+shell1:  ./test_chorus.sh
+
+shell2:  ./run_relay_tester.sh

test_with_relay_tester/run_relay_tester.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+if [ ! -x relay-tester ] ; then
+    echo "You must build https://github.com/mikedilge/relay-tester and copy the "
+    echo "resultant target/release/relay-tester binary into this directory."
+    exit 1
+fi
+
+./relay-tester \
+    ws://localhost:8080/ \
+    nsec16xfd467kyd3xpu9x5u4933u00v73xrl0jyq9rk5ktd9t2j38k20qtwxuj3 \
+    nsec1l50yuf6uxm2l5qxm87fkm56z3m7g88jnfy5s6az5wscxpu5l2yqq6qwk88

test_with_relay_tester/test_chorus.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+echo "Use ws://localhost:8080/ as the relay url"
+
+cargo build --release
+rm -rf ./data/
+../target/release/chorus_init ./test_chorus.toml
+../target/release/chorus_cmd ./test_chorus.toml add_user de16d3ed2d5ceb91d33e39dbe30585164e0c19f3f2e2a5b121def086b447a2e5 0
+../target/release/chorus_cmd ./test_chorus.toml add_user 35d6bbcf17fc31a9c4f7a2f68aa40ad32c8f9de1ae77505dc5eb3722d8b2987d 0
+../target/release/chorus ./test_chorus.toml

test_with_relay_tester/test_chorus.toml

@@ -0,0 +1,40 @@
+# See contrib/chorus.toml for a documented config file
+
+data_directory = "./data"
+ip_address = "127.0.0.1"
+port = 8080
+hostname = "localhost"
+chorus_is_behind_a_proxy = false
+use_tls = false
+certchain_pem_path = "tls/fullchain.pem"
+key_pem_path = "tls/privkey.pem"
+name = "Chorus Sample"
+description = "A sample run of the Chorus relay"
+#icon_url =
+open_relay = false
+admin_hex_keys = [
+  "ee11a5dff40c19a555f41fe42b48f00e618c91225622ae37b6c2bb67b76c4e49",
+
+  # npub1mctd8mfdtn4er5e788d7xpv9ze8qcx0n7t32tvfpmmcgddz85tjsuyxe7z
+  "de16d3ed2d5ceb91d33e39dbe30585164e0c19f3f2e2a5b121def086b447a2e5",
+
+  # npub1xhtthnchlsc6n38h5tmg4fq26vkgl80p4em4qhw9avmj9k9jnp7sm78ql6
+  "35d6bbcf17fc31a9c4f7a2f68aa40ad32c8f9de1ae77505dc5eb3722d8b2987d"
+]
+verify_events = true
+allow_scraping = false
+allow_scrape_if_limited_to = 100
+allow_scrape_if_max_seconds = 7200
+max_subscriptions = 128
+serve_ephemeral = true
+serve_relay_lists = true
+server_log_level = "Info"
+library_log_level = "Info"
+client_log_level = "Debug"
+enable_ip_blocking = false
+minimum_ban_seconds = 1
+timeout_seconds = 60
+max_connections_per_ip = 5
+throttling_bytes_per_second = 131072
+throttling_burst = 4194304
+enable_negentropy = true