- Since we have important fixes deployed that we would like to get effected
immediately.
Tests:
- Not tested.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Add Full Text Search capability to Dovecot.
- Add 'dovecot-fts-xapian' to the list of packages for the email app.
- Add relevant configs for both dovecot 2.3 and 2.4
- Add a systemd timer to periodically clean search indexes
Configurations taken from plugin's upstream documentation:
https://github.com/grosjo/fts-xapian
Sunil:
- Tweak the dovecot 2.4 configuration. Remove explicit configuration same as or
close to default values.
- Drop the timer service for cleaning up the index. Dovecot documentation that
FTS plugins do it themselves.
- Drop the re-indexing command on setup. This could not be properly tested. On
first search, indexes will be created for mailboxes that don't have them.
Tests done:
- Perform a fresh install, on both Bookworm and Trixie, confirm the install is
successful, confirm the systemd service runs with exit 0.
- On Bookworm, apply the patches on an existing setup, confirm the patches apply
as expected.
- On a production like setup, set dovecot 2.4 to debug mode and check the
journal logs while receiving an email: The logs confirm that the fts module is
loaded and that it automatically creates a db for the indexes. I also opened the
newly created db file with less and confirmed that the human readable parts
contain my recent email.
- Using Sogo, perform a full search (including headers and body). Search works
and indexes are freshly created on all the folders.
Signed-off-by: Benedek Nagy <contact@nbenedek.me>
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- This helps during distribution upgrade from dovecot 2.3 to 2.4. Dovecot will
stop running due to dovecot server 2.4 not understanding version 2.3
configuration files. When setup is re-run, starting the daemons again is the
right thing to do.
Tests:
- With email app installed, upgrade from bookworm to trixie. Dovecot is stopped
during distribution upgrade but after freedombox service runs, it recovers and
starts running again.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Sunil:
- When dovecot package is upgrade from 2.3 to 2.4 during distribution upgrade,
automatically re-run setup.
- Upgrade existing setups to new scheme by re-running setup with incremented app
version.
- Don't query dovecot version during app initialization. Instead overwrite the
DropinConfigs component to query dovecot version during setup and enable
operations.
- Use apt.Cache() to retrieve the installed version of dovecot package. Use
plinth.utils.Version to parse the version and perform a comparison.
- Split even configuration files that have not changed for simplicity.
- Add/update links in Dovecot configuration files.
Tests:
- Install email app on a testing container. Ensure that all files in
/etc/dovecot/conf.d/ are linked properly to 2.4 versions. TLS configuration is
accurate. Use Sogo to test login and sending mails.
- User with LDAP account and correct password is able to login.
- User without LDAP account or incorrect password is unable to login.
- Send mail with Sogo to another account on the server. Notice that mails are
stored in /var/mail/{user}/mail/ with mail:mail ownership in mbox format.
- Logging in with email such as user@example.com works. Capital letters are
allowed.
- "Archive", "Drafts", "Sent", "Junk", "Trash" folders are automatically
created and are marked with special flags. Creating additional folders such
as "Sent Items" also results in them having special flags.
- Thunderbird is able to connect via SSL with a self-signed certificate
exception.
- When an example spam message is sent, it is automatically moved to "Junk"
folder after getting marked by rspamd.
- When a message is moved to Junk folder, it is learned as spam by rspamd as
seen in its admin console.
- When a message is moved out of Junk folder (to other than "Trash" folder),
it is learned as not-spam by rspamd as seen in its admin console.
- Install email app on a stable container with patches. Ensure that all files in
/etc/dovecot/conf.d/ are linked properly to 2.3 versions. TLS configuration is
accurate. Use Sogo to test login and sending mails.
- Install email app on a stable container without patches. Apply patches. Ensure
that all files in /etc/dovecot/conf.d/ are linked properly to 2.3 versions. TLS
configuration is accurate for dovecot 2.3. Use Sogo to test login and sending
mails. Perform distribution upgrade to testing. Ensure that all files in
/etc/dovecot/conf.d/ are linked properly to 2.3 versions. TLS configuration is
accurate for dovecot 2.4. Use Sogo to test login and sending mails.
Signed-off-by: Benedek Nagy <contact@nbenedek.me>
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
The changes introduced in https://salsa.debian.org/freedombox-team/freedombox/-/merge_requests/2616
make the email app installation fail if it's installed for the first
time.
This patch ensures that 95-freedombox-sieve.conf is present and that
dovecot is restarted before privileged.setup_spam() is run. This way,
the sievec command can run.
Signed-off-by: Benedek Nagy <contact@nbenedek.me>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Add two sieve scripts for spam/ham learning. When the user moves a mail
from anywhere to junk, or from junk to anywhere (except for trash) the
mail is piped into the respective rspamc learn_spam/learn_ham command.
The rspamc command is run as the mail user and the command requires that
the user can connect to localhost:11334. Because of that, add the mail
user to the allowed users that can access protected services.
The sievec compilation of the new scripts requre the dovecot-antispam
package, so install it and increment the email version number.
Closes: #2487
Imroves: #56
Tests done:
1. Apply the patches on an existing install
2. Confirm the firewall and the email app get updated
3. Move a mail from inbox to junk and confirm that rspamd statistics for
"Learned" mails increment by one.
4. Move back the mail from junk to inbox and confirm the number
increments again.
5. Move the mail to trash and confirm the script doesn't execute.
6. Repeat steps 3-5 with mail_debug = yes in /etc/dovecot/dovecot.conf
and confirm the script esxecution further by reading the debug logs.
[Sunil]
- Split the configuration file 90-freedombox-sieve.conf into
90-freedombox-imap.conf and merge the remaining with 95-freedombox-sieve.conf.
- These changes do not need dovecot-anitspam package. Remove it from packages
list for the app.
Signed-off-by: Benedek Nagy <contact@nbenedek.me>
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- In tags cleanup code, the front page shortcut for email uses different tags
than those provided in the manifest. When the app is instantiated in privileged
code without setting up Django, this causes an error even though gettext_noop is
used. Workaround by using a custom method for this.
Tests:
- Installing, enabling/disabling email apps works.
- 'make update-translations' output shows that django.pot has been updated and
the tags in question are part of the POT file.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Add typing information for init methods Info, Shortcut, and Menu to easily
identify problems.
- Update docstrings for these components.
- Updates test cases to deal with tags instead of short description.
- Update custom shortcuts code to read tags and ignore short description.
- Update API to send tags instead of custom shortcuts.
- OpenVPN special treatment of info.description in Shortcut
Tests:
- All unit tests pass and type checking succeeds.
- All apps show icons with tags in apps and system section.
- In help section cards don't show tags.
- In front page, enabled apps show shortcuts with tags.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@riseup.net>
See the spam score for an email sent with these settings:
https://www.mail-tester.com/test-jy6unbdzu
Tests done:
1. Install the email app with version 4, then check out the files:
```
ls -la /var/lib/rspamd/dkim/ ; cat /etc/rspamd/local.d/dkim_signing.conf
total 4
dr-x------ 1 _rspamd _rspamd 50 Jan 1 19:14 .
drwxr-x--- 1 _rspamd _rspamd 16936 Jan 1 20:21 ..
-r-------- 1 root root 1704 Jan 1 19:14 freedombox.local.dkim.key
cat: /etc/rspamd/local.d/dkim_signing.conf: No such file or directory
```
2. Apply the patches and update the app:
```
ls -la /var/lib/rspamd/dkim/ ; cat /etc/rspamd/local.d/dkim_signing.conf
total 4
dr-x------ 1 _rspamd _rspamd 50 Jan 1 19:14 .
drwxr-x--- 1 _rspamd _rspamd 16936 Jan 1 20:22 ..
-r-------- 1 _rspamd _rspamd 1704 Jan 1 19:14 freedombox.local.dkim.key
allow_username_mismatch = true;
sign_authenticated = true;
use_domain = "header";
domain {
freedombox.local {
path = "/var/lib/rspamd/dkim/freedombox.local.dkim.key";
selector = "dkim";
}
}
```
3. Configure example.com as a domain under Name Services, then also change the primary domain in the email app and confirm it completes without errors.
Also see:
```
ls -la /var/lib/rspamd/dkim/ ; cat /etc/rspamd/local.d/dkim_signing.conf
total 8
dr-x------ 1 _rspamd _rspamd 90 Jan 1 21:15 .
drwxr-x--- 1 _rspamd _rspamd 16936 Jan 1 21:17 ..
-r-------- 1 _rspamd _rspamd 1704 Jan 1 21:15 example.com.dkim.key
-r-------- 1 _rspamd _rspamd 1704 Jan 1 19:14 freedombox.local.dkim.key
allow_username_mismatch = true;
sign_authenticated = true;
use_domain = "header";
domain {
example.com {
path = "/var/lib/rspamd/dkim/example.com.dkim.key";
selector = "dkim";
}
}
```
4. Uninstall the app and perform a fresh install, confirm it completes succesfully.
The configurations in /etc/rspamd/local.d/dkim_signing.conf have been
verified to work on a throw-away VPS setup.
Once merged, this should be mentioned here: https://discuss.freedombox.org/t/solved-email-messages-not-signed-with-dkim/2387
Signed-off-by: Benedek Nagy <contact@nbenedek.me>
[sunil: Add comment explaining allow_username_mismatch option]
[sunil: Drop an unused variable, added docstrings]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Fixes#2460
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
[sunil: Update the style to provide spacing after tags lines]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Add tags to Info component of apps. Use only English tags for all operations.
Localized tags are used for presentation to the user only. Add tags to all the
apps. Conventions (English):
1. Tags describing use cases should be in kebab case.
2. Protocols in tag names should be in their canonical format.
3. Tags needn't be 100% technically correct. This can get in the way of
comparing apps using a tag. Words that describe use cases that users can
easily understand should be preferred over being pedantic.
4. Tags should be short, ideally not more than 2 words. Avoid conjunctions
like "and", "or" in tags.
5. Avoid redundant words like "server", or "web-clients". Most apps on
FreedomBox are either servers or web clients.
6. Keep your nouns singular in tags.
- Use query
params to filter the Apps page by tags. When all tags are removed, redirect to /apps.
- Add UI elements to add and remove tag filters in the Apps page. Make the UI
similar to GitLab issue tags. Since there are 40 apps, there will be at least 40
tags. Selecting a tag from a dropdown will be difficult on mobile devices. A
fuzzy search is useful to find tags to add to the filter. Allow user to find the
best match for the search term and highlight it visually. The user can then
press Enter to select the highlighted tag. Make tag search case-insensitive.
Make the dropdown menu scrollable with a fixed size. User input is debounced by
300 ms during search.
- tests: Add missing mock in test_module_loader.py
- Add functional tests
[sunil]
- 'list' can be used instead of 'List' for typing in recent Python versions.
- Reserve tripe-quoted strings for docstrings.
- Undo some changes in module initialization, use module_name for logging
errors.
- isort and yapf changes.
- Encode parameters before adding them to the URL.
Tests:
- Tested the functionality of filtering by tag with one tag and two tags.
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Tests:
- Config app description is as expected.
- Config form does not show domain name field anymore.
- Submitting the form with changes works.
- Names app has correct link for configuring static domain name. Clicking it
takes to page for setting domain name.
- On startup, static domian name signal is sent properly if set. Otherwise no
signal is send.
- Change domain name form shows correct value for current domain name.
- Change domain name form sets the value for domain name properly.
- Page title is correct.
- Validations works.
- Add/remove domain name signals are sent properly.
- Success message as shown expected
- /etc/hosts is updated as expected.
- Unit tests work.
- Functional tests on ejabberd, letsencrypt, matrix, email, jsxc, openvpn
- After freshly starting the service. Visiting names app shows correct list of
domains.
- ejabberd:
- Installs works as expected. Currently set domain_name is setup properly.
Copy certificate happens on proper domain.
- Changing the domain sets the domain properly in ejabberd configuration.
- Ejabberd app page shows link to name services instead of config app.
Clicking works as expected.
- letsencrypt:
- When no domains are configured, the link to 'Configure domains' is to the
names app.
- matrix-synapse:
- Domain name is properly shown in the status.
- email:
- Primary domain name is shows properly in the app page.
- Setting new primary domain works.
- When installing, domain set as static domain name is prioritized as primary
domain.
- jsxc:
- Show the current static domain name in the domain field. BOSH server is
available.
- openvpn:
- Show the current static domain in profile is set otherwise show the current
hostname.
- If domain name is not set, downloaded OpenVPN profile shows hostname.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
- This is so that the methods will be checked by mypy. This should help identify
any incorrect initialization of components.
- Remove unused self.repos in GitwebApp.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This includes the envelope size, so the actual size of the message and
attachments is somewhat lower.
Tests:
- Install the app freshly with the patch. Notice that message_size_limit is set
to 102400000 by running 'postconf message_size_limit'.
- Install the app without the patch. Apply patche and notice that the app is
updated. Notice that message_size_limit is set to 102400000 by running 'postconf
message_size_limit'.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
During installation of email app, sievec is run on
/etc/dovecot/freedombox-sieve-after/sort-spam.sieve file to produce a binary
version of the file. However, this file is not available until after the app is
enabled. This is due to the newly introduced drop-in configuration mechanism. To
fix this, during setup, enable the drop-in configuration component responsible
for this file. The component is newly split from the component that deals with
all drop-in components so that not all the drop-in configuration files are
symlinked.
There is no change needed to the app when the drop-in configuration component is
split into multiple components. This is true for all three state of the app;
not-installed, installed-enabled, and installed-disabled.
Tests:
- Install latest code with ./setup.py install. Install the app and it works.
- Uninstall the app and reinstall. It works. There are no warnings that config
file have been replaced with symlinks.
- /etc/dovecot/freedombox-sieve-after has the sort-span.svbin binary sievec
compiled file.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Warning that exim4 will be removed is incorrectly not shown during email app
installation. Fix that.
Tests:
- Unit tests pass.
- On a fresh testing container, trying to install email app shows the warning
message that exim will be removed. Installing the email app succeeds. exim is
removed during installation.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- When app is freshly installed, nft rules are inserted.
- Trying to connect to local daemon from fbx user fails.
- Functional tests pass.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- DONE: Unit tests work
- DONE: Transmission
- DONE: Enabling/disabling an app with a daemon works: transmission
- DONE: Showing the status of whether the app is enabled with daemon
is-enabled works.
- DONE: A message is shown if app is enabled and service is not running
- DONE: Service is stopped and re-started during backup
- DONE: Adding user to share group during initial setup restarts the service
- Not tested: Enabling/disabling a service with alias works (no such apps)
- DONE: Restarting/try-restarting a service works
- DONE: Masking/unmasking works
- DONE: rsyslog is masked after initial setup
- DONE: systemd-journald is try-restarted during initial setup
- DONE: Avahi, email, security initial setup works
- DONE: Fail2ban is unmasked and enabled
- DONE: Enabling/disabling fail2ban is security app works
- DONE: Enabling/disabling password authentication in SSH works
- ?? Let's encrypt
- Services are try-restarted during certificate setup, obtain, renew
- Not tested: upgrade pagekite from version 1
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Functional tests work (uninstall test does not work)
- Initial setup works
- Domains are setup
- Home is setup (others don't have permission for /var/mail)
- Aliases configuration is setup
- Postfix is setup
- rspamd is setup
- Changing primary domain works
- Adding/removing domains works
- Error during operations is handle properly: getting dkim key
- Setting up DKIM key when changing, adding/removing domain works
- Showing DKIM key in app page works
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
module.app property usage is greatly reduced because setup() and force_upgrade()
method are now part of App class instead of at the module level. Remove the
remaining minor cases of usage and drop the property altogether.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Primary purpose is to complete the App API and allow for multiple apps to be
present in a module without a single clashing setup() method. Secondary
objective is to get rid of SetupHelper instance simple use App instance instead.
- This brings us closer to not needing to implement setup() method for some of
the typical apps.
- Remove default value None for old_version parameter.
- A valid integer value is always passed to this call.
- The value of None is undefined.
- Simplifies the App API slightly.
- Drop setting 'pre', 'post' values to indicate the stage of setup for the App.
- Simplifies the setup methods significantly. Eliminates a class of
bugs (some of them seen earlier).
- The UI can show a simple 'installing...' or progress spinner instead of
individual stages.
- There are currently many inconsistencies where many operations are not
wrapped in helper.call() calls.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Discussed during progress call.
- More testing has happened on the app since making it available.
- Will document difficulties with setting up domain name and ISP outgoing
traffic on port 25.
Tests:
- App is available without the advance flag.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
As discussed during progress calls, this is a pre-condition for making the app
generally available (without advanced flag).
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #1797.
Tests:
- For each affected app, install the app, visit frontpage. Select the shortcut.
Notice that Learn More... link is shown and clicking on the link takes us to the
manual page of the app.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Since FreedomBox does not allow any users but those belonging to 'admin' group
to login, using passwd driver for auth means that only admin can login to
postfix/dovecot. Fix this by reverting to using LDAP driver.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
They are unused. postfix-ldap is needed for LDAP based maps. In dovecot,
authentication happens using PAM.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
There is nothing related to LDAP in the module. It actually configures postfix.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
The module does not interact with kernel audit system or is not related to
security logging. The 'privileged' makes us pay more attension.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
A fresh install of clamav-daemon takes up about 1GiB or RAM. Most of this is the
virus signature database and is used regularly for a scan. This makes ClamAV
unsuitable for FreedomBox running on many single board computers. Drop ClamAV
until we start recommending/requiring at least 2GiB of RAM.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- managesieve plugin is already enabled due to installation of
dovecot-managesieve.
- Using ManageSieve protocol, clients like Thunderbird (with managesieve addon)
will be able to edit mail filters on the server. The server will perform the
filtering enabling all clients to share the benefits.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Many modules in rspamd including the bayes built-in module depend on
redis-server. Install, run and configure redis-server.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
For simplicity and consistency. Eliminate '_' in the name.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
