- Fetch all pages included in manual
- Also fetch pages with + in name
- Generate raw xml from raw wiki files
- Don't fetch raw xml for manual pages
- Delete manual raw xml at end of fetch
- Iterate manual pages over raw wiki files instead of raw xml
- Skip remove-footer step for generated manual xml pages
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
[sunil: Don't remove intermediate targets during build]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
data/var/log and data/var/run were not being used for a while.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Inspired by: https://salsa.debian.org/vexch/plinth/snippets/391
This script creates a simple container using systemd-nspawn for developing
FreedomBox. It has many advantages over running a VM using Vagrant. RAM is
allocated to processes in the container as needed without any fixed limit. Also
RAM does not have to be statically allocated so it is typically much lighter
than running an VM. There is no hardware emulation when running a container with
same architecture, so processes run as fast as they would on the host machine.
Environment: The script will run only run on hosts having systemd-nspawn and
network-manager installed, typical GNU/Linux distributions. It has been
primarily developed and tested on Debian Buster but should work on most modern
GNU/Linux distributions.
Disk image: systemd-nspawn accepts not only a directory for starting a container
but also a disk image. This disk image is loop-back mounted and container is
started from that mounted directory. The partition to use is determined by
looking at the boot flag in the partition table. This happens to work well with
all existing FreedomBox images. In future, we may be able to run different
architectures in this manner.
After downloading, the disk image is expanded along with the partition and file
system inside so that development can be done without running into disk space
issues. Expanding the disk does not immediately consume disk space because it
will be a sparse file. As data is written to the disk, it will occupy more and
more space but the upper limit is the size to which disk has been expanded.
Downloading images: Images are downloaded from FreedomBox download server using
fixed URLs for each distribution. Signature is verified for the download images.
The fingerprint of the allowed signing key is hard-coded in this script.
Downloaded images are kept even after destroying the extracted raw image along
with container. This allows for quickly resetting the container without
downloading again.
Booting: systemd-nspawn is run in 'boot' mode. This means that init process
(happens to be systemd) is started inside the container. It then spawns all the
other necessary daemons including openssh-server, firewalld and network-manager.
A login terminal can be opened using 'machinectl login' because container is
running systemd. SSH into the container is possible because network is up,
configured by network-manager, and openssh server is running.
Shared folder: Using systemd-nspawn, the project directory is mounted as
/freedombox inside the container. The project directory is determined as
directory in which this script resides. The project folder from the container
point of view will be read-only. Container should be able to write various files
such as build files, FreedomBox sqlite3 database and session files into the
/freedombox folder. To enable writing, an additional read-write folder is
overlayed onto /freedombox folder in the container. This directory can't be
created under the project folder and is created instead in
$XDG_DATA_HOME/freedombox-container/overlay/$DISTRIBUTION. If XDG_DATA_HOME is
not set, it is assumed to be $HOME/.local/shared/. Whenever data is written into
/freedombox directory inside the container, this directory on the host receives
the changes. See documentation for Overlay filesystem for further details. When
container is destroyed, this overlay folder is destroyed to ensure clean state
after bringing up the container again.
Users: PrivateUsers configuration flag for systemd-nspawn is currently off. This
means that each user's UID on the host is also the same UID in the container as
along as there is an entry in the container's password database. In future, we
may explore using private users inside the container.
'fbx' is the development user and its UID is changed during setup phase to 10000
hoping it would not match anything on the host system. 'fbx' user has full sudo
access inside the container without needing a password. Password for this user
is not set by default, but can be set if needed. If there is no access to the
container in any way, one can run 'sudo machinectl shell' and then run 'passwd
fbx' to set the password for the 'fbx' user.
'plinth' user's UID in the container is also changed and set to the UID of
whichever user owns the project directory. This allows the files to written by
'plinth' container user in the project directory because UID of the owner of the
directory is same as the 'plinth' user's UID in container.
Network: A private network is created inside the container using systemd-nspawn
feature. Network interfaces from the host are not available inside the
container. A new network interface called 'host0' is configured inside the
container which is automatically configured by network-manager. On the host a
new network interface is created. This script creates configuration for a
'shared' network using network-manager. When bringing up the container, this
network connection is also brought up. A DHCP server and a DNS server are
started network-manager on the host side so that DHCP and DNS client functions
work inside the container. Traffic from the container is also masqueraded so
that Internet connectivity inside the container works if the host has one.
If necessary, the network interface on host side can be differently configured.
For example, it can be bridged with another interface to expose the container on
a network that the host machine participates in.
The network IP address inside the container can be queried using machinectl.
This script queries that IP address and presents the address in its console
messages. All ports in the container can be reached from the host using this IP
address as long as the firewall inside the container allows it. There is no need
to perform port forwarding or mapping.
SSH: It is assumed that openssh-server is installed inside the container. SSH
server keys in the container are created if missing. Client side keys are
created in .container/ssh directory and the public key is installed in the
authorized keys file of the 'fbx' user. The 'ssh' sub-command to this script is
simply a convenience mechanism for quick launch of ssh with the right IP
address, user name and identity file.
Role of machinectl: Most of the work is done by systemd-nspawn. machinectl is
useful for running systemd-nspawn in the background and querying its current
state. It also helps with providing the IP address of the container. machinectl
is made to recognize the container by creating a link in /var/lib/machines/ to
the image file. systemd-nspawn options are added by creating a temporary file in
/run/systemd/nspawn. All machinectl commands should work.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
- Secret is important for various functions of Django. There is no impact on
existing installations due to the change. Improves the security of existing
functions in minor ways and will be useful in future usage of Django.
- Create the file in /var/lib/plinth/ with 0o600 permissions.
- Make git ignore the file in code folder.
- Don't copy the file during './setup.py install' operation.
Impact to users after upgrade:
- All existing sessions will get logged out. This is because SECRET_KEY is used
to generate user session hash that is used to logout users when their password
changes.
Tests performed:
- Run development version of service. File should get created in
data/var/lib/plinth/django-secret.key. Permissions should be 0o600.
- Run again, the file should not be overwritten. Printing
django.conf.settings.SECRET_KEY should match the one in the file.
- Run `setup.py install`. This should not install django-secret.key in
/var/lib/plinth.
- Run `sudo -u plinth plinth`. This should create the secret key file in
/var/lib/plinth/django-secret.key. Permissions on the file should be 0o600.
Ownership should be plinth:plinth.
- Remove the file in both cases, a fresh new file should get created with new key.
- Truncate the file to less than 128 chars, the existing file should get
overwritten with new key.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This is completely reworked but based on /Developer page in the FreedomBox
Manual.
- This documentation can be made available as static site on
https://docs.freedombox.org and the /Developer page in the FreedomBox Manual
can be dropped.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
[sunil@medhas.org Have common top level makefile for building manual]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Create option --include-functional to run functional tests. Otherwise, they
are disabled by default. If pytest-bdd is not installed, functional tests are
not discovered at all.
- Make pytest-django discover the setting files by creating dummy manage.py in
top level directory.
- Make pytest run as './setup.py pytest'. Add alias from './setup.py test'. This
requires pytest-runner package.
- Merge .gitignore files from functional_tests/
- Update gitlab-ci.yml to run tests with coverage using pytest.
- Update HACKING.md to suggest using py.test-3 instead of old way of running.
Merge functional tests README.md into HACKING.md.
- Remove execution wrapper runtests.py as pytest-django is able to configure
Django settings before execution of tests. Update tests to explicitly ask for
Django database as database access is denied by default.
- Replace usage of python3-coverage with python3-pytest-coverage. Execution
wrappers are not required.
- Add build dependencies on pytest modules.
- Let all warnings be shown after running tests.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Created basic plinth app which starts an introducer and a storage
node on the FreedomBox.
- Prompt user to set a domain name before creating Tahoe-LAFS nodes.
- Support adding and removing of introducers to the storage node.
- Serve Tahoe-LAFS from a different port.
- Start all nodes and introducers at system startup.
- Add utility class YAMLFile with test cases.
- Install mod_auth_pubtkt and generate public/private key-pair.
- Redirect user to login page if no cookie is presented.
- Add check for authenticated user for login page.
- Temporarily switched to DSA because of a bug in mod_auth_pubtkt
which causes it to accept only DSA and not RSA. Also had to use SHA1
instead of SHA256.
- Enabled SSO for Syncthing, Repro and TT-RSS.
- Using tokens to authorize by user groups.
- Generate keys during first boot.
- Rename AdminMiddleware to AdminRequiredMiddleware to be consistent
with stronghold/Django terminology
- Simplify .gitignore pattern
- Format single line docstrings as per PEP8.
- Add missing docstrings.
- Restrict lines to 79 characters.
The generated TODO file is very limted in scope and gives a wrong
impression to a potential contributor. We have worked towards a central
TODO list for the entire project scrapping all other TODO lists.
Redirect the contributor to that list.
- Implement coverage functionality as a setuptools Command subclass
(in plinth/tests/coverage/test_coverage.py)
- Register the new 'test_coverage' command in setup.py
- Modify the INSTALL file to specify python3-coverage as a new dependency
- Modify the HACKING file to document the new 'test_coverage' command
- Have git (via .gitignore) ignore the '.coverage' output data file and
the contents of the 'plinth/tests/coverage/report' directory
- Store users using Django user/group/permission model
- Database is data/plinth.sqlite3 instead of data/user.sqlite3
- Use Django auth context processors in templates
The makefile has three new targets:
1. hosting: Publish repository and automagically build the archives to
serve to clients.
2. current-checkout.tar.gz: Create an archive of the current project
directory.
3. current-repository.tar.gz: Create an archive of the current project
directory along with the source repository metadata so that the
archive is a full checkout of the project.
