- setuptools aims to a build library instead of being invoked by setup.py.
Launching setup tools using ./setup.py is deprecated. Launching it using
generic build tools that use pyproject.toml is recommended.
- With the new approach customizing the build is not possible to the earlier
extent. So, introduce is a simple and sufficient build system using 'make'.
Tests:
- Check the pyproject.toml using validate-pyproject tool.
- Run diffoscope on old and new packages and verify that no unexpected changes
were introduced by the build system change.
- None of the files part of .deb package have different file permissions
compared to before.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Check for errors when running the provisioning script.
- Check for errors when installing missing packages.
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
The script detects the system architecture of the Debian machine and
picks the appropriate container images to download and run.
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
This is for arm64 boards like Raspberry Pi and cross-arch VMs.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #2324.
The arm64 image contains two partitions in a GPT partition table. First is the
EFI partition and the second is the root partition. The container script
currently assumes that there will only be one partition in the image file. Fix
this by picking up the partition number of the last partition and resizing that.
GPT partition table also requires relocating the second copy of the partition
table to the end before partition resize can succeed.
Tests:
- Create testing containers in arm64 and amd64 architectures.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- DONE: Functional tests work
- DONE: Initial setup works
- DONE: Automatic upgrades are enable by default
- DONE: apt preferences have been updated
- DONE: Enabling backports works
- DONE: Configuration file is created
- DONE: Correct status is shown in the app page
- DONE: Enabling/disabling automatic upgrades works
- DONE: Configuration file is updated
- DONE: Correct status is shown in the app page
- DONE: Manual triggering of updates work
- DONE: Log is shown properly in the app page
- DONE: Checking for distribution upgrade works
- DONE: Distribution upgrade from stable to testing works
- DONE: When running on btrfs distribution, snapshot is created before.
- DONE: Snapshots will be disable before upgrade and re-enabled later.
- DONE: When searx is enabled before upgrade, it's uwsgi will be disabled and
re-enabled later.
- Failures due to freedombox package not being the latest version (with the
changes).
- DONE: Development Vagrant box
- DONE: Automatic updates are disabled during development setup
- DONE: Development Container
- DONE: Automatic updates are disabled during development setup
- DONE: On stable, backports are enabled when running tests
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Currently privileged actions use stdout for returning the results. If any of the
sub-processes accidentally output to stdout, decoding errors occur. Prevent this
by opening a pipe to the privileged action and returning the output in that
pipe.
Tests:
- Run unit tests
- Functional tests for other apps pass
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
[sunil: Use the default formatter on all the subparsers]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Invoke ssh with the 'IdentitiesOnly` option enabled in order to
force the use of the configured authentication identity. This is
needed in situations where ssh-agent offers many different identities.
Closes#2243
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
This is recommended by PEP-0597: https://peps.python.org/pep-0597/
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This allows the user to understand the wrapper script and skip/adapt it when
necessary.
- Debug any issues with the script.
- Maintain consistency with the philosophy of the rest of the container script.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
This removes the dependencies xvfb and pytest-xvfb.
--splinter-headless can be used as a substitute for running tests in
headless mode.
[sunil: Edit description as running run-tests starts plinth in container]
[sunil: Retain the xauth command]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Currently, with recent version of systemd on host machine, running './container
up' results in the following warning thrown many times. Fix this.
Console mode 'pipe' selected, but standard input/output are connected to an
interactive TTY. Most likely you want to use 'interactive' console mode for
proper interactivity and shell job control. Proceeding anyway.
Tests:
- With systemd version 248 on host machine, run './container up' and notice that
warning is no longer printed.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Avoid flake8 warnings.
- Makes the call more explicitly readable in case an exception is expected but
check=True is not passed by mistake.
Tests:
- Many tests are skipped since the changes are considered trivial.
check=False is already the default for subprocess.run() method.
- actions/package: Install an app when it is not installed.
- actions/upgrade: Run manual upgrades.
- actions/users: Change a user password. Login. Create/remove a user.
- actions/zoph: Restore a database.
- container: On a fresh repository, run ./container up,ssh,stop,destroy for a
testing container.
- plinth/action_utils.py: Enable/disable an app that has a running service.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #2109.
We moved from Nginx to Apache on ftp.freedombox.org. This changed the datetime
format in the index pages we were relying on to find the difference with local
image. Update this datetime format.
Tests:
- Run ./container update with an old image already in the .container directory.
New image will be downloaded and verified.
- Run ./container update immediately after downloading the latest image. No new
download is done.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
[sunil: Use the bullseye/ directory for more URL stability]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Closes: #2091.
To avoid current issues with keys.gnupg.net.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Allow distribution to be passed as an environment variable.
Fixes#1914
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Install bash-completion when provisioning container, makes terminal
usage more comfortable.
- Increase default container image size to 16G, so that a distribution
upgrade fits well.
- Check free disk space on host before expanding disk image.
- Make pytest coverage reports writable to the fbx user, closes #2010.
- run-tests command:
- Use DEBIAN_FRONTEND=noninteractive when upgrading packages.
- Do not install sshpass as provision script already installs it.
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Add './container ip' command to print current IP address of the container.
Useful when running container related scripts on host machine.
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
- Add new subcommand `run-tests` to the ./container script and the --pytest-args
argument to pass additional arguments to the pytest command.
- Runs tests as root.
- Add new parameters to the SSH command:
- -Y Enable X11 forwarding to be able to run a browser GUI on a container.
- -C Use traffic compression so the X11 GUI is not laggy.
- -t Force pseudo-terminal allocation to enable colors on terminal.
- Use the LogLevel=error option to suppress some warning messages.
- Update module markers in the pytest.ini file.
- Increase the default container disk size to 12G.
- Update HACKING.md to run unit and functional tests.
PROVISION_SCRIPT:
- Explicitly install sshpass as requirement for tests, needed on Debian
stable.
- Ensure that geckodriver.log and .pytest_cache/ exist and are others-writable.
Closes#1901
Tests performed:
- Run unit tests on stable, testing and unstable containers, no failed tests.
- Run functional tests on stable and testing containers, no regressions.
[ fioddor: Some failing tests. All of them unrelated to the changes
introduced. Impact restricted to development tools; the product
hasn't been changed.]
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: Fioddor Superconcentrado <fioddor@gmail.com>
If firewalld is running, the virtual network interface created by
systemd-nspawn gets assigned to the home zone by default. Because
of this, DHCP server is not availabe for the container and most
of the incoming ports are closed.
This commit assigns the network interface created by systemd-nspawn to
the trusted network zone if firewalld is running, so that all network
connections are accepted.
Signed-off-by: Veiko Aasa veiko17@disroot.org
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Fix update command failing when image files do not exist.
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
systemd-nspawn versions < 245 seem to truncate the interface name from
've-fbx-unstable' to 've-fbx-unstabl'.
See:
- https://github.com/systemd/systemd/pull/12865
- https://salsa.debian.org/freedombox-team/freedombox/-/merge_requests/1856
Tests:
- On Debian Buster with systemd version 241, start testing and unstable
containers. For testing the interface name is ve-fbx-testing and for unstable it
is ve-fbx-unstabl. The containers are brought up and provisioned successfully.
Running 'nmcli c show fbx-${distribution}-shared' shows the correct interface
name assigned.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Used a function from the standard library instead.
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
- Only effects develop mode.
- To primarily avoid writing to the source code directory. Multiple containers
or VMs using the source folder won't fight with the database file (the overlay
file system plan is not working out well for containers #1873).
- In the earlier days, we used to allow running from source code directory
without even doing ./setup.py install. Currently it is not possible anyway. We
pretty much install freedombox package before running from source directory.
- If the build process itself learns not to write to source directory, then
containers/VMs won't have to write to source directory at all.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
During --list-dependencies if an old sqlite3 file is present with gitweb
enabled, then a2enconf -c gitweb-freedombox-auth get executed. In this case,
setting up apache2 module fails because authpubtkt tokens are not yet generated
but they are being referred to in the configuration files.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Inspired by: https://salsa.debian.org/vexch/plinth/snippets/391
This script creates a simple container using systemd-nspawn for developing
FreedomBox. It has many advantages over running a VM using Vagrant. RAM is
allocated to processes in the container as needed without any fixed limit. Also
RAM does not have to be statically allocated so it is typically much lighter
than running an VM. There is no hardware emulation when running a container with
same architecture, so processes run as fast as they would on the host machine.
Environment: The script will run only run on hosts having systemd-nspawn and
network-manager installed, typical GNU/Linux distributions. It has been
primarily developed and tested on Debian Buster but should work on most modern
GNU/Linux distributions.
Disk image: systemd-nspawn accepts not only a directory for starting a container
but also a disk image. This disk image is loop-back mounted and container is
started from that mounted directory. The partition to use is determined by
looking at the boot flag in the partition table. This happens to work well with
all existing FreedomBox images. In future, we may be able to run different
architectures in this manner.
After downloading, the disk image is expanded along with the partition and file
system inside so that development can be done without running into disk space
issues. Expanding the disk does not immediately consume disk space because it
will be a sparse file. As data is written to the disk, it will occupy more and
more space but the upper limit is the size to which disk has been expanded.
Downloading images: Images are downloaded from FreedomBox download server using
fixed URLs for each distribution. Signature is verified for the download images.
The fingerprint of the allowed signing key is hard-coded in this script.
Downloaded images are kept even after destroying the extracted raw image along
with container. This allows for quickly resetting the container without
downloading again.
Booting: systemd-nspawn is run in 'boot' mode. This means that init process
(happens to be systemd) is started inside the container. It then spawns all the
other necessary daemons including openssh-server, firewalld and network-manager.
A login terminal can be opened using 'machinectl login' because container is
running systemd. SSH into the container is possible because network is up,
configured by network-manager, and openssh server is running.
Shared folder: Using systemd-nspawn, the project directory is mounted as
/freedombox inside the container. The project directory is determined as
directory in which this script resides. The project folder from the container
point of view will be read-only. Container should be able to write various files
such as build files, FreedomBox sqlite3 database and session files into the
/freedombox folder. To enable writing, an additional read-write folder is
overlayed onto /freedombox folder in the container. This directory can't be
created under the project folder and is created instead in
$XDG_DATA_HOME/freedombox-container/overlay/$DISTRIBUTION. If XDG_DATA_HOME is
not set, it is assumed to be $HOME/.local/shared/. Whenever data is written into
/freedombox directory inside the container, this directory on the host receives
the changes. See documentation for Overlay filesystem for further details. When
container is destroyed, this overlay folder is destroyed to ensure clean state
after bringing up the container again.
Users: PrivateUsers configuration flag for systemd-nspawn is currently off. This
means that each user's UID on the host is also the same UID in the container as
along as there is an entry in the container's password database. In future, we
may explore using private users inside the container.
'fbx' is the development user and its UID is changed during setup phase to 10000
hoping it would not match anything on the host system. 'fbx' user has full sudo
access inside the container without needing a password. Password for this user
is not set by default, but can be set if needed. If there is no access to the
container in any way, one can run 'sudo machinectl shell' and then run 'passwd
fbx' to set the password for the 'fbx' user.
'plinth' user's UID in the container is also changed and set to the UID of
whichever user owns the project directory. This allows the files to written by
'plinth' container user in the project directory because UID of the owner of the
directory is same as the 'plinth' user's UID in container.
Network: A private network is created inside the container using systemd-nspawn
feature. Network interfaces from the host are not available inside the
container. A new network interface called 'host0' is configured inside the
container which is automatically configured by network-manager. On the host a
new network interface is created. This script creates configuration for a
'shared' network using network-manager. When bringing up the container, this
network connection is also brought up. A DHCP server and a DNS server are
started network-manager on the host side so that DHCP and DNS client functions
work inside the container. Traffic from the container is also masqueraded so
that Internet connectivity inside the container works if the host has one.
If necessary, the network interface on host side can be differently configured.
For example, it can be bridged with another interface to expose the container on
a network that the host machine participates in.
The network IP address inside the container can be queried using machinectl.
This script queries that IP address and presents the address in its console
messages. All ports in the container can be reached from the host using this IP
address as long as the firewall inside the container allows it. There is no need
to perform port forwarding or mapping.
SSH: It is assumed that openssh-server is installed inside the container. SSH
server keys in the container are created if missing. Client side keys are
created in .container/ssh directory and the public key is installed in the
authorized keys file of the 'fbx' user. The 'ssh' sub-command to this script is
simply a convenience mechanism for quick launch of ssh with the right IP
address, user name and identity file.
Role of machinectl: Most of the work is done by systemd-nspawn. machinectl is
useful for running systemd-nspawn in the background and querying its current
state. It also helps with providing the IP address of the container. machinectl
is made to recognize the container by creating a link in /var/lib/machines/ to
the image file. systemd-nspawn options are added by creating a temporary file in
/run/systemd/nspawn. All machinectl commands should work.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
