For uniformity with the more often used ./container script. Developers may have
to destroy and recreate their vagrant machines for this to take effect.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Added a separate command to run unit tests only
- Clarified the fact that the application should be started before
running functional tests
- Add one troubleshooting tip
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
This removes the dependencies xvfb and pytest-xvfb.
--splinter-headless can be used as a substitute for running tests in
headless mode.
[sunil: Edit description as running run-tests starts plinth in container]
[sunil: Retain the xauth command]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Create the 'heavy' mark for tests that take long to run.
- Skip the tests marked as 'heavy' by default.
- Enable functional and heavy tests by setting the environment variable
EXTENDED_TESTING=1.
- Update HACKING.md accordingly.
Signed-off-by: Fioddor Superconcentrado <fioddor@gmail.com>
[sunil: Minor indentation]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- ugettext functions will be removed in Django 4.0. Each use emits a warning
when running with Django 3.2. Since we have warnings enabled in developer mode,
we see quite a few messages because of this.
- ugettext is already a simple alias of gettext. So, no regressions are
expected.
Tests:
- Accessing an affected app in UI with Django 3.2 and Django 2.2 works fine.
- Using Django 3.2 there are no warnings related to removal of ugettext
functions.
- Ran regular unit tests.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Allow distribution to be passed as an environment variable.
Fixes#1914
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Set default SSH and Samba port for testing a container.
- Set default URL for running functional tests from within container.
- Update instructions.
Closes: #2015.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
Reviewed-by: Fioddor Superconcentrado <fioddor@gmail.com>
- Add new subcommand `run-tests` to the ./container script and the --pytest-args
argument to pass additional arguments to the pytest command.
- Runs tests as root.
- Add new parameters to the SSH command:
- -Y Enable X11 forwarding to be able to run a browser GUI on a container.
- -C Use traffic compression so the X11 GUI is not laggy.
- -t Force pseudo-terminal allocation to enable colors on terminal.
- Use the LogLevel=error option to suppress some warning messages.
- Update module markers in the pytest.ini file.
- Increase the default container disk size to 12G.
- Update HACKING.md to run unit and functional tests.
PROVISION_SCRIPT:
- Explicitly install sshpass as requirement for tests, needed on Debian
stable.
- Ensure that geckodriver.log and .pytest_cache/ exist and are others-writable.
Closes#1901
Tests performed:
- Run unit tests on stable, testing and unstable containers, no failed tests.
- Run functional tests on stable and testing containers, no regressions.
[ fioddor: Some failing tests. All of them unrelated to the changes
introduced. Impact restricted to development tools; the product
hasn't been changed.]
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: Fioddor Superconcentrado <fioddor@gmail.com>
See MR#1919.note_194567.
Since the documents and some chapters are long a table of contents has been
added at the beginning along with navigation links pointing back to it at the
end of main sections and long chapters.
Signed-off-by: Fioddor Superconcentrado <fioddor@gmail.com>
[sunil: Drop periods at the end of TOC entries]
[sunil: Re-title some sections]
[sunil: Add links to Django i18n documentation and IRC chatroom]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
This change prevents the plinth user to become a superuser without
knowing an admin password.
Users module and action script:
- User credentials are now required for the subcommands: create-user,
set-user-password, add-user-to-group (if the group is admin),
remove-user-from-group (if the group is admin), set-user-status,
remove-user (if the removed user is the last admin user.
Note: the web UI doesn't allow to delete last admin user).
- subcommand remove-users requires authentication if the user is last
admin user. Password must be provided through standard input.
- subcommand remove-group: do not allow to remove group 'admin'
- User credentials must be provided using the argument
--auth-user and a passsword must be provided through standard input.
- If there are no users in the admin group, no admin password is
required and if the --auth-user argument is required, it can be an
empty string.
Users web UI:
- An admin needs to enter current password to create and edit a user
and to change user's password.
- Show more detailed error text on exceptions when submitting forms.
- Show page title on the edit and create user pages.
Users unit and functional tests:
- Added a configuration parameters to the pytest configuration file
to set current admin user/password.
- Added a configuration parameter 'ssh_port' to the functional tests.
You can overwrite this with the FREEDOMBOX_SSH_PORT environment
variable. Modified HACKING.md accordingly.
- Added an unit test:
- test changing the password as a non-admin user.
- test invalid admin password input.
- test that removing the admin group fails.
- Capture stdout and stderr in the unit tests when calling an action
script to be able to see more info on exceptions.
- Added functional tests for setting ssh keys and changing passwords
for admin and non-admin users.
- Added a functional test for setting a user as active/inactive.
Changes during review [sunil]:
- Move uncommon functional step definitions to users module from global. This is
keep the common functional step definitions to minimal level and promote when
needed.
- Minor styling changes, flake8 fixes.
- Don't require pampy module when running non-admin tests. This allows tests to
be run from outside the container on the host machine without python3-pam
installed.
- Call the confirm password field 'Authorization Password'. This avoid confusion
with a very common field 'Confirm Password' which essentially means retype
your password to ensure you didn't get it wrong. Add label explaining why the
field exists.
- Don't hard-code /tmp path in test_actions.py. Use tmp_path_factory fixture
provided by pytest.
- Remove unused _get_password_hash() from actions/users.
- Undo splitting ldapgid output before parsing. It does not seem correct and
could introduce problems when field values contain spaces.
Tests performed:
- No failed unit tests (run with and without sudo).
- All 'users' functional tests pass.
- Creating an admin user during the first boot wizard succeeds.
- Creating a user using the web UI with an empty or wrong admin
password fails and with the correct admin password succeeds.
- Editing a user using the web UI with an empty or wrong admin
password fails and with the correct admin password succeeds.
- Changing user's password using the web UI with an empty or wrong
admin password fails and with the correct admin password succeeds.
- Above mentioned user action script commands can't be run without
correct credentials.
- Adding the daemon user to the freedombox-share group succeeds when
installing certain apps (deluge, mldonkey, syncthing, transmission).
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
[sunil: Move uncommon functional step definitions to users module from global]
[sunil: Minor styling changes, flake8 fixes]
[sunil: Don't require pampy module when running non-admin tests]
[sunil: Call the confirm password field 'Authorization Password']
[sunil: Don't hard-code /tmp path in test_actions.py]
[sunil: Remove unused _get_password_hash() from actions/users]
[sunil: Undo splitting ldapgid output before parsing]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
[sunil: Reword the extra requirements to explain when they are needed]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
A 'Troubleshooting' section added under 'Using Containers' chapter with 2 hints.
References to files and directories were sometimes marked as code sometimes not.
I've standardized them. Bullets in 'Requirements for Development OS' section
visually stress the alternatives.
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Inspired by: https://salsa.debian.org/vexch/plinth/snippets/391
This script creates a simple container using systemd-nspawn for developing
FreedomBox. It has many advantages over running a VM using Vagrant. RAM is
allocated to processes in the container as needed without any fixed limit. Also
RAM does not have to be statically allocated so it is typically much lighter
than running an VM. There is no hardware emulation when running a container with
same architecture, so processes run as fast as they would on the host machine.
Environment: The script will run only run on hosts having systemd-nspawn and
network-manager installed, typical GNU/Linux distributions. It has been
primarily developed and tested on Debian Buster but should work on most modern
GNU/Linux distributions.
Disk image: systemd-nspawn accepts not only a directory for starting a container
but also a disk image. This disk image is loop-back mounted and container is
started from that mounted directory. The partition to use is determined by
looking at the boot flag in the partition table. This happens to work well with
all existing FreedomBox images. In future, we may be able to run different
architectures in this manner.
After downloading, the disk image is expanded along with the partition and file
system inside so that development can be done without running into disk space
issues. Expanding the disk does not immediately consume disk space because it
will be a sparse file. As data is written to the disk, it will occupy more and
more space but the upper limit is the size to which disk has been expanded.
Downloading images: Images are downloaded from FreedomBox download server using
fixed URLs for each distribution. Signature is verified for the download images.
The fingerprint of the allowed signing key is hard-coded in this script.
Downloaded images are kept even after destroying the extracted raw image along
with container. This allows for quickly resetting the container without
downloading again.
Booting: systemd-nspawn is run in 'boot' mode. This means that init process
(happens to be systemd) is started inside the container. It then spawns all the
other necessary daemons including openssh-server, firewalld and network-manager.
A login terminal can be opened using 'machinectl login' because container is
running systemd. SSH into the container is possible because network is up,
configured by network-manager, and openssh server is running.
Shared folder: Using systemd-nspawn, the project directory is mounted as
/freedombox inside the container. The project directory is determined as
directory in which this script resides. The project folder from the container
point of view will be read-only. Container should be able to write various files
such as build files, FreedomBox sqlite3 database and session files into the
/freedombox folder. To enable writing, an additional read-write folder is
overlayed onto /freedombox folder in the container. This directory can't be
created under the project folder and is created instead in
$XDG_DATA_HOME/freedombox-container/overlay/$DISTRIBUTION. If XDG_DATA_HOME is
not set, it is assumed to be $HOME/.local/shared/. Whenever data is written into
/freedombox directory inside the container, this directory on the host receives
the changes. See documentation for Overlay filesystem for further details. When
container is destroyed, this overlay folder is destroyed to ensure clean state
after bringing up the container again.
Users: PrivateUsers configuration flag for systemd-nspawn is currently off. This
means that each user's UID on the host is also the same UID in the container as
along as there is an entry in the container's password database. In future, we
may explore using private users inside the container.
'fbx' is the development user and its UID is changed during setup phase to 10000
hoping it would not match anything on the host system. 'fbx' user has full sudo
access inside the container without needing a password. Password for this user
is not set by default, but can be set if needed. If there is no access to the
container in any way, one can run 'sudo machinectl shell' and then run 'passwd
fbx' to set the password for the 'fbx' user.
'plinth' user's UID in the container is also changed and set to the UID of
whichever user owns the project directory. This allows the files to written by
'plinth' container user in the project directory because UID of the owner of the
directory is same as the 'plinth' user's UID in container.
Network: A private network is created inside the container using systemd-nspawn
feature. Network interfaces from the host are not available inside the
container. A new network interface called 'host0' is configured inside the
container which is automatically configured by network-manager. On the host a
new network interface is created. This script creates configuration for a
'shared' network using network-manager. When bringing up the container, this
network connection is also brought up. A DHCP server and a DNS server are
started network-manager on the host side so that DHCP and DNS client functions
work inside the container. Traffic from the container is also masqueraded so
that Internet connectivity inside the container works if the host has one.
If necessary, the network interface on host side can be differently configured.
For example, it can be bridged with another interface to expose the container on
a network that the host machine participates in.
The network IP address inside the container can be queried using machinectl.
This script queries that IP address and presents the address in its console
messages. All ports in the container can be reached from the host using this IP
address as long as the firewall inside the container allows it. There is no need
to perform port forwarding or mapping.
SSH: It is assumed that openssh-server is installed inside the container. SSH
server keys in the container are created if missing. Client side keys are
created in .container/ssh directory and the public key is installed in the
authorized keys file of the 'fbx' user. The 'ssh' sub-command to this script is
simply a convenience mechanism for quick launch of ssh with the right IP
address, user name and identity file.
Role of machinectl: Most of the work is done by systemd-nspawn. machinectl is
useful for running systemd-nspawn in the background and querying its current
state. It also helps with providing the IP address of the container. machinectl
is made to recognize the container by creating a link in /var/lib/machines/ to
the image file. systemd-nspawn options are added by creating a temporary file in
/run/systemd/nspawn. All machinectl commands should work.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Homebrew is the correct name of the package manager, not Brew.
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Add more detailed instructions for installing VirtualBox manually on
Debian and its derivatives.
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- The repository has been renamed from 'plinth' to 'freedombox'.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@riseup.net>
- Add functional tests
- Add unit tests for samba views
- New dependency for running functional tests: smbclient
- Make port configurable for the smbclient
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: Joseph Nuthalapati <njoseph@riseup.net>
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
- Add instructions on how to install Git, Vagrant and VirtualBox for other OSes.
Instructions for other OSes are untested by me.
- Add instruction to checkout the source code.
- Remove instructions for installing on the host machine, these have not been
tested for a while and could lead to confusion. 'setup.py develop' option is
not being used anymore.
- Installing firefox using apt provides a sufficient version of functional
tests. Simplify the instructions.
- Update functional tests instructions to use the '-m' mark option instead of of
'-k' keyword.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
It's now possible to run the tests in the VM more easily
without copy-pasting too much from HACKING.md.
`vagrant provision --provisiion-with tests` will take care of the dependencies.
freedombox/plinth#1521 - Provision VM to run function functional tests
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Running plinth as root is causing preventable permission errors in production.
This commit makes the necessary changes to run plinth as the user plinth so that
permission errors can be caught early and prevented.
- Give ownership of all files in the mounted directory to plinth
- Add permissions in sudoers file so that plinth can run scripts in
/vagrant/actions as superuser
- Move scripts that run on vagrant triggers to a separate directory
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Create option --include-functional to run functional tests. Otherwise, they
are disabled by default. If pytest-bdd is not installed, functional tests are
not discovered at all.
- Make pytest-django discover the setting files by creating dummy manage.py in
top level directory.
- Make pytest run as './setup.py pytest'. Add alias from './setup.py test'. This
requires pytest-runner package.
- Merge .gitignore files from functional_tests/
- Update gitlab-ci.yml to run tests with coverage using pytest.
- Update HACKING.md to suggest using py.test-3 instead of old way of running.
Merge functional tests README.md into HACKING.md.
- Remove execution wrapper runtests.py as pytest-django is able to configure
Django settings before execution of tests. Update tests to explicitly ask for
Django database as database access is denied by default.
- Replace usage of python3-coverage with python3-pytest-coverage. Execution
wrappers are not required.
- Add build dependencies on pytest modules.
- Let all warnings be shown after running tests.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Moved some frequently required operations to the Vagrantfile.
Newcomers to the project almost always face the issue of port 8000 not being
available. Disabling system Plinth and running only the development version
solves this issue.
Automatic upgrades are a frequent blocker for development and an unnecessary
annoyance on a development setup.
- Update documentation about vagrant file changes
- Remove plinth restart step
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
For longer lists with various paragraphs per item, (gitlab?) markdown
needs four spaces of indentation to work correctly:
https://docs.gitlab.com/ee/user/markdown.html#lists
Signed-off-by: Michael Pimmer <info@fonfon.at>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- use markdown more explicitly
- update information on i18n and translation
- move or delete redundant info from HACKING into CONTRIBUTING
Signed-off-by: Johannes Keyser <johanneskeyser@posteo.de>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
