storage: Auto-mount disks, notify of failing disks

- Remove freedombox-udiskie.service file. Don't run udiskie anymore. Use our own
implementation of auto-mounting.

- Schedule disk failure checking to 3 seconds after application initialization.
Also perform auto-mounting at that time.

- Listen to new filesystems added and auto-mount them.

- Listen to disk failing attribute and report to user via a notification.

- Add rules to polkit-1 to allow plinth user to mount drives.

- Add simple abstractions over DBusProxy objects make accessing properties
simpler.

- Replicate udiskie's approach to mounting disks.

- Mount as root user for now using command line instead of DBus API. This is to
keep compatibility with older code that mounted under /media/root with relaxed
permissions.

Udiskie analysis:

  - On device added, media added, perform auto_add
  - On device changed and is addable and old state is not addable or removeable
  - Automount condition:
    - Matches configuration
    - Not ignored
    - is_filesystem and not mounted -> mount
    - crypto device -> try unlock -> if success, mount
    - is partition table
      - Get all non-ignored devices, if partition then mount
  - Mount condition:
    - Is not ignored
    - Is filesystem
    - Find device with path
    - Get options from configuration
    - Is ntfs and executable ntfs-3g is not available
    - Call mount
  - No support for udisks1
  - Built-in rules
    - {'symlinks': '/dev/mapper/docker-*', 'ignore': True}
    - {'symlinks': '/dev/disk/by-id/dm-name-docker-*', 'ignore': True}
    - {'is_loop': True, 'is_ignored': False, 'loop_file': '/*', 'ignore': False}
    - {'is_block': False, 'ignore': True}
    - {'is_external': False, 'is_toplevel': True, 'ignore': True}
    - {'is_ignored': True, 'ignore': True}

Tests performed:
  - Create a CDROM in VM, inject media. Disk should get mounted.
  - Create a temp file. mkfs.ext4 it at top level. losetup it. It should not get
    auto mounted as it is a top level internal device.
  - Create a temp file. Create two partitions and format the partitions. kpartx
    -a on it. Both the file systems should get mounted.
  - Create a temp file. luksformat it. Create a filesystem. luksopen the file.
    It should get auto mounted.
  - Checking for disk space repeatedly happens every 3 minutes.
  - Drives are checked for healthy status only once, 3 seconds after FreedomBox is started.
  - FreedomBox is able to mount disks while running as 'plinth' user with
    policykit-1 version 0.105-26.
  - FreedomBox is able to mount disks while running as 'plinth' user with
    policykit-1 version 0.116-2 from experimental.
  - Temporarily flip the is_failing condition in report_failing_drive. When
    FreedomBox is restarted, notification about drives failing show up. When the
    condition is reverted to normal, the notification is withdrawn.
  - Build new Debian package and upgrade system with 20.8 installed. Two files
    should be removed:
    /var/lib/systemd/deb-systemd-helper-enabled/freedombox-udiskie.service.dsh-also
    /etc/systemd/system/multi-user.target.wants/freedombox-udiskie.service .
    systemctl status freedombox-udiskie.service should report no such unit.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>

actions/storage

@@ -34,6 +34,10 @@ def parse_arguments():
     subparser.add_argument('device',
                            help='Partition which needs to be resized')
 
+    subparser = subparsers.add_parser('mount', help='Mount a filesystem')
+    subparser.add_argument('--block-device',
+                           help='Block device of the filesystem to mount')
+
     subparser = subparsers.add_parser('eject', help='Eject a storage device')
     subparser.add_argument('device', help='Path of the device to eject')
 
@@ -232,6 +236,26 @@ def _interpret_unit(value):
     return int(value)
 
 
+def subcommand_mount(arguments):
+    """Mount a disk are root user.
+
+    XXX: This is primarily to provide compatibility with older code that used
+    udiskie to auto-mount all partitions as root user under /media/root/
+    directory. We are setting special permissions for the directory /media/root
+    and users have set shared folders using this path. This can be removed in
+    favor of using DBus API once we have a migration plan in place. Disks can
+    be mounted directly /mount without ACL restrictions that apply to
+    /mount/<user> directories. This can be done by setting udev flag
+    UDISKS_FILESYSTEM_SHARED=1 by writing a udev rule.
+
+    """
+    process = subprocess.run([
+        'udisksctl', 'mount', '--block-device', arguments.block_device,
+        '--no-user-interaction'
+    ])
+    sys.exit(process.returncode)
+
+
 def subcommand_eject(arguments):
     """Eject a device by its path."""
     device_path = arguments.device

debian/freedombox.preinst

@@ -19,6 +19,22 @@ case "$1" in
                 systemctl daemon-reload
             fi
         fi
+
+        # Handle removing freedombox-udiskie.service from 20.9.
+        if dpkg --compare-versions "$2" le 20.9; then
+            if [ -x "/usr/bin/deb-systemd-invoke" ]; then
+                deb-systemd-invoke stop freedombox-udiskie.service >/dev/null 2>/dev/null || true
+            fi
+
+            if [ -x "/usr/bin/deb-systemd-helper" ]; then
+                deb-systemd-helper purge freedombox-udiskie.service >/dev/null || true
+                deb-systemd-helper unmask freedombox-udiskie.service >/dev/null || true
+            fi
+
+            if [ -d /run/systemd/system ]; then
+                systemctl daemon-reload
+            fi
+        fi
         ;;
 esac
 

plinth/modules/storage/__init__.py

@@ -3,6 +3,7 @@
 FreedomBox app to manage storage.
 """
 
+import base64
 import logging
 import subprocess
 
@@ -13,17 +14,15 @@ from django.utils.translation import ugettext_noop
 from plinth import actions
 from plinth import app as app_module
 from plinth import cfg, glib, menu, utils
-from plinth.daemon import Daemon
 from plinth.errors import ActionError, PlinthError
 from plinth.utils import format_lazy, import_from_gi
 
+from . import udisks2
 from .manifest import backup  # noqa, pylint: disable=unused-import
 
 version = 4
 
-managed_services = ['freedombox-udiskie']
+managed_packages = ['parted', 'udisks2', 'gir1.2-udisks-2.0']
-
-managed_packages = ['parted', 'udiskie', 'gir1.2-udisks-2.0']
 
 _description = [
     format_lazy(
@@ -60,13 +59,13 @@ class StorageApp(app_module.App):
                               'storage:index', parent_url_name='system')
         self.add(menu_item)
 
-        daemon = Daemon('daemon-udiskie', managed_services[0])
-        self.add(daemon)
-
         # Check every hour for low disk space, every 3 minutes in debug mode
         interval = 180 if cfg.develop else 3600
         glib.schedule(interval, warn_about_low_disk_space)
 
+        # Schedule initialization of UDisks2 initialization
+        glib.schedule(3, udisks2.init, repeat=False)
+
 
 def init():
     """Initialize the module."""
@@ -334,3 +333,22 @@ def warn_about_low_disk_space(request):
                                       title=title, message=message,
                                       actions=actions, data=data,
                                       group='admin')
+
+
+def report_failing_drive(id, is_failing):
+    """Show or withdraw notification about failing drive."""
+    notification_id = 'storage-disk-failure-' + base64.b32encode(
+        id.encode()).decode()
+
+    from plinth.notification import Notification
+    title = ugettext_noop('Disk failure imminent')
+    message = ugettext_noop(
+        'Disk {id} is reporting that it is likely to fail in the near future. '
+        'Copy any data while you still can and replace the drive.')
+    data = {'id': id}
+    note = Notification.update_or_create(id=notification_id, app_id='storage',
+                                         severity='error', title=title,
+                                         message=message, actions=[{
+                                             'type': 'dismiss'
+                                         }], data=data, group='admin')
+    note.dismiss(should_dismiss=not is_failing)

plinth/modules/storage/data/lib/systemd/system/freedombox-udiskie.service

@@ -1,22 +0,0 @@
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
-[Unit]
-Description=handle automounting
-Documentation=man:udiskie(1)
-
-[Service]
-ExecStart=/usr/bin/udiskie
-LockPersonality=yes
-PrivateTmp=yes
-ProtectControlGroups=yes
-ProtectHome=yes
-ProtectKernelLogs=yes
-ProtectKernelModules=yes
-ProtectKernelTunables=yes
-ProtectSystem=full
-RestrictAddressFamilies=AF_UNIX
-RestrictRealtime=yes
-SystemCallArchitectures=native
-
-[Install]
-WantedBy=multi-user.target

plinth/modules/storage/data/usr/share/polkit-1/rules.d/50-freedombox-udisks2.rules

@@ -0,0 +1,18 @@
+/*
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+This file is used only by policykit-1 version > 0.105. A corresponding .pkla
+file is used by policykit-1 <= 0.105. See:
+https://davidz25.blogspot.com/2012/06/authorization-rules-in-polkit.html
+
+*/
+
+polkit.addRule(function(action, subject) {
+    if ((action.id == "org.freedesktop.udisks2.filesystem-mount" ||
+         action.id == "org.freedesktop.udisks2.filesystem-mount-system" ||
+         action.id == "org.freedesktop.udisks2.filesystem-mount-other-seat" ||
+         action.id == "org.freedesktop.udisks2.filesystem-fstab") &&
+        subject.user == "plinth") {
+        return polkit.Result.YES;
+    }
+});

plinth/modules/storage/data/var/lib/polkit-1/localauthority/10-vendor.d/org.freedombox.UDisks2.pkla

@@ -0,0 +1,4 @@
+[Allow FreedomBox to manage UDisks2]
+Identity=unix-user:plinth
+Action=org.freedesktop.udisks2.filesystem-mount;org.freedesktop.udisks2.filesystem-mount-system;org.freedesktop.udisks2.filesystem-mount-other-seat;org.freedesktop.udisks2.filesystem-fstab
+ResultAny=yes

plinth/modules/storage/udisks2.py

@@ -0,0 +1,340 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+"""
+Handle disk operations using UDisk2 DBus API.
+"""
+
+import logging
+import threading
+
+from plinth import actions
+from plinth.errors import ActionError
+from plinth.utils import import_from_gi
+
+glib = import_from_gi('GLib', '2.0')
+gio = import_from_gi('Gio', '2.0')
+
+_DBUS_NAME = 'org.freedesktop.UDisks2'
+
+_INTERFACES = {
+    'Ata': 'org.freedesktop.UDisks2.Drive.Ata',
+    'Block': 'org.freedesktop.UDisks2.Block',
+    'Drive': 'org.freedesktop.UDisks2.Drive',
+    'Filesystem': 'org.freedesktop.UDisks2.Filesystem',
+    'Job': 'org.freedesktop.UDisks2.Job',
+    'Manager': 'org.freedesktop.UDisks2.Manager',
+    'ObjectManager': 'org.freedesktop.DBus.ObjectManager',
+    'Partition': 'org.freedesktop.UDisks2.Partition',
+    'Properties': 'org.freedesktop.DBus.Properties',
+    'UDisks2': 'org.freedesktop.UDisks2',
+}
+
+_OBJECTS = {
+    'drives': '/org/freedesktop/UDisks2/drives/',
+    'jobs': '/org/freedesktop/UDisks2/jobs/',
+    'Manager': '/org/freedesktop/UDisks2/Manager',
+    'UDisks2': '/org/freedesktop/UDisks2',
+}
+
+_ERRORS = {
+    'AlreadyMounted': 'org.freedesktop.UDisks2.Error.AlreadyMounted',
+    'Failed': 'org.freedesktop.UDisks2.Error.Failed',
+}
+
+_jobs = {}
+
+logger = logging.getLogger(__name__)
+
+
+def _get_dbus_proxy(object_, interface):
+    """Return a DBusProxy for a given UDisks2 object and interface."""
+    connection = gio.bus_get_sync(gio.BusType.SYSTEM)
+    return gio.DBusProxy.new_sync(connection, gio.DBusProxyFlags.NONE, None,
+                                  _DBUS_NAME, object_, interface)
+
+
+class Proxy:
+    """Base methods for abstraction over UDisks2 DBus proxy objects."""
+    interface = None
+    properties = {}
+
+    def __init__(self, object_path):
+        """Return an object instance."""
+        self.object_path = object_path
+        self._proxy = _get_dbus_proxy(object_path, self.interface)
+
+    def __getattr__(self, name):
+        """Retrieve a property from underlying proxy or delegate."""
+        if name not in self.properties:
+            return getattr(self._proxy, name)
+
+        signature, original_name = self.properties[name]
+        value = self._proxy.get_cached_property(original_name)
+
+        if value is None:
+            return value
+
+        if signature == 'ay':
+            return bytes(value)[:-1].decode()
+
+        if signature == 'aay':
+            return [bytes(value_item).decode()[:-1] for value_item in value]
+
+        if signature in ('s', 'b', 'o', 'u'):
+            return glib.Variant.unpack(value)
+
+        raise ValueError('Unhandled type')
+
+
+class Drive(Proxy):
+    """Abstraction for UDisks2 Drive."""
+    interface = _INTERFACES['Drive']
+    properties = {'id': ('s', 'Id')}
+
+
+class BlockDevice(Proxy):
+    """Abstraction for UDisks2 Block device."""
+    interface = _INTERFACES['Block']
+    properties = {
+        'crypto_backing_device': ('o', 'CryptoBackingDevice'),
+        'device': ('ay', 'Device'),
+        'hint_ignore': ('b', 'HintIgnore'),
+        'hint_system': ('b', 'HintSystem'),
+        'id': ('s', 'Id'),
+        'preferred_device': ('ay', 'PreferredDevice'),
+        'symlinks': ('aay', 'Symlinks'),
+    }
+
+
+class Partition(Proxy):
+    """Abstraction for UDisks2 Partition."""
+    interface = _INTERFACES['Partition']
+    properties = {
+        'number': ('u', 'Number'),
+    }
+
+
+class Filesystem(Proxy):
+    """Abstraction for UDisks2 Filesystem."""
+    interface = _INTERFACES['Filesystem']
+    properties = {'mount_points': ('aay', 'MountPoints')}
+
+
+def _mount(object_path):
+    """Start the mount operation on an block device.
+
+    Runs in a separate thread from glib due to blocking operations.
+
+    """
+    filesystem = Filesystem(object_path)
+    block_device = BlockDevice(object_path)
+    if filesystem.mount_points:
+        logger.info('Ignoring auto-mount on already mounted device: %s %s',
+                    block_device.id, block_device.preferred_device)
+        return
+
+    logger.info('Auto-mounting device: %s %s', block_device.id,
+                block_device.preferred_device)
+    try:
+        actions.superuser_run(
+            'storage',
+            ['mount', '--block-device', block_device.preferred_device],
+            log_error=False)
+    except ActionError as exception:
+        parts = exception.args[2].split(':')
+        if parts[1].strip() != 'GDBus.Error':
+            raise
+
+        if parts[2].strip() == _ERRORS['AlreadyMounted']:
+            logger.warning('Device is already mounted: %s %s', block_device.id,
+                           block_device.preferred_device)
+        elif parts[2].strip() == _ERRORS['Failed']:
+            logger.warning('Mount operation failed: %s %s: %s',
+                           block_device.id, block_device.preferred_device,
+                           exception)
+        else:
+            raise
+
+
+def _on_job_created(object_path, interfaces_created):
+    """Called when a job is created.
+
+    Runs in glib thread. No blocking operations.
+
+    """
+    job = interfaces_created[_INTERFACES['Job']]
+    if job['Operation'] == 'filesystem-mount':
+        logger.info('Mounting operation started on disk: %s',
+                    ', '.join(job['Objects']))
+        _jobs[object_path] = job
+
+
+def _on_job_removed(object_path):
+    """Called when a job is completed.
+
+    Runs in glib thread. No blocking operations.
+
+    """
+    if object_path in _jobs:
+        logger.info('Mounting operation completed on disk: %s',
+                    ', '.join(_jobs[object_path]['Objects']))
+
+
+def _on_filesystem_added(object_path, _interfaces):
+    """Called when a filesystem is added.
+
+    Runs in glib thread. No blocking operations.
+
+    """
+    threading.Thread(target=_consider_for_mounting, args=[object_path]).start()
+
+
+def _consider_for_mounting(object_path):
+    """Check if the block device needs mounting and mount it."""
+    block_device = BlockDevice(object_path)
+
+    # Ignore non-block devices.
+    if not block_device.device:
+        logger.info('Ignoring non-block device, not auto-mounting %s',
+                    object_path)
+        return
+
+    logger.info('New filesystem found: %s %s', block_device.id,
+                block_device.preferred_device)
+
+    # Ignore devices that are hinted by udev to ignore.
+    if block_device.hint_ignore:
+        logger.info(
+            'Ignoring auto-mount of device due to udev ignore hint: %s %s',
+            block_device.id, block_device.preferred_device)
+        return
+
+    # Ignore docker devices.
+    for symlink in block_device.symlinks:
+        if symlink.startswith('/dev/mapper/docker-') or \
+           symlink.startswith('/dev/disk/by-id/dm-name-docker-'):
+            logger.info('Ignoring auto-mount of docker device: %s %s',
+                        block_device.id, block_device.preferred_device)
+            return
+
+    # Ignore non-external devices that don't have partition table (top-level
+    # filesystem). If the device is backed by a crypto device, still handle it.
+    # XXX: This rule is from udiskie. Should we keep it?
+    partition = Partition(object_path)
+    if block_device.hint_system and not partition.number and \
+       block_device.crypto_backing_device == '/':
+        logger.info('Ignoring auto-mount of top-level internal device: %s %s',
+                    block_device.id, block_device.preferred_device)
+        return
+
+    _mount(object_path)
+
+
+def _on_interfaces_added(_connection, _sender_name, _object_path,
+                         _interface_name, _signal_name, parameters, _user_data,
+                         _unknown):
+    """Called when objects/interfaces have been added.
+
+    Runs in glib thread. No blocking operations.
+
+    """
+    object_path, interfaces = parameters
+    if object_path.startswith(_OBJECTS['jobs']):
+        _on_job_created(object_path, interfaces)
+
+    if _INTERFACES['Filesystem'] in interfaces:
+        _on_filesystem_added(object_path, interfaces)
+
+
+def _on_interfaces_removed(_connection, _sender_name, _object_path,
+                           _interface_name, _signal_name, parameters,
+                           _user_data, _unknown):
+    """Called when objects/interfaces have been removed.
+
+    Runs in glib thread. No blocking operations.
+
+    """
+    object_path, _interfaces = parameters
+    if object_path.startswith(_OBJECTS['jobs']):
+        _on_job_removed(object_path)
+
+
+def _on_properties_changed(_connection, _sender_name, object_path,
+                           _interface_name, _signal_name, parameters,
+                           _user_data, _unknown):
+    """Called when properties change on matching objects.
+
+    Runs in glib thread. No blocking operations.
+
+    """
+    interface_changed, properties_changed, _properties_invalided = parameters
+    if interface_changed == _INTERFACES['Ata'] and \
+       'SmartFailing' in properties_changed:
+        drive = Drive(object_path)
+        thread = threading.Thread(
+            target=_report_failing_drive,
+            args=[drive.id, properties_changed['SmartFailing']])
+        thread.start()
+
+
+def _report_failing_drive(id_, is_failing):
+    """Show or withdraw notification about failing drive."""
+    if is_failing:
+        logger.info('Drive %s is failing', id_)
+    else:
+        logger.info('Drive %s appears healthy', id_)
+
+    from . import report_failing_drive
+    report_failing_drive(id_, is_failing)
+
+
+def _connect():
+    """Connect to all necessary signals from UDisks2."""
+    udisks = _get_dbus_proxy(_OBJECTS['UDisks2'], _INTERFACES['UDisks2'])
+    connection = udisks.get_connection()
+
+    connection.signal_subscribe(None, _INTERFACES['ObjectManager'],
+                                'InterfacesAdded', _OBJECTS['UDisks2'], None,
+                                gio.DBusSignalFlags.NONE, _on_interfaces_added,
+                                None, None)
+    connection.signal_subscribe(None, _INTERFACES['ObjectManager'],
+                                'InterfacesRemoved', _OBJECTS['UDisks2'], None,
+                                gio.DBusSignalFlags.NONE,
+                                _on_interfaces_removed, None, None)
+    connection.signal_subscribe(udisks.get_name(), _INTERFACES['Properties'],
+                                'PropertiesChanged', None, None,
+                                gio.DBusSignalFlags.NONE,
+                                _on_properties_changed, None, None)
+
+
+def _check_failing_drives():
+    """Check if any of the drives are failing and report."""
+    manager = _get_dbus_proxy(_OBJECTS['UDisks2'],
+                              _INTERFACES['ObjectManager'])
+    objects = manager.GetManagedObjects()
+    for _, interface_and_properties in objects.items():
+        if _INTERFACES['Drive'] in interface_and_properties and \
+           _INTERFACES['Ata'] in interface_and_properties:
+            _report_failing_drive(
+                interface_and_properties[_INTERFACES['Drive']]['Id'],
+                interface_and_properties[_INTERFACES['Ata']]['SmartFailing'])
+
+
+def _mount_initial_devices():
+    """Check if any of the block devices need mounting."""
+    manager = _get_dbus_proxy(_OBJECTS['UDisks2'],
+                              _INTERFACES['ObjectManager'])
+    objects = manager.GetManagedObjects()
+    for object_, interface_and_properties in objects.items():
+        if _INTERFACES['Filesystem'] in interface_and_properties:
+            _consider_for_mounting(object_)
+
+
+def init(_data):
+    """Subscribe to signals from UDisks2 and check for failing drives.
+
+    Runs in a separate thread from glib thread due to blocking operations.
+
+    """
+    _connect()
+    _check_failing_drives()
+    _mount_initial_devices()