davical/testing/README.regression_tests

Regression Testing
==================

These tests were basically written to work in Andrew's environment.
Along with the database and PHP timezone handling code in DAViCal,
they have been improved to a point where they should be able to pass
on any system. But should you encounter timezone-related issues, make
sure all components interpret floating timezones to be Pacific/Auckland.

You may also need to add a regression.conf file, see the provided example.


TESTING SETUP
-------------

By default, the tests are run on a database called 'regression' and a
number of hosts called 'regression', 'mycaldav' and 'myempty'. To set
this up locally, add

  127.0.1.1       regression mycaldav myempty

to /etc/hosts and a VirtualHost to Apache with

  Listen 127.0.1.1:80
  <VirtualHost 127.0.1.1:80>
    ServerName regression
    ServerAlias myempty mycaldav
    DocumentRoot /path/to/davical/htdocs
    CustomLog ${APACHE_LOG_DIR}/regression-access.log combined
    ErrorLog  ${APACHE_LOG_DIR}/regression-error.log

    Alias /davical /path/to/davical/htdocs
    <Directory /path/to/davical/htdocs>
      Require all granted
      DirectoryIndex index.php index.html
      php_value include_path /path/to/awl/inc:/path/to/davical/testing
    </Directory>

    RewriteEngine On
    # Redirect /.well-known URLs
    RewriteRule ^/\.well-known/(.*)$ /caldav.php/.well-known/$1 [NC,L]
    # and other special URLs
    RewriteRule ^/principals/users/(.*)$ /caldav.php/$1 [NC,L]
    RewriteRule ^/principals/resources/(.*)$ /caldav.php/$1 [NC,L]
    RewriteRule ^/calendars/__uids__/(.*)$ /caldav.php/$1 [NC,L]
    RewriteRule ^/addressbooks/__uids__/(.*)$ /caldav.php/$1 [NC,L]
    # let caldav.php handle "anything else"
    RewriteCond %{REQUEST_URI} !^/$
    RewriteCond %{REQUEST_URI} !\.(php|css|png|gif|js|jpg|ico)
    RewriteRule ^(.*)$ /caldav.php$1  [NC,L]
  </VirtualHost>

For DAViCal, create a regression-conf.php (as well as identical
mycaldav-conf.php and myempty-conf.php, which can be symlinks to the
former) with

  <?php
    $c->pg_connect[] = 'dbname=regression user=davical_app';

    // use strict result ordering for regression testing only
    $c->strict_result_ordering = true;
    // PUT: 405 Method Not Allowed
    $c->readonly_webdav_collections = false;
    // fix ETag differences (CRLF) in 0218-Moz-REPORT and later
    $c->hide_alarm = true;
    // some freebusy queries will be 404 otherwise
    $c->public_freebusy_url = true;
    // helps with 1035-GET-mashup
    $c->get_includes_subcollections = true;
    // helps with 1036-REPORT-sync-initial-bound
    $c->hide_TODO = false;

    // for ischedule suite; also needs DKIM set up?
    $c->enable_scheduling = true;

    // enable for debugging, as needed
    //$c->dbg['ALL'] = 1;
  ?>

Make the user running the tests known to postgres and allow it to create
databases and roles (e.g. run the dba/create-database.sh script):

  sudo -u postgres createuser <USERNAME> --createdb --createrole --role=davical_dba

and let everyone locally access the regression database at will by adding the
following line to pg_hba.conf:

  local   regression    all           trust

Finally, don't forget to restart Apache and Postgres for all these
changes to take effekt.


RUNNING THE TESTSUITE
---------------------

You can then run the entire test suite with the command

  ./run_regressions.sh

To run a number of specific suites in addition to the standard
regression-suite, use

  ./run_regressions.sh all

When a test fails to produce the expected output, you are prompted

  Accept new result [e/s/r/v/f/m/x/y/w/N]?

the options being

  y = make new result the expected result
  x = print command for running test in debug mode and exit testing
  v = show test
  s = side-by-side diff of actual and expected result
  w = colourized wdiff
  m = side-by-side 'meld'
  f = show full details of test and result
  e = edit test (vi)
  r = re-run test
  N = run next test

This should allow for convenient updating of tests and expected results.


WRITING TESTS
-------------

run_regressions.sh looks for test definitions in tests/<testsuite>/*.test,
which it will pass to dav_test for execution. The result is filtered through
normalise_result and then compared to the expected result in
tests/<testsuite>/<testname>.result

The test definition file will include lines like:

    # This is an example
    URL=http://mycaldav/caldav.php/andrew/
    HEADER=Depth: 0
    HEADER=Content-type: text/xml
    TYPE=PROPFIND
    HEAD
    DATA=OTHERTEST
    # This will let you use ##somename## for this value after this
    GETSQL=somename
    SELECT column FROM table WHERE criteria
    ENDSQL
    # The data can be included in line
    BEGINDATA
    ... data content ...
    ENDDATA
    # The result could be some SQL output
    QUERY
    SELECT something, or, other FROM table ...
    ENDQUERY
    #
    REPLACE=/pattern/replacement/options

You can use the following fields:

URL      The URL to request from.
HEADER   An additional header for the request
TYPE     The type of request (e.g. GET/PUT/POST/REPORT/...)
HEAD     Whether to include the headers in the recorded response
VERBOSE  Whether to provide the full request / response headers.
DATA     The name of a different test in this suite to use data from.
REPLACE  A perl regex replacement to post-process the result through.

Additionally, if a file 'tests/<testsuite>/<testname>.data' exists
the contents of that file will be sent in the body of the request.