Warning : NEVER run tests on a production system. The tests WILL delete and modify data in the database configured in config.php, and some tests rewrite config.php itself. Use a separate database for testing and make a backup of important data first.

Hint: Every time you fix a bug, add a regression test so that it does not come back.

Introduction

LimeSurvey has two kinds of PHPUnit tests in the tests/ folder:

Unit tests (tests/unit) test PHP code directly. Many of them use the database, but none needs a browser.
Functional tests (tests/functional) control a real Firefox browser through WebDriver (geckodriver or Selenium) and click through the admin interface and surveys.

All tests run automatically on GitHub Actions for every pull request, see Continuous integration. This page explains how to run them on your own computer and how to write new tests.

The tests are only included in the Git repository, not in the release downloads. Clone the repository to run them.

Requirements

A Git clone of LimeSurvey, installed and reachable through a local web server (for functional tests)
PHP on the command line, the same version as your web server. CI tests with PHP 8.1 and 8.5.
Composer
A database for testing only (MySQL/MariaDB, PostgreSQL or Microsoft SQL Server)
For functional tests: Firefox and geckodriver

Setup

All commands are run in the LimeSurvey root folder.

Install the PHP dependencies, including PHPUnit:
```
composer install
```
Create an empty test database and set its credentials in application/config/config.php.
Install LimeSurvey from the command line. The admin user and password are the ones the functional tests log in with (see Environment variables):
```
php application/commands/console.php install admin password Admin admin@example.com verbose
```
Enable the tests by creating the file enabletests. Without it, the test bootstrap refuses to run, as a protection against running tests on a production system.
```
touch enabletests
```
On Windows (PowerShell): New-Item -ItemType File -Name "enabletests"

Unit tests can run now. For functional tests, also set up the browser driver as described below.

Browser driver for functional tests

The PHP WebDriver library connects to a WebDriver server on port 4444. The simplest way is to run geckodriver directly. No Java and no Selenium Server are needed.

Linux and macOS

Install geckodriver, for example from the releases page, with your Linux package manager, or with brew install geckodriver on macOS. Make sure it is on your PATH.
Start it in a separate terminal. It listens on port 4444 by default:
```
geckodriver --port 4444
```
To run Firefox without a visible window:
```
MOZ_HEADLESS=1 geckodriver --port 4444
```

Check that it is ready:

curl http://localhost:4444/status
# Expected: {"value":{"message":"","ready":true}}

Windows

(All commands are run in PowerShell.)

Download geckodriver from the releases page and place geckodriver.exe somewhere on your PATH (e.g. C:\tools\). If you use XAMPP, also add C:\xampp\php to your PATH.
Open a separate PowerShell window and start geckodriver. It listens on port 4444 by default:
```
geckodriver
```
To run headless (no Firefox window):
```
$env:MOZ_HEADLESS = 1
geckodriver
```

Check that it is ready:

Invoke-WebRequest "http://localhost:4444/status" -UseBasicParsing | Select-Object -ExpandProperty Content
# Expected: {"value":{"message":"","ready":true}}

Alternative: Selenium Server

CI uses the Selenium Server 3.9 JAR, which needs Java and runs at the path /wd/hub. If you use Selenium Server (or the selenium/standalone-firefox Docker image), do not set WEBDRIVERSUFFIX. With Docker, set WEBDRIVERHOST to the container's host name, and DOMAIN to an address that the browser inside the container can reach.

Running the tests

Unit tests only (no browser needed):

php vendor/bin/phpunit --testdox tests/unit

Functional tests with geckodriver (Linux/macOS):

export DOMAIN=localhost/limesurvey   # URL of your installation, without http://
export WEBDRIVERSUFFIX=none          # required for geckodriver
php vendor/bin/phpunit --testdox tests/functional

The same on Windows:

$env:DOMAIN = "localhost/limesurvey"
$env:WEBDRIVERSUFFIX = "none"
php vendor/bin/phpunit --testdox tests/functional

Other useful commands:

Task	Command
One test file	`php vendor/bin/phpunit --testdox tests/functional/backend/AdminViewsTest.php`
One test method	`php vendor/bin/phpunit --filter testStartingValuesPrefilled tests/functional/frontend/PrefillStartingValuesTest.php`
One functional test suite as in CI	`php vendor/bin/phpunit --testsuite theme` (suites are defined in `phpunit.xml`)
Show PHP warnings as errors, as in CI	`LIMESURVEY_DEBUG=2 php vendor/bin/phpunit --exclude-group debugoff tests/unit`
On PostgreSQL or MSSQL	Add `--exclude-group mysql`
Stop at the first failure	Add `--stop-on-failure`

Environment variables

Variable	Default	Description
`DOMAIN`	(required for functional tests)	URL of your installation without `http://`, e.g. `localhost` or `localhost/limesurvey`
`WEBDRIVERSUFFIX`	`/wd/hub`	Set to `none` when using geckodriver directly. Leave it unset for Selenium Server.
`WEBDRIVERHOST`	`localhost`	Host where geckodriver or Selenium is running
`ADMINUSERNAME`	`admin`	LimeSurvey admin user name
`PASSWORD`	`password`	LimeSurvey admin password
`MOZ_HEADLESS`	(unset)	Set to `1` before starting geckodriver to run Firefox without a window
`KEEP_BROWSER_OPEN`	(unset)	Set to `1` to keep the browser open after a test class, to see where it stopped
`LIMESURVEY_DEBUG`	(debug setting from `config.php`)	Debug level for the tests. `2` reports all PHP warnings and notices.

Troubleshooting

phpunit disabled. NEVER run tests on a production system: The file enabletests is missing in the LimeSurvey root folder.

HTTP method not allowed: You use geckodriver but did not set WEBDRIVERSUFFIX=none. The default path /wd/hub only exists in Selenium Server.

Could not connect to remote web driver: geckodriver is not running, or runs on another port. Check that http://localhost:4444/status returns "ready":true.

Element could not be scrolled into view: The browser window may be too small. CI uses 1280×1024 (MOZ_HEADLESS_WIDTH/MOZ_HEADLESS_HEIGHT). For a quick check, maximize the window by adding self::$webDriver->manage()->window()->maximize(); in the test's setUpBeforeClass().

A test fails only sometimes: This is usually a timing problem: the test looks for an element before the page has finished loading. Wait for the element with self::$webDriver->wait(10)->until(...) instead of using sleep().

Xdebug timeout warnings in the output: Harmless. Set xdebug.mode=off in php.ini when you are not debugging.

Writing tests

What should be tested?

Everything that can break and be reported as a bug. Most important is that answers given by participants are saved correctly, and that the Expression Manager calculates the right values.

Where tests go

Folder	Base class	Use for
`tests/unit/` (subfolders such as `models`, `helpers`, `services`, `api`)	`ls\tests\TestBaseClass`	Testing PHP code without a browser
`tests/functional/backend/`	`ls\tests\TestBaseClassWeb`	Admin interface in the browser
`tests/functional/frontend/`	`ls\tests\TestBaseClassWeb`	Taking a survey in the browser
`tests/data/surveys/`	–	Survey files (.lss) used by the tests

Useful helpers in the base classes:

self::importSurvey($file) imports a survey in setUpBeforeClass(). The survey is deleted automatically in tearDownAfterClass(). self::$surveysFolder points to tests/data/surveys.
$this->getAllSurveyQuestions() returns the questions of the imported survey by question code.
self::adminLogin() logs in to the admin interface (functional tests).
self::createUserWithPermissions(), self::installAndActivatePlugin() and self::dispatchPluginEvent() help with permission and plugin tests.

Adding a regression test

Create a minimal survey that shows the bug, export it as .lss file and save it in tests/data/surveys/.
Create a test class in the namespace ls\tests. The file name must match the class name, and the class name must end with Test, e.g. PrefillStartingValuesTest.php.
Import the survey in setUpBeforeClass() and write one test...() method per case.
For a functional test: add the file to one of the functional test suites in phpunit.xml. Otherwise it never runs in CI.

Example (shortened from tests/functional/frontend/PrefillStartingValuesTest.php):

<?php

namespace ls\tests;

use Facebook\WebDriver\WebDriverBy;

class PrefillStartingValuesTest extends TestBaseClassWeb
{
    public static function setUpBeforeClass(): void
    {
        parent::setUpBeforeClass();
        self::importSurvey(self::$surveysFolder . '/limesurvey_survey_PrefillStartingValues.lss');
    }

    public function testStartingValuesPrefilled()
    {
        $urlMan = \Yii::app()->urlManager;
        $urlMan->setBaseUrl('http://' . self::$domain . '/index.php');
        $url = $urlMan->createUrl('survey/index', [
            'sid' => self::$surveyId,
            'newtest' => 'Y',
            'PREFILL1' => 'A3',
        ]);
        $questions = $this->getAllSurveyQuestions();

        self::$webDriver->get($url);
        self::$webDriver->next();

        $sgqa = self::$surveyId . 'X' . $questions['PREFILL1']->gid . 'X' . $questions['PREFILL1']->qid;
        $dropdown = self::$webDriver->findElement(WebDriverBy::id('answer' . $sgqa));
        $this->assertEquals('A3', $dropdown->getAttribute('value'));
    }
}

Rules

Never use markTestSkipped(). CI runs with --fail-on-skipped, so a skipped test fails the build.
Tests that only work on MySQL get @group mysql in the class or method DocBlock.
Tests that cannot run with debug mode on get @group debugoff.
Clean up what you create (surveys, users, plugins, files), so that the next test starts from a clean state.

For the PHPUnit basics, see the PHPUnit 9.6 manual. For the browser API, see php-webdriver.

Main chapters

Regression and unit tests

From LimeSurvey Manual

Contents