OpenQA-agnostic testing

[a proposal]

Andrea Manzini

• https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/data/security/testPolkit

ingredients

• metadata enabled test runner script
• testPolkit.go main test program logic
• reusable utility libraries for
  • process execution
  • output format
• (optional) a Perl module to run in openQA

The main dish

an Array of TestCase

	testCases := map[int]tap.TestCase{
		1: {"Save original hostname", nil},
		2: {"Check polkit rules directory permissions (root:polkitd)", checkPermissions},
		3: {"Add polkit rule and restart service", addRuleAndRestart},
		4: {"Change hostname without authentication", changeHostnameWithAuth},
		5: {"Verify hostname was changed", verifyHostnameChanged},
		6: {"Remove polkit rule and restart service", removeRuleAndRestart},
		7: {"Hostname change should fail without authentication", changeHostnameShouldFail},
		8: {"Verify hostname was not changed", verifyHostnameUnchanged},
		9: {"Restore original hostname", nil},
	}

• Small , testable functions with simple, straightforward logic.
• You can see the test plan without reading the code.

side dish: Auxiliary libs

• TAP (Test Anything Protocol) runner and formatter:

  receives an array of tests and run it, properly formatting the output

• exec helper:

  run a command in background, reading stdout/stderr and manage timeout

dessert

• metadata embedded !

---
test: policykit rules
desc: Verifies functionality of policykit rules
steps:
  - check folder permissions /etc/polkit-1/rules.d/ , bail out if !root:polkitd
  - save original hostname
  - add a permissive polkit rule in /etc/polkit-1/rules.d/ and restart polkit service
  - try to change hostname, should succeed (do not ask root password)
  - ensure hostname has been changed
  - remove polkit rule and restart service
  - try to change hostname, should fail (will ask root password)
  - ensure hostname has NOT been changed
  - restore original hostname
author: <andrea.manzini@suse.com>
maintainer: QE Security <none@suse.de>
expected: no errors raised, user is allowed/denied to change hostname according to the rule
platform: Tumbleweed
tags: security polkit bsc#1249581

how do I run it outside of openQA ?

• [start a Tumbleweed vm and access it]
• install packages: zypper in go git polkit
• git clone --depth 1 https://github.com/os-autoinst/os-autoinst-distri-opensuse
• cd os-autoinst-distri-opensuse/data/security/testPolkit
• sudo ./runtest
• cat testPolkit.tap

what openQA Perl module does:

• installs Go compiler
• download 'data' files (the real test program)
• creates directories and puts Go libraries in place
• run actual test
• export results in TAP format to openQA, cleanup
• example run : https://openqa.opensuse.org/tests/5404762#step/polkit_rules/27

(see 'external results')

sub run {
    select_serial_terminal;

    my @files = qw(runtest go.mod testPolkit.go utils/utils.go tap/tap.go);
    # install go and download test files
    zypper_call 'in go';
    assert_script_run 'mkdir -p ~/testPolkit && cd ~/testPolkit';
    my $url = data_url("security/testPolkit/");
    assert_script_run 'curl -s --create-dirs ' . join ' ', map { "-O $url/$_" } @files;
    assert_script_run 'mkdir utils tap && mv utils.go utils/ && mv tap.go tap/';

    # run test and generate result file
    assert_script_run("chmod +x ./runtest && ./runtest && mv testPolkit.tap /tmp/polkit_rules.tap");

    #cleanup after test
    assert_script_run("cd ~ && rm -rf testPolkit");
    parse_extra_log('TAP', '/tmp/polkit_rules.tap');
}

Pro and cons

strongly typed, catch errors at compile time

can be run independently (e.g. give it to a customer or stakeholder)

compatible with other automated test runners (Jenkins, ArgoCD, Travis CI , Github CI, AWS/Azure ... )

Can't see the test source in openQA webui

Less features than os-autoinst (needles ?)

Next steps

• collect feedback, explore the idea adding more tests
• decide files location and project layout
• make it scalable, for example converting Tap type to an interface{}
• concurrent/parallel testing ?

Thanks for watching

OpenQA-agnostic testing

[a proposal]

Andrea Manzini

Question time!