Parameterized Test

Getting Started with Parameterized Test

The Cangjie unittest framework supports parameterized tests in the data DSL format. The framework automatically inserts input parameters to the test.

The following is a complex code example that sorts an array using a function.

package example

func sort(array: Array<Int64>): Unit {
    for (i in 0..array.size) {
        var minIndex = i
        for (j in i..array.size) {
            if (array[j] < array[minIndex]) {
                minIndex = j
            }
        }
        (array[i], array[minIndex]) = (array[minIndex], array[i])
    }
}

This function is not the optimal and most efficient sorting implementation, but it serves the purpose. Let's test it.

package example

@Test
func testSort() {
    let arr = [45, 5, -1, 0, 1, 2]
    sort(arr)
    @Expect(arr, [-1, 0, 1, 2, 5, 45])
}

The test results show that the function works with a single input. Now test whether the sorting function still works if the array contains only equal elements.

@Test
func testAllEqual() {
    let arr = [0, 0, 0, 0]
    let expected = [0, 0, 0, 0]
    sort(arr)
    @Expect(expected, arr)
}

The test results show that the function works, but it is still unclear whether it works for arrays of all sizes. Next, test parameterized arrays of different sizes.

@Test[size in [0, 1, 50, 100, 1000]]
func testAllEqual(size: Int64) {
    let arr = Array(size, item: 0)
    let expected = Array(size, item: 0)
    sort(arr)
    @Expect(expected, arr)
}

Now, the parameterized test is complete. This is the simplest form of parameterized testing, called value-driven testing, where the values are listed directly in the code. Multiple parameters are allowed in a parameterized test. We can specify not only the size of the sorting function but also the elements to be tested.

@Test[
    size in [0, 1, 50, 100, 1000],
    item in (-1..20)
]
func testAllEqual(size: Int64, item: Int64) {
    let arr = Array(size, item: item)
    let expected = Array(size, item: item)
    sort(arr)
    @Expect(expected, arr)
}

Note that the value range of item is -1..20, which is not an array. So, how will the results look when this test is run? The values of size and item will be combined to form test cases and results will be returned. Therefore, be cautious not to configure too many parameters for a function; otherwise, the number of combinations may grow too large, making the tests slow. In the preceding example, the size parameter has five values and item has 21 values, resulting in a total of 21 x 5=105 combinations.

Note that value-driven testing is not limited to integer or built-in types. It can be used with any Cangjie types. Consider the following test:

@Test[
    array in [
        [],
        [1, 2, 3],
        [1, 2, 3, 4, 5],
        [5, 4, 3, 2, 1],
        [-1, 0],
        [-20]
    ]
]
func testDifferentArrays(array: Array<Int64>) {
    // Test whether the array is sorted.
    for (i in 0..(array.size - 1)) {
        @Expect(array[i] <= array[i + 1])
    }
}

Here, the test data is provided directly as parameters.

Testing code with arrays can be too bulky. In some cases, it may be easier to randomly generate data for such a generic function. The following shows a more advanced form of parameterized testing: random testing. You can create a random test by replacing the value-driven arrays or ranges with the unittest.random<T>() function:

@Test[
    array in random()
]
func testRandomArrays(array: Array<Int64>) {
    // Test whether the array is sorted.
    for (i in 0..(array.size - 1)) {
        @Expect(array[i] <= array[i + 1])
    }
}

This test essentially generates a large number of completely random values (200 by default) and uses them to test the code. The values are not entirely random but tend to include boundary values, such as zero, maximum and minimum values of specific types, and empty sets.

Note: It is recommended that random testing and manually written tests be used together, as practices show that they complement each other.

To better describe random testing, put the sorting function aside and write the following test code:

@Test[
    array in random()
]
func testNonsense(array: Array<Int64>) {
    if (array.size < 2) { return }
    @Expect(array[0] <= array[1] + 500)
}

Running this test generates output similar to the following:

[ FAILED ] CASE: testNonsense (1159229 ns)
    REASON: After 4 generation steps and 200 reduction steps:
        array = [0, -453923686263, 0]
    with randomSeed = 1702373121372171563

    Expect Failed: `(array [ 0 ] <= array [ 1 ] + 500 == true)`
       left: false
      right: true

The result shows that the test with the array [0, -453923686263, 0] fails. Run the test again:

[ FAILED ] CASE: testNonsense (1904718 ns)
    REASON: After 5 generation steps and 200 reduction steps:
        array = [0, -1196768422]
    with randomSeed = 1702373320782980551

    Expect Failed: `(array [ 0 ] <= array [ 1 ] + 500 == true)`
       left: false
      right: true

The test fails again, but the values are different. Why does this happen? Because random tests are inherently random, new random values are generated each time. It may be easy to test functions with various data, but for some tests, this means that the test may pass sometimes and fail at others, making it difficult to share test results. Random testing is a powerful tool but comes with these drawbacks which we need to be clear of when using it.

How do we show results to other developers when the test results differ each time? The answer is simple—it is in the output of running the test:

with randomSeed = 1702373320782980551

A random seed is provided above, which can be used as a configuration in the test. Modify the test as follows:

@Test[
    array in random()
]
@Configure[randomSeed: 1702373320782980551]
func testNonsense(array: Array<Int64>) {
    if (array.size < 2) { return }
    @Expect(array[0] <= array[1] + 500)
}

The output will now be:

[ FAILED ] CASE: testNonsense (1910109 ns)
    REASON: After 5 generation steps and 200 reduction steps:
        array = [0, -1196768422]
    with randomSeed = 1702373320782980551

    Expect Failed: `(array [ 0 ] <= array [ 1 ] + 500 == true)`
       left: false
      right: true

Note that the values generated, the steps used to generate the values, and the random seed used in this run are the same as those in the previous run. This mechanism makes random tests reproducible, allowing them to be shared in a test suite with other developers. You can also just extract data from random tests (in this example, the array values [0, -1196768422]) and write new tests using those values.

Take a quick look at the output from the failed test.

    REASON: After 5 generation steps and 200 reduction steps:
        array = [0, -1196768422]
    with randomSeed = 1702373320782980551

The output contains the following important information:

• Number of generation steps before the test fail: This is the number of iterations running before the test fails.
• Number of reduction steps : Random tests have a mechanism for reducing test data, which makes it easier to work with and improves the readability by minimizing the data set.
• Actual data that causes the test failure: All parameters are list in order (in this case,, only one parameter array) along with the values that actually cause the test failure. The parameters must implement the ToString interface. Otherwise, only placeholders are printed.
• The random seed: Used to reproduce the random test.

Some tests are tricky and require adjustments in the random generation steps. You can control this with the following two configuration parameters: generationSteps and reductionSteps. A significant advantage of random testing is that it can run for a long time if enough steps are set, testing millions of values.

However, to avoid excessively long tests, the maximum values for both generationSteps and reductionSteps are 200 by default. These two configuration parameters allow you to set the maximum number of steps that the framework runs. For example, for a small test like the one mentioned earlier, setting a large value for the parameters may not make much sense. Previous runs have shown that the test typically fails in fewer than 10 steps.

Type-Parameterized Test

Although the current sorting implementation is designed specially for integer arrays, it could, in essence, sort any type. You can modify the sort function to make it generic, allowing it to sort arrays of any type. Note that elements must be comparable in order to be sorted.

package example

func sort<T>(array: Array<T>): Unit where T <: Comparable<T> {
    for (i in 0..array.size) {
        var minIndex = i
        for (j in i..array.size) {
            if (array[j] < array[minIndex]) {
                minIndex = j
            }
        }
        (array[i], array[minIndex]) = (array[minIndex], array[i])
    }
}

All tests continue to run normally, showing that the sorting function is widely applicable. But does it really work for types other than integers?

To check this, write a new test based on the testDifferentArrays function in the previous example and use it to test other types (such as strings):

@Test[
    array in [
        [],
        ["1","2","3"],
        ["1","2","3","4","5"],
        ["5","4","3","2","1"]
    ]
]
func testDifferentArraysString(array: Array<String>) {
    // Test whether the array is sorted.
    let sorted = array.clone()
    sort(sorted)
    for (i in 0..(sorted.size - 1)) {
        @Expect(sorted[i] <= sorted[i + 1])
    }
}

The test result shows that it works correctly. Note that the subjects and assertions of the two tests are the same. Imagine how convenient it would be if the test for a generic function could also be generic.

Revisit the previous random testing example:

@Test[array in random()]
func testRandomArrays(array: Array<Int64>) {
    let sorted = array.clone()
    sort(sorted)
    for (i in 0..(sorted.size - 1)) {
        @Expect(sorted[i] <= sorted[i + 1])
    }
}

The test is widely applicable. The element type is not limited to Int64, but can be any type T. For example, the elements:

• Are comparable: They need to implement Comparable<T>.
• Support random instance generation: They need to implement Arbitrary<T>.

Now, rewrite this test as a generic function:

@Test[array in random()]
func testRandomArrays<T>(array: Array<T>) where T <: Comparable<T> & Arbitrary<T> {
    let sorted = array.clone()
    sort(sorted)
    for (i in 0..(sorted.size - 1)) {
        @Expect(sorted[i] <= sorted[i + 1])
    }
}

After compiling and running, we encounter an issue:

An exception has occurred:
MacroException: Generic function testRandomArrays requires a @Types macro to set up generic arguments

Naturally, to run the test, we need to provide types for testing. We can set the types using the @Types macro, allowing the test to run with Int64, Float64, and String.

@Test[array in random()]
@Types[T in <Int64, Float64, String>]
func testRandomArrays<T>(array: Array<T>) where T <: Comparable<T> & Arbitrary<T> {
    let sorted = array.clone()
    sort(sorted)
    for (i in 0..(sorted.size - 1)) {
        @Expect(sorted[i] <= sorted[i + 1])
    }
}

Now, when you run the test, it compiles and generates the following output:

    TCS: TestCase_testRandomArrays, time elapsed: 2491737752 ns, RESULT:
    [ PASSED ] CASE: testRandomArrays<Int64> (208846446 ns)
    [ PASSED ] CASE: testRandomArrays<Float64> (172845910 ns)
    [ PASSED ] CASE: testRandomArrays<String> (2110037787 ns)

As you can see, each type test runs separately, as the behavior may vary greatly between types. The @Types macro can be used for any parameterized test cases, including test functions and test classes.

Reusing, Combining, and Mapping Parameter Strategies

Take HashSet as an example. Start by testing the contains function.

@Test[data in random()]
func testHashSetContains(data: Array<Int64>) {
    let hashSet = HashSet(len)
    hashSet.putAll(data)
 
    for (element in data){
        @Assert(hashSet.contains(element))
    }
}

This works well. Now try testing the remove function.

@Test[data in random()]
func testHashSetRemove(data: Array<Int64>) {
    let hashSet = HashSet(len)
    hashSet.putAll(data)
 
    for (element in data){
        @Assert(hashSet.remove(element))
    }
}

At first glance, it seems like the code should work fine. However, you will soon notice that it does not work correctly because the randomly generated array may contain duplicate items, which will cause the second call to remove to fail. What we need is to generate arrays without duplicates.

var counter = 0
@OverflowWrapping
func generateUniqArray(len: Int64, start: Int64){
    let rng = Random(start)
    let step = Int64.Max / len
    counter = 0
    Array(len, { _ => 
        counter += rng.nextInt64()%step 
        counter
    } )
}

@Test[len in random(), start in random()]
func testHashSetRemove(len: Int64, start: Int64) {
    let data = generateUniqArray(len,start)
    let hashSet = HashSet(len)
    hashSet.putAll(data)
 
    for (element in data){
        @Assert(hashSet.remove(element))
    }
}

However, even though the data generation has been moved to a separate function, there is still a considerable amount of duplication if it is expected to be reused across different tests. In addition, the distinction between preparation code and test code is unclear. To solve this issue, the test framework provides a convenient API in the form of the @Strategy macro, allowing existing strategies to be combined and mapped.

var counter = 0
@Strategy[len in random(), start in random()]
@OverflowWrapping
func generateUniqArray(len: Int64, start: Int64): Array<Int64>{
    let rng = Random(start)
    let step = Int64.Max / len
    counter = 0
    Array(len, { _ => 
        counter += rng.nextInt64()%step 
        counter
    } )
}

Now, you can use the strategy for this input:

@Test[data in generateUniqArray]
func testHashSetRemove(data: Array<Int64>) {
    let hashSet = HashSet()
    hashSet.putAll(data)
 
    for (element in data){
        @Assert(hashSet.remove(element))
    }
}
 
@Test[data in generateUniqArray]
func testHashSetToArray(data: Array<Int64>) {
    let hashSet = HashSet()
    hashSet.putAll(data)
 
    let result = hashSet.toArray()
    result.sort()
    data.sort()
    @Assert(result == data)
}