Verification API of Mock Framework

The verification API is part of the mock framework and provides the following functionality:

  • Verifying whether certain calls are executed
  • Verifying the number of specific calls
  • Verifying whether specified parameters are used for calling
  • Verifying whether calls follow a specified order

Verify whether assertions can be run by checking the call logs created during the test. The call logs cover all calls that make mock and spy objects (as well as static members, top-level functions, and top-level variables) accessible in the test. Only calls made on mock and spy objects (as well as static members, top-level functions, and top-level variables) can be verified.

The Verify class provides access to the verification API. The @Called macro is used to construct assertions about code.

The @Called macro call constructs a verification statement, that is, checks a single assertion of the code against the call log. The Verify class is a collection of static methods. Methods such as that, ordered, and unordered can be called to construct verification blocks.

Example

let foo = mock<Foo>()
//Configure foo.
@On(foo.bar()).returns()
foo.bar()
Verify.that(@Called(foo.bar())) //Verify that bar is called at least once.

Verification Statements and @CalledMacro

Verification statements are represented by the VerifyStatement class. The VerifyStatement instance is created by the @Called macro.

Similar to the @On macro, the @Called macro call accepts stub signatures, and complies with the rules of parameter matchers.

Examples:

@Called(foo.bar(1, _)) //Match the verification statement called by the bar method, where the first parameter is '1'.
@Called(Foo.baz)       //Match the verification statement called by the baz static attribute getter.

The API provided by the VerifyStatement class is similar to the cardinality specifier available during stub configuration.

Cardinality functions are as follows:

  • once()
  • atLeastOnce()
  • times(expectedTimes: Int64)
  • times(min!: Int64, max!: Int64)
  • atLeastTimes(minTimesExpected: Int64)
  • never()

Calling these functions returns the same VerifyStatement instance. Cardinality cannot be reset for one statement and must be set before the statement is passed to the verification block generator function. If cardinality is not explicitly set, the default cardinality is used.

Verify.that(@Called(foo.bar()).atLeastOnce())
Verify.that(@Called(foo.bar()).once())

Verification Block

A verification block usually contains one or more verification statements, and checking the statements the check block will construct more complex assertions.

A call to a verification block is verified immediately, but subsequent calls are not verified. Verification blocks do not change the state of the call log. Each call in the log can be checked by any number of blocks. Each block is checked independently. No dependency exists between blocks. The order of calling verification blocks is insignificant unless some calls occur between blocks or the call log is manually cleared.

A verification statement itself does not perform any type of verification but must be passed to a verification block for verification.

A single verification block only checks the calls to mock or spy objects mentioned by verification statements within the block, and calls to other objects are ignored.

The Verify class contains several static methods for constructing verification blocks. Ordered verification blocks are used to check the exact order of calls. Unordered verification blocks verify only the number of calls.

Ordered

To check the call sequence of one or more objects, the ordered verification block generator is used.

The static function ordered receives an array of verification statements.

for (i in 0..4) {
    foo.bar(i % 2)
}

Verify.ordered(
    @Called(foo.bar(0)),
    @Called(foo.bar(1)),
    @Called(foo.bar(0)),
    @Called(foo.bar(1))
)

The call sequence of multiple objects can be checked.

for (i in 0..4) {
    if (i % 2 == 0) {
        fooEven.bar(i)
    }
    else {
        forOdd.bar(i)
    }
}

Verify.ordered(
    @Called(fooEven.bar(0)),
    @Called(fooOdd.bar(1)),
    @Called(fooEven.bar(2)),
    @Called(fooOdd.bar(3)),
)

The default cardinality specifier for ordered verification is once(). Other cardinality specifiers can be used if necessary.

for (i in 0..4) {
    foo1.bar(i)
}

for (i in 0..4) {
    foo2.bar(i)
}

Verify.ordered(
    @Called(foo1.bar(_).times(4)),
    @Called(foo2.bar(_).times(4))
)

For ordered verification, all calls to mock or spy objects (mentioned in the block) must be listed. Any unlisted call will cause a validation failure.

foo.bar(0)
foo.bar(10)
foo.bar(1000)

Verify.ordered(
    @Called(foo.bar(0)),
    @Called(foo.bar(10))
)

Output:

Verification failed
    No statement is matched for the call below:
        foo.bar(...) at example_test.cj:6

Unordered

Unordered verification checks only the number of times its verification statement is called.

For unordered verification, the cardinality atLeastOnce() is used unless explicitly specified, that is, the statement called at least once is checked.

for (i in 0..4) {
    foo.bar(i % 2)
}

//Verifies whether the bar with the parameter of 0 and 1 is called at least once.
Verify.unordered(
    @Called(foo.bar(0)),
    @Called(foo.bar(1))
)

//Verifies whether the bar with the parameter of 0 and 1 is called twice.
Verify.unordered(
    @Called(foo.bar(0)).times(2),
    @Called(foo.bar(1)).times(2)
)

//Verifies whether the bar is called four times in total.
Verify.unordered(
    @Called(foo.bar(_)).times(4)
)

Unordered verification falls into Partial and Exhaustive verifications.

Exhaustive verification is used by default. All calls to the mock or spy object mentioned in the verification statement must be listed. For Partial verification, only part of calls are listed.

for (i in 0..4) {
    foo.bar(i)
}

//Failed. Two calls to foo.bar() are not listed in the block.
Verify.unordered(
    @Called(foo.bar(0)).once(),
    @Called(foo.bar(1)).once()
)

//Unrelated calls are ignored.
Verify.unordered(Partial,
    @Called(foo.bar(0)).once(),
    @Called(foo.bar(1)).once()
)

Dynamic Construction of Verification Blocks

The ordered and unordered functions are overloaded functions supporting lambda. The checkThat(statement: VerifyStatement) function can be used to dynamically add statements.

Examples:

let totalTimes = 40
for (i in 0..totalTimes) {
    foo.bar(i % 2)
}

Verify.ordered { v =>
    for (j in 0..totalTimes) {
        v.checkThat(@Called(foo.bar(eq(j % 2))))
    }
}

Other APIs

The Verify class further provides the following tools:

  • that(statement: VerifyStatement) is the alias of Verify.unordered(Paritial, statement), which is used to check a single statement. It is unnecessary to list all calls to the corresponding mock or spy object.
  • noInteractions(mocks: Array) is used to check the mock or spy objects that are not called.
  • clearInvocationLog() resets the log to the empty state. This affects all subsequent verification blocks, but does not affect stub expectations.

    • Examples:

    foo.bar()
Verify.that(@Called(foo.bar())) // OK
Verify.noInteractions(foo)      // Failed. The call to foo.bar() is recorded in the log.
Verify.clearInvocationLog()     // Clear the log.
Verify.noInteractions(foo)      // Clear all interactions with foo from the log.
Verify.that(@Called(foo.bar())) // Failed.

    Verify APIs

    public static func that(statement: VerifyStatement): Unit

public static func unordered(
    exhaustive: Exhaustiveness,
    collectStatements: (UnorderedVerifier) -> Unit
): Unit

public static func unordered(
    collectStatements: (UnorderedVerifier) -> Unit
): Unit

public static func unordered(statements: Array<VerifyStatement>): Unit

public static func unordered(
    exhaustive: Exhaustiveness,
    statements: Array<VerifyStatement>
): Unit

public static func ordered(
    collectStatements: (OrderedVerifier) -> Unit
): Unit

public static func ordered(statements: Array<VerifyStatement>): Unit

public static func clearInvocationLog(): Unit

public static func noInteractions(mocks: Array<Object>): Unit

    Verification Errors

    If a verification fails, VerificationFailedException is thrown and the mock framework generates a report. Do not capture such exception.

    The failure types are as follows:

    • Excessively few or many calls: The number of calls does not match the statement in the block.
    • Statement mismatch: A statement in the block does not match the number of calls recorded in the log.
    • Call mismatch: The log contains a call that does not match the statement in the block.
    • Unexpected call: An ordered verification block requires other calls.
    • Useless interaction: noInteractions detects an unexpected call.

    Another failure type is disjoint statements. Therefore, a failure may not be caused by an error in test code. This failure type is reported when multiple statements match a call. This error may be caused if a statement with a disjoint parameter matcher is used in a single verification block. Fuzzy match is not allowed between statements and calls.

    Examples and Patterns

    Usually, the verification API is used to verify the interaction between test code (including functions, classes, and entire packages) and external objects. Details are as follows:

    • creating spy objects;
    • passing these spy objects to the test code; and
    • verifying the interaction between code and the spy objects.

    Verifying the Number of Calls

    func testDataCaching() {
    // Creates required spy or mock objects.
    let uncachedRepo = spy(Repository())
    let invalidationTracker = mock<InvalidationTracker>()
    @On(invalidationTracker.getTimestamp()).returns(0)

    // Prepares test data.
    let cachedRepo = CachedRepistory(uncachedRepo, invalidationTracker)

    // Runs test code.
    for (i in 0..10) {
        cachedRepo.get(TEST_ID)
    }

    // According to verification results, only the basic repo is queried once, and other calls to uncached repos do not exist.
    Verify.unordered(Exhaustive,
        @Called(uncachedRepo.get(TEST_ID)).once()
    )

    // Clears the log.
    Verify.clearInvocationLog()

    // Sets other behavior.
    @On(invalidationTracker.getTimestamp()).returns(1)

    for (i in 0..10) {
        cachedRepo.get(TEST_ID)
    }

    // Only one call is executed since the last clearing.
    Verify.unordered(Exhaustive,
        @Called(uncachedRepo.get(TEST_ID)).once()
    )
}

    Verifying Calls with Specified Parameters

    func testDrawingTriangle() {
    // Creates required spy or mock objects.
    let canvas = spy(Canvas())

    // Runs test code.
    canvas.draw(Triangle())

    // Tests that a triangle consists of three lines and three points.

    // Uses "that" block.
    Verify.that(
        @Called(canvas.draw(ofType<Dot>())).times(3)
    )
    Verify.that(
        @Called(canvas.draw(ofType<Line>())).times(3)
    )

    //Uses part of the unordered verification block instead.

    Verify.unordered(Partial, // The actual sequence of drawing lines and points is unknown.
        @Called(canvas.draw(ofType<Dot>())).times(3),
        @Called(canvas.draw(ofType<Line>())).times(3)
    )

    // During use of an enumeration block, all calls must be enumerated.
    Verify.unordered(Exhaustive,
        @Called(canvas.draw(ofType<Triangle>())).once(),
        @Called(canvas.draw(ofType<Dot>())).times(3),
        @Called(canvas.draw(ofType<Line>())).times(3)
    )

    // Verifies that the draw function with an input parameter of the Square type is never called.
    Verify.that(
        @Called(canvas.draw(ofType<Square>)).never()
    )

    // To distinguish parameters by using more complex conditions,
    // use the pattern below:
    let isDot = { f: Figure =>
        f is Dot //This is a more complex logic.
    }

    Verify.that(
        @Called(canvas.draw(argThat(isDot))).times(3)
    )

    // Note: It must be specified that statements in one block matches only one call.
    // In the counterexample below, some calls match two statements.
    Verify.unordered(
        @Called(canvas.draw(_)).times(7),
        @Called(canvas.draw(ofType<Dot>())).times(3)
    )
}

    Verifying the Call Sequence

    func testBuildFlight() {
    let plane = spy(Plane())

    FlightBuilder(plane).planFlight(Shenzhen, Shanghai, Beijing).execute()

    Verify.ordered(
        @Called(plane.takeOffAt(Shenzhen)),
        @Called(plane.landAt(Shanghai)),
        @Called(plane.takeOffAt(Shanghai)),
        @Called(plane.landAt(Beijing))
    )
}

    Expectation and Verification API

    When configuring a stub, you can set expectations and verification API to cover some assertions in the test code. In this case, you have to select a method that can better reflect the test intention.

    Generally, it is recommended that you avoid repeating the configuration steps in the verification block.

    let foo = mock<Foo>()
@On(foo.bar(_)).returns() //If this stub is never used, the test fails.

foo.bar(1)
foo.bar(2)

Verify.that(
    //Unnecessary. Automatic verification is used.
    @Called(foo.bar(_)).atLeastOnce()
)

//The number of calls and specific parameters can be examined instead.
Verify.unordered(
    @Called(foo.bar(1)).once(),
    @Called(foo.bar(2)).once()
)

    The example above can be rewritten using an expectation:

    let foo = mock<Foo>()
@On(foo.bar(1)).returns().once() //It is expected to be called once. The parameter is '1'.
@On(foo.bar(2)).returns().once() //It is expected to be called once. The parameter is '2'.

foo.bar(1)
foo.bar(2)

//If no stub is triggered, the test fails.