Introduction to Perl

Testing

G. Wade Johnson

Houston.pm

notes

What is Testing?

A way to ...

The word testing means different things to different people. In many cases, testing is just a means to an end. When we are talking about testing in Perl, we are actually talking more about a development philosophy.

Ad Hoc Testing

Exercise the code by hand to see if it breaks

There is nothing wrong with ad hoc testing, unless it is the only form of testing you use. This approach (obviously) requires a lot of person-hours to perform. It is also not really suitable for use in verifying code repeatedly. It is very repetitious, which generally means error-prone when executed by humans.

Automated Testing

Have the computer do the boring work.

Most of the testing we want to talk about is automated. When the procedure is to run the same test day after day, you really want to hand the test off to a computer. This makes the test more reproducable and, quite frankly, the computer does not get bored and wander off topic.

How to Test?

Many different people have created testing frameworks in hopes of automating away the tedium of tests. Some are complicated. Some are simple. Others are incredibly flexible and customizable.

In keeping with Perl's philosophy, there are many ways to test Perl code as well. We will focus on the most widely used and standardized system in the Perl ecosystem: TAP.

Test Anything Protocol (TAP)

    1..4
    ok 1 - Initial sanity verified
    ok 2 - Sanity still exists
    not ok 3 - Sanity has left the building
    #   Failed test 'Sanity has left the building'
    #   at examples/sanity.t line 11.
    #          got: '4'
    #     expected: '5'
    ok 4 - Sanity has been restored
    # Looks like you failed 1 test of 4.

This is a very simple example of the output from a TAP-based test. It shows all of the major parts of a TAP test.

  • Marker indicating the number of tests to run
  • A series of test assertions.
    • Success indicator: ok or not ok
    • Assertion number
    • Assertion name/label
  • Diagnostic information preceded by a #

Test::Simple

    #!/usr/bin/perl
    use Test::Simple tests => 4;

    use strict;
    use warnings;

    my $var = 1;
    ok( $var == 1, 'Initial sanity verified' );
    ok( $var =~ m/^\d+$/, 'Sanity still exists' );
    ok( 2+2 == 5, 'Sanity has left the building' );
    ok( 2+2 != 5, 'Sanity has been restored' );

The simplest Perl module supporting TAP is Test::Simple. It only supplies the ability to set the test plan and a single function ok(). The ok() function tests a scalar value, printing ok on true and not ok for a false value.

It also provides the assertion number automatically, and adds a label if provided as a second argument.

Test::More

    #!/usr/bin/perl
    use Test::More tests => 4;

    use strict;
    use warnings;

    my $var = 1;
    is( $var, 1, 'Initial sanity verified' );
    like( $var, qr/^\d+$/, 'Sanity still exists' );
    is( 2+2, 5, 'Sanity has left the building' );
    isnt( 2+2, 5, 'Sanity has been restored' );

The more advanced Test::More module provides a more complete set of assertion functions. These don't really do much more than the straight ok() function (which is also provided) if the assertion succeeds. But, if the assertion fails, they provide more useful diagnostic output.

Test::More Assertions

These are the major test assertion functions provided by Test::More. The module does provide other functionality, but we will not be exploring that in an introduction.

More Test Modules

You are not limited to Test::Simple and Test::More. There are a large number of specialized modules that provide other assertions and extend functionality. All of these still output TAP, so they are compatible with any tools that would recognize that output.

More Advanced Testing

There are also test modules that go far beyond just testing what your methods return.

Why Test?

notes

Robust Code

Testing cannot find all bugs, but not testing cannot find any bugs.

notes

Documentation

Well-written tests show how the module should be used.

notes

Interface Design

Writing tests makes you use an interface.

notes

Regression Protection

Any bug that is covered by a unit test will not come back
(without you knowing).

notes

What to Test

The old Extreme Programming methodology used the first approach to decide what should be tested.

Most people remember to test the obvious success case. It is equally important to look for places on the edges of good behavior or changes in behavior and test around those areas as well. Edges are a rich source of bugs.

Fewer developers remember to test the failure cases. Do you know how the code acts if it is passed a bad parameter? How about if a file it depends on does not exist? How about permission failures? Testing these areas makes certain code is in place to handle the sorts of surprises that happen in the real world. These corners of code are often never tested except by unit tests. So the first time they may execute is in production.

Almost nobody remembers to test their assumptions. These assertions almost never fail. When they do, they tend to generate very strange behavior that may take days to unravel.

Example Unit Tests

Look at test code from SVG::Sparkline.

These files are real unit tests copied from the SVG::Sparkline module. They are tests for real code, but are not much more complex than the simple examples from before.

Kinds of Testing

notes

Unit Tests for Coverage

The Devel::Cover module helps determine what code is executed.

Together with a good test suite, we can increase confidence in our code.

notes