Test specs

Test specs allow any Test Result coming from either results command directly or from sync tests with --poll or --wait options to be asserted by comparing the actual result with expected results defined by a spec JSON string or file.

JSON Test Specs

Specs are defined following Test Results JSON output whereas response.data tree structure returned from results command is traversed and all matching leaves from test specs definition are compared. e.g.:

testspecs.json:

{
  "median": {
    "firstView": {
      "requests": 20
      "render": 400,
      "loadTime": 3000,
      "score_gzip": {
        "min": 90
      }
    }
  }
}

By running:

$ webpagetest test http://staging.example.com --first --poll --spec testspecs.json

After tested and returning the following test results:

{
  "response": {
    ...
    "data": {
    ...
      "median": {
        "firstView": {
          ...
          "requests": 15
          "render": 500,
          "loadTime": 2500,
          "score_gzip": 70
          ...
        }
      }
    }
  ...
  }
}

It is compared to testspecs.json and the output would be:

WebPageTest
    ✓ median.firstView.requests: 15 should be less than 20 
    1) median.firstView.render: 500 should be less than 400
    ✓ median.firstView.loadTime: 2500 should be less than 3000 
    2) median.firstView.score_gzip: 70 should be greater than 90

  2 passing (3 ms)
  2 failing

With exit status:

$ echo $?
2

Comparison

By default all comparisons operations are < (lower than), except when an object is informed with min and/or max values, in this case the operations used for comparison are > (greater than) and < (lower than) when both min and max are informed than a range comparison is used.

Examples

Lower than comparison

{ "median": { "firstView": {
  "render": 400
}}}

or

{ "median": { "firstView": {
  "render": { "max": 400 }
}}}

Greater than comparison

{ "median": { "firstView": {
  "score_gzip": { "min": 75 }
}}}

Range comparison

{ "median": { "firstView": {
  "requests": { "min": 10, "max": 30 }
}}}

Defaults

It is possible to optionally define default operations and label templates inside defaults property in the specs JSON file:

{
  "defaults": {
    "suiteName": "Performance Test Suite for example.com",
    "text": ": {actual} should be {operation} {expected} for {metric}",
    "operation": ">"
  },
  "median": { "firstView": {
    "score_gzip": 80,
    "score_keep-alive": 80
  }}
}

Test suite name and specs texts will be used in the test output and both scores should be greater than 80 as follows:

Performance Test Suite for example.com
    1) 70 should be greater than 80 for median.firstView.score_gzip
    ✓ 100 should be greater than 80 for median.firstView.score_keep-alive

  1 passing (3 ms)
  1 failing

If defaults property is omitted, the following properties are used:

  "defaults": {
    "suiteName": "WebPageTest",
    "text": "{metric}: {actual} should be {operation} {expected}",
    "operation": "<"
  }

Text template tags

• {metric}: metric name, eg: median.firstView.loadTime
• {actual}: the value returned from the actual test results, eg: 300
• {operation}: the long operation name, eg: lower than
• {expected}: the defined expected value, eg: 200

Operations

• "<" → lower than
• ">" → greater than
• "<>" → greater than and lower than (range)
• "=" → equal to

Single overrides

Overriding individual specs labels is also possible by providing text in the spec object:

{ "median": { "firstView": {
  "loadTime": {
    "text": "page load time took {actual}ms and should be no more than {expected}ms",
    "max": 3000
  }
}}}

Which outputs:

WebPageTest
    ✓ page load time took 2500ms and should be no more than 3000ms

  1 passing (2 ms)

Reporter

WebPageTest API Wrapper Test Specs use Mocha to build and run test suite. The following reporters are available:

• dot (default)
• spec
• tap
• xunit
• list
• progress
• min
• nyan
• landing
• json
• doc
• markdown
• teamcity

Jenkins Integration

Integration with Jenkins and other CI tools is seamless: Using a sync test command with either --poll or --wait (if Jenkins server is accessible from private instance of WebPageTest server) and specifying --specs file or JSON string with either tap or xunit as --reporter.

Configuration

TAP

* TAP plugin installed from Jenkins Plugin Manager

JUnit

Reports

TAP

JUnit