Back in 2003, Brian Marick published an influential series of articles on agile testing. He was attempting to point the way forward for agile testers. But, in the process, he came up with an elegant method of cataloguing testing methods that has become known as “The Marick Test Matrix.”

I’d like to introduce the matrix here in the hopes of fostering a discussion about what we test and how we test it.

Brian’s work categorized tests by asking two questions:

  • Is the test business facing or technology facing?
  • Does the test support engineering or critique a product?

When you combine the two questions (or axes), you get this grid (or matrix):

Let's take a look at each side of the matrix to explore their meaning...

Business Facing Tests

A business facing test is one that is expressed in terms that are well understood by a business expert. For example:

  • If you withdraw more money that you have in your account, the system should automatically extend you a loan for the differenct.

Business facing tests are best authored by people who understand the business (e.g. product owner, business analyst, etc.).

Technology Facing Tests

A technology facing test is one that is expressed in terms that are well understood by a technology expert. For example:

  • Different browsers implement Javascript differently, so we test whether our product works with the most important browsers.

Technology facing tests are best authored by people who understand the technology (e.g. developer, tester, etc.).

Tests that Support Programming

A test that supports programming is one that defines what the software should do. For example:

  • Clicking on the Account Details link should take the user to the Account Details screen.
  • Calling the Add method with 2 and 2 should return 4.

These tests may be written before the software exists. These tests are often automated and executed after a change is made to the software to ensure that the software still works as it should (i.e. regression testing). Once one of these tests passes, it should never be allowed to fail again.

Tests that Critique a Product

A test that critiques a product is one that tries to identify problems in completed software. In other words, this is the class of tests where the tester is actively trying to break the software in order to find bugs. For example:

  • When I logged on as Joe, I saw Tom’s data.
  • When I clicked the blue button after clicking the red button 400 times, the system threw an error.
  • When I configured the load testing tool to send 1,000 simultaneous users to the site, average response times increased to over 10 seconds.

In general, these tests are not automated – until a problem is identified, at which point a test that clearly reproduces the problem can be added to the tests that support programming.

Let’s take a look at where some different types of tests might go on the matrix:

Unit Tests

Unit tests are used by developers to ensure that the code they are writing does what they expect it to. In essence, these tests form a specification for a single unit of code. By that definition, unit tests support programming. These tests are also (or should be) very close to the code under test, which makes them technology facing. So, unit tests belong in the lower left quadrant of the matrix. The benefit of automating these tests is very high.

Functional Tests

Functional tests are used by development teams to ensure that the application they are writing does what they expect it to do. In essence, these tests form a specification for the entire system. That means that these tests support programming. But, functional tests are (or should be) written in a way that business users understand, making them business facing. So, functional tests belong in the upper left quadrant of the matrix. The benefit of automating these tests is high.

Exploratory Tests

Exploratory testing is the practice of trying to identify problems in an application. By that definition, these tests critique a product. Exploratory tests are considered business facing due to the fact that the testers are using the software the same way a real end-user might. So, exploratory tests belong in the upper, right quadrant of the matrix. There is no benefit to automating exploratory tests – until a defect is identified – at which point, a new functional test can be added to ensure that the defect remains closed.

Performance Tests

Performance tests are used to determine the speed of an application within a specific set of parameters. Specialized tools are used to perform this testing. As such, these tests require a good deal of technical knowledge and are therefore technology facing. Performance tests require working software, and are therefore tests that critique a product. So, performance tests belong in the lower, right quadrant of the matrix.

My editor now tells me that I'm rapidly approaching 1000 words. So, to prove the axiom, here's a summary of everything I just wrote:

That’s enough for now. In a subsequent post, I’ll dive into why it’s important to cover all four quadrants.

For more information, check out Brian Marick’s original Agile Testing posts, or Google Agile Test Quadrants.