Let’s face it—design engineers are never really “done” improving their designs. But at some point, the product’s design must be declared complete, the product must be built, and finally prepared for shipment. However, before a product can ever transition from the design engineer’s desk to production, and before it can leave the production floor for shipping to the customer, assurance must be provided that the product is ready to transition to its next phase.

These assurance activities include many activities across the product lifecycle but testing of the product plays a large role. Too little testing can result in unsatisfied customers if they receive defective products, but too much (or unnecessary) testing can reduce profit and lead to unsatisfied management.

Finding the right balance is achieved through Test Case Coverage Analysis when creating the testing strategies for the product. In this blog post, we’ll be sharing what you need to know for test case coverage analysis, considerations for production and design assurance testing, and how to find the right testing balance.

Key Information Required for Test Case Coverage Analysis

There are two main points in a product’s lifecycle where testing occurs:

• Before going into production, where the design of the product is demonstrated to be correct (i.e., to confirm that you “built the right thing”)

• After the product is fully assembled, where you confirm that you “built it right” (and that there are no defective components or latent manufacturing issues)

The tests performed at each of these events will typically not be the same, but each should be accompanied with a set of requirements (or specifications) describing the product, its capabilities, and how it is built.

Test Case Coverage Analysis ensures that the requirements are shown to have been satisfied via Testing.

While this description is short, simple, and easy to say—it is not always easy to do—especially for products with multiple features, each of which perform complex algorithms. Regardless of whether it is a simple or a complex product, Test Case Coverage Analysis must start with well-documented and detailed requirements. These serve as the reference point for determining how many tests are necessary to ensure that the product will satisfy its intended purpose.

Production Testing vs. Design Assurance Testing

As noted above, the goal of production testing is to ensure that the product was “built right” and contains no latent defects. These tests check for missing or wrong components, mis-wiring, bad solder joints, etc. Test Case Coverage Analysis uses the specifications, schematics, and manufacturing drawings to determine which tests to perform in order to efficiently detect these kinds of manufacturing and component defects. For some mission-critical or high-reliability products, production testing includes environmental stress testing of the product (such as voltage transients, temperature cycling, or shock and vibration cycles) to force incipient defects to be exposed in order to minimize the risk of such failures occurring for the end user.

In contrast, design assurance testing does not look for manufacturing or component defects but, instead, is intended to show that the product’s integrated hardware and software perform their intended purposes (as expressed in the product’s requirements). For complex products, design assurance testing takes significantly more time to execute than production testing because much of the functionality of the product is accomplished in software. While some of the test resources used for design assurance testing may also be used for production testing, the type and depth of tests performed post-production only need to demonstrate hardware integrity and do not need to re-confirm any functional capabilities that are implemented in software beyond confirming that the correct software was properly loaded.

Having Enough Tests vs. Having Too Many Tests

The more you test, the better your product—right? Not necessarily. Many test cases could be considered as simply repeating other tests by applying test conditions that are functionally indistinguishable from each other. You must have just enough tests to be thorough, while still meeting test cost and efficiency goals.

Products typically have analog or digital sensors that cover a broad range of signal levels and any attempt to perform exhaustive testing (covering every possible input value and combinations of input values) faces the daunting reality that testing time would be impractically long. Thus, shortcuts must be taken to determine a good, representative set of test input conditions to completely cover the product’s operating modes.

An analysis of the requirements helps determine the set of test conditions by identifying transition points in behavior (or edge conditions). The regions of operation at and between these boundaries are used to determine the Test Cases that will ensure the design has been satisfied. Any specific test condition between these boundaries can be considered as equally capable of representing all other test conditions in that region and, collectively, the entire set of such possible test conditions are said to fall into the same Equivalency Class.

For example, we can use the case of an oil pressure warning indicator that has 3 states: Green, Yellow, and Red, where Green indicates pressures between 10 psi and 12 psi, Yellow is indicated for pressures between 8 and 10 psi and between 12 psi and 14 psi, and Red is indicated for pressures below 8 and above 14 psi.

In this case there are 5 edge conditions: 8, 10, 12, and 14 psi, and 5 Equivalency Class Regions (1 Green, 2 Yellow, and 2 Red). A Test Coverage Analysis would reveal that there should be a Minimum of 8 Test Cases to cover both sides of each edge condition. Note: testing at an edge is fraught with test stimulus accuracy concerns so testing near each edge is typical and also covers at least two test conditions for each equivalency class region.

One set of possible test conditions is listed below:

• • • RED region test conditions: 7.9 psi, and 14.1 psi

    • YELLOW region test conditions: 8.1 psi, 9.9 psi, 12.1 psi, and 13.9 psi

    • GREEN region test conditions: 10.1 psi, and 11.9 psi

Adding any MORE test conditions (aside from stress-testing scenarios) may appear to provide more thorough testing, but it won’t reveal any new information that could expose product issues in either the sensors, the design, or the software. If nothing new can be learned about the product by adding more tests—then they only add time and tie up valuable test resources which could otherwise be used for testing more product.

Planning your testing? Contact Genuen today to discuss test case coverage analysis for your products.