—— How We Use the VMM Planner to Track CDV progress

Abstract

A constraint-driven random environment in VMM requires the coverage data to help answer the question “Is everything verified?” Then the verification plan needs to define with the coverage feature. The VMM Planner is definitely a good start point because it provides the template and executing flow for users, which means the introduction of VMM Planner makes the CDV progress track possible. This paper documents valuable VMM Planner features, such as hierarchy verification plan, automatic annotation and etc. This paper also talks about how we integrate the VMM Planner features into our legacy tools. We will explain what we concerned in the integration, and we will conclude the benefits that the VMM Planner brings to us.

1. Introduction

The ever-increasing complexity of the DUV has necessitated a shift in the verification approach from a collection of directed tests to a constraint-driven random verification environment. This shift has introduced a new challenge: how to track the progress of verification projects. VMM Planner is a technology that allows you to create a comprehensive model to track the progress of verification projects.

In the past projects, we used a uniform verification progress track tool called Verification Integrity Form. It lists features, TestPoints and Testcases, and we can track all the coverage manually. It’s a good structure for the Direct Test, but is not suitable for CDV.

In the new project, we have integrated the VMM Planner into Verification Integrity Form to build a new verification planning tool. With the VMM Planner, the verification plan becomes an executable plan. You can trace the coverage data automatically which can reduce the manual mistakes. It is a technology that allows the users to think about the verification process at a high level.

In this paper, we will talk about how we integrate the VMM Planner to the Verification Integrity Form, and how the integration benefits our project.

2. Original Verification Integrity Form

2.1 Introduction of the original Verification Integrity Form

Verification Integrity Form is a Spreadsheet XML file, which is composed of the Design Spec sheet, Feature sheet, Test Case sheet, Test Record sheet, Fault Report sheet, Test Status Report sheet, Test Report sheet and other sheets. Some of these sheets are edited as Verification Plan, and others are automatically generated according the digits you filled in the sheets. It’s a comprehensive structure for directed test, but it is not suitable for CDV.

2.2 Disadvantages

Disadvantages of the original Verification Integrity Form are as follow:

• Manual launch of tests

• Manual tracking of feature status

• Subjective judgment of “is my project on track”

• Not easy to reuse between block level verification and system level verification

3. Solution with VMM Planner

3.1 Introduction of VMM Planner

VMM Planner is a verification planning tool incorporated into several Synopsys products. It is a technology that allows you to think about the verification process at a high-level while working with the real objective of low level verification data. With VMM Planner, you can convert the low-level data into useful information to plan and track the progress of verification projects. VMM Planner uses HVP (Hierarchical Verification Plan) language to describe the verification plan. HVP language is a comprehensive language that allows you to describe a verification plan hierarchically in a text file. There is no graphical user interface for entering or viewing the plan at the beginning, but by following a few simple rules, you can define the plan in an XML Spreadsheet and the annotate command (hvp annotate) will generate the .hvp file that it needs to back-annotate the Spreadsheet with coverage and other metric data. And we choose the XML Spreadsheet mode. Managers would like it because it is easy to get the coverage information.

And now there is a graphical user interface for entering or viewing the plan yet. It’s an interactive editor built-in to DVE. It is also good to help the users create and modify their verification plan.

Figure 1 shows the data sources used by VMM Planner-enabled applications such as the hvp application we used that is supplied with VCS. The .hvp file is the verification plan. The Synopsys Unified Coverage Database can include Code Coverage data (line, condition, FSM and toggle), assertion coverage (OVA, SVA, and PSL), and Functional (Testbench) Coverage data. The external data is formatted into a text file to annotate user-defined metrics. As mentioned before, a number of VMM Planner applications are currently planned or under development. We are using the Spreadsheet annotator flow. When using the Spreadsheet approach, the .hvp file is automatically produced from the spreadsheet-based plan as shown in Figure 1.

Figure 1: VMM Planner Spreadsheet Annotator

3.2 Highlights

The solution with VMM Planner is preferred because the VMM Planner is a predictable planning which enables you to define the goal, and speed the creation of the environment to get you there. It defines standards that can be deployed across teams for consistent, reliable collaboration. It maximizes verification reuse to reduce effort needed to develop new code.

The solution with VMM Planner also introduce the feature of predictable execution which enables you to easily manage and scale the size of your verification resources to meet unexpected challenges, and to increase the rate of coverage generation through higher levels of automation.

And the solution with VMM Planner also introduce the feature of predictable analysis which provides comprehensive visibility into the status of verification

4. How we integrate VMM Planner into Verification Integrity Form

4.1 What we concerned in the integration

What we concerned mostly is whether some good features of legacy tools will lose during the integration, because the good features and good structure of the legacy tools is proofed to be useful. Luckily, the VMM Planner can be just embedded into our legacy tool.

4.2 How we rebuild the xml file

Firstly, we deleted the Feature sheet, Test Record sheet, Fault Report sheet and Test Status Report sheet. Secondly, we added Feature & VRL sheet, vmmp Metrics sheet and vmmp Attribute sheet to build the new Verification Integrity Form. The contents in the deleted sheets were moved to new added Feature & VRL sheet, and rearranged.

4.2.1 Introduction of Feature & VRL sheet

It’s an “hvp plan” sheet, named dummy, as shown in Figure 2, Figure 3 and Figure 4.

Figure 2: Feature & VRL sheet(1)

What each meta-tag column means:

• Column B is an include meta-tag column, and sub plan xml file name can be put into the B3 cell, and the other cell in this row will be ignored.

• Column C is a feature meta-tag column, and the feature value will be block name.

• Column D is s subfeature meta-tag column, and the subfeature value will Feature List items, the value also can be sub block name.

• Column E is s skip meta-tag column, and the value in this column is Design Spec Track Information, such as Design Spec ID.

• Column F is also a skip meta-tag column, and the value in this column is TestPoints Description.

• Column G is a subfeature meta-tag column, and the subfeature value is Coverage Layer 1s, such as Code Coverage, Function Coverage, Assertion Coverage and Tests Coverage.

• Column H is also a subfeature meta-tag column, and the subfeature value Coverage layer 2s, which used to branch the Tests into Random Tests and Directed Tests.

• Column I is also an subfeature meta-tag column, and the subfeature value is Coverage tag, used to comment the Coverage.

• Column J is a measure m1.source meta-tag column, and measure m1 is the Synopsys build-in Coverage measurement, and its source is the Coverage Scope.

• Column K is a measure m2.source meta-tag column, and measure m2 is the bugs measurement, and its source is User Data.

• Column L is a value m2.bugs meta-tag column, and its value is the bugs Measurement value.

• Column M is a value m1.Group meta-tag column, and its value is the Function Coverage Measurement value.

• Column N is a value m1.Assert meta-tag column, and its value is the Asserting Coverage Measurement value.

• Column O is a value m1.Line meta-tag column, and its value is the Line Coverage Measurement value.

• Column P is a value m1.Cond meta-tag column, and its value is the Condition Coverage Measurement value.

• Column Q is a goal.test meta-tag column, and its value is the goal of test.

• Column R is a measure m3.source meta-tag column, and its value is the source of test measurement.

• Column S is a value m3.test, meta-tag column, and its value is the test measurement value.

Figure 3: Feature & VRL sheet(2)

Figure 4: Feature & VRL sheet(3)

The Feature & VRL sheet depicts the feature-subfeature hierarchy and plan-subplan hierarchy, and subplans can be easily reused between different projects.

4.2.2 Introduction of vmmp Metrics sheet

It’s an “hvp metric” sheet, and its plan named dummy, as shown in Figure 5.

What each meta-tag column means:

• Column B is a name meta-tag column, and it means the name of user defined metric, for example, B2 Cell means a metric of bugs.

• Column C is a type meta-tag column.

• Column D is an aggregator meta-tag column.

• Column E is a goal meta-tag column.

The “bug” metric is used for bug rate annotation, while tracking the rate is a key function of verification plan.

4.2.3 Introduction of vmmp Attributes sheet

Figure 6: vmmp Attributes sheet

It’s an “hvp attribute” sheet, and its plan named dummy, as shown in Figure 6.

What each meta-tag column means:

• Column B is a name meta-tag column, and it means the name of user defined attribute.

• Column C is a name meta-tag column, and it means the type of user defined attribute.

• Column D is a default meta-tag column, and it means the default value of user defined attribute.

The “phase” attribute is used in reusing the plan between different block verification phases and reusing the plan between block level verification and system level verification. And the “version” attribute is used to tag the feature version update, and can be used to filter the feature form different version if it’s required.

4.3 How we fill the verification plan

Since 100 percents Coverage is the goal of the CDV, firstly, we classified the Coverage Measurements. All the Coverage Measurements can be classified into Code Coverage Measurement, Testbench (Group) Coverage Measurement, Assertion Coverage Measurement and Tests Coverage Measurement.

Code Coverage Measurement

Code coverage has no relation with TestPoints. It’s structure coverage information of the circuit. We partition measurement of it into block or sub block, and do not map them to any feature or TestPoint.

Testbench (Group) Coverage Measurement

It’s the major part of all the coverage, and measure how much coverage space is hit by constraint-random tests.

Assertion Coverage Measurement

Some bus protocol check point or some exception will be measured by assertion coverage.

Tests Coverage Measurement

Test coverage measured by the mean of pass or failed. It’s a traditional method of functional coverage.

We regard all the kinds of Coverage above as verification plan goals, and we fill them in the Feature & VRL sheet separately.

Secondly, we write down the TestPoints to capture the Function Specification, Architecture Specification and other standard documents, and then map the TestPoints to all kinds of the Coverage Measurement. It’s our Systematic Way from Spec to an Executable Verification Plan

TestPoints can be mapped to Random Test and Directed Test. So TestPoints will be mapped to Group Coverage and Tests Coverage. Our approach is to have each paragraph/items/bullet of the test plan to be accompanied with verification code (assertions, coverage code and etc.) And every TestPoints will have at least one associated Functional Coverage item (bin or cross), which is compiled into the Testbench or bound into design. But not all verification items require constraint-random testing, and therefore assertion/coverage development, some of them require just direct testing. Register access verification is a good example of such item. So, test coverage should also be measured.

After we classify the Coverage, write down the TestPoints, and map the TestPoints the Coverage, the verification plan begins to work. It doesn’t guarantee that the test plan text description matches the test implementation — in fact, no one can guarantee that — but by keeping them together makes it more likely that nothing slip through the cracks. Even better, coverage results are gathered for each test plan item, and presented alongside it. For management’s point of view, a verification plan simply breaks the overall verification task into smaller task. Therefore, the plan’s key aspect is to assure that completion of all these smaller tasks will achieve the overall verification goal. If this is too hard, the plan should also present the achievable overall verification goal. Another advantage of this is that verification engineers can start developing the test plan and the Testbench implementation (check and coverage) at the same time.

5. Conclusion and Future Works

It’s our try to integrate the VMM Planner into our legacy tools and it is a smooth operation. The good structure of the legacy tools is maintained, and new feature of VMM Planner is added. By now, we can track the progress of verification visually. We can predict the verification schedule with better quality because we know what has already been done by the VMM Planner report. Figure 7 is a VMM Planner report example.

Figure 7: VMM Planner Report Example

In the future, we will delved deeply into the VMM Planner, and take the advantage of the new feature of VMM Planner, such as DVE VMM Planner editor, incompleteness checks and etc. In another side, we continue to look for a better systematic way from Spec to verification plan.

6. Acknowledge

We would like to thank the support team of Synopsys. They are who introduce the VMM Planner to us, and help us to comprehend and integrate the VMM Planner.

7. References

[1] VMM Planner User Guide, B-2008.12 -2, Synopsys, September 2008