Multi-million Set-Top Box Static Timing Analysis using Distributed Multi-scenario Analysis


HOU Hua Min,LI Ying
Thomson Broadband R&D (Beijing) Co., Ltd

Abstract

In deep submicron technologies, verifying a chip design requires several PrimeTime analysis runs under different chip operating modes, different operating conditions and different RC parasitic corners. All these combined STA scenarios must be checked one by one. With Distributed Multi-scenario Analysis (DMSA), we can let PrimeTime analyze the scenarios in parallel. The merged reporting capability of DMSA can reduce the debug time. We use two enhanced merged reporting: Quality of Result and the timing violations distribution histogram in shell mode. Especially the timing violations distribution histogram report, it significantly helps us identify the critical issues during a short time.

An efficient experiment is ECO hold-fixing under DMSA. Without DMSA, any ECO change in one corner will cause one-more-time verification of all other corners. If the change results in any violation increase in one corner, it must be refreshed and verified in all corners again. With DMSA feature, we can quickly evaluate the effects of ECO changes and obtain merged reporting on the active scenarios. In this paper, we share our experience using DMSA in our design.

1.0 Motivation

In complex SoC design, there are multiple chip operating modes, for example, function, MBIST, atpg shift, atpg capture, at-speed test and so on. It also requires analysis under different operating conditions (extreme temperatures and operating voltage). With the advent of deep submicron technologies (90nm or below), shrinking of process geometries adds many additional process corners that require additional analysis in order to close timing. A specific combination of operating mode, operating condition, and parasitic corner can be called a scenario. The figure 1 shows the combined STA scenarios matrix. We need to analyze all scenarios for the successful sign-off. This turns timing analysis into a time-consuming job.

Figure 1. STA Scenarios Matrix

In our Set-Top Box design, it has more than 6 million logic gates and 700K Bytes memories. It is a hierarchical design. Except the top level, there are 6 sub-blocks which layout separately. In each sub-blocks, it has mission mode (functional and mbist), test_shift and test_capture operation mode. In top level, it includes functional mode, mbist mode, test_shift, test_capture, test_at_speed and JTAG operating mode. The TSMC tsmc90ghp technology process is used in our design. It has 3 PVT operating condition corners (Best, Typical, and Worst) and 5 RC parasitic corners (Typical, Best C, Worst C, Best RC and Worst RC). Totally, we should check 3*3*5=45 scenarios in each sub-blocks, 6*3*5=90 scenarios in top under STA.
Clearly, the STA run-time is long. And it is a time-consuming job to check the reports under so many scenarios.

2.0 Distributed Multi-Scenario Analysis

2.1 Introduction

Primetime has the capability of Distributed Multi-Scenario Analysis to process multiple scenarios in parallel. Instead of analysis each scenario separately, DMSA provides efficient unified analysis of multiple scenarios from a single master. The master setups, executes and controls multiple slave processes, one for each scenario. This capability does not use multithreading (breaking a single process into pieces that can be executed in parallel). Instead, it uses an efficient way to specify the analysis of different operating conditions and operating modes onto different hosts.
To start PrimeTime in DMSA mode, use the pt_shell command with the -multi_scenario option.
% pt_shell -multi_scenario

2.2 Unified Multiple Analysis

A multi-scenario analysis is carried out by one master PrimeTime process and multiple PrimeTime slave processes. The master and slave processes interact with each other using full-duplex network communications. The Figure 2 is the unified multiple analysis from a single DMSA master. We can work with multiple analyses as easily as with a single PrimeTime analysis run.

Figure 2. Unified Multiple Analysis from a Single DMSA Master

The master process generates analysis tasks and manages the distribution of those tasks to the slave processes. Each slave process does timing analysis under specific scenario. The master process collects the slave process reports and submits the merged report.

2.3 Directory Structure

The multi_scenario_working_directory variable specifies the DMSA work directory in which to store the working files. The multi_scenario_merged_error_log variable specifies the file in which to save all error, warning, and information messages issued by the slaves.
set multi_scenario_merged_error_log error_log.txt
The error_log.txt file appears in the directory specified by the multi_scenario_working_directory variable. In Default, a maximum of 100 RC messages will be written to the merged error log file per task. The multi_scenario_merged_error_limit variable can control the maximum messages.
All slave log information will be piped to the out.log file in the scenario directory in the DMSA working directory. The Figure 3 shows the detail directory structure.

Figure 3. DMSA Directory Structure

3.0 DMSA STA

3.1 Environment Configuration

The following is the configuration file which we are using.
file delete -force ${TEV_DMSA_WORK_DIR}
set multi_scenario_working_directory ${TEV_DMSA_WORK_DIR}
set multi_scenario_merged_error_log ${TEV_DMSA_WORK_DIR}/error_log.txt
set_distributed_parameters -shell ssh
foreach machine_host $TEV_NOW_MACHINE_HOSTS {
add_distributed_hosts -farm now [lindex $machine_host 0] \
-num_of_hosts [lindex $machine_host 1]
}
## licenses will only be pulled from the license server as they are needed
set_multi_scenario_license_limit -feature PrimeTime ${TEV_NUM_OF_HOSTS}
set_multi_scenario_license_limit -feature PrimeTime-SI ${TEV_NUM_OF_HOSTS}
foreach TEV_SCENARIO $scenario_list {
create_scenario -name $TEV_SCENARIO \
-specific_variables {TEV_SCENARIO PILOT_ENV} \
-specific_data “$GEV(gscript_dir)/sta/pt_dmsa.tcl”

}
create_distributed_farm
current_session -all
remote_execute {}

tproc_msg -info “Reprot the DMSA merged reports.”
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_timing.max {
report_timing -derate -net -input -capa -tran
}
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_timing.min {
report_timing -delay_type min -derate -net -input -capa -tran
}
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_violations.setup {
report_violations
}
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_violations.hold {
report_violations -type hold
}
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_qor {report_qor}

Figure 4. DMSA Configuration File

● Slave Process Allocation
The slave hosts can be allocated with add_distributed_hosts in the virtual farm. The unmanaged compute resources are described as a Network Of Workstations (NOW). These are internally load balanced during multi-scenario analysis. The distributed environment can be configured by the following command.
set_distributed_parameters -shell ssh
● Licenses
The set_multi_scenario_license_limit command sets the upper limit licenses which can be used in DMSA. Any licenses that the master checks out are made available for slave usage. We run the timing analysis and noise analysis in DMSA mode.
● Session and Scenario
The create_scenario command defines the scenario name and the names of the variables and scripts that the slave processes use. The following is the command usage.
create_scenario # create a scenario for multi_scenario analysis
-name scenario_name (scenario name)
[-common_data file_list] (list of files with common data)
[-common_variables variables_list] (list of common variables)
[-specific_data file_list] (list of files with specific data)
[-specific_variables variable_list] (list of variables with specific data)
[-image scenario image] (scenario image)
The current session is the set of scenarios which we want to analyze in parallel. We can select the scenarios for the current session using the current_session command. All defined scenarios can be the current session with the -all option.
current_session -all
By default, all scenarios in the current session are the active scenarios. We also can further narrow the active scenarios with the current_scenario command.
● Remove Execution
From the master, it creates a command buffer to be executed in the slave context.

3.2 Merged Reporting

Executing reporting in multi-scenario analysis can generate large amounts of report about each scenario. The quantity of data can make it difficult to identify the critical information. In order to manage these large reports, and to identify critical issues, PrimeTime provides the merged reporting capability in DMSA mode. Merged reporting can automatically eliminate redundant data and sort the data in order of criticality. This effectively allows you to treat all the scenarios in command focus as if they are a single virtual PrimeTime instance.
The main merged reporting is timing report. We can issue the report_timing command at the master as we are in regular PrimeTime. This results in the master and slave processes working together to execute the command in all scenarios. The figure 5 shows the merged timing report mechanism in DMSA mode. It produces a final merged report in master console. It includes the scenario name in each timing report. The figure 6 shows a typical merged timing report.

Figure 5. Merged Report Mechanism

Startpoint: u_out_stage_top/ycbcr_sd16_reg_2_
(rising edge-triggered flip-flop clocked by pix_clk)
Endpoint: ccir656[2] (output port clocked by vo_pix_clk)
Path Group: vo_pix_clk
Path Type: min
Scenario: func.OC_WC.RC_WC_1
Figure 6. Merged Timing Report

3.3 Quality of Results Report

Synopsys provides an additional procedure which reports an overview of the quality of a design (QoR == Quality of Results), including the length of the worst paths, total negative slack, and other informations, including setup/hold summaries and a complete, detailed DRC summary. We can find the report_qor procedure in Solvnet Doc 008602.
The script has been updated to work in DMSA mode. When run from a DMSA master console, the script will gather design QoR metrics across every active scenarios, and generate a report based on the worst violations at each endpoint or pin across every scenario. The report_qor script will show the worst violation like figure 7.
****************************************
Report : qor
Design : multi-scenario design
Version: A-2007.12-SP1
Date : Mon Apr 7 21:52:13 2008
****************************************

Timing Path Group ‘pix_clk’ (max_delay/setup)
———————————————
Levels of Logic: 16 (func.OC_WC.RC_BRC_1)
Critical Path Length: 21.93 (func.OC_WC.RC_BRC_1)
Critical Path Slack: 15.92 (func.OC_WC.RC_BRC_1)
Total Negative Slack: 0.00
No. of Violating Paths: 0
———————————————

Figure 7. QoR Report

3.4 Violation Distribution Histogram Report in DMSA

In PrimeTime GUI mode, there is the histogram menu which can show the distribution of analysis values in bar-graph form. It is intended to aid us in better understanding of timing violation in the design. Synopsys provides a useful tcl procedure (report_violations) to report the timing violation (setup or hold) histogram for each path group in shell mode. The procedure can be found in SolvNet Doc 019948.
The script provides the flexibility to report a specific path group rather then generating reports for all violating path groups. But it doesn’t support DMSA mode. So we update the procedure to support DMSA mode. PrimeTime provides the pt_shell_mode variable to specify the working mode: primetime, primetime_master or primetime_slave. We use the variable to identify the working mode in the procedure. It is primetime_master in DMSA master mode.
With get_distributed_variables command, we can get the timing violations in each scenarios. The returned values are saved into the master array. Then we can retrieve the violation in histogram manner in shell mode.
get_distributed_variables -pre_commands \
{redirect -var rpt_setup_vlt {report_constraint -all_violators -max_delay -nosplit -sig 5}} \
rpt_setup_vlt
get_distributed_variables -pre_commands \
{redirect -var rpt_hold_vlt {report_constraint -all_violators -min_delay -nosplit -sig 5}} \
rpt_hold_vlt
The Figure 8 shows the DMSA timing violations distribution histogram. Here the path groups number is the sum of all scenarios’ path groups number. With a quick glance on the histogram, we can know the timing violation distribution histogram in different scenario. It can significantly reduce the time identify the critical information.
The appendix is the main scripts of the procedure.
****************************************
Report : Violation Summary
Design : multi-scenario design
Version: A-2007.12-SP1
Type : hold
Date : Mon Apr 7 21:07:47 2008
****************************************

Total number of path groups in design : 155

Total number of path groups with violations (max_delay/setup) : 5
Total number of violations (max_delay/setup) in design : 1130

Total number of path groups with violations (min_delay/hold) : 56

Total number of violations (min_delay/hold) in design : 2517

*************************************
Scenario : test_shift.OC_BC.RC_BC_1
Path Group Name : scan_clock1
Total Violations : 93
Worst Violator : -0.04119
*************************************

Graph Data:
===================================================
Violation Range for Bar 1 : -0.000000 to -0.008238
Violation Range for Bar 2 : -0.008238 to -0.016476
Violation Range for Bar 3 : -0.016476 to -0.024714
Violation Range for Bar 4 : -0.024714 to -0.032952
Violation Range for Bar 5 : > -0.032952
===================================================

Histogram of violations on ’scan_clock1′ (bins = 5)
| (% of WNS) (violations) –>
V
0-20% : ************************************************** (50)
20-40% : ******************************* (31)
40-60% : ******* (7)
60-80% : **** (4)
> 80% : * (1)
Figure 8. DMSA Timing Violation Distribution Histogram Report

3.5 Save and Restore Session

The sessions can be saved from the DMSA master. The different scenarios are saved in different directory. Each scenario session is same as the regular PrimeTime session. So we can reload single scenario session into a regular PrimeTime envionment.
Any saved sessions can be read into a DMSA scenario. A tcl procedure restore_dmsa_session is available in Synopsys SolvNet. The SolvNet Doc ID is 018039. With the procedure, we can load any PrimeTime session that has been produced by the save_session command (either inside or outside of DMSA). The procedure is invoked with the directory name where the saved sessions are stored. Since all of these sessions are previously analyzed, it saves a lot of time to analyze the design in DMSA mode.

3.6 Comparison between Regular STA and DMSA

In DMSA, we use 4 PrimeTime and PrimeTime-SI licenses. We distribute the DMSA slaves on 4 machines. Totally, it has 45 scenarios in each sub-block STA of our design. We run STA in regular mode and DMSA mode. The Figure 9 is the results. We get the setup and hold violation distribution histogram from 2 report_violations files in DMSA mode. But it needs to check 90 files in regular STA mode. The debug time can be significantly reduced in DMSA mode. At the same time, the run-time is shorter.

Figure 9. Compararison between Regular and DMSA STA

4.0 ECO Hold-fixing in DMSA Mode

We may have setup or hold violations in our design. In the early time of the schedule, we can redo the layout. But in the very late time of the schedule, we do not have enough time to re-run the place and routing. ECOs (Engineering Change Order) only add minor changes which can save time.

When fixing setup violations, it requires a complex logic function changes according to tradeoff of intrinsic and extrinsic delay, signal routes and other factors. It is hard to fix it in PrimeTime only. So we put the ECO setup fixing into layout tool in our design.

Fixing hold violations is normally a straightforward process of inserting a delay at a pin that has a negative hold slack. It is much easier than setup fixing. It is also easy to fix the hold violation with automation scripts. Choose the amount of delay so that it improves or resolves the hold violation without incurring a setup violation. This is normally straightforward within a single analysis; however, under many different scenarios (operating modes, PVT corners, RC corners), it can become extremely tricky to resolve the hold violations in a way that satisfies all scenarios.

DMSA provides a way to fix the hold violation in all scenarios. We can quickly evaluate the effects of ECO changes on active scenarios and obtain merged reporting. For example, in our sub-blocks, it has three operating modes, analyzed at three PVT corners and five RC extraction corners. A fully exhaustive analysis of this design would require 3*3*5=45 separate PrimeTime runs. Evaluating the affect of hold fixing across all 45 scenarios in our design is challenging and time-consuming.

Across different scenarios:
● Rise/fall delay characteristics of the hold-fixing buffers and delay cells are different
● Pins have different setup/hold slacks

● Pins have different propagated transition times and signal integrity effects

In each of these runs, inserting the same buffer or delay cell, because of different constraints (input slew, signal integrity effect, and RC loading), there has 45 different rise/fall setup/hold effects. Synopsys provides a DMSA hold-fixing script. The script can find an optimal set of buffer/delay cell insertions to resolve the hold violations under all scenarios. We can find the dmsa_fix_hold procedure in SolvNet Doc. 018510.

4.1 dmsa_fix_hold Procedure

The major flow of the hold-fixing algorithm is shown in Figure 10 and described as follows:
● Estimate rise and fall delay characteristics of provided buffer cells across all scenarios.
● Obtain all failing hold paths across all scenarios. Exit if the design is clean.
● Based on the timing paths obtained across all scenarios, estimate the optimal buffer set to resolve the hold violations.
● Perform the actual buffer insertion in all scenarios
● Check the resulting timing across all scenarios, tweaking the sizes to further improve any remaining hold violations and resolve any accidental setup violations.
● Go back to Step 2 and perform another iteration.

Figure 10. The Algorithm of dmsa_fix_hold Procedure

The following is the script usage.
Usage:
dmsa_fix_hold # fix hold violations across DMSA scenarios
[-prefix name] (Instance name prefix for inserted buffers)
[-allow_outputs] (Consider buffer insertion at output pins)
[-slack_lesser_than slack_limit] (Slack threshold for hold criticality)
[-verbose] (Show additional verbose information)
[-hold_margin value] (Hold margin used during buffer selection)
[-setup_margin value] (Setup margin used during buffer selection)
[-effort level] (Specify runtime versus repair quality: Values: low, high)
[-allow_list value] (Allow buffer insertion at these hierarchical levels)
[-avoid_list value] (Avoid buffer insertion at these hierarchical levels)
buffers (List of buffers/delay cells to use)
Providing many different drive strengths of cell is not recommended, and will result in excessive runtime. For best results, provide a small set of cells which have consistent drive strengths and a good spread of cell delays.
It is recommended to provide a set of cells of similar drive strength. The reason is that the algorithms are designed to work with cells of different intrinsic (load-independent) delay, not different extrinsic (load-dependent) delays.
Based on the suggestion of the procedure’s author, Chris Papademetrious, we use the following buffers or delays to fix the hold in our design.
BUFFD4 BUFFHVTD4 DEL005 DELHVT005 DEL01 DELHVT01
The script estimates the intrinsic delay of the provided cells. The following is the sorted delay values.
Sorted buffer cells:
BUFFD4 37ps
BUFFHVTD4 47ps
DEL005 94ps
DELHVT005 120ps
DEL01 168ps
DELHVT01 220ps
This script uses the first one as its initial baseline. If the cell delay failed to meet the requirement, the script will choose the next one. With the small set of provided cells, it can choose right cell without trying too more cells.
We are using version v1.6 which released in June 2007. It requires some feature of PT Z-2007.06 or later.
Another requirement needs to keep in mind. When we run the PrimeTime analysis that we want to do hold fixing, we must set the variable timing_save_pin_arrival_and_slack to true. It is necessary to save the arrival time and slack value for each pin when we save the session. The dmsa_fix_hold script uses these values in its analysis.

4.2 Fix the Hold Violation

The figure 11 is the hold-fixing result in our design. It shows the multiple runs to know what happen during the hold-fixing iterations and the run-time. We can find that among the 548 hold-critical endpoints found with -nworst 100, there are 555 unique paths. For almost all of the endpoints, each endpoint only has a single startpoint. We also can find that it doesn’t have setup-critical pins. We are most likely dealing with simple flop-to-flop hold violations in our case.
Note the total runtime is not equal to the amount of all iteration runtime. The reason is the runtime are wall clock times rather than CPU times. The Distributed Multi-Scenario Analysis spends much of its time idling while the remote processes perform the job.

Figure 11. Hold-fixing Result with DMSA

5.0 Pilot DMSA Enhancement

We are using Pilot design flow 2.2 which developed by Synopsys Professional Services in our design. The Pilot Design Environment is a complete RTL-to-GDSII design system based on Synopsys Galaxy and Discovery platform. It provides a comprehensive, production-ready design flow with built-in methodologies and utilities to improve designer productivity and accelerate the tape-out of system-on-chip designs.
With Pilot flow, we are able to more quickly produce our chip and with lower risk. But the DMSA STA mode is not supported in Standard Pilot flow. We integrate the DMSA STA into our customized pilot flow. We can choose the regular or DMSA STA with the different Makefile task. The following is the requirement of the integration:
● Pilot DMSA launch command task
● DMSA master distribute script: pt_dmsa_distribute.tcl
● DMSA slave wrapper script: pt_dmsa.tcl
● Merged report based on previous STA run session: pt_dmsa_report.tcl
● DMSA hold-fixing wrapper script: pt_dmsa_fix_hold.tcl

5.1 Launch Task

Since the DMSA mode needs the -multi_scenario option in pt_shell, we create the new launch task ptdmsa in tool_launch.gmake file.

5.2 Master Distribution

The figure 4 shows the DMSA master script pt_dmsa_distribute.tcl. It sets up the master PrimeTime process and controls multiple slave processes. Here we use some TEV variables which are read from the Makefile task. The DMSA work directory location, distributed NOW host machine and the upper license limit can be configured in the Makefile task. The following is an example.
$(LOG_DIR)/dmsa_sta/dmsa_sta.pass:
@rm -f $@
@$(MAKE_CMD) ptdmsa \
GEV_SRC=outputs \
GEV_DST=dmsa_sta \
GEV_SCRIPT=$(GEV_GSCRIPT_DIR)/sta/pt_dmsa_distribute.tcl \
GEV_LOG_FILE=dmsa_sta.log \
TEV_NUM_OF_HOSTS=4 \
TEV_NOW_MACHINE_HOSTS=”{beijslxtsc28 1} {beijslxtsc27 1} {beijslxtsc26 1} {beijslxtsc24 1}” \
TEV_HIER_MODE=FLAT
@$(GEV_CHECK) \
-task dmsa_sta \
-log $(LOG_DIR)/dmsa_sta/dmsa_sta.log

5.3 Slave Wrapper

The execute_scenario.tcl is the main regular STA script in Pilot flow. For the unified environment, we also use the same STA script as the DMSA slave mode. A slave wrapper is created to setup the slave environment. The following is the wrapper script -pt_dmsa.tcl. The TEV_SAVE_SESSION variable specifies to save the session or not. In DMSA mode, the saving session is supported in master mode. So we keep it as 0 (do not saved).
source $env(GEV_GSCRIPT_DIR)/misc/utilities.tcl
tproc_source -file $GEV(tscript_dir)/conf/node.tcl
tproc_source -file $GEV(tscript_dir)/conf/node_setup.tcl
tproc_source -file $GEV(gscript_dir)/conf/project.tcl
tproc_source -file $GEV(gscript_dir)/conf/project_setup.tcl
tproc_source -file $GEV(lscript_dir)/conf/block.tcl
tproc_source -file $GEV(lscript_dir)/conf/block_setup.tcl

set env(TEV_SAVE_SESSION) “0″
set env(TEV_SCENARIO) $TEV_SCENARIO

tproc_metric_time -start
tproc_metric_system -start_of_script
tproc_source -file $GEV(gscript_dir)/sta/execute_scenario.tcl
tproc_metric_time -stop
tproc_metric_system -end_of_script

5.4 Merged Reporting Based on Previous STA Run

Because of the limit of CPU and license, some users run the STA in regular mode. The sessions are saved in the same directory. We provide a simple way to retrieve the merged report from the saved session. The saved session can be restored into DMSA environment with restore_dmsa_session. It has been introduced in section 3.4. Since the session has been run before, it is faster to get the merged reports in DMSA mode.
The following is the pt_dmsa_report.tcl. The setup is same as the pt_dmsa_distribute.tcl. The sessions are restored from previous saved session. We can get the report what we want.
If the sessions are saved by DMSA master, the script is also available.
file delete -force ${TEV_DMSA_WORK_DIR}
set multi_scenario_working_directory ${TEV_DMSA_WORK_DIR}
set multi_scenario_merged_error_log ${TEV_DMSA_WORK_DIR}/error_log.txt
set_distributed_parameters -shell ssh
foreach machine_host $TEV_NOW_MACHINE_HOSTS {
add_distributed_hosts -farm now [lindex $machine_host 0] \
-num_of_hosts [lindex $machine_host 1]
}
## licenses will only be pulled from the license server as they are needed
set_multi_scenario_license_limit -feature PrimeTime ${TEV_NUM_OF_HOSTS}
set_multi_scenario_license_limit -feature PrimeTime-SI ${TEV_NUM_OF_HOSTS}
restore_dmsa_session $TEV_SESSION_DIR
create_distributed_farm
current_session -all
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_violations.setup {
report_violations
}
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_violations.hold {
report_violations -type hold
}
redirect -file $GEV(rpt_dir)/$GEV(block).dmsa.report_qor \
{report_qor}

5.5 Hold-fixing Wrapper

The wrapper script, pt_dmsa_fix_hold.tcl, is created to set up and invoke the DMSA fix hold procedure. The following is the detail. We ignore the TEV variables setting in the script. These variables include the buffers to use for hold-fixing. The DMSA environment is same to the pt_dmsa_report.tcl. The saved sessions are restored for hold-fixing. When the scenarios are started on the distributed hosts, the dmsa_fix_hold procedure is invoked. After the hold-fixing complete, the netlist will have the proper cells added to fix the hold violations in the memory. An ECO changed script is written to implement these changes under the PNR tool.
file delete -force ${TEV_DMSA_WORK_DIR}
set multi_scenario_working_directory ${TEV_DMSA_WORK_DIR}
set multi_scenario_merged_error_log ${TEV_DMSA_WORK_DIR}/error_log.txt
set_distributed_parameters -shell ssh
foreach machine_host $TEV_NOW_MACHINE_HOSTS {
add_distributed_hosts -farm now [lindex $machine_host 0] \
-num_of_hosts [lindex $machine_host 1]
}
## licenses will only be pulled from the license server as they are needed
set_multi_scenario_license_limit -feature PrimeTime ${TEV_NUM_OF_HOSTS}
set_multi_scenario_license_limit -feature PrimeTime-SI ${TEV_NUM_OF_HOSTS}
restore_dmsa_session $TEV_SESSION_DIR
create_distributed_farm