Reporting Overview

dsClient Desktop offers several different types of reports, each geared towards different goals. All summarize the Operational Measurements (OMs) that they report - some by individual measurements, some by measurement categories - while also providing the lowest level details. All reports may be generated from dsTest OM databases and some may be viewed while the test is running.

Chart Report: Plotting OM values over time in charts, Chart Reports excel at exposing trends and timing. Chart Reports can be integrated into the Workspace with your test configuration and configured to start automatically after you load the test. You decide which OMs to include in your charts, you can define custom measurements, and you can include multiple charts in one report. Stack your time-synchronized charts to compare timing or lay them out side-by-side to compare volume. The most recent values are displayed at the end of the chart and you can display values for any point in time by rolling over that report interval with your mouse. You can run a Chart Report while the test is running or generate the report from the OM database to see the values for the entire test. In either case you can export report values to CSV-formatted data.

Tabular Report: When you're interested in seeing numeric values, choose Tabular Report. It displays measurement values by interval in a spreadsheet-like format in real time or from a database. You can page through results, navigate to a specific report interval, or jump to the final, or most recent, data. Here again you can define custom measurements, export report data to a CSV file, or integrate these reports into Workspaces.

SmartReport: SmartReport is always generated from OM databases after the test is complete and it analyzes the success or failure of the test based on criteria that you define. All measurement data is summarized by category - application status, test actions, TPS, data throughput, etc. - in a test summary that includes top-level charts and customizable dashboard blocks. The dashboards compare your criteria with report values to determine pass/fail status for the categories you choose. In addition, you can drill down into detailed data and view statistical analysis of measurement variability to aid in troubleshooting a failing test.

HTML Test Results: Designed to fit into your automated testing, HTML Test Results produces a pass/fail report in an HTML file that conforms to the template you designed. Your template may contain a description of the test goals, a topology graphic, or any other information you desire. Within it you designate where, and which, SmartReport dashboard blocks will be written. The resulting file can then be emailed to interested parties. For this report you launch dsClient Desktop from the command line while referencing the OM database(s), the SmartReport profile containing the test's criteria, and your report template. dsClient runs in "headless" mode to generate this report, meaning that a display is not necessary and you can run the application on a server.

All types of reports can span multiple OM databases. The databases must all be from the same test run but it is not required that the first database be included. The databases are validated by checking a field within each database that is incremented as OM database files roll over, and by ensuring that the interval IDs are sequential from one database to the next. If either of these criteria are not met, a fatal error is displayed and/or logged and report generation ceases.

Many of the features in dsClient's reporting infrastructure are available or applicable to all types of reports. The remainder of this topic will explore those features.

When you choose to create or open a report from the graphic interface you will be asked to identify the source of the report data.

You can view an active test with a Chart Report or Tabular Report by setting Report Source to Report from Server (Live) and then selecting the dsTest platform that is hosting the test.

After a test is complete you can view the results by setting Report Source to Report from Database(Static) and then identifying the database(s) to include in the report. Choose Local File(s) and then click Browse... to navigate to those files if they are already accessible to your file system.

If the databases are still on the server, select Download from Server and a dsTest platform. A list of the OM databases in the current working directory on that platform is displayed, as shown in the example, and you can select the databases to be included in your report. Set a Destination directory using the adjacent Browse... button and dsClient will first download the files and then create the report.

When you're satisfied with your configuration, click OK to open the report screen.

In a live report, dsClient learns about the objects and possible OMs resulting from your test configuration by querying dsTest. Therefore, you must load your configuration prior to opening a live report.

As discussed in Working with OM Databases, statistical information is written to the database after the database has been closed. You can safely download databases for your reports using dsClient, however, as it is aware of the statistical process and will wait for a database to become available before downloading it.

dsTest reports OMs in a hierarchical structure which dsClient Desktop visually displays as a tree. Node emulators, subscriber databases, Packet Capture, and system diagnostics are the top-level objects in that hierarchy as you can see displayed under the Detail Measurements tree node in the example to the right.

Tree nodes that represent hierarchical objects are displayed with the complex element icon (

) and are labeled with the objects' XML element names along with any instance names you defined in your test configuration. Top-level objects will always have instance names. Subordinate objects that may appear more than once under the same parent object will also display instance names. In the example shown there are multiple gNodeB node emulators (gnb), each with a unique instance name.

Under node emulator tree nodes you'll find tree nodes for the applications that were executed by that node emulator as shown under the Summary Measurements node. Physical interfaces that may be shared appear under the node emulator while interfaces that can only be used by one application typically appear under that application's node. As you expand application and interface nodes you'll begin to see tree nodes representing measurements generated by that object along with tree nodes for any subordinate objects.

Subscriber databases report measurements that are applicable to subscribers - SmartEvents and CDR processing OMs, for example - and they will be grouped under the applicable subscriber profile tree node. These sets of OMs will include measurements that are unique to the states and event handlers you've configured in SmartEvents, enabling you to view detailed measurements that match your configuration.

System diagnostics OMs appear directly under the diag tree node. These OMs are typically only used when investigating the cause of a failing test. They report resource availability and utilization, current aggregate TPS on the platform, and transaction throttling due to inadequate licensed TPS availability.

When your test configuration involves more that one object of the same type, dsClient summarizes the individual OMs across those objects. You'll find those measurements under the Summary Measurements tree node. In the example shown, all of the OMs generated for a gNodeB node are summarized into one gNodeB structure in the summary section. Likewise, OMs for the two SPR nodes are summarized into one. If different types of node emulators are running the same interface application or utilize the same type of physical interface, those objects will be summarized across all types of nodes as was the IP interface used by both SPR and gNodeB nodes in the example.

If your test configuration includes 1,000 or more instances of the same type of node, dsTest may automatically summarize the measurement values into one structure for that node type. In this case that structure will appear under the Detail Measurements tree node since those are the most detailed values available.

Measurement tree nodes are labeled with OM names, with error OMs shown in a red font, and are displayed with icons that indicate the measurement type. Accumulator OMs that natively report cumulative values display the ever-increasing value icon (

) while OMs that natively report interval values display the saw-tooth value icon (

There are two ways to add OMs to a report, and you can choose your preferred method in the Preferences screen.

Drag-n-drop: The OM tree is displayed in a collapsible panel on the left side of the report screen. Drag an OM node from the OM tree and drop it on one of the charts in a Chart Report or on a Tabular Report. This is the default method.

Select tree nodes: Use the checkboxes displayed on OM tree nodes, as shown in the example, to select the OMs for a particular chart in a Chart Report or for a Tabular Report. In a Chart Report each chart owns an OM tree and it is displayed in a pop-up dialog, allowing you to select multiple OMs to be added after you click the Update Chart button. In a Tabular Report the OM tree is always visible and the OMs are added to the report as you select the checkboxes.

The names of OMs you've added will be shown in bold in the OM tree as will the entire hierarchy that ascends from those OMs. This allows you to quickly find you way back to a structure to add sibling OMs.

To remove an OM from a report, click on the OM's legend in a Chart Report or on the OM's row in a Tabular Report and then click the delete icon (

). To remove all OMs from a report, click the delete all icon (

). If you're using the checkbox method to add OMs you can clear checkboxes to remove OMs, and you can both add and remove OMs at the same time when using a chart's pop-up dialog.

Since the same sets of OMs appear under like objects and the same measurement names - Transaction Attempts, for example - are used in many different applications, breadcrumb-like OM paths are used in chart legends, row headers, and anywhere else the identities of specific OMs are displayed. The paths for summary measurements always begin with Sum> followed by the element name of the object that generated the OM: Sum>n2>Transaction Successful, for example. The paths for detail measurements use instance names where available and element names otherwise, and trace the entire hierarchical path to the OM: demo-ds-gnb>n2>interface>sctp>Connections Initiated would identify a measurement from one of the gNodeB nodes in the OM tree depicted above.

When you roll your mouse over a chart legend or the OM label in a Tabular Report the full path for that OM is displayed in a tool tip.

When you save a Chart Report or Tabular report, the full, absolute path of each OM is written to the report configuration file, and when you open that report dsClient attempts to locate those paths in the report source you identified. If you change the instance names of any named objects in an OM's ancestry, that OM will be excluded from the report as its path will have changed. Summary measurements would not be affected since their paths consist of only element names. A pop-up alert will inform you of any measurements that could not be located.

dsTest includes in its Interval Notifications the number of errors that have occurred during the test, enabling dsClient to alert you to those anomalies as they occur. It does so by displaying the number of errors in the toolbar of a Chart Report or a Tabular Report. When you open a report from a database dsClient sums the final values of all error OMs in order to provide that information.

Clicking the View... button adjacent to the error counter opens a pop-up dialog, shown here, that displays all of the error OMs that have non-zero values, along with the latest value for each. You can use this dialog to add those measurements to your report.

All of the measurements are selected by default and you can use the Clear All and Select All buttons to clear the current selections or re-select all OMs respectively. Clearing checkboxes in this dialog has no effect on OMs that you've already included in your report.

In a Tabular Report, clicking Add Selections to Report appends the selected measurements to the report. In a Chart Report, you have the additional options to specify to which chart the OMs will be added. The default option, Add OMs to new chart adds a chart to the report while Add OMs to selected chart will cause the OMs to be added to the chart you specify.

You can revisit this dialog and continue to add more error OMs to your report. Any OMs you have already added to your report will be shown in bold and the dialog will prevent you from adding them again unless you had removed them.

Native Mode: All measurements report values as they are reported by dsTest. Accumulator OMs report cumulative values while rate, duration, and value log OMs report interval values.

Interval Values: In this mode accumulator OMs report their incremental values for each interval. OMs that report interval values report as they do in Native Mode.

Change-per-Second: This mode is useful for viewing how rapidly values are changing for all types of OMs - particularly when the report interval has been extended.

In a SmartReport or a Chart Report, Report Mode is set on the individual charts. In a Tabular Report the mode is applied to all OMs in the report.

You can export the values from any visual report to a CSV file that you can then import to a spreadsheet or database to either archive test results or to produce your own graphical representation.

From a Chart Report or a Tabular Report, click the To CSV... button to open the export dialog. This button will be disabled when a live report is running and you will need to stop the report before you can export the data.

From SmartReport you can use the context menu (

) on any chart to access the Export to CSV option.

No matter how you initiate the export the options are those shown in the example.

Export To: Use the Browse... button to select a destination and to set the name of the resulting CSV file. A .csv extension will be appended to the file name if needed.

Time Frame: You can export the all report intervals, only the intervals currently displayed in a chart (this option is only available when you have "zoomed in" to a subset of intervals on a chart), or a range of report intervals specified by the first and last interval ID.

Measurement Values: Select the Report Mode for the exported values and choose which measurements to include. When exporting from a Chart Report or Tabular Report all OMs in the report may be exported while an export from SmartReport will only include the measurements from the selected chart.

Click Export Report to generate the file and a pop-up alert will inform you when the file is ready for use.