Testwell CTC++ is a powerful instrumentation-based test coverage and dynamic analysis tool for C and C++ code. As a coverage tool, CTC++ shows the coverage all the way to the Modified Condition/Decision Coverage (MC/DC) level as required by DO-178B projects. As a dynamic analysis tool, CTC++ shows the execution counters in the code, i.e. more than plain boolean coverage information. You can also use CTC++ to measure function execution costs (normally time) and to enable function entry/exit tracing at test time.

CTC++ is easy to use. When used in command-line mode (by makefiles or other build scripts), the instrumentation is just a front end phase at the compile/link command. No changes to the source files or build scripts are needed. Test runs are done with the instrumented program version, in the same way as with the original program. Coverage and other execution profiling reports can be obtained easily in straight text, HTML, XML and Excel input form. On some environments, e.g. Microsoft Visua Studio, CTC++ use  is possible directly from the compiler IDE.

CTC++'s overhead on the size and execution speed of the instrumented code is very reasonable. CTC++'s reporting is informative and well-organised. The reports give both the top-level summary view and the detailed view mapped all the way to the actual source code.

Commercially CTC++ is available as follows:

• "CTC++": the basic package of the tool, so called "host-only", on one of the following environment: Windows, Linux, Solaris, HP-UX, see  CTC++ availability. The tool utilities are run at the selected host environment. The used C/C++ compiler is one of the supported compilers that generate code for the selected host environment. The tests with the instrumented code are run on the same host environment.
• "CTC++ Host-Target add-on": an add-on package to CTC++, which gives certain additional components facilitating cross-compiling the instrumented code for a target, running tests on the target, getting the coverage data back to the host, and viewing the coverage reports at the host. The cross-compiler can effectively be "whatever" (except it needs to run at the host where the basic CTC++ runs). The target machine architecture and its operating system (if any) can be effectively "whatever".
  
• "CTC++ for Symbian Target Devices add-on": an add-on package on Windows-hosted CTC++, which supports CTC++ usage (test runs) on the Symbian OS target devices, i.e. on the mobile phones

CTC++ is an industry-strength tool, which has been used about 20 years in the IT industry.

Here are some quick links on some of the CTC++ capabilities.

• How to instrument, compile and link
• How to run the instrumented program
• How to get the reports
• Example of HTML form coverage report (started in a new window)
• Examples of textual form Execution Profile, Untested Code, Coverage Summary, Execution Time reports
• Example of XML report
  
• Integration to Visual Studio
• Integration to makefiles
  
• Support for target testing
• Support for Symbian OS testing (both on the EPOC emulator and on the device)
  

CTC++ facilitates

• Measuring test coverage => ensure thorough testing, you know when to stop testing, etc.
• Function coverage (functions called)
• Decision coverage (conditional expressions true and false in program branches, case branches in switch statements, catch exception handlers in C++, control transfers)
• Condition coverage (additionally to decision coverage, elementary conditions true and false in conditional expressions -- roughly corresponds to Modified Condition/Desicion Coverage)
• Multicondition coverage (additionally to decision coverage, all possible ways exercised to evaluate a conditional expression -- slightly stronger than Modified Condition/Desicion Coverage)
• Searching execution bottlenecks => no more guessing in algorithm tunings
• Function execution timing (if needed, the user may introduce his own time-taking function for measuring whatever is interesting in regard of function resource consumption)
• Execution counters
• Displaying function call trace => helps in analyzing program behavior
• You can provide the function, which makes the call tracing (perhaps displaying on the screen). At instrumentation phase CTC++ is made to call your trace function at the entry/exit of each function under test.
  
• Conveniently presented test results
• Hierarchical, color-coded, HTML-browsable coverage reports
• Pure textual reports
• Coverage data can be converted to Excel input file
• XML report
  
• Ease of use
• Instrumentation phase as a "front end" to the compilation command => very simple to use
• No changes are needed to the C/C++ source files to be measured
  
• "Ctc-builds" can be done with your existing build makefiles, which normally can be used as is
• automated script-based use from command line
  
• Usable "in the large"
• Instrumentation overhead very reasonable
• You can select what source files to instrument and with what instrumentation options
• Besides full executables, libraries and DLLs can also be measured
• Measuring of never-ending processes is conveniently supported (without any source code modifications CTC++  can link to the enstrumented code a thread, which gets called, and which periodically takes care of the coverage data writing from memory)
  
• On some environments usable via IDE
• See description of the CTC++/Visual Studio integration (standard add-on on Windows version of CTC++)
• Usable at embedded targets (CTC++ Host-Target add-on is needed)
  
• The target can be effectively "whatever"
  
• Usable for Symbian code testing
• When used on the EPOC emulator on Windows (standard add-on on Windows version of CTC++)
• When used on Symbian target devices (mobile phones, CTC++ for Symbian Target Devices add-on is needed)
  
• Good management and visibility of testing
• Easy to read listings (textual and HTML)
• In terms of the original source code
• Untested code highlighted
• Various summary level reports
• TER-% (test effectiveness ratio) calculated per function, source file, directory, and overall

There are basically three steps in the use of CTC++ (command-line mode of use):

• You use the CTC++ Preprocessor (ctc) utility for instrumenting and compiling the C or C++ source files of interest and for linking the instrumented program with the CTC++ run-time library. At this phase ctc maintains a symbol file, MON.sym by default, where it remembers the names of the instrumented files and what they contained.
• You execute the test runs with the instrumented program as you see necessary. When the instrumented code portions are executed, CTC++ collects the coverage and function timing history in memory. Normally at the end of the program, and automatically by CTC++, the collected counters are written  to a data file, MON.dat by default. If there was previous counters in the data file, they are summed up.
  
• You use the CTC++ Postprocessor (ctcpost) utility for putting one or more symbol files and data files together and produce the human readable textual reports. One of them, the Execution Profile Listing, can be further processed with ctc2html utility for getting an easy-to-view hierarchical and color-coded HTML representation of the coverage information. You can obtain the coverage information also in XML and Excel form.

Use of ctc is connected to the command by which you compile and link your programs. Adding 'ctc' and possible 'ctc-options'  in front of the original compilation command is all that is needed for instrumenting and compiling the source file. If the command also links, ctc automatically adds the CTC++ run-time library to the linkage. The ctc on-line help is:

Usage:
ctc [ctc-options] comp/link-command comp/link-options-and-files

[ctc-options]:
   [-i [f][d][m][t]] [-n symbolfile] [-2comp] [-no-comp] [-no-templates]
   [-c conf-file[;conf-file]...]... [-C conf-param{=value|+value}]...
   [-no-warnings] [-v] [-V] [-k] [-h] [@optionsfile]...

On Windows with these actual source files with Visual C++ compiler in command-line mode we could give command:

ctc -i mt cl -Feprime.exe calc.c io.c prime.c

which instruments these three C files with multicondition and timing instrumentation modes, compiles the instrumented code with the 'cl' compiler, and links the instrumented target 'prime.exe' with 'cl' (the CTC++ run-time library is automatically added to the linkage). The same could have been obtained also with the following sequence of commands:

ctc -i mt cl -c calc.c
ctc -i mt cl -c io.c
ctc -i mt cl -c prime.c
ctc link -out:prime.exe calc.obj io.obj prime.obj

The first three commands instrument their argument source file. ctc also invokes the compiler on the instrumented version of the source file resulting an object file to the same place as the original compilation would have generated it. No changes to the source file is needed by the user and it remains intact.

In the last command ctc just repeats the linking command and adds CTC++ run-time library to it. The result is an instrumented executable in the same place as the original linking would have generated it.

Normally real programs are not built by explicitly issuing compile and link commands. Instead they are built by a makefile, for example as follows

nmake -f Makefile

which uses its own ruling to emit the elementary compile and link commands.  In this case the Makefile can be modified to emit the compile and link commands prefixed with ctc, for example

nmake -f Makefile "CC=ctc -i mt cl" "LNK=ctc link"

Or a simpler way is to use ctcwrap command to make the "build" to be a "ctc-build", as follows:

ctcwrap -i mt nmake -f Makefile

The ctcwrap command executes its argument command (here 'nmake') in a special context. In it all compile and link commands are changed to behave "ctc-wise" with the given instrumentation options (here with '-i mt'), i.e. they are run as if they would have 'ctc -i mt' in front of them. The net effect is that the building is "with CTC++".  The makefiles do not need modifications for the sake of using with CTC++.

You run the tests with the instrumented program. Due to instrumentation it is marginally bigger and marginally slower than originally. How much? It depends on what instrumentation modes you have selected and how many files you have instrumented in your program. With the largest coverage instrumentation in a non-trivial case the instrumentation increases the program size c. 10 % (plus the fixed size CTC++ run-time library, which in some environments is linked statically to the instrumented code, in some environments it comes from a DLL or from a shared library). Instrumenting with "lower" instrumentation modes or instrumenting only some files of the program gives lower overheads. Besides those points, the functionality of the instrumented program is as with the original program. CTC++'s impact to the execution speed has been found to be very modest.

prime
Enter a number (0 for stop program): 2
2 IS a prime.

Enter a number (0 for stop program): 5
5 IS a prime.

Enter a number (0 for stop program): 20
20 IS NOT a prime.

Enter a number (0 for stop program): 0

The program was used and it behaved just like the original program. At the program end the CTC++ run-time system, which  has been linked to program, wrote the collected execution counters data to a data file, here to MON.dat.

Finally you use the ctcpost and ctc2html  utilities to get the results for the analysis, i.e. the various types of listings showing the information you initially asked for. ctcpost  is used first. Its on-line help is:

Usage:
ctcpost [general-options] [-T threshold] [-w listing-width] [-ff|-fd|-fc]
        [-f source-file[;source-file]...]... [symbolfile]... [datafile]...
        [-p|-s|-u|-t outputfile]...
ctcpost [general-options] [datafile]... -a target-datafile
ctcpost [general-options] {-l|-L} {symbolfile|datafile}...

[general-options]:
   [-c conf-file[;conf-file]...]... [-C conf-param{=value|+value}]...
   [-V] [-h] [@optionsfile]...

and Execution Profile Listing could be obtained as follows:

ctcpost MON.sym MON.dat -p profile.txt

With ctcpost you can get the following textual reports:

• Execution Profile Listing shows the missing coverage as well as how many times each code location has been visited. This is the primary CTC++ report, normally worked onwards to HTML form.
  
• Untested Code Listing is similar to execution profile listing but shows only the places where test coverage is inadequate. The HTML form report shows also the untested information, at summary levels (TER%) and individual code locations.
  
• Coverage Summary Listing shows summary of coverage percentages: overall, per each source file, and per each function. The HTML form report shows also this information, and perhaps in a more easy-to-read.
  
• Execution Time Listing shows the cumulative and average execution times of functions.
• XML report contains the information that is in Execution Profile Listing and in Execution Time Listing, but is in XML form. This report is meant for postprocessing of the coverage data by your own XML-utility.
  

Normally the  Execution Profile Listing is straight away worked onwards to HTML form using ctc2html utility. Its on-line help is:

Usage:
  ctc2html [-i inputfile] [-t threshold] [-s source-dir]...
           [-o output-dir] [-p prefix] [-nsb] [-no-sources]
           [-no-javascript] [-li number]
  ctc2html -h

Command-line options:
  -i inputfile   The input Execution Profile Listing file. Default is STDIN.
  -t threshold   Set the threshold percent (0-100). Default is 100.
  -s source-dir  Source files are searched also from this directory.
  -o output-dir  The output HTML directory. Default is CTCHTML.
  -p prefix      Specify a file name prefix for the generated HTML files.
                 Default prefix is "index".
  -nsb           Do not start HTML browser automatically (only Windows).
  -no-sources    Do not convert sources to HTML, use only ctcpost listing.
  -no-javascript Do not generate javascript on HTML pages.
  -li number     Limit Index frame at Execution Profile page to contain only the
                 the file names, if Index would be > 'number' lines.
                 Default is 2000.
  -h             Displays this help text.
Start browsing from the file CTCHTML/index.html

An example:

ctc2html -i profile.txt -t 85
start CTCHTML\index.html

ctc2html  converts the Execution Profile Listing information to hierarchical, easily navigable, color-coded HTML representation. Also the actual source files are incorporated to the HTML. The generated HTML files can be viewed with whatever commonly used web browsers.

The HTML representation is called "CTC++ Coverage Report". It is hierarchical and has four levels:

• Directory Summary: General header information (from what data generated, when generated, etc.), directory TERs shown in histograms and numerically, coverage-% not meeting the suggested threshold percent (-t option) are shown in red color. TER over all directories.
• Files Summary: Zoom-in to the files in the directories. Similar TERs and color-coding are shown but at file levels.
• Functions Summary: Zoom-in to the methods and functions in the files. Similar TERs and color-coding are shown but at function levels.
• Execution Profile: Zoom-in to the detailed view where the execution counters are shown with the source code. Lines that are not fully exercised with respect to the selected coverage criteria are highlighted in red.
  

See the example  HTML report (started in a new window).

When same instrumented program is run multiple times, CTC++ automatically accumulates the execution coverage data to the results of the previous test runs.

When you have many independently instrumented programs and wish to get a combined coverage report of them, you can do it for example as follows:

ctcpost MON.sym MON.dat ..\test2\MON.sym ..\test2\MON.dat -p profile.txt

In getting the Execution Profile Listing you can specify that only selected files are shown in the report. Further, you can specify that the coverage information is shown in a lower coverage measure (e.g. in a compact function coverage view only) than the file was actually instrumented with.

The coverage data from an Execution Profile Listing can be converted to converted to a TSV (tab separated values) file, suitable input to Excel (or another spreadsheet application). The ctc2excel utility is used here. Its on-line help is:

Usage:
  ctc2excel [-i inputfile] [-o outputfile]
            [-full] [-nopath]
  ctc2excel -h

Command-line options:
  -i inputfile          Specify the input CTC++ execution profile listing.
                        By default input is read from STDIN.
  -o outputfile         Specify the output Excel TSV file.
                        Different linetypes are tagged with '1', '2', etc.
                        By default only one line (with tag '2') of each
                        function is copied to the output file.
                        By default output is written to STDOUT.
  -full                 Generate all Excel TSV output linetypes tagged with
                        '1', '2', etc.
                        By default only function specific linetype tagged with
                        '2' is generated.
  -nopath               Show no file path.
                        By default file path is shown.
  -h                    Displays this help text.

See man ctc2excel or %CTCHOME%\DOC directory for more help

And an example of the use:

ctc2excel -i profile.txt -o excelinput.txt
start excel excelinput.txt

#include <stdio.h>
void mytrace(char* fname, char* start_or_end) {
   printf("mytrace: function %s %s\n", fname, start_or_end);
}

• CTC++ Set... : for setting "instrumentation mode on" and specifying the ctc-options under which the coming builds in the IDE will be done. Later this command is used for setting "instrumentation mode off", i.e.  the builds to normal/ctc-free again.
  
• CTC++ Execute...: for running the instrumented code. Or, in some integrations, the instrumented executable is run just in the same way as the original non-instrumented executable is run, without any CTC++ involvement in its starting.
• CTC++ Report...: for getting the various coverage reports and starting some appropriate viewer (like notepad, html browser, Excel) on them.

See CTC++/Visual Studio Integration on how the integration looks at Visual Studios 6.0 / 2003 / 2005 / 2008 / 2010.

Some IDEs support that the build can be commanded from command line. With many of them, in the next example with Visual Studio 2003, the"ctc-build" can be made with ctcwrap  as follows:

ctcwrap -i d devenv mySolution.sln /rebuild debug

Information in this chapter corresponds to the CTC++ Host-Target add-on version 4.0

With "target testing" it is here meant that you have a host environment, where you do builds for some target machine, typically an embedded system. The used C/C++ cross-compiler and the target machine can be something with which CTC++ may not have been used before. You anyway want to instrument the code, compile it with the cross-compiler, run it at the target machine, and get the coverage data back to the host for reporting. The "CTC++ Host-Target add-on" (HoTa) package  provides this capability.

Technically the arrangement goes roughly as follows:

• At the host you need to have normal CTC++ (host-only) license copy. Additionally you need this HoTa package.
  
• You "teach" to CTC++ the cross-compiler/linker command names and options. This is a one-time job. If the cross-compiler is a decent one and follows the common conventions, there is no bigger problem in this.
  
• The instrumented code needs a little CTC++ run-time support layer at the target. The HoTa package contains it in C source code form. It is "vanilla C", about 1000 lines of well-commented code, and it compiles with any C compiler, also with your cross-compiler. You need not touch that code.
  
• However, into the run-time layer's use you need to implement the low-level data transfer layer, with which the coverage data is transferred to the host. The coverage data is in CTC++-internal encoded form as a sequence of printable ascii characters. No CTC++ internal knowledge is needed in its handling, just writing it one char at a time to somewhere.  Ultimately the char stream needs to be transferred to the host machine to a text file. If you can write at the target machine the data to a text file (and separately move it to the host later), the work is simple. For this alternative the delivery package has a compile-ready implementation, which uses normal C text file I/O. In some cases the coverage data needs to be written to some communication channel, which  your program in the host listens and writes to a file. Developing this low-level data transfer layer is a one-time job. At test time the coverage data writing is normally a one-time act at the end of the instrumented program execution.
  
• The instrumentation phase at the host for the target is similar as instrumentation for the host. Only the cross-compiler is used (not the one that compiles for the host) and the slightly modified target-specific run-time layer is linked to the instrumented code (not the one that is linked in the ctc-builds to the host).
• You run the tests at the target. Depending how you have arranged to data transfer, you get the coverage data to host side where you feed it CTC++'s ctc2dat  utility and you get MON.dat file. That after the reports can be taken normally at the host by ctcpost and ctc2html utilities.
  

• what hardware architecture the target has
• what operating system, if any, the target runs
• what brand is the C/C++ cross compiler for the target
• do the host and target machines use the same endianness in binary data
• do the target machine (cross-compiler) use same amount of bits for basic C integral types for storing the execution counters
  

The Host-Target add-on package is also usable when the code under test is operating system kernel code. Read more from  Kernelcoverage.

CTC++ is a versatile tool to be used in testing and tuning of all kinds of applications written in C or C++.

Testing becomes more efficient

The execution profile listing reveals the parts of code which have not yet been executed. The coverage information helps to design the missing test cases. On the other hand, CTC++ helps to determine when to stop testing (from code coverage point of view), thus preventing the waste of the costly human resources.

Testing becomes a measurable, well-managed activity

The summary listing with TER-% histograms characterizing the reached test coverage gives valuable information for project management purposes at a glance.

Usable for program dynamic analysis and performance tuning

When CTC++ is used for improving the performance of a program, it can easily show where the bottlenecks of the program are (functions that are executed most often and functions that consume most time). Also production level applications can be monitored when they are used in their real environment. The function call tracing capability can be very useful in analyzing the program behavior.

Use of CTC++ is easy and fast

CTC++ is easy to use. No modifications of the user's code is needed. Instrumentation takes place by just adding 'ctc' in front of the compilation/link command. Enforcing the existing makefiles to build instrumented targets instead of the original non-instrumented ones is very straightforward. This does not need any changes to the makefiles itself [especially this is true in Unixes, where the ctcwrap utility can be used].  Browsing the coverage results in HTML is very easy. The overall picture is shown in color-coded histograms of coverage percentages. Zooming to the detailed level can be be done with only a few mouse clicks and the untested code locations are clearly shown mapped to the original source code.

 Independent instrumentation of source file

The source files of the executable program are instrumented independently of each other, or left as non-instrumented. The source files can be instrumented with such instrumentation modes that are appropriate in the situation at hand. Instrumented source files can be linked to different executables and yet the merged coverage of their executions in different executables can be obtained.

Configurability

CTC++ is easy to configure for specific needs by simply editing the textual configuration file. At the time instrumentation and getting the reports there are powerful means to fine-tune the process to obtain the desired result.

Usable for many purposes

CTC++ can be used for many purposes: measuring code coverage at various testing phases (module testing, integration testing, system testing), performance testing, optimization, comparing efficiency of algorithms, locating dead code, ...

Support for host-target and kernel code testing

CTC++ has powerful support for measuring code coverage at embedded targets. The instrumentation overhead is very moderate. The used C/C++ cross-compiler for the target, the target operating system and the target hardware type can be virtually of any brand and type. Applying on kernel code is something unique among coverage tools.

Supports both C and C++

With one CTC++ tool you can work both with C and C++ code.

Work motivation

The programmers/testers are likely to design more and better test cases to achieve higher test coverage when they have an easy-to-use tool for measuring it.

Usable with CTA++

CTC++ can be used together with Testwell's test harnessing tools CTA++ (C++ Test Aider), see  CTA++ description. Such usage combines the "black box" (behavioral, functional) and "white box" (structural) testing strategies for purposes of systematic module testing and reducing of testing costs.

Usable with other vendors' test execution frameworks

CTC++ can be seamlessly used with other vendors' unit test and system (GUI) test execution frameworks.

CTC++ has been compared to US Army Jeep...

Simple to use. Works. Can be driven on almost whatever terrain (especially when Host-Target add-on package is used).

CTC++ is available on several machine / operating system / C/C++ compiler environments, see the detailed list: CTC++ availability