Testwell CTC++
Test Coverage Analyzer for C/C++
Information in this document corresponds to the CTC++
version 7.1.2
1. INTRODUCTION
Testwell CTC++ is a powerful instrumentation-based code
coverage and dynamic analysis tool for C and C++ code. As a code
coverage tool, CTC++ shows the coverage all the way to the Modified
Condition/Decision Coverage (MC/DC) level as required by DO-178B
projects. As a dynamic analysis tool, CTC++ shows the execution
counters in the code, i.e. more than plain boolean coverage
information. You can also use CTC++ to measure function execution
costs (normally time) and to enable function entry/exit tracing at
test time.
CTC++ is easy to use. When used in command-line
mode (by makefiles or other build scripts), the instrumentation is
just a front end phase at the compile/link command. No changes to
the source files or build scripts are needed. Test runs are done
with the instrumented program version, in the same way as with the
original program. Coverage and other execution profiling reports can
be obtained easily in straight text, HTML, XML and Excel input form.
On some environments, e.g. Microsoft Visua Studio, CTC++ use
is possible directly from the compiler IDE.
CTC++'s overhead on
the size and execution speed of the instrumented code is very reasonable. CTC++'s reporting is informative and well-organised. The
reports give both a top-level view, which show the coverage
percentages at various summary levels, and a detailed view, where the executed/not
executed information is mapped to the actual source code locations.
Commercially
CTC++ is available as follows:
- "CTC++ host-only": This is the basic package of the tool,
so called "host-only". It is available Windows, Linux, Solaris and
HP-UX, see CTC++ availability.
The tool utilities are run at the selected host environment. The
used C/C++ compiler is one of the supported compilers that
generate code for the selected host environment. The tests with
the instrumented code are run at the same host environment.
- "CTC++ with
Host-Target add-on": This contains the basic CTC++ host-only on the
selected host environment and Host-Target add-on package (HOTA).
HOTA gives certain additional components, which facilitate
cross-compiling the instrumented code for a target, running tests
at the target, getting the coverage data back to the host,
and viewing the coverage reports at the host. The cross-compiler can
effectively be "whatever" (except it needs to run at the
host where the basic CTC++ runs). The target machine architecture
and its operating system (if any) can be effectively "whatever".
- "CTC++ with Host-Target
add-on and Bitcov add-on": This is like "CTC++ with Host-Target add-on" but contains additionally the Bitcov add-on package
(Bitcov). BItcov is a variant of HOTA, and
facilitates CTC++ use in targets, which have very limited memory. The Bitcov add-on
package is developed by Verifysoft
Technology.
Additionally there are
separate add-on packages supporting C# and Java code. From CTC++
point of view C# and Java are seen as special dialects of C++, and
with certain arrangements CTC++ instrumentation is connected to the
C#/Java compilation phase, similarly as when instrumenting and
compiling C/C++ code. The C#/Java run-time context is modeled as a
special type of target for which the CTC++ support
library has been implemented of the HOTA components, i.e.
rewritten in C#/Java. The net result is that CTC++ gives
of C#/Java code similar coverage and dynamic analysis information as
here described for C/C++ code. The C# and Java add-on packages are
developed by Verifysoft Technology.
CTC++ is an industry-strength tool, which has
been used over 20 years in the IT industry.
Here are some
quick links on some of the CTC++ capabilities.
CTC++ facilitates
- Measuring code coverage = >
ensure thorough testing, you know when to stop testing, etc.
- Function coverage (instrumentation mode: functions called)
- Decision coverage (instrumentation mode:
additionally to function coverage, conditional expressions true
and false in program branches, case branches in switch
statements, catch exception handlers in C++, control transfers)
-
Statement coverage (analyzed from program
flow: percent of statements executed in functions/files/overall)
- Multicondition coverage (instrumentation
mode: additionally to decision coverage, in program
branches have all the possible ways to evaluate a
conditional expressions having && and || operators been
exercised)
- MC/DC coverage (like multicondition
coverage, but it suffices that each elementary condition is
shown to independently affect the conditional expression outcome--not
so demanding than full multicondition coverage)
- Condition coverage (like
multicondition coverage, but it suffices that each elementary
condition is shown to have been evaluated to true and false--not
so demanding than full multicondition coverage)
- Searching execution bottlenecks => no more guessing in
algorithm tunings
- Function execution timing
(instrumentation mode: total, average, maximum time--if needed, the user may introduce
his own time-taking function for measuring whatever is
interesting in regard of function resource consumption)
- Execution counters (how many times the
CTC++ instrumentation probes have been executed)
- Displaying function call trace => helps in analyzing
program behavior
- You can provide the function, which makes the call tracing
(perhaps displaying on the screen). At instrumentation phase
CTC++ is made to call your trace function at the entry/exit of
each function under test.
- Conveniently presented test results
- Hierarchical, color-coded, HTML-browsable coverage reports
- Pure textual reports
- Coverage data can be converted to Excel input file
- XML report
- Ease of use
- Instrumentation phase as a "front end" to the compilation
command => very simple to use
- No changes are needed to the C/C++ source files to be
measured
- "Ctc-builds" can be done with your existing build makefiles,
which normally can be used as is
- automated script-based use from command line
- Mature product
- Has been in demanding use in
the IT industry for over 20 years
- Long use experience with the most
commonly used C/C++ compilers (VC++, gcc/g++,...)
and with their versions. Many of them have their "extreme corner
specialities", but which the tool can handle.
- Also a proven
record of working with numerous (~30+) cross-compilers
- Full C and C++ support, including
the new C++11 additions (lambda functions, trailing return type, range-based loop, etc.)
- Usable "in the large"
- Instrumentation overhead very reasonable
- You can select what source files to instrument and with what
instrumentation options
- Besides full executables, static and dynamically loaded
libraries (.lib/.a/.dll/.so) can also be measured
-
Capturing coverage data of never-ending processes is conveniently
supported
- Combined coverage report can be
obtained of test runs of different programs and
there are powerful means to select the source files that will
be reported and in what coverage view they will be
reported.
-
Combined coverage report can be obtained of test runs that are run
at different machines.
- On some environments usable via IDE
- Usable at embedded targets (CTC++
Host-Target add-on is needed)
- The target can be effectively "whatever"
- Good management and visibility of testing
- Easy to read listings (textual and HTML)
- In terms of the original source code
- Untested code highlighted
- Various summary level reports (in HTML)
- TER-% (test effectiveness ratio) calculated per function,
source file, directory, and overall
2. HOW CTC++ WORKS
There are basically three steps in the use of CTC++
(command-line mode of use):
- You use the CTC++ Preprocessor (ctc) utility for
instrumenting and compiling the C or C++ source files of interest
and for linking the instrumented program with the CTC++ run-time
library. At this phase ctc maintains a symbol file, MON.sym
by default, where it remembers the names of the instrumented files
and what they contained. If you build your program by a makefile,
you just prepend the make command by ctcwrap
ctc-options
, and all the emitted compile and link commands that
the make emits will be done "in ctc control".
- You execute the test runs with the instrumented program as you
see necessary. When the instrumented code portions are executed,
CTC++ collects the coverage and function timing history in memory.
Normally at the end of the program, and automatically by CTC++,
the collected counters are written to a data file, MON.dat
by default. If there was previous counters in the data file, they
are summed up.
- You use the CTC++ Postprocessor (ctcpost) utility for
putting one or more symbol files and data files together and
produce the human readable textual reports. One of them, the
Execution Profile Listing, can be further processed with
ctc2html utility for getting an easy-to-view hierarchical
and color-coded HTML representation of the coverage information.
You can obtain the coverage information also in XML and Excel
form.
2.1. Building the Instrumented
Program
Use of
ctc is connected to the command by
which you compile and link your programs. Adding 'ctc' and possible
'ctc-options' in front of the original compilation command is
all that is needed for instrumenting and compiling the source file.
If the command also links,
ctc automatically adds the CTC++
run-time library to the linkage. The
ctc on-line help
is:
Usage:
ctc [ctc-options] comp/link-command comp/link-options-and-files
[ctc-options]:
[-i {f|d|m|te|ti}...] [-n symbolfile]
[-v] [-V] [-k] [-h] [@optionsfile]...
[-c conf-file[;conf-file]...]... [-C conf-param{=value|+value}]...
[-2comp] [-no-comp] [-no-templates] [-no-warnings]
On
Windows with
these actual source files
with Visual C++ compiler in command-line mode we could give
command:
ctc -i mte cl -Feprime.exe calc.c io.c prime.c
which
instruments
these three C files with multicondition and (exclusive) timing
instrumentation modes, compiles the instrumented code with the 'cl'
compiler, and links the instrumented target 'prime.exe' with 'cl'
(the CTC++ run-time library is automatically added to the linkage).
The same could have been obtained also with the following sequence
of commands:
ctc -i mte cl -c calc.c
ctc -i mte cl -c io.c
ctc -i mte cl -c prime.c
ctc link -out:prime.exe calc.obj io.obj prime.obj
The
first three commands instrument their argument source file.
ctc also invokes the compiler on the instrumented version
of the source file resulting an object file to the same place as the
original compilation would have generated it. No changes to the
source file is needed by the user and it remains intact.
In
the last command ctc just repeats the linking command and adds CTC++
run-time library to it. The result is an instrumented executable in
the same place as the original linking would have generated
it.
Normally real programs are
not built by explicitly issuing compile and link commands. Instead
they are built by a makefile, for example as follows
nmake -f Makefile
which
uses its own ruling to emit the elementary compile and link
commands. In this case the Makefile can be modified to emit
the compile and link commands prefixed with
ctc, for
example
nmake -f Makefile "CC=ctc -i mte cl" "LNK=ctc link"
Or
a simpler way is to use
ctcwrap command to make the "build"
to be a "ctc-build", as follows:
ctcwrap -i mte nmake -f Makefile
The
ctcwrap command executes its argument
command (here 'nmake') in a special context. In it all compile and
link commands are changed to behave "ctc-wise" with the given
instrumentation options (here with '-i mte'), i.e. they are run as
if they would have
'ctc -i mte' in front of them. The net effect is that the building is
"with CTC++". The makefiles do not need modifications for the
sake of using with CTC++. There are means to specify to CTC++ which
files only will be instrumented of all the files that the
makefile compiles.
2.2. Running the Tests with the
Instrumented Program
You run the tests
with the instrumented program. Due to
instrumentation it is somewhat bigger and slower than originally. How
much? It depends on what kind program control structures you
have in your code, what instrumentation mode you have
selected, what compiler optimization has been used, and have you
instrumented all code files of your program. Increase to program size is normally a concern
only in some embedded target cases, which have limited memory. When
the largest multicondition instrumentation mode is used, a size
increase of 50-70% could result on the instrumented code portions. But note
that normally the programs have also non-instrumented portions like
system libraries that are linked to it. Lower instrumentation gives
lower overhead. In the Windows example Cube.exe case, which contains
many system libraries and where the CTC++ run-time library is in a
separate DLL (see
HTML form
coverage report ) the
actual program size grew only
with 6%.
CTC++'s
impact to the execution speed has been found to be
very modest.
When the program logic executes the code in the
instrumented files, the inserted instrumentation probes collect
execution history in main memory. When the program ends (or at some
explicit user-determined places), the execution counters are
automatically written to a data file on disk. If the instrumented
program is a never-ending process, there are simple means to add to
the instrumented executable an auxiliary thread, which periodically
writes the coverage data to the disk. Multiple executions accumulate
the counters in the file as long as the instrumented file is the
same as before.
To continue the example, the instrumented executable
could be run as follows:
prime
Enter a number (0 for stop program): 2
2 IS a prime.
Enter a number (0 for stop program): 5
5 IS a prime.
Enter a number (0 for stop program): 20
20 IS NOT a prime.
Enter a number (0 for stop program): 0
The program was used and it behaved just like the
original program. At the program end the CTC++ run-time system,
which has been linked to program, wrote the collected
execution counters data to a data file, here to
MON.dat.
2.3. Getting the Results of the
Test Runs
Finally you use the
ctcpost and
ctc2html utilities to get the results for the analysis,
i.e. the various types of listings showing the information you
initially asked for.
ctcpost is used first. Its
on-line help is:
Usage:
ctcpost [general-options] [symbolfile]... [datafile]...
[-ff|-fd|-fc|-fmcdc] [-w report-width] {{-p|-u|-t|-x} reportfile}...
ctcpost [general-options] datafile... -a target-datafile
ctcpost [general-options] symbolfile... -a target-symbolfile
ctcpost [general-options] {-l|-L} {symbolfile|datafile}...
[general-options]:
[-h] [-V] [@optionsfile]...
[-c conf-file[;conf-file]...]... [-C conf-param{=value|+value}]...
[-f source-file[;source-file]...]... [-nf source-file[;source-file]...]...
and
Execution Profile Listing could be obtained as follows:
ctcpost MON.sym MON.dat -p profile.txt
Here
is a more complex example where coverage report is generated of two
independently instrumented progams, and some files, which
however have been instrumented, are excluded from the report:
ctcpost MON.sym MON.dat ..\prj2\MON.sym ..\prj2\MON.dat \
-nf "*\harnessdir\*" -nf "*\moc_*.cpp" -p profile.txt
With
ctcpost you can get the following textual reports:
- Execution Profile Listing shows the missing coverage as well
as how many times each code location has been visited. This is the
primary CTC++ report, normally worked onwards to HTML form. The
report can be also generated with somewhat reduced coverage
information compared how the code was instrumented. See examples
- Untested Code Listing is similar to execution profile listing
but shows only the places where test coverage is
inadequate. The HTML form report shows also the untested information, at
summary levels (TER%) and individual code locations.
- Execution Time Listing shows the
cumulative and average execution times of functions.
- XML report contains the information
that is in Execution Profile Listing and in Execution Time
Listing, but is in XML form. This report is meant for
postprocessing of the coverage data by your own
XML-utility.
Normally the Execution Profile Listing
is straight away worked onwards to
HTML form using
ctc2html utility. Its
on-line help is:
Usage:
ctc2html [-i inputfile] [-o outputdir] [-s sourcedir][;sourcedir]...]...
[-t threshold] [-p prefix] [-nsb] [-no-sources]
[-no-javascript] [-li number]
ctc2html [-h] [--enable-help]
Command-line options:
-i inputfile Input Execution Profile Listing file. Default 'stdin'.
-o outputdir Output HTML directory. Default 'CTCHTML'.
-s sourcedir Source files are searched also from this directory.
-t threshold Set the threshold percent (0-100). Default 100.
-p prefix (Deprecated) File name prefix in HTML files. Default 'index'.
-nsb Do not start HTML browser automatically (only Windows).
-no-sources Do not convert sources to HTML, use only ctcpost listing.
-no-javascript Do not generate javascript on HTML pages.
-li number For load speed reasons reduce Index frame at Execution
Profile page to contain max 'number' lines. Default 2000.
-h Display this command line help.
--enable-help Display help of advanced --enable-XXX options.
Start browsing from file 'outputdir'/'prefix'.html (CTCHTML/index.html).
An
example:
ctc2html -i profile.txt -t 85 -nsb
start CTCHTML\index.html
ctc2html
converts the Execution Profile
Listing information to hierarchical, easily navigable, color-coded HTML
representation. Also the actual source files are incorporated to the HTML.
The generated HTML files can be viewed with normal web browsers.
The
HTML representation is called "CTC++ Coverage Report". It is
hierarchical and has four levels:
- Directory Summary: General header information (from what data
generated, when generated, etc.), directory TERs shown in
histograms and numerically, coverage-% not meeting the suggested
threshold percent (-t option) are shown in red color. TER over all
directories.
- Files Summary: Zoom-in to the files in the directories.
Similar TERs and color-coding are shown but at file levels.
- Functions Summary: Zoom-in to the methods and functions in the files.
Similar TERs and color-coding are shown but at function levels.
- Untested Code: Compact listing of untested code locations
with links to the actual source file to the pertinent
line.
- Execution Profile: Zoom-in to the detailed view where the
execution counters are shown with the source code. Lines that are
not fully exercised with respect to the selected coverage criteria
are highlighted in red.
See the example
HTML report (started in
a new window).
2.4. Getting combined coverage reports
When same instrumented program is run multiple
times, CTC++ automatically accumulates the execution coverage data
to the results of the previous test runs.
When you have many
independently instrumented programs and wish to get a combined
coverage report of them, you can do it for example as
follows:
ctcpost MON.sym ..\test2\MON.sym MON.dat ..\test2\MON.dat -p profile.txt
If your independently instrumented
separate programs contain partially same source files, the
above kind of ctcpost command will give you combined coverage of
them, too. I.e. in the profile.txt file there is one entry of the
file and it contains aggregated coverage of the
file execution in different programs. The requirement on
this to succeed is that the common files really represent the
same level as CTC++ sees them, e.g. compiled with same flags,
and the files have been instrumented in the same way.
In getting the coverage report you can specify that
some selected files only are shown in the report and some others
not. Further, you can specify that the coverage information is shown
in a lower coverage measure (e.g. in a compact function
coverage view only) than the file was actually instrumented
with.
2.5. Connection to Excel
The coverage data from an Execution Profile Listing
can be converted to converted to a TSV (tab separated values) file,
suitable input to Excel (or another spreadsheet application). The
ctc2excel utility is used here. Its on-line help is:
Usage:
ctc2excel [-i inputfile] [-o outputfile]
[-full] [-nopath]
ctc2excel -h
Command-line options:
-i inputfile Specify the input CTC++ execution profile listing.
By default input is read from STDIN.
-o outputfile Specify the output Excel TSV file.
Different linetypes are tagged with '1', '2', etc.
By default only one line (with tag '2') of each
function is copied to the output file.
By default output is written to STDOUT.
-full Generate all Excel TSV output linetypes tagged with
'1', '2', etc.
By default only function specific linetype tagged with
'2' is generated.
-nopath Show no file path.
By default file path is shown.
-h Displays this help text.
And
an example of the use:
ctc2excel -i profile.txt -o excelinput.txt
start excel excelinput.txt
2.6. Function call trace
With certain simple arrangements you can instrument
your code to produce a function call trace. It means that you
provide your own trace function (and link to the executable), which
CTC++ calls at the begin and return of each instrumented function.
As a parameter the trace function gets the name of the function that
was just called. Your trace function, then, can do whatever you see
necessary, for example display the called function name on the
screen.
As an example, the trace function might be the
following:
#include <stdio.h>
void mytrace(char* fname, char* start_or_end) {
printf("mytrace: function %s %s\n", fname, start_or_end);
}
You may find this useful when wanting to analyze the
dynamic behavior of your program. Another situation might be when
your program crashes and you do not know where it happens. You just
instrument the code for producing this "brute force call trace" and
you see how far your program managed to get.
3. IDE INTEGRATIONS
CTC++ has been integrated to a couple
of compilation system IDEs, all at Windows platform. The integration
means that in the IDE Tools menu there has been added new commands,
like
- CTC++ Set... : for setting "instrumentation mode on" and specifying the ctc-options under
which the coming builds in the IDE will be done.
Later this command is used for setting "instrumentation mode
off", i.e. the builds to normal/ctc-free again.
- CTC++ Report...: for getting the various coverage reports and
starting some appropriate viewer (like notepad, html browser,
Excel) on them.
See
CTC++/Visual
Studio Integration
on how the integration looks at Visual Studio IDE.
Some IDEs support
that the build can be commanded from command line. With many of
them, in the next example with Visual Studio 2003, the"ctc-build"
can be made with
ctcwrap as follows:
ctcwrap -i d devenv mySolution.sln /rebuild debug
4. SUPPORT FOR
TARGET TESTING
HOTA:
Information in this chapter corresponds to the CTC++ Host-Target
add-on (HOTA) version 5.0
With "target testing" it is
here meant that you have a host environment, where you do builds for
some target machine, typically an embedded system. The used C/C++
cross-compiler and the target machine can be something with which
CTC++ may not have been used before. You anyway want to instrument
the code, compile it with the cross-compiler, run it at the target
machine, and get the coverage data back to the host for reporting.
The "CTC++ Host-Target add-on" (HoTa) package provides this
capability.
Technically the arrangement goes roughly as
follows:
- At the host you need to have normal CTC++ (host-only) license
copy. Additionally you need this HoTa package.
- You "teach" to CTC++ the cross-compiler/linker command names
and options. This is a one-time job. If the cross-compiler is a
decent one and follows the common conventions, there is no bigger
problem in this.
- The instrumented code needs a little CTC++ run-time support
layer at the target. The HoTa package contains it in C source code
form. It is "vanilla C", about 1000 lines of well-commented code,
and it compiles with any C compiler, also with your
cross-compiler. You need not touch that code.
- However, into the run-time layer's use you need to implement
the low-level data transfer layer, with which the coverage data is
transferred to the host. The coverage data is in CTC++-internal
encoded form as a sequence of printable ascii characters. No CTC++
internal knowledge is needed in its handling, just writing it one
char at a time to somewhere. Ultimately the char stream
needs to be transferred to the host machine to a text file. If you
can write at the target machine the data to a text file (and
separately move it to the host later), the work is simple. For
this alternative the delivery package has a compile-ready
implementation, which uses normal C text file I/O. In some cases
the coverage data needs to be written to some communication
channel, which your program in the host listens and writes
to a file. Developing this low-level data transfer layer is a
one-time job. At test time the coverage data writing is normally a
one-time act at the end of the instrumented program execution.
- The instrumentation phase at the host for the target is
similar as instrumentation for the host. Only the cross-compiler
is used (not the one that compiles for the host) and the slightly
modified target-specific run-time layer is linked to the
instrumented code (not the one that is linked in the ctc-builds to
the host).
- You run the tests at the target. Depending how you have
arranged to data transfer, you get the coverage data to host side
where you feed it CTC++'s ctc2dat utility and you
get MON.dat file. That after the reports can be taken normally at
the host by ctcpost and ctc2html
utilities.
CTC++ Host-Target is designed to cope with situations
where it is not known
- what hardware architecture the target has
- what operating system, if any, the target runs
- what brand is the C/C++ cross compiler for the target
- do the host and target machines use the same endianness in
binary data
- do the target machine (cross-compiler)
use same amount of bits for basic C integral types for storing the
execution counters
The Host-Target add-on package is also usable
when the code under test is operating system kernel code. Read more
from
Kernelcoverage.
BITCOV:
Bitcov is a derivative work based on HOTA.
It is meant for small embedded micro-controller kind
of targets, which have very little free data memory for CTC++'s use,
or where the HOTA style to transfer the coverage data as an
encoded ASCII stream to the host is difficult.
In Bitcov there is one global bit array in the
target main memory where the execution hits are recorded, one bit
per probe. For example with a 1000 byte array 8000 probes could
be recorded. It might be well enough for a reasonable size
instrumented program, around 30000 lines of instrumented code. In
normal instrumentation CTC++ run-time data area consumption
would be in a similar case 8000 * 4 bytes (one counter is
normally 4 bytes) = 32000 bytes + something more
for CTC++'s internal control data needs. At the target
there essentially is no CTC++ run-time layer at
all.
After the test run the bit array is
captured to the host to a file, either so that the initiative is at
the host who pulls the coverage data (e.g. by debugger means)
from the target, or so that the initiative is at the target side
who pushes the coverage data to the host (as it is possible at
the target, e.g. embedded in the instrumented code). At the host the coverage data
is converted to a form, which is suitable to the HOTA tool chain for
further processing (ctc2dat, ctcpost,
ctc2html).
In Bitcov timing instrumentation is not
supported. In coverage reports the counters
are reduced to 0 (not executed) and 1 (executed) while in normal
coverage reports the counter value tells also how many times the
code at the probe location was
executed.
