Testwell CTC++ / Kernelcoverage
This capability is a special use case of
the Testwell CTC++ Host-Target add-on package. While in the general
Host-Target case there are two computers, the host and the target
computers, in Kernelcoverage the issue is primarily of user-space code
and kernel-space code. Kernelcoverage describes and suggests a
technique how the collected execution counter data can be transferred
from kernel space to user space.
Measuring coverage in kernel-space code is
generally a challenge to instrumentation-based coverage tools. In
kernel-space code the instrumented probes cannot use any library
functions or systems calls while in user-space code they can be used.
The CTC++ way of doing the instrumentation and the Host-Target run-time
support layer (provided in C source code form) assumes only basic C
with no usage of system services. Thus it needs minimal adaptations so
that it can be compiled for and execued at the operating system kernel
space.
[Remark 13 November 2003: The below
exercise was done in 2001. Our plan it to re-do this with current CTC++
version and now applying CTC++ on the new Linux kernel 2.6...] As
a proof of concept for Kernelcoverage we have instrumented a complete
Linux kernel and have run a certain test suite on it. Here we describe
the use case. You can also view the resultant CTC++ coverage
report.
The starting point was a Linux 2.4.0
kernel, consisting of 227 C code files (158000 source lines) plus the
necessary header files. Originally their compilation took about 9
minutes (wall clock time) resulting in a 545 kB kernel.
When all these 227 code files were
instrumented (CTC++'s widest coverage instrumentation mode
'multicondition coverage' was used) and the instrumented files
compiled, it took about 16 minutes (+78%). The resultant instrumented
kernel was 916 kB (+68%). [Remark 13 November 2003: This was taken
from compressed images. Comparing the uncompressed images, the size
increase was 44%.]
The publicly available Linux kernel test
suite from http://oss.sgi.com/projects/ltp/
version ltp-20001012 was executed both with the original
non-instrumented kernel and with the instrumented kernel. No
human-noticeable slowing down of performance could be observed. When
measuring with time(1) it seemed that 16% more time was spent
in the instrumented kernel than in the original kernel when executing
this test suite.
The test suite itself did not bring very
high test coverage over all these 227 C files. Booting and login made
already 21% and running the test suite increased it with 4% to 25%. The
coverage percentage, however, is not the point here (and CTC++ has no
control over it). The point is to show that a complete kernel can be
instrumented with CTC++, even its most time-critical portions, and the
execution profile can be obtained.
Also, this exercise gives you an idea what
overhead CTC++ brings on the instrumentation/compilation time, on the
instrumented executable size, and on the instrumented code execution
time. Even though we did the instrumentation so that the CTC++ overhead
would be maximal, it is still quite reasonable. And it may well be that
we are not interested in all those 227 files but measuring only a
couple of files would suffice, or a lesser instrumentation mode than
'multicondition coverage' would suffice, in which case these overheads
would be lower.
See Kernelcoverage report for
viewing the CTC++ coverage report of this exercise [Remark 18 May 2007: The report is worked
up with old ctc2html version. In current CTC++ the HTML report
looks a bit different] . Two browser windows are opened, two
frames in each. The left frame in the first window shows the coverage
summaries on file level. Clicking on a file name focuses the right
frame on the clicked file and gives you a zoom-in view of the functions
of the file. Clicking the links in the right frame focuses the second
window on the detailed execution profile listing of the pertinent file
or function. Finally, in the right frame of the second window you can
view the actual source file. The top frame shows all the 227 files. The
other frames contain c. 8 files each, that is, you need to start
navigating from the top frame again, for seeing the files that are not
near enough to your current focus point.
Portions for Kernelcoverage and this
demonstration of it has been done at our former German distributor. The
special work for Kernelcoverage is now continued at our current German
distributor Verifysoft Technology Gmbh.