Testwell CTC++ / Kernelcoverage

Measuring coverage in kernel-space code is generally a challenge to instrumentation-based coverage tools. In kernel-space code the instrumented probes cannot use any library functions or systems calls while in user-space code they can be used. The CTC++ way of doing the instrumentation and the Host-Target run-time support layer (provided in C source code form) assumes only basic C with no usage of system services. Thus it needs minimal adaptations so that it can be compiled for and execued at the operating system kernel space.

[Remark 13 November 2003: The below exercise was done in 2001. Our plan it to re-do this with current CTC++ version and now applying CTC++ on the new Linux kernel 2.6...] As a proof of concept for Kernelcoverage we have instrumented a complete Linux kernel and have run a certain test suite on it. Here we describe the use case. You can also view the resultant CTC++ coverage report. 

The starting point was a Linux 2.4.0 kernel, consisting of 227 C code files (158000 source lines) plus the necessary header files. Originally their compilation took about 9 minutes (wall clock time) resulting in a 545 kB kernel.

When all these 227 code files were instrumented (CTC++'s widest coverage instrumentation mode 'multicondition coverage' was used) and the instrumented files compiled, it took about 16 minutes (+78%). The resultant instrumented kernel was 916 kB (+68%). [Remark 13 November 2003: This was taken from compressed images. Comparing the uncompressed images, the size increase was 44%.]

The publicly available Linux kernel test suite from http://oss.sgi.com/projects/ltp/ version ltp-20001012 was executed both with the original non-instrumented kernel and with the instrumented kernel. No human-noticeable slowing down of performance could be observed. When measuring with time(1) it seemed that 16% more time was spent in the instrumented kernel than in the original kernel when executing this test suite.

The test suite itself did not bring very high test coverage over all these 227 C files. Booting and login made already 21% and running the test suite increased it with 4% to 25%. The coverage percentage, however, is not the point here (and CTC++ has no control over it). The point is to show that a complete kernel can be instrumented with CTC++, even its most time-critical portions, and the execution profile can be obtained.

Also, this exercise gives you an idea what overhead CTC++ brings on the instrumentation/compilation time, on the instrumented executable size, and on the instrumented code execution time. Even though we did the instrumentation so that the CTC++ overhead would be maximal, it is still quite reasonable. And it may well be that we are not interested in all those 227 files but measuring only a couple of files would suffice, or a lesser instrumentation mode than 'multicondition coverage' would suffice, in which case these overheads would be lower.

See Kernelcoverage report for viewing the CTC++ coverage report of this exercise [Remark 18 May 2007: The report is worked up with old ctc2html version. In current CTC++ the HTML report  looks a bit different] . Two browser windows are opened, two frames in each. The left frame in the first window shows the coverage summaries on file level. Clicking on a file name focuses the right frame on the clicked file and gives you a zoom-in view of the functions of the file. Clicking the links in the right frame focuses the second window on the detailed execution profile listing of the pertinent file or function. Finally, in the right frame of the second window you can view the actual source file. The top frame shows all the 227 files. The other frames contain c. 8 files each, that is, you need to start navigating from the top frame again, for seeing the files that are not near enough to your current focus point.

Portions for Kernelcoverage and this demonstration of it has been done at our former German distributor. The special work for Kernelcoverage is now continued at our current German distributor Verifysoft Technology Gmbh.