Problem: SystemTap fails to find kernel tracepoints

UPDATE: The bug is fixed now, click here to see more.

I just encountered a problem about systemtap on my box, it couldn't find any static kernel tracepoints:

kdr2@Debian-X230:~$ stap -vv -l 'kernel.trace("*")'
Systemtap translator/driver (version 2.6/0.159, Debian version 2.6-0.2)
Copyright (C) 2005-2014 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS
Created temporary directory "/tmp/stapsAsdb0"
Session arch: x86_64 release: 3.10.53
Searched for library macro files: "/usr/share/systemtap/tapset/linux/*.stpm", found: 3, processed: 3
Searched for library macro files: "/usr/share/systemtap/tapset/*.stpm", found: 5, processed: 5
Searched: "/usr/share/systemtap/tapset/linux/x86_64/*.stp", found: 3, processed: 3
Searched: "/usr/share/systemtap/tapset/linux/*.stp", found: 70, processed: 70
Searched: "/usr/share/systemtap/tapset/x86_64/*.stp", found: 1, processed: 1
Searched: "/usr/share/systemtap/tapset/*.stp", found: 28, processed: 28
Pass 1: parsed user script and 110 library script(s) using 94292virt/30644res/2608shr/28524data kb, in 110usr/20sys/125rea$ ms.
Attempting to extract kernel debuginfo build ID from /lib/modules/3.10.53/build/vmlinux.id
Attempting to extract kernel debuginfo build ID from /sys/kernel/notes
semantic error: while resolving probe point: identifier 'kernel' at <input>:1:7
   thrown from: elaborate.cxx:1062
        source: probe kernel.trace("*") {}

semantic error: no match
   thrown from: elaborate.cxx:1025
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 95384virt/32340res/3096shr/29616data kb,
in 30usr/180sys/205real ms.
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
Running rm -rf /tmp/stapsAsdb0
Spawn waitpid result (0x0): 0
Removed temporary directory "/tmp/stapsAsdb0"                      ^

But my kernel is correctly configured with debuginfo and tracepoint supporting:

# cat /sys/kernel/debug/tracing/events/enable
X
# less /sys/kernel/debug/tracing/available_events
drm:drm_vblank_event
drm:drm_vblank_event_queued
drm:drm_vblank_event_delivered
i2c:i2c_write
i2c:i2c_read
...
...
...
skb:skb_copy_datagram_iovec
skb:consume_skb
skb:kfree_skb

That really confused me. After having a disscussion on the systemtap maillist, I got the reason:

I built my kernel in a separate diredtory from the kernel source tree directory, by setting a environment variable named BUILDDIR (export BUILDDIR=/opt/kernel/build/3.10). So my /lib/modules/$(uname -r)/build and /lib/modules/$(uname -r)/source refer to two different directories:

kdr2@Debian-X230:~$ ls -l /lib/modules/$(uname -r)/{build,source}
lrwxrwxrwx 1 root root 28 Sep  3 08:59 /lib/modules/3.10.53/build -> /opt/kernel/build/3.10/build
lrwxrwxrwx 1 root root 29 Sep  3 08:59 /lib/modules/3.10.53/source -> /home/kdr2/Work/opensrc/linux
kdr2@Debian-X230:~$

Under this situation, systemtap fails to find the tracepoints. I recompiled my kernel directly in the kernel source tree(not in a separate BUILDDIR), and the problem fixed.

But on debian, these two directories are indeed two different directories:

kdr2@Debian-X230:~$ dpkg -l|grep -e 'linux-.*-3.16'|awk '{print $2}'
linux-headers-3.16.0-4-amd64
linux-headers-3.16.0-4-common
linux-image-3.16.0-4-amd64
linux-image-3.16.0-4-amd64-dbg
linux-kbuild-3.16
kdr2@Debian-X230:~$ ls -l /lib/modules/3.16.0-4-amd64/{build,source}
lrwxrwxrwx 1 root root 37 Nov  7 07:30 /lib/modules/3.16.0-4-amd64/build -> /usr/src/linux-headers-3.16.0-4-amd64
lrwxrwxrwx 1 root root 38 Nov  7 07:30 /lib/modules/3.16.0-4-amd64/source -> /usr/src/linux-headers-3.16.0-4-common

So systemtap doesn't work well with these kernel packages to find the tracepoits, you need kernel built directly in the kernel source tree by yourself :)

On Fedora, build and source directory is a same one, stap works well.

Have few questions or feedback? Feel free to send me(killian.zhuo📧gmail.com) an email!