libbpf:
- Docs: https://www.kernel.org/doc/html/latest/bpf/libbpf/
- Example: https://github.com/libbpf/libbpf-bootstrap/blob/master/examples/c/profile.bpf.c
- From BCC: https://nakryiko.com/posts/bcc-to-libbpf-howto-guide/
Examples from Linux sources:
- User space: https://github.com/torvalds/linux/blob/b0546776ad3f332e215cebc0b063ba4351971cca/samples/bpf/trace_event_user.c
- Kernel space: https://github.com/torvalds/linux/blob/b0546776ad3f332e215cebc0b063ba4351971cca/samples/bpf/trace_event_kern.c
Perf UAPI and eBPF user libraries:
perf_event_attrstruct: https://github.com/torvalds/linux/blob/b0546776ad3f332e215cebc0b063ba4351971cca/include/uapi/linux/perf_event.h#L389perf_event_opensyscall: https://man7.org/linux/man-pages/man2/perf_event_open.2.html- perf: https://perf.wiki.kernel.org/index.php/Main_Page
- How to attach per events with Cilium:
Blogs:
- https://www.brendangregg.com/blog/2016-10-21/linux-efficient-profiler.html
- https://blog.px.dev/cpu-profiling/ (a series)
- https://github.com/pixie-io/pixie-demos/blob/main/ebpf-profiler (with BCC)
- https://www.airplane.dev/blog/building-an-ebpf-based-profiler (with libbpf)
Other material:
- Perf ring buffer and lost events
- BPF ring buffer
BPF_MAP_TYPE_STACK- Brendan Gregg’s stack trace hack
- [Official Linux Offwaketime:
- https://stackoverflow.com/questions/23618743/implementing-an-interrupt-driven-sampling-profiler
Stack walking:
- x86 assembly
- https://c9x.me/x86/html/file_module_x86_id_154.html
- https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf
- DataDog’s Stack traces in Go
- https://stackoverflow.com/questions/57268045/how-to-read-stack-trace-kernelside-in-ebpf
- NEW: https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
Symbolization:
- https://jvns.ca/blog/2018/01/09/resolving-symbol-addresses/
- https://www.polarsignals.com/blog/posts/2022/01/13/fantastic-symbols-and-where-to-find-them
- http://web.cse.ohio-state.edu/~reeves.92/CSE2421au12/SlidesDay52.pdf
- https://mpatrol.sourceforge.net/doc/Call-stacks-and-symbol-tables.html
- Go binary structure: https://utcc.utoronto.ca/%7Ecks/space/blog/programming/GoBinaryStructureNotes
Beyond frame pointers:
Extra:
- Off-CPU profiling: https://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html