This talk by Brendan Gregg is a fantastic introduction to the tools available in linux for performance instrumentation, monitoring, and debugging.
More to come on which of these tools I use and documenting how I like to use them. (e.g. bcantrill's dtrace, ebpf, metrics, etc.)