Foreword xxi Preface About the Authors Acknowledgments PART ONE: Observability Methods Chapter 1 Introduction to Observability Tools 1.1 Observability Tools 1.2 Drill-Down Analysis 1.3 About Part One Chapter 2: CPUs 2.1 Tools for CPU Analysis 2.2 vmstat Tool 2.3 CPU Utilization 2.4 CPU Saturation 2.5 psrinfo Command 2.6 uptime Command 2.7 sar Command 2.8 Clock Tick Woes 2.9 mpstat Command 2.10 Who Is Using the CPU? 2.11 CPU Run Queue Latency 2.12 CPU Statistics Internals 2.13 Using DTrace to Explain Events from Performance Tools 2.14 DTrace Versions of runq-sz, %runocc 2.15 DTrace Probes for CPU States Chapter 3: Processes 3.1 Tools for Process Analysis 3.2 Process Statistics Summary: prstat 3.3 Process Status: ps 3.4 Tools for Listing and Controlling Processes 3.5 Process Introspection Commands 3.6 Examining User-Level Locks in a Process 3.7 Tracing Processes 3.8 Java Processes Chapter 4: Disk Behavior and Analysis 4.1 Terms for Disk Analysis 4.2 Random vs. Sequential I/O 4.3 Storage Arrays 4.4 Sector Zoning 4.5 Max I/O Size 4.6 iostat Utility 4.7 Disk Utilization 4.8 Disk Saturation 4.9 Disk Throughput 4.10 iostat Reference 4.11 Reading iostat 4.12 iostat Internals 4.13 sar -d 4.14 Trace Normal Form (TNF) Tracing for I/O 4.15 DTrace for I/O 4.16 Disk I/O Time 4.17 DTraceToolkit Commands 4.18 DTraceTazTool Chapter 5: File Systems 5.1 Layers of File System and I/O 5.2 Observing Physical I/O 5.3 File System Latency 5.4 Causes of Read/Write File System Latency 5.5 Observing File System “Top End” Activity 5.6 File System Caches 5.7 NFS Statistics Chapter 6: Memory 6.1 Tools for Memory Analysis 6.2 vmstat(1M) Command 6.3 Types of Paging 6.4 Physical Memory Allocation 6.5 Relieving Memory Pressure 6.6 Scan Rate as a Memory Health Indicator 6.7 Process Virtual and Resident Set Size 6.8 Using pmap to Inspect Process Memory Usage 6.9 Calculating Process Memory Usage with ps and pmap 6.10 Displaying Page-Size Information with pmap 6.11 Using DTrace for Memory Analysis 6.12 Obtaining Memory Kstats 6.13 Using the Perl Kstat API to Look at Memory Statistics 6.14 System Memory Allocation Kstats 6.15 Kernel Memory with kstat 6.16 System Paging Kstats 6.17 Observing MMU Performance Impact with trapstat 6.18 Swap Space Chapter 7: Networks 7.1 Terms for Network Analysis 7.2 Packets Are Not Bytes 7.3 Network Utilization 7.4 Network Saturation 7.5 Network Errors 7.6 Misconfigurations 7.7 Systemwide Statistics 7.8 Per-Process Network Statistics 7.9 TCP Statistics 7.10 IP Statistics 7.11 ICMP Statistics Chapter 8: Performance Counters 8.1 Introducing CPU Caches 8.2 cpustat Command 8.3 cputrack Command 8.4 busstat Command Chapter 9: Kernel Monitoring 9.1 Tools for Kernel Monitoring 9.2 Profiling the Kernel and Drivers 9.3 Analyzing Kernel Locks 9.4 DTrace lockstat Provider 9.5 DTrace Kernel Profiling 9.6 Interrupt Statistics: vmstat -i 9.7 Interrupt Analysis: intrstat PART TWO: Observability Infrastructure Chapter 10: Dynamic Tracing 10.1 Introduction to DTrace 10.2 The Basics 10.3 Inspecting Java Applications with DTrace 10.4 DTrace Architecture 10.5 Summary 10.6 Probe Reference 10.7 MDB Reference Chapter 11: Kernel Statistics 11.1 C-Level Kstat Interface 11.2 Command-Line Interface 11.3 Using Perl to Access kstats 11.4 Snooping a Program’s kstat Use with DTrace 11.5 Adding Statistics to the Solaris Kernel 11.6 Additional Information PART THREE: Debugging Chapter 12: The Modular Debugger 12.1 Introduction to the Modular Debugger 12.2 MDB Concepts Chapter 13: An MDB Tutorial 335 13.1 Invoking MDB 335 13.2 MDB Command Syntax 336 13.3 Working with Debugging Targets 13.4 GDB-to-MDB Reference 13.5 dcmd and Walker Reference Chapter 14: Debugging Kernels 14.1 Working with Kernel Cores 14.2 Examining User Process Stacks within a Kernel Image 14.3 Switching MDB to Debug a Specific Process 14.4 kmdb, the Kernel Modular Debugger 14.5 Kernel Built-In MDB dcmds APPENDICES Appendix A Tunables and Settings Appendix B DTrace One-Liners Appendix C Java DTrace Scripts Appendix D Sample Perl Kstat Utilities Bibliography Index
