# MEMORY FRAGMENTATION (4K pages) #Node Zone 1Pg 2Pgs 4Pgs 8Pgs 16Pgs 32Pgs 64Pgs 128Pgs 256Pgs 512Pgs 1024Pgs
# SINGLE CPU STATISTICS # Cpu User Nice Sys Wait IRQ Soft Steal Guest NiceG Idle INTRPT
| CPU | The CPU number which the stats are associated with |
| User | Time spent in User mode, not including time spend in "nice" mode. |
| Nice | Time spent in Nice mode, that is lower priority as adjusted by the nice command and have the "N" status flag set when examined with "ps". |
| Sys | This is time spent in "pure" system time. |
| Wait | Also known as "iowait", this is the time the CPU was idle during an outstanding disk I/O request. This is not considered to be part of the total or system times reported in brief mode. |
| Irq | Time spent processing interrupts and also considered to be part of the summary system time reported in "brief" mode. |
| Soft | Time spent processing soft interrupts and also considered to be part of the summary system time reported in "brief" mode. |
| Steal | Time spend in involuntary wait state while the hypervisor was servicing another virtual processor. |
| Guest | Time spent running a virtual CPU for guest operating systems under the control of the Linux kernel, new since 2.6.24 |
| NiceG | Time spent running a niced guest (virtual CPU for guest operating systems under the control of the Linux kernel), new since 2.6.33 |
| Idle | Time spent idle, nothing that since CPU numbers are rounded off, they may not always add up to 100% |
| Intrpt | If the interrupt summary stats were requested at the same time, this will be included which is the aggregate number of interrupts for each CPU. |
If you specify filtering with --dskfilt, the disk names that match the pattern(s) will either be included or excluded from the the summary data. However, the data will still be collected so if recorded to a file can later be viewed. Note: if you specify --dskopts f, fractional values will be reported for some of the fields for more precision.
# DISK STATISTICS (/sec) # <---------reads---------><---------writes---------><--------averages--------> Pct #Name KBytes Merged IOs Size KBytes Merged IOs Size RWSize QLen Wait SvcTim Util
| Name | Name of the disk the statistics are being reported for. |
| KBytes | KB read/sec |
| Merged | Read requests merged per second when being dequeued. |
| IOs | Number of reads/sec |
| Size | Average read I/O size in KBytes |
| KBytes | KB written/sec |
| Merged | Write requests merged per second when being dequeued. |
| IOs | Number of writes/sec |
| Size | Average write I/O size in KBytes |
| RWSize | Average combined read and write I/O size in KBytes. This is not the average of the read and write sizes but rather the sum of the reads/write divided by the number of I/Os |
| QLen | Average number of requests queued |
| Wait | Average time in msec for a request has been waiting in the queue |
| SvcTim | Average time in msec for a request to be serviced by the device |
| Util | Percentage of CPU time during which I/O requests were issued |
# INFINIBAND STATISTICS (/sec) #HCA KBIn PktIn SizeIn KBOut PktOut SizeOut Errors
| HCA | HCA instance name |
| KBIn | KB received/sec. |
| PktIn | Received packets/sec. |
| SizeIn | Average incoming packet size in KB |
| KBOut | KB transmitted/sec. |
| PktOut | Transmitted packets/sec. |
| SizeOut | Average outgoing packet size in KB |
| Errs | Count of current errors. Since these are typically infrequent, it is felt that reporting them as a rate would result in either not seeing them OR round-off hiding their values. |
# INTERRUPT DETAILS # Int Cpu0 [Cpu...] Type Device(s)
| Int | Interrupt number within the range 0-255. Note that only those interrupts that have had any activity since the last monitoring interval will be reported |
| CPUn... | The CPU for which the interrupt count is being reported. There will be one column/CPU |
| Type | Interrupt type, whitespace removed |
| Device | The names of the devices which are generating this interrut as a comma separated list |
There are several formats the lustre detail data can take based on whether you're looking at a client or an OSS (there is not any MDS specific detail data, though it does share the same disk-level buffer size data as the OSS). Furthermore, if one specifies the -sLL form of the detail switch OST level details will be reported where appropriate.
Lustre Client, collectl -sL
# LUSTRE CLIENT DETAIL (/sec) #Fils KBRead Reads SizeKB KBWrite Writes SizeKB
| Filsys | Name of the filesystem these stats apply to |
| KBRead | KBs read/sec |
| SizeKB | Average read size |
| Reads | Reads/sec |
| KBWrite | KBs written/sec |
| Writes | Writes/sec |
| SizeKB | Average write size |
Lustre Client, collectl --lustops O
# LUSTRE CLIENT DETAIL (/sec) #Fils Ost KBRead Reads SizeKB KBWrite Writes SizeKB
| Filsys | Name of the filesystem these stats apply to |
| Ost | OST name within the filesystem |
| KBRead | KBs read/sec |
| Reads | Reads/sec |
| SizeKB | Average read size |
| KBWrite | KBs written/sec |
| Writes | Writes/sec |
| SizeKB | Average write size |
Lustre Client RPB-Buffer Stats, collectl --lustopts B
# LUSTRE CLIENT DETAIL: RPC-BUFFERS (pages) #Filsys Ost RdK Rds 1K 2K ... WrtK Wrts 1K 2K ...
| Filsys | Name of the filesystem these stats apply to |
| Ost | OST name within the filesystem |
| RdK | KBs read/sec |
| Rds | Reads/sec |
| nK | Number of pages of of this size read |
| WrtK | KBs written/sec |
| Wrts | Writes/sec |
| nK | Number of pages of of this size written |
Lustre Client Metadata, collectl -sL --lustopts M
# LUSTRE CLIENT DETAIL: METADATA #Filsys KBRead Reads KBWrite Writes Open Close GAttr SAttr Seek Fsync DrtHit DrtMis
| Filsys | Name of the filesystem these stats apply to |
| KBRead | KBs read/sec |
| Reads | Reads/sec |
| KBWrite | KBs written/sec |
| Writes | Writes/sec |
| Open | Opens/sec |
| Close | Closes/sec |
| GAttr | Get Attributes/sec |
| SAttr | Set Attributes/sec |
| Seek | Seeks/sec |
| Fsync | FSyncs/sex |
| DrtHit | Dirty Hits/sec |
| DrtMis | Dirty Misses/sec |
Lustre Client Readhead, collectl -sL --lustopts R
# LUSTRE CLIENT DETAIL: READAHEAD #Filsys KBRead Reads KBWrite Writes Pend Hits Misses NotCon MisWin LckFal Discrd ZFile ZerWin RA2Eof HitMax
| Filsys | Name of the filesystem these stats apply to |
| KBRead | KBs read/sec |
| Reads | Reads/sec |
| KBWrite | KBs written/sec |
| Writes | Writes/sec |
| Pend | Pending issued pages |
| Hits | Hits |
| Misses | Misses |
| NotCon | Readpage not consecutive |
| MisWin | Miss inside window |
| LckFal | Failed lock match |
| Discrd | Read but discarded |
| ZFile | Zero length file |
| ZerWin | Zero size window |
| RA2Eof | Read-ahead to EOF |
| HitMax | Hit max r-a issue |
Lustre OSS, collectl -sL
# LUSTRE FILESYSTEM SINGLE OST STATISTICS (/sec) #Ost KBRead Reads SizeKB KBWrite Writes SizeKB
| Ost | OST name |
| KBRead | KBs read/sec |
| Reads | Reads/sec |
| SizeKB | Average read size |
| KBWrite | KBs written/sec |
| Writes | Writes/sec |
| SizeKB | Average write size |
Lustre OSS RPC Buffers, collectl -sL --lustopts B
# LUSTRE FILESYSTEM SINGLE OST STATISTICS #Ost RdK Rds 1P 2P ... WrtK Wrts 1P 2P ...
| Filsys | Name of the filesystem these stats apply to |
| Ost | OST name within the filesystem |
| RdK | KBs read/sec |
| Rds | Reads/sec |
| nP | Number of pages of of this size read |
| WrtK | KBs written/sec |
| Wrts | Writes/sec |
| nP | Number of pages of of this size written |
Lustre OSS and MDS Disk Buffers, collectl -sL --lustopts D
This display is very similar the the RPC buffers in that the sizes of different size I/O requests are reported. In this case there are requests send to the disk driver. Note that this report is only available for HP's SFS.
# LUSTRE DISK BLOCK LEVEL DETAIL (units are 512 bytes) #DISK RdK Rds 0.5K 1K 2K ... WrtK Wrts 0.5K 1K 2K ...
| Disk | Name of the disk these stats apply to |
| RdK | Reads/sec |
| Rds | KBs read/sec |
| nK | Number of blocks of of this size read |
| WrtK | Writes/sec |
| Wrts | KBs written/sec |
| nK | Number of blocks of of this size written |
This is also known as numa data and provides detail information about memory utilization in each numa node.
# MEMORY STATISTICS # Node Total Used Free Slab Mapped Anon Locked Inact HitPct
| Node | Numa node number, which is usually the same as a physical socket |
| Total | Total physical memory |
| Used | Used physical memory. This does not include memory used by the kernel itself. |
| Free | Unallocated memory |
| Slab | Memory used for slabs, see collectl -sY |
| Mapped | Memory mapped by processes |
| Anon | Anonymous memory |
| Locked | Locked memory |
| Inactive | Inactive pages, which is the sum of Inactive(anon) and Inactive(file). Note that Inactive(anon) is not considered nor included in the previous anonynous memory field. |
| Hit% | It is currenlty not entirely clear how useful this number actually is as it refers to the hit percentages for both local and foreign memory as a single number. Most importat, it does not refer to memory access but rather memory allocation. In other words, it does not differentiate between one failing to allocation memory and only referencing it a small number of time vs a very large number of times. Clearly the latter would be more interesting from a performance perspective. |
If you specify filtering with --netfilt, the names that match the pattern(s) will either be included or excluded from the the summary data. However, the data will still be collected so if recorded to a file can later be viewed.
# NETWORK STATISTICS (/sec) #Num Name KBIn PktIn SizeIn MultI CmpI ErrsI KBOut PktOut SizeO CmpO ErrsO
| Num | Each network interface is numbered, starting with 0 |
| Name | Name of the interface |
| KBIn | Incoming KB/sec |
| PktIn | Incoming packets/sec |
| SizeI | Average incoming packet size in bytes |
| MultI | Incoming multicast packets/sec |
| CmpI | Incoming compressed packets/sec |
| ErrsI | Total incoming errors/sec. This is an aggregration of incoming errors. To see explicit error counters use --netopts e |
| KBOut | Outgoing KB/sec |
| PktOut | Outgoing packets/sec |
| SizeO | Average outgoing packet size in bytes |
| CmpO | Outgoing compressed packets/sec |
| ErrsO | Total outgoing errors/sec. This is an aggregation of outgoing errors. To see explicit error counters use --netopts e |
# NETWORK ERRORS SUMMARY (/sec) #Num Name ErrIn DropIn FifoIn FrameIn ErrOut DropOut FifoOut CollOut CarrOut
| Num | Each network interface is numbered, starting with 0 |
| Name | Name of the interface |
| ErrIn | Receive errors/sec detected by the device driver |
| DropIn | Receive packets dropped/sec |
| FifoIn | Receive packet FIFO buffer errors/sec |
| FrameIn | Receive packet framing errors/sec |
| ErrOut | Transmit errors/sec detected by the device driver |
| DropOut | Transmit packets dropped/sec |
| FifoOut | Transmit packet FIFO buffer errors/sec |
| CollOut | Transmit collisions/sec detected on the interface |
| CarrOut | Transmit packet carrier loss errors detected/sec |
By default, collectl will report all 6 types of nfs data (clients and servers for all 3 versions of nfs) unless one has limited the reporting with --nfsfilt. For versions prior to 3.2.1 which only collected a single type of data, collectl will only report that single type.
Also note that nfs V2 records 2 fields that V3 doesn't record and V4 records many more. At this time and detail data is standardized on the V3 format and fields not collected will be left blank. These fields map onto those reported by nfsstat as indicated and are reported as rates.
# NFS SERVER/CLIENT DETAILS (/sec) #Type Read Writ Comm Look Accs Gttr Sttr Rdir Cre8 Rmov Rnam Link Rlnk Null Syml Mkdr Rmdr Fsta Finf Path Mknd Rdr+
| Type | a combination of Clt or Svr and one of 2, 3 or 4 |
| Read | Reads |
| Writ | Writes |
| Comm | Commits |
| Look | Lookups |
| Accs | Accesses |
| Gttr | Getattrs |
| Sttr | Setattrs |
| Rdir | Readdirs |
| Cre8 | Creates |
| Rmov | Removes |
| Rnam | Renames |
| Link | Links |
| Rlnk | Readlinks |
| Null | Nulls |
| Syml | Symlinks |
| Mkdr | Mkdirs |
| Rmdr | Rmdirs |
| Fsta | Fsstats |
| Finf | Fsinfos |
| Path | Pathconfs |
| Mknd | Mknods |
| Rdr+ | Readdirpluses |
There are actually multiple formats process data can be displayed in, the default being the one shown immediately below. By using --procopts as shown in later examples, you can change what is displayed, noting that when playing back the data from a raw file, all information has been recorded and so you can actually play it back multiple times and see different views.
These switched can also be use in conjunction with --top.
# PROCESS SUMMARY (faults are /sec) # PID User PR PPID S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command
| PID | Pid of the process |
| User | Name of user which this process is running under. In playback mode on a different machine, use -oP to direct collectl to use the password file named in collectl.conf (default is /etc/passwd) to lookup the corresponding username. Otherwise the UID will be reported instead. |
| PR | Process priority |
| PPID | PID of this process's parent |
| S | Process State: S - Sleeping, D - Uninterruptable Sleep, R - Running, Z - Zombie or T - Stopped/Traced |
| VSZ | This is the amount of VS memory used by this process |
| RSS | This is the amount of RSS memory used by this process |
| CP | CPU number this process is currently running on |
| SysT | The amount of System Time this process used during this interval |
| UsrT | The amount of User Time this process used during this interval |
| Pct | Percentage of the current interval taken up by this task (the User and System time are used for this calculation) |
| AccuTime | Total accumulated System and User time since the process began execution |
| RKB | This is the number of kilobytes of data written by each process. Both this and the WKB field are only present if the kernel had proces I/O monitoring enabled which is not the default as of 2.6.23. |
| WKB | This is the number of kilobytes of data read by each process |
| MajF | Major Page Faults per second |
| MinF | Minor Page Faults per second |
| Command | Command that is running. Path and command line options are NOT included unless --procopts w |
This format is essentially idential to the last except that it adds extended information to the display, specifically VCtx and NCtx.
# PROCESS SUMMARY (counters are /sec) # PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB VCtx NCtx MajF MinF Command
| VCtx | Voluntary context switches |
| NCtx | Non-voluntary context switches |
# PID User PPID S SysT UsrT Pct AccuTime RKB WKB RKBC WKBC RSys WSys Cncl Command
| PID | Pid of the process |
| User | Name of user which this process is running under. In playback mode on a different machine, use -oP to direct collectl to use the password file named in collectl.conf (default is /etc/passwd) to lookup the corresponding username. Otherwise the UID will be reported instead. |
| PPID | PID of this process's parent |
| S | Process State: S - Sleeping, D - Uninterruptable Sleep, R - Running, Z - Zombie or T - Stopped/Traced |
| SysT | The amount of System Time this process used during this interval |
| UsrT | The amount of User Time this process used during this interval |
| Pct | Percentage of the current interval taken up by this task (the User and System time are used for this calculation) |
| AccuTime | Total accumulated System and User time since the process began execution |
| RKB | Attempt to count the number of bytes which this process really did cause to be fetched from the storage layer by doing calls to read_bytes. This is done at the submit_bio() level, so it is accurate for block-backed filesystems. |
| WKB | Attempt to count the number of bytes which this process caused to be sent to the storage layer by doing calls to write_bytes. This is done at page-dirtying time. |
| RKBC | Number of bytes which were read via read, readv, pread and sendfile. Since these requests are satisfied from kernel pagecache they won't be accounted for by RKB, because they didn't require any I/O. |
| WKBC | Number of bytes which were written via write, writev, pwrite and sendfile. Like RKBC, since the I/O uses the pagecache these values won't be accounted for by WKB. |
| RSys | Number of read syscalls, specifically: read, pread, readv and sendfile |
| WSys | Number of write syscalls, specifically: write, pwrite, writev and sendfile |
| Cncl | Number of cancelled write bytes. |
# PID User S VmSize VmLck VmRSS VmData VmStk VmExe VmLib VmSwp MajF MinF Command
| PID | Pid of the process |
| User | Name of user which this process is running under. In playback mode on a different machine, use -oP to direct collectl to use the password file named in collectl.conf (default is /etc/passwd) to lookup the corresponding username. Otherwise the UID will be reported instead. |
| S | Process State: S - Sleeping, D - Uninterruptable Sleep, R - Running, Z - Zombie or T - Stopped/Traced |
| VmSize | Size of Virtual memory used by the entire process |
| VmLck | Size of Locked Virtual Memory |
| VmRSS | Size of Resident Virtual Memory |
| VmData | Size of Virtual Memory used for heap |
| VmStk | Size of Virtual Memory used for stack |
| VmExe | Size of Virtual Memory used for exe and statically linked libraries |
| VmLib | Size of Virtual Memory used for dynamically linked libraries |
| VmSwp | Size of Virtual Memory used for swapping. This does not necessarily mean a process is actively swapping but only that the memory has been mapped. |
| MajF | Major Page Faults per second |
| MinF | Minor Page Faults per second |
| Command | Command that is running. Path and command line options are NOT included unless --procopt w |
There are actualy 3 different formats for slab data. The first applies to all kernels prior to 2.6.22 and contains the same fields as the Summary report for each named slab and loses the caches fields.
# SLAB DETAIL # <-----------Objects----------><---------Slab Allocation------><----Change--> #Name InUse Bytes Alloc Bytes InUse Bytes Total Bytes Diff Pct
| Objects | |
| InUse | Total number of objects that are currently in use. |
| Bytes | Total size of all the objects in use. |
| Alloc | Total number of objects that have been allocated but not necessarily in use. |
| Bytes | Total size of all the allocated objects whether in use or not. |
| Slab Allocation | |
| InUse | Number of slabs that have at least one active object in them. |
| Bytes | Total size of all the slabs. |
| Total | Total number of slabs that have been allocated whether in use or not. |
| Bytes | Total size of all the slabs that have been allocted whether in use or not. |
| Diff | Change in size of this slab since last sample in bytes |
| Pct | Percentage change in size of this slab |
This second format applies to the new SLUB allocator starting with the 2.6.22 kernel. As with the old format slab detail report, the same fields as are found in the Slab Summary Report are shown for each named slab.
# SLAB DETAIL # <----------- objects --------><--- slabs ---><---------allocated memory--------> #Slab Name Size /slab In Use Avail SizeK Number UsedK TotalK Change Pct
| Objects | |
| Size | Size of a single slab object |
| /Slab | The number of objecs in a single slab |
| InUse | The total number of objects that have been allocated to processes. |
| Avail | The total number of objects that are available in the currently allocated slabs. This includes those that have already been allocated toprocesses. |
| Slabs | |
| SizeK | The size of one slab, which typically contains multiple objects |
| Number | This is the number of individual slabs that have been allocated and taking physical memory. |
| Memory | |
| UsedK | Memory used by those objects that have been allocated to processes. |
| TotalK | Total physical memory allocated to processes. When there is no filtering in effect, this number will be equal to the Slabs field reported by -sm. |
| Change | Change in size of this slab since last sample in bytes |
| Pct | Percentage change in size of this slab |
The third format uses a common format based on the slabtop utility for displaying top slab data sorted by any of the listed column headers. All on need to do is use one of these names as the argument to the --top switch in lower case (use --showtopopts for full list of options to the --top switch). All the same rules apply for controlling the number of lines in the data section, mixing in subsystem data and even using with playback mode.
# TOP SLABS 15:38:08 #NumObj ActObj ObjSize NumSlab Obj/Slab TotSize TotChg TotPct Name
| NumObj | Total number of objects that are available, which includes those in use |
| ActObj | Number of objects that are in use |
| ObjSize | Size of an individual slab object |
| NumSlab | Total number of slabs that have been allocated |
| Obj/Slab | This is the constant number of objects that fit into one slab |
| TotSize | Amount of memory consumed by this slab even if not all objects are actualy allocated |
| TotChg | Change in size of this slab since last sample in bytes |
| TotPct | Percentage change in size of this slab |
| Name | Slab Name |
| updated July 23, 2014 |