Plot Files

One of collectl's main features is its ability to generate output in a ready-to-plot format and write that data to one or more files which are compatible with what gnuplot expects and there are actually 2 main types of files that it generates. The first, which has an extension of tab, represents a table of all the summary data. What makes this file unique is that all data elements are in a fixed set of columns - some columns may get added over time, but for all intents and purposes, the set of data for say CPUs do not change regardless of how many CPUs are in the system. The second type of files deal with detail data, the amount of which changes with the number of instances so a 4 CPU system will have 1/2 the data an 8 CPU system has. There is one file for each type of detail data and like raw files you tell collectl where to put the plot files with -f.

Plot files can be generated in 2 ways and each has its own advantages as well as disadvantages.

Caution - though one can leave off -f and have collectl write its plot formatted output to the terminal, this should be avoided unless there is a specific need. In fact, since multiple detail files from multiple systems can actually contain different numbers of columns, collectl explcitly only allows generating detail plot data from a single raw file.

At first glance, it sounds like you'd always want to generate plot files directly since you avoid the need for the conversion step, but you should also realize a few things about this methodology:

Generating Plot Files On-The-Fly

While generating files this way is as easy as appending -P to the collectl command either when run interactively or in /etc/collectl.conf, there are a couple of things to keep in mind:

Generating Plot Files from RAW Files

Collectl has the capability to play back a single file or multiple once but in either case the first thing collectl does is examine the raw file header to get the source host name and creation date. There will always be a new set of data generated for each unique combination of host and creation date. Note that depending on the subsystems chosen there may be multiple output files generated. This also means a single raw file that spans multiple dates will result in a single set of data.

By default, the name of the plot file contains only the date and a test is made to see if a file with that name already exists. If not, it is created in append mode. This means that multiple raw data files for the same host on the same date will result in a single set of data. However, if that file already exists, collectl will NOT process any data, and request you specify -oc to tell it to perform the first open in create mode so that subsequent files can be appended. If you specify -oa all files will be appended to the original one which may not be what you want. Collectl cannot read your mind so to be safe, be explicit. If you want to generate a unique set of data files for each raw file use -ou which causes the time to be included in file names, resulting in a unique output file name for each raw file.

This certainly maximizes your flexibility for all the reasons listed earlier. However, this now puts the responsibility of managing your data more squarely on your shoulders. Some of the questions you need to answer include:

Having answered these questions and perhaps others, it now just becomes a matter of executing the appropriate copy and/or collectl commands, which can be relatively easily scripted.

TIP - If you rsync raw files to another server and then process them using a wildcard in your playback command, you will probably end up processing some of today's files too! If you then later copy over the rest of today's file(s) you will need to recreate today's plot file since collectl will not overwrite an exiting file by default. But if you specify the -oc switch with a wild card you will end up recreating all the plot files which will result in a lot more processing than you were planning on. Collectl supports a special syntax that allows you to playback just the files from yesterday by replacing that string with yesterday's date as in the following:

collectl -p "YESTERDAY*" etc...
noting that all uppercase characters are required and you can include other characters in the string such as a host name if need be.

TIP - If you want to create multiple sets of plot files from the same raw file, you can always include a unique qualifier along with the directory name with the -f switch to give each set a different prefix.

Daily vs Unique Plot Files

Collectl raw files are created every time a new instance of collectl is run or whenever collectl is instructed to create new one via -r such as when running as a Daemon. This is why each file name include a time as well as a date. However trying to plot multiple files for any given day can be problematic even for an automation script that might help generate plots for you and so by default collectl creates non-timestamped, daily plot files.

Whether you choose to create plot files on-the-fly or manually (by playing back existing files), if you've not instructed collectl to do anything with unique file, it will simply append new data onto an existing file. One the other hand if you explicitly ask for unique files, whenever a new raw is processed or a new instance of collect is run, a new plot file will be created that includes a corresponding timestamp.

The obvious question then becomes, why would you ever choose to create unique files when they're such a pain to plot? and there are actually several good reasons you might choose to do so:

Dealing with configuration changes
Consider the following situation: you run collectl twice during the same day with the following commands:
collectl -scd  -P -f/tmp
collectl -scdm -P -f/tmp
or perhaps you even generated raw files first and later play them back, converting them to plottable files. In either case you now want to plot the data. Since it's all in the same file, the headers that are initially written will only tell you there is cpu and disk data in the file! Collectl will have written a second set of headers in the file at the time the second instance was run, but do you really want to have to make a pass through the whole file every time you want to generate a plot looking for additional data? Furthermore, there will be less columns of data in the first part of the file that the latter, a condition that will probably cause most plotting packages to blow up, so you really need unique files.

Another situation that can cause this is when dealing with detail data for dynamic subsystems such as Lustre and disks. Dynamic change detection for Lustre has always been a part collectl but support for dynamic disk configuration changes has been added to collectl V3.3.4. When a configuration change is detected, it forces collectl to create a new file, whether generating data in raw or plot format. Furthermore, if you later try to combine disk data from multiple raw files into a single plot file that has disk configuration change data in it, collectl won't let you unless you specific -ou forcing the generation of unique files. Lustre changes can be combined into a single plot file by adding in any missing columns and 0-filling them.

If you are generating non-unique plot files on-the-fly and a configuration change occurs, that new data will simply be appended to the single file. There is nothing collectl can do about this, because it wants to keep all files in a consistent name format and will not switch to unique name formatting without explicitly being told to do so.

Configuration changes do not happen often and this is only an issue when generating plot files in real-time. Furthermore, since this only effects detail, because the summary data will always accurately reflect the sum of the instance data, this typically will not effect anyone but is being stated for completeness.

If after all this you choose to generate real-time, non-unique detailed plot files and find yourself in a situation where you should have, you can always write a script to split the plot files back into individual ones since there is sufficient data in the internal headers to do so. If you have multiple unique files and find single files easier to deal with, you can also choose to write a post-processing script that merges these into a single file with zero-filled columns where there is missing data.
updated June 9, 2017