Piece of Unix Hints

I’ve not used RRDtool for a while and put back my attention on it few weeks ago. I found out that lots of new cool stuff are avalaible, like LSLSLOPE, LSLINT. These function return the parameters of the Least Squares Line (y = ax +b) approximating a dataset (LSLSLOPE return a, LSLINT return b).
This is interesting because with the function approximating your data you can graph a prediction of future data. Of course a Least Squares Line function will work best to approximate a dataset that tend to grow or shrink (like filesystem usage, memory usage, …) but not for data like temperature. I would say that if your data can be expressed in a percentage, an Least Squares Line can be fine. For data not tending to grow or shrink rrdtool provide some other function like TREND and PREDICT.

I will show how to use LSLSLOPE and LSLINT taking memory usage of a device as an example. My exemple will produce a graph like the following :

As you see, the graph show trend using two Least Squares Line function, one generated from the full dataset (dataset is starting 24 Oct 2009) and one generated only from last week data. Projection on time axis is done from 90% to 100% of memory usage and the date resulting of calculation for 90% and 100% of usage is displayed. I’ve seen lots of question asking how to do this but did not found any answer, so I hope that my example will provide an answer.

Here is the perl code I’m using to generate this graph. There is no Perl specific code, so it can be converted to a normal rrdtool command.

#! /usr/bin/perl
use RRDs;

$rrd_file = 'MEMORY.rrd';

RRDs::graph "MEMORY_Trend.png",
'--start', "10/24/2009",
'--end', "12/31/2009 00:00am",
'--title', "Memory Usage",
'--interlace', '--width=620', '--height=200',
"--color","ARROW#009900",
'--vertical-label', "Memory used (%)",
'--lower-limit', '0',
'--upper-limit', '100',
'--border','0',
'--rigid',

"DEF:used1=$rrd_file:used:AVERAGE",
"DEF:used2=$rrd_file:used:AVERAGE:start=10/24/2009",
"DEF:used3=$rrd_file:used:AVERAGE:start=-1w",
"DEF:used4=$rrd_file:used:AVERAGE:start=-2w",
"DEF:used5=$rrd_file:used:AVERAGE:start=-4w",
"DEF:free1=$rrd_file:free:AVERAGE",
"DEF:free2=$rrd_file:free:AVERAGE:start=10/24/2009",
"DEF:free3=$rrd_file:free:AVERAGE:start=-1w",
"DEF:free4=$rrd_file:free:AVERAGE:start=-2w",
"DEF:free5=$rrd_file:free:AVERAGE:start=-4w",

"CDEF:pused1=used1,100,*,used1,free1,+,/",
"CDEF:pused2=used2,100,*,used2,free2,+,/",
"CDEF:pused3=used3,100,*,used3,free3,+,/",
"CDEF:pused4=used4,100,*,used4,free4,+,/",
"CDEF:pused5=used5,100,*,used5,free5,+,/",

“LINE1:90″,
“AREA:5#FF000022::STACK”,
“AREA:5#FF000044::STACK”,

"COMMENT:                         Now          Min             Avg             Max\\n",
"AREA:pused1#00880077:Memory Used",
'GPRINT:pused1:LAST:%12.0lf%s',
'GPRINT:pused1:MIN:%10.0lf%s',
'GPRINT:pused1:AVERAGE:%13.0lf%s',
'GPRINT:pused1:MAX:%13.0lf%s' . "\\n",
"COMMENT: \\n",

'VDEF:D2=pused2,LSLSLOPE',
'VDEF:H2=pused2,LSLINT',
'CDEF:avg2=pused2,POP,D2,COUNT,*,H2,+',
'CDEF:abc2=avg2,90,100,LIMIT',
'VDEF:minabc2=abc2,FIRST',
'VDEF:maxabc2=abc2,LAST',

'VDEF:D3=pused3,LSLSLOPE',
'VDEF:H3=pused3,LSLINT',
'CDEF:avg3=pused3,POP,D3,COUNT,*,H3,+',
'CDEF:abc3=avg3,90,100,LIMIT',
'VDEF:minabc3=abc3,FIRST',
'VDEF:maxabc3=abc3,LAST',

"AREA:abc2#FFBB0077",
"AREA:abc3#0077FF77",
"LINE2:abc2#FFBB00",
"LINE2:abc3#0077FF",

"LINE1:avg2#FFBB00:Trend since 24 Oct 2009                      :dashes=10",
"LINE1:avg3#0077FF:Trend since 1 week\\n:dashes=10",
"GPRINT:minabc2:  Reach  90% @ %c :strftime",
"GPRINT:minabc3:  Reach  90% @ %c \\n:strftime",
"GPRINT:maxabc2:  Reach 100% @ %c :strftime",
"GPRINT:maxabc3:  Reach 100% @ %c \\n:strftime",

;

my $ERR=RRDs::error;
die "ERROR : $ERR" if $ERR;

You have a .rrd file but you don’t remember how much time RRAs can store data ? all right, no problem.

First save this Perl code into a .pl file.

$step = $1 if (m/step = (\d+)/);
$rows = $1 if (m/rra.*\.rows = (\d+)/);
if (m/(.*)\.pdp_per_row = (\d+)/) {
$pdp = $2; $time = $step*$rows*$pdp;
if ($time > 31536000) { $time = sprintf("%.2f year",$time/31536000) }
elsif ($time > 86400) { $time = sprintf("%.2f days",$time/86400) }
elsif ($time > 3600) { $time = sprintf("%.2f hours",$time/3600) }
print "$1: $step*$rows*$pdp = $time\n";
}

Assuming the filename of the script is rrd_info.pl, run this Shell command:

rrdtool info file.rrd | perl -n rrd_info.pl

Here is an output example:

rra[0]: 300*864*1 = 3.00 days
rra[1]: 300*864*5 = 15.00 days
rra[2]: 300*702*25 = 60.94 days
rra[3]: 300*840*125 = 364.58 days
rra[4]: 300*840*625 = 4.99 year
rra[5]: 300*864*1 = 3.00 days
rra[6]: 300*864*5 = 15.00 days
rra[7]: 300*702*25 = 60.94 days
rra[8]: 300*840*125 = 364.58 days
rra[9]: 300*840*625 = 4.99 year
rra[10]: 300*864*1 = 3.00 days
rra[11]: 300*864*5 = 15.00 days
rra[12]: 300*702*25 = 60.94 days
rra[13]: 300*840*125 = 364.58 days
rra[14]: 300*840*625 = 4.99 year

The formula is update step time * numbers of rows in the rra * pdp_per_row

A common problem with rrdtool is to have a rrd file with one or more datasource configured as COUNTER, but your counter reset and you get an ugly high peak.

There is 2 solutions to solve this

• In the “script” you use to generate the graph file you have something like DEF:A=/path/to/your/file.rrd:dsname:AVERAGE to get data, then you graph it with something like LINE1:A#…..
  Try to add CDEF:B=A,MINVALUE,GT,A,MAXVALUE,LT,A,UNKN,IF,UNKN,IF and graph B instead of A.
  Replace MINVALUE with the minimun value you want to graph (typically 0) and MAXVALUE with the maximum value you want on the graph (100000000 or something like this).
  This will suppress ugly peak on the graph.

• The second (better) solution is to change from COUNTER DS to DERIVE DS, to do so follow theses commands:
  # cp file.rrd file.rrd.bkup # this is to backup the file !
# rrdtool tune file.rrd
# rrdtool tune file.rrd --data-source-type dsname:DERIVE # this change DS dsname from COUNTER to DERIVE
# rrdtool tune file.rrd --minimum dsname:0 --maximum dsname:100000000 # Set the min and max value of data you have to graph
# rrdtool tune file.rrd
# rrdtool dump file.rrd > file.xml # Dump data to xml file
# rrdtool restore -r file.xml file.rrd.new # This restore xml data in rrd file with a check on value
# to suppress value higher than --maximum you choose