Trend / Prediction with RRDtool

December 4th, 2009 17 comments

I’ve not used RRDtool for a while and put back my attention on it few weeks ago. I found out that lots of new cool stuff are avalaible, like LSLSLOPE, LSLINT. These function return the parameters of the Least Squares Line (y = ax +b) approximating a dataset (LSLSLOPE return a, LSLINT return b).
This is interesting because with the function approximating your data you can graph a prediction of future data. Of course a Least Squares Line function will work best to approximate a dataset that tend to grow or shrink (like filesystem usage, memory usage, …) but not for data like temperature. I would say that if your data can be expressed in a percentage, an Least Squares Line can be fine. For data not tending to grow or shrink rrdtool provide some other function like TREND and PREDICT.

I will show how to use LSLSLOPE and LSLINT taking memory usage of a device as an example. My exemple will produce a graph like the following :
MemoryTrend

As you see, the graph show trend using two Least Squares Line function, one generated from the full dataset (dataset is starting 24 Oct 2009) and one generated only from last week data. Projection on time axis is done from 90% to 100% of memory usage and the date resulting of calculation for 90% and 100% of usage is displayed. I’ve seen lots of question asking how to do this but did not found any answer, so I hope that my example will provide an answer.

Here is the perl code I’m using to generate this graph. There is no Perl specific code, so it can be converted to a normal rrdtool command.

#! /usr/bin/perl
use RRDs;

$rrd_file = 'MEMORY.rrd';

RRDs::graph "MEMORY_Trend.png",
'--start', "10/24/2009",
'--end', "12/31/2009 00:00am",
'--title', "Memory Usage",
'--interlace', '--width=620', '--height=200',
"--color","ARROW#009900",
'--vertical-label', "Memory used (%)",
'--lower-limit', '0',
'--upper-limit', '100',
'--border','0',
'--rigid',

"DEF:used1=$rrd_file:used:AVERAGE",
"DEF:used2=$rrd_file:used:AVERAGE:start=10/24/2009",
"DEF:used3=$rrd_file:used:AVERAGE:start=-1w",
"DEF:used4=$rrd_file:used:AVERAGE:start=-2w",
"DEF:used5=$rrd_file:used:AVERAGE:start=-4w",
"DEF:free1=$rrd_file:free:AVERAGE",
"DEF:free2=$rrd_file:free:AVERAGE:start=10/24/2009",
"DEF:free3=$rrd_file:free:AVERAGE:start=-1w",
"DEF:free4=$rrd_file:free:AVERAGE:start=-2w",
"DEF:free5=$rrd_file:free:AVERAGE:start=-4w",

"CDEF:pused1=used1,100,*,used1,free1,+,/",
"CDEF:pused2=used2,100,*,used2,free2,+,/",
"CDEF:pused3=used3,100,*,used3,free3,+,/",
"CDEF:pused4=used4,100,*,used4,free4,+,/",
"CDEF:pused5=used5,100,*,used5,free5,+,/",

“LINE1:90″,
“AREA:5#FF000022::STACK”,
“AREA:5#FF000044::STACK”,

"COMMENT:                         Now          Min             Avg             Max\\n",
"AREA:pused1#00880077:Memory Used",
'GPRINT:pused1:LAST:%12.0lf%s',
'GPRINT:pused1:MIN:%10.0lf%s',
'GPRINT:pused1:AVERAGE:%13.0lf%s',
'GPRINT:pused1:MAX:%13.0lf%s' . "\\n",
"COMMENT: \\n",

'VDEF:D2=pused2,LSLSLOPE',
'VDEF:H2=pused2,LSLINT',
'CDEF:avg2=pused2,POP,D2,COUNT,*,H2,+',
'CDEF:abc2=avg2,90,100,LIMIT',
'VDEF:minabc2=abc2,FIRST',
'VDEF:maxabc2=abc2,LAST',

'VDEF:D3=pused3,LSLSLOPE',
'VDEF:H3=pused3,LSLINT',
'CDEF:avg3=pused3,POP,D3,COUNT,*,H3,+',
'CDEF:abc3=avg3,90,100,LIMIT',
'VDEF:minabc3=abc3,FIRST',
'VDEF:maxabc3=abc3,LAST',

"AREA:abc2#FFBB0077",
"AREA:abc3#0077FF77",
"LINE2:abc2#FFBB00",
"LINE2:abc3#0077FF",

"LINE1:avg2#FFBB00:Trend since 24 Oct 2009                      :dashes=10",
"LINE1:avg3#0077FF:Trend since 1 week\\n:dashes=10",
"GPRINT:minabc2:  Reach  90% @ %c :strftime",
"GPRINT:minabc3:  Reach  90% @ %c \\n:strftime",
"GPRINT:maxabc2:  Reach 100% @ %c :strftime",
"GPRINT:maxabc3:  Reach 100% @ %c \\n:strftime",

;

my $ERR=RRDs::error;
die "ERROR : $ERR" if $ERR;
Categories: RRDTool Tags: ,

Tiff to JPEG

December 20th, 2008 No comments
for file in `ls *.tiff`; do file2=`basename $file .tiff`;/bla/bin/tifftopnm "$file" | /bla/bin/pnmtojpeg > "$file2.jpg"; done
Categories: Unix Tags:

Per virtual user sa-learn training

December 17th, 2008 3 comments

Context

I use a LDA that use Virtual User, and store email in /some/path/mail/<domaine.tld>/<user>/, this is a quite standard way to do.
I also use spamassassin but wanted to have a per user bayes database and configuration. It’s still simple with spamc/spamd by running spamd -c --virtual-config-dir=/some/path/to/spamassassin/%d/%l ... and invoking spamc from you MTA with spamc -u ${recipient} -f -e /path/to/your/LDA so that i have user preference in /some/path/saconf/<domaine.tld>/<user>/.
Now I would like to provide 2 imap folder to users, LearnSpam and LearnHam so that they could train their bayes database.
Here the problem start, especially if you are not using one of the latest spamassassin version.

The bad way

What sa-learn command will you run to take care of LearnSpam and LearnHam folders ? sa-lean has an –username option, you may want to use that but this is not intended to be use in this case, it’s to be used when bayes database are stored in an SQL database instead of file (this is correctly documented in latest SA version). So don’t try sa-learn --username=<user>@<domaine.tld> --spam /some/path/mail/<domaine.tld>/<user>/.INBOX.LearnSpam/cur/* it will not work. Imagine how this can work ? it can’t, how sa-learn could convert <user>@<domaine.tld> to /some/path/saconf/<domaine.tld>/<user>/ ?

The good way

So the right command to use is sa-learn -p /some/path/saconf/<domaine.tld>/<user>/user_prefs --spam /some/path/mail/<domaine.tld>/<user>/.INBOX.LearnSpam/cur/* Using the -D (debug) option could be very helpfull to check if it’s work correctly, you must see dbg: bayes: tie-ing to DB file R/O /some/path/saconf/<domaine.tld>/<user>/bayes_toks

Categories: Unix Tags:

Backup file of multiple user with rsync

December 17th, 2008 No comments

root problem with rsync

Imagine that you want to backup the /home directory of server ‘A’ to server ‘B’ using rsync.

There is two way to do this :

  • You can run rsync on the server ‘A’, but if you want to correctly backup (I mean, having correct uid/gid/.. on backuped files) files you should connect to the server ‘B’ as root. I’m sure you don’t want to do that.
  • You can run rsync on the server ‘B’, but you should connect to ‘A’ with an user that can read all file in /home. This could be complicated depending of your gid managment.

When Tar start to be your best friend

So how can you do ? the solution would be to store (uid/gid/permission/..) information in a dedicated file, so that you can apply them if you need to restore data.
How can you do that ? I’m sure you are too lazy to write a shell/perl/python/.. script to do that. You’re right ! Use tar.
What ? What ? You want me to tar /home and rsync it ? Are you mad ? I don’t use rsync to transfer 20Go at each backup.

When 1 option and 2 lines can save you

Tar as an incremental option. This mean that you can make a 1st tar file with /home then you can do a 2nd tar file with only modified file since previous tar. This option is -g.
Here is a 2 lines shell script to do the job

gtar -g /var/backup/home/home-backup.snar -cpvzf /var/backup/home/home-backup.`/bin/date +%s`.tgz /home/
rsync --delay-updates -avz -e ssh /var/backup/home backupuser@'B':/var/backup/

–delay-updates is very important because if you don’t use it if ‘A’ crash when rsync is copying the .snar file (used to store incrementation information) you will miss it on ‘B’ and can’t retore tar file correctly.
-g only exist in GNU Tar. You may have to install it if you’re running *BSD. First check if you have a gtar binnary

Categories: Backup, Unix Tags: , ,

Misconfigured Perl install path

December 17th, 2008 No comments

I got a strange problem on a server. CPAN was installing module outside of perl @INC, this is quite a nightmare.
I found what was wrong and it was quite easy to solve.
Run this command:

# perl -V:'install.*'

Look at installprivlib and installarchlib. If they don’t match @INC you can change them at the end of perl Config.pm (in tie %Config, ‘Config’, {..})

To find your Config.pm file type

# perl -MConfig -le 'print $INC{"Config.pm"};'

If you only use CPAN to install module I guess you can just change PREFIX in the CPAN Config.pm (but I haven’t tested this)

N.B.: If you can’t find installprivlib and installarchlib in tie %Config just add them.

Categories: Perl Tags: ,

Getting human reading time info from RRD files

December 17th, 2008 No comments

You have a .rrd file but you don’t remember how much time RRAs can store data ? all right, no problem.

First save this Perl code into a .pl file.

$step = $1 if (m/step = (\d+)/);
$rows = $1 if (m/rra.*\.rows = (\d+)/);
if (m/(.*)\.pdp_per_row = (\d+)/) {
$pdp = $2; $time = $step*$rows*$pdp;
if ($time > 31536000) { $time = sprintf("%.2f year",$time/31536000) }
elsif ($time > 86400) { $time = sprintf("%.2f days",$time/86400) }
elsif ($time > 3600) { $time = sprintf("%.2f hours",$time/3600) }
print "$1: $step*$rows*$pdp = $time\n";
}

Assuming the filename of the script is rrd_info.pl, run this Shell command:

rrdtool info file.rrd | perl -n rrd_info.pl

Here is an output example:

rra[0]: 300*864*1 = 3.00 days
rra[1]: 300*864*5 = 15.00 days
rra[2]: 300*702*25 = 60.94 days
rra[3]: 300*840*125 = 364.58 days
rra[4]: 300*840*625 = 4.99 year
rra[5]: 300*864*1 = 3.00 days
rra[6]: 300*864*5 = 15.00 days
rra[7]: 300*702*25 = 60.94 days
rra[8]: 300*840*125 = 364.58 days
rra[9]: 300*840*625 = 4.99 year
rra[10]: 300*864*1 = 3.00 days
rra[11]: 300*864*5 = 15.00 days
rra[12]: 300*702*25 = 60.94 days
rra[13]: 300*840*125 = 364.58 days
rra[14]: 300*840*625 = 4.99 year

The formula is update step time * numbers of rows in the rra * pdp_per_row

Categories: Perl, RRDTool Tags: , ,

Booting NetBSD on Mac (iMac DV+)

December 17th, 2008 No comments

This is what to use to boot NetBSD MacPPC CD on an iMac DV+ (at least):

boot cd:,ofwboot.xcf netbsd.macppc

or

boot cd:0,ofwboot.xcf netbsd.macppc

You may not need to put netbsd.macppc depending of the CD


To boot on HD try this:

boot cd:,ofwboot.xcf hd:/nebtsd

or

boot cd:,ofwboot.xcf hd:13/nebtsd

Where 13 is the number of the netbsd partition (usualy near 13)

Categories: NetBSD Tags: ,

PPP Over SSH

December 17th, 2008 No comments

You may want to establish a full IP connection to a remote host (or remote lan) but you may not have any VPN software on the remote host, or on your host.
There is a solution using SSH and PPP with the command

pppd pty 'ssh -x -t -e none user@server /usr/sbin/pppd passive noauth 9600' noauth 10.0.0.1:10.0.0.2

You have to use key authentication because the tty is redirected to pppd so you can’t be prompted for a password.
With this command, you can reach server at IP 10.0.0.1.
By playing with pppd and route tables you can extend the IP tunnel to the entire remote LAN.

This has been tested between Mac OS X and NetBSD but should work with any system.
It works but it’s very slow.

On some system you may have TTY problem, if it’s your case take a look to http://shinythings.com/pty-redir/ or http://www.ishiboo.com/~nirva/Projects/vpn/

Categories: Unix Tags: ,

Best hints for Mac OS X 10.3 (Panther)

December 17th, 2008 No comments
Categories: Mac OS X Tags: ,

Get HTTP arguments when using PHP as CGI

December 17th, 2008 No comments

Normaly when you use PHP as CGI, $_GET and $_POST should work as usual.
But in certain configuration, they may not be set. In this case this code is the base to get args

foreach (split('&',$QUERY_STRING) as $tmp) {
$tmp2 = split('=',$tmp);
$$tmp2[0]=$tmp2[1];
}

Categories: Uncategorized Tags: ,