Pluto

or how to make Perl juggle with billions

Niclas Lundborg, Erwan Lemonnier

lead developers

Premiepensionsmyndigheten (PPM)

$Revision: 1.7 $

Introducing Pluto

• The pension crisis
• A solution: let users choose themselves in which funds to save
• Users take risk but achieve higher profits
• Premiepensionmyndigheten (PPM)

Introducing Pluto

• Pluto:
• PPM's financial platform (its brain!)
• holds one saving account per person
• performs all fund transactions (buy/sell/switch/dividend/merge/split...)
• implements the mathematical insurance model
• manages pension payments
• and more!

Pluto's story

• A large american computer consultant tried and failed
• Pluto was the backup system
• Started small: first task
• Addons later
• First money in the system (Summer 2000)
• First fund selection (Autumn 2000)
• First payment (Mid 2001)

Challenges

• Huge volumes of data
• No bugs allowed
• one bug triggered once for every 1 million persons means at least 5 bugs a day! (Probably more)
• High availability: running 24/7
• Tight development deadlines
• Politicians!
• User requirements: fast transactions

Pluto in numbers (2006-10)

• Loads of code:
• 320 000 lines of Perl code (.pl, .pm, .t)
• 68 000 lines of SQL script
• 27 000 lines of shell script
• 26 000 lines of HTML (Mason)
• 230 database tables
• Largest table has 500 000 000 entries
• 750 gigabytes of data in an Oracle database
• Managing 250 000 000 000 SEK (23 billion euros)
• 5.5 million users
• Up to 30 developers over 7 years

Pluto's architecture

Pluto's architecture (1/3)

The Programs

• A batch system
• Many small programs doing simple things (More than 100 individual programs)
• Perl programs wrapped in shell scripts
• Simple interfaces between them:
• Inbox model
• Database tables
• A daily run schedule

Pluto's architecture (2/3)

The Database

• A huge Oracle database
• Always add, never remove
• The database schema evolves in small steps (migrations)
• Keep a history log of every change
• Each program running gets a unique ID, to which every change refers
• Every program logs its actions to the database
• Every database change is made by a program
• To update a row, add a new one with a higher sequence ID

Pluto's architecture (3/3)

• Design decisions:
• Perl
• Originally: functional design, no object orientation
• No multithreading
• No use of floats or other number formats with decimals!
• Beside batch programs, we have:
• a Pluto server for fetching user data in realtime
• used by: the telephone interface (CTI), the customer support staff, statistic engines, etc...
• a couple of administrator web interfaces to various parts of the system

How was it built?

The Beginning

• One strong leader
• Lots of consultant developers
• Focus on a few areas at the time
• Do only what we need right now, the rest we take another day
• When we had spare time, we guessed what new demands there would be and acted accordingly

Code Standard

• C / Pascal like code!
• No fancy Perl coding!
• No smart programming!
• Comment where needed!
• KISS (Keep It Simple Stupid)
• Use standard SQL-commands
• Use placeholders whenever possible!

Paranoid Coding

• Defensive programming: trust no one
• Assert the impossible
• Crash, don't trash
• Log everything
• Contract programming

Peering

• All code changes are reviewed and tested by another developer
• Check that:
• the bug is solved
• the code does not do anything illegal
• tests pass
• code standard is followed

Pluto a few years ago

• A stable production system
• BUT!
• A procedural system (no object orientation)
• Lots of cargo code
• Lots of hidden dependencies
• Lack of automated regression tests

Testing

Component tests / unit tests (1/4)

• test-driven development (kind of)
• using Perl test files (*.t) and the TAP protocol
• fast growth:
• as of 2004-04: less than 10 test files, 0 unit tests
• as of 2006-10 (after 1.5 years): 260 test files, 5000 unit tests
• as of 2007-04 (6 months later): 350 test files, 10000 unit tests
• tests are automated
• all tests are run every night against each cvs branch

Component tests / unit tests (2/4)

• Different sorts of automated tests:
• test a sub
• test a program/batch against a specific database
• test a server
• test sequences of batches
• We use code coverage
• We want continuous integration (coming soon)

Test tools (3/4)

• Most tests involve a database filled with proper data
• We have a tool to extract all data related to a person from one database and dump it into a file...
• ...and a tool to inject the data from this file into an other database
• Our test sequences do that all the time
• Useful to reproduce bugs from other databases in our test databases
• A tool to record and rerun sequences of batches and simulate the daily run schedule
• Various simulators, simulating systems to which Pluto is connected

More testing

• Once the code is released, the test unit takes over:
• 15 persons
• plenty of test environments
• lists of requirements to validate
• databases filled with well chosen test data
• Black box testing at various levels:
• application test
• integration test
• acceptance test
• pre-prod test

Performance

• Code profilers
• Cache profiler
• We had a SQL/DBI profiler when none existed for Perl
• All executing programs are timed
• Multiprocessing (fork) at critical places
• Optimised database (indexes, smart SQL queries)

Pluto today

• Pluto is undergoing massive refactoring:
• *Modulizing* the system
• Hunt down redundancy
• Gather code into Perl modules (classes)
• Use CPAN
• Write unit tests for all new code
• Follow code standard
• Use object orientation where it fits
• Try new programming techniques...

The bleeding edge

What we are experimenting with...

Contract Programming

• Perl is a weakly typed language
• We compensate that by using contracts
• ie dynamically compiled subroutine wrapers that validate input arguments and results
• This is in fact much more accurate than strong typing:
• You get type controls AND code assertion, all in one

Intelligent debug messages

• Code is full with debug output, off by default
• Debug output is added to the code during early development, and stays there.
• We define a way to selectively turn on debug output in specific parts of the code, or based on specific conditions
• Like an intelligent filter on top of debug logging
• See Hook::Filter on CPAN

Open Sourcing Code

• Myndighet = code is public
• Part of our code base is generic enough to be used by others
• We put it on sourceforge/CPAN
• It gets tested by others
• CPAN testers
• Improvements are suggested
• The code gets better

Various

• SCRUM
• Continuous Integration
• Ambient Orb!!!
• coming soon...

Perl and Pluto

Issues about using a dynamic language?

Advantages of Perl

• High level language
• Regular expressions
• Extremly fast development timeline
• Integrates well with UNIX/Linux and Oracle
• Clean code (with code standard!)
• Advanced language features (closures, runtime hacks...)
• CPAN

Drawbacks of Perl

• Ugly syntax
• versus code standard
• Weakly typed
• versus contracts
• Sometimes slow
• but not the slowest link (database)
• Integrates weakly with Java
• No serious obstacles...

The End

• Yeah... we have a cool job
• Quality comes first: a dream situation for a developer
• Active development, even after 7 years
• And with the incoming restructuration of the Swedish pension system, the fun will go on!
• Looking for (ex)job?

Questions?

Links

Henrik Sandell's presentation of Pluto, shown at the Nordic Perl Workshop 2003:

http://www.perlworkshop.dk/2003/presentations/henrik_sandell/pluto/PerlMongerPluto.htm