I'm on my way back from the European Common Lisp Meeting in Hamburg. Edi Weitz and Arthur Lemmens put together an incredible conference. Everything was well organized, the location at the Gastwerk hotel was a beautiful setting, the food was delicious and the speakers gave some amazing talks. I've recorded all talks on DAT and will make them available for download as soon as the speakers have reviewed the recordings of their talks. I've put a couple of images up at Flickr.com.

1. 9:30 Jans Aasman: AllegroGraph

• The semantic web
• Intelligent search depends on meta data
• Why we need a triple database
• XML is actually not relevant, but reasoning is
• Missing constraints for IDF documents and contained definitions: OWL
• An IDF reasoner can reason about IDF documents
• The intelligence community was first do demand reasoning tools
• Other customers: telecom companies, biotech
• ISP (configuration mgt): 2 billion triples
• Telecom datamining over VOIP CDRs: Is there a circle of friends? -> almost impossible to do with relational databases!
• Movie database: Kevin Baken number, relationship mining
• First Franz product using Lisp inside a black box with a http and Java interface (no interface for other Lisps?)
• Reification:
  • put a trust metric as meta data on the graph
  • meta data about triples
• burst rate 14000 triples per second (saving and indexing)
• treat AllegroGraph as a Prolog database
• for governments it's important to say things about triples (trust metric)
• you can use this meta information inside of queries
Questions:
• How do you add new/correcting information to the triple
• fuzzy queries? Yes, every triple comes with a trust value and you can use a reasoning engine to do queries.

2. 11:12 James Anderson: Time series analysis for financial markets

• First get prediction with internal market data working
• You cannot beat the market with internal data only
• mix in external data (news headlines)
• timestamp the data on the feeds in multiple time dimensions so you can go back in time an analyze again
• 250MB per day input feed (one minute resolution) = 5 megapixel snapshot of the market
• evaluating several million ticks per day
• solve problems with differences in timing/timestamping between different servers
• develop the aibility to go back in time an clean up data (in case of interruption in the feed)
• Now predicting "market sentiment" not "nearest neighbour"
• used for ESF trading (future of Dow Jones), traded in Chicago 24h/day
• you need to do synchronous analysis to get rid of the discrepancies btw. servers
• impedance: code complexity issues even with Lisp
• Create a DSL with special operators to deal with the problems
• do data introspection at a higher level
• create a "generic function graph"
• give the analysts a language to express the way they think about the problem set
• visualize data in 3D using an abstraction layer that is device independant
• our target marktet is day-traders

3. 12:00 Arthur Lemmens: Rucksack

• persistence library for CL, because no one else was doing it
• deal with serializing shared objects
• write your own serializer to speed up serialization (orders of magnitude faster)
• use the MOP to check when a slot is referenced/accessed
• use a proxy for referenced objects that have not yet been accessed (memory forwarding pointer)
• cache: put loaded objects into a hash table, put changed objects into a separate hash table
• manual memory managment is a sin and it isn't even a pleasant one
• there should not be a delete instance in Rucksack= no dangling pointers!
• build a disk based garbage collector for the store
• copying collector for structured data
• mark and sweep collector more efficient for big blobs
• use a commit file to recover from crashes
• simple query language
• support schema evolution: class definitions are immutable, keep a list of versions of class definitions
• all schemas are kept in the schema table and the table is kept on disk
• about 2MM effort implementing it

7. 14:30 Jim Newton, Thomas F. Burdick, Peter Herth, Björn Lindberg (Cadence): Electronic Design Automation

Jim Newton:

• 5 billion per year industry
• 25% market share by Cadence
• physical design kit at the core
• Lisp library "Skill"
• Can't keep Cadence IP and customer IP in same repository for revision control (because of IP law)
• develop a "Meta-Version Control" to separate the code
• PCMan database is an implementation of this

Björn Lindberg: Database Design

• client server model of the PCman system (pronounced Pacman)
• create a CLOS wrapper for the database scheme

Thomas F. Burdick: Server

• assume that the clients are hostile
• only the server knows about the database
• clients only know about objects
• we have a first class transaction method
• first committer wins and all other clients are forced to update to the latest version of an object

Peter Herth: Database Editor

• use Ltk to connect Lisp to Tk GUI
• use the Ltk remote extension to start the Gui remotely
• Generic Browsers

Questions:

• How do you pick the modules to be used for a release? A release engineer decides which modules go into a release.
• Show the macro expansion of the database stuff...

3. 15:28 Martin Cracauer (ITA): QPX - Low fare search engine

• the airline industry separates the prices from travel information
• the booking problem is complicated by the number of prices created by the airline companies
• there's a combinatoric explosion of possible intiniaries
• Boston-Hamburg -> search 2.5 mio prices
• 10^28 solutions is not unusual!
• "In a way we have the worlds most sophisticated compression program for this data"
• "We need the raw speed from Lisp..."
• unecessary initialization of data structures is a problem with dynamic languages (and Lisp)
• we need bitfields
• 5 GigaBytes of maximally compressed flight data
• efficient datastructures are key
• we need to mmap anyway (because of the size of the data structure)
• share the data between instances of the search processes (one instance per processor)
• we need to read the 32bit of a C-struct directly from Lisp without a function call in between
• manual control of inlining is a very important feature for us (for 32 bit integer operations)
• one of the big benefits of Lisp to us is, that you can compile with different presets for speed and safety
• regression tests use the range check in safe compiled test versions to catch overflows
• we always leave array bounds checking on (because it is very hard to debug problems with array overruns) even when compiled in fast mode

Questions:

• Why still use Lisp? Because we have very complicated algorithms in the actual search engine. We use macros (several million lines of code after macro expansion)
• how do you deal with negotiated fares? We use dedicated machines for customers with negotiated fares.
• How do you deal with availability checks?

4. 16:34 Klaus Harbo (Mu): cl-muproc

• Algorithmic trading platforms
• arbitrage in online sports betting markets
• MP is here to stay
• implement the important aspects of Erlang in Lisp
• muprocs exchange mumsg messages
• everything goes on in a single Lisp instance
• clients block on reads until they get a timeout
• we can generate all the message passing and the client only has to call a function
• supervisor for server processes enforce the policy (like automatic restarting services)
• 2500 lines of code
• BSD licensed

6.David McClain (Refined Audiometric Laboratory): Signal Processing SigLab

• Lisp-based numerical signal processing
• connect blocks using virtual wires
• we can send functional closures down the wire
• run simulations of signal processing
• model systems in Lisp
• Why Lisp?
• Exceedingly powerful macrology!
• most elegant OOP: CLOS
• dynamically typed
• ad-hoc design of new networks
• performance
• using Lisp augmentation (C-array substrate) to speed up processing
• hand coded high performance vectorized math routines
• aspect oriented programming extensions
• I've reverse engineered the BBE-Processing filter (~40 lines of code)
• discovered the EarSpring Equation which describes normal human hearing
• might make it available for people who are interested