Software

Software on this page is provided in the hope that some people will find them useful. Of course, use all code at your own risk, etc.All code here is LGPLd.
Critics, bug-reports, and suggestions to be sent to: 

• RLFAPP: Value Function Approximation methods in Reinforcement Learning 0.01 - C++ code
  Code used to compare Soft State Aggregation (Singh et. al, NIPS94 paper), Kernel-Based Reinforcement Learning (Ormoneit and Sen, TR version of Machine Learning paper here) and Interpolation-based Reinforcement Learning algorithms introduced in  my ICML'04 paper written together with Bill Smart. Simulations run on the Mountain car domain, other domains should not be that hard to add, though the code is not (yet) in a shape that would make this particularly easy.. The main goal was simply to provide efficient (well, not very inefficient) implementations of the 3 algorithms to make the comparisions.The code is provided here for people interested in the implementation details or who'd like to extend the scope of experiments, etc.  
  Developers: Bill Smart and Csaba Szepesvári

• Funspec - Octave code.
  Allows inline function specifications for octave. Imagine for example that you want to find the root of 
  y = -2*x^2 + 3*x + 4*sin(x) - 6;
  Then you simply type
  [x, info] = fsolve (funspec("x","y","y=-2*x**2+3*x+4*sin(x)-4")
  What does it do? funspec defines a temporary function that can be passed to other octave functions.
  Download in tar.gz or zip format! Due to popular demand, matlab "versions" are found here: tar.gz, zip. The matlab 'version' uses 'end' uniformly instead of the more expressive 'endif','endwhile', etc. Otherwise it is the same (I could not test it, but some people did - they have also corrected the numeruous incompatibility problems!)

• Monte-Carlo package - Octave code
  The idea is to have a number of generic Monte-Carlo based sampling and integration routines that can accept functions and return the results. Note that this is an experimental version (say version 0.1) with a rather limited functionality (but it worked for me). Main functionality is provided by the following methods:
  impint - integration using importance sampling
  impint_stat - tests impint
  rejection_rnd - rejection sampling
  rejection_rnd_test - tests the previous method on two problems; some graphics.
  You will need the statistics package of octave-forge installed on your system and the above Funspec package in order to make this package work. Funspec is used only in rejection_rnd_test. The plan is to extend this package to other sampling methods. Contributions are warmly welcome!
  Download in tar.gz or zip format! Matlab versions: tar.gz, zip. The matlab version needs the statistics package.

• My (Biased) Doxygen Quick Reference Guide.
  It is only 2 pages long! You can get it in pdf format. Versions: Single page, 2 pages.

Last update: 05/28/2004