wiki:Developer/ModuleSystem/Tutorial

Module creation tutorial

Concepts

Modules in the Bisque system are nothing more than URLs that adhere to the Bisque Module API. This means that they need to provide some entry points in that URL and be able to talk back and forth Bisque documents. Read about this in detail in the module system description. In summary a module for Bisque system is a URL like this:

http://bisque.ece.ucsb.edu/module_service/Botanicam

This URL must also have additional entry points like:

http://bisque.ece.ucsb.edu/module_service/Botanicam/definition
http://bisque.ece.ucsb.edu/module_service/Botanicam/execute
...

The '/definition' should be returning module definition XML. The '/execute' should be able to receive MEX document and start the actual execution of the provided code with parameters received in the MEX.

This means that Bisque can talk to many modules that may exist on many separate machines and even better they can use some special local hardware required by that module.

Most of this is already done for developers by a Bisque package Engine Service. This package can wrap local files and give them URLs and provide all necessary entry points. Bisque can talk to many Engines physically located on many different machines. Engine Service is rather smart and allows running code locally using a thread pool or can submit jobs to Condor job management system. In the case of using Condor, machine where Engine Service is installed must also be a Condor submit node.

Once code is executed it needs to be able to fetch data from Bisque and ultimately post some results back. These can be done either by using Engine Service functionality to fetch data locally and pass it to executables on the command line or the executables themselves should be able to fetch data as needed using Bisque APIs. Some glue code might be needed if using local fetching in order to convert data from Bisque format into a format desired by the executable.

Once the module is running and serving necessary entry points it can be added to Bisque to be available for users. Currently this is done by the system's admin using the provided module URL. We are adding functionality for user to share their modules automatically. (More about this later)

Steps

Now that we understand how it all works let's enumerate steps to get a working module.

  1. Create module definition XML
  2. Modify module code to run with Bisque, writing a wrapper or conversion glue code
  3. Create a web server or run Engine Server
    1. Engine Service - write configuration file on how Engine Service will run your code
  4. Register module with the Bisque system

Personally I feel it's always easier to get an existing module and modify its contents to fit my needs, here are some of the examples:

All of the files required for these to work will be describe in the following.

Creating module

If you are using Engine Service then all you need to do is to create a directory with the module's name in the "modules" directory in the Engine Service. You will see other modules that are submitted to the Bisque repository.

Let's call our module MyModule and start by creating MyModule directory in modules. The next step is to populate that directory with module stuff like actual code, module description XML and other needed files.

Describing module

The first step is to describe your module to the Bisque system. You can do that by creating an XML file with the same name as the module itself in the module's directory. In our case that would be MyModule.xml. The module definition file must describe the inputs the module will take, its outputs, ways to run it, its interface and other things like description and help.

You can find a detailed definition of every definition element in the module definition XML.

For now let's simply copy an existing definition and modify it. source:bisque/modules/Botanicam/Botanicam.xml

The easiest way is to describe required inputs and outputs and let Bisque create its UI automatically. Though it is possible to modify automatic UI or even completely replace it with custom UI. This would require understanding of web programming, html5, css, javascript and possibly ExtJS.

Creating code

Now that we have described our module we need to make code that would run and take Bisque data as input. Here we have many ways of doing that, we'll look at !Matlab, command line and python.

Matlab

It's rather easy to make Matlab code available to Bisque. I suggest using Bisque Matlab API and write a simple matlab function that would wrap your code and get and send data to/from Bisque. Take a look at the example of the NuclearDetector3D module: source:bisque/modules/NuclearDetector3D/NuclearDetector3D.m

Here image pixels in the Matlab ND matrix form are fetched in the line:

imm = image.slice([],current_t).remap(membrane_channel).fetch();

and then the actual function is run:

np{current_t} = BONuclearDetector3D(imn, ns(1:3), t, session, timetext);   

and later outputs are send back to Bisque in lines marked by "%% Store results".

Bisque Matlab API

To understand the Bisque Matlab API a bit better you can look at the examples provided within the API as well as its source: source:bisque/tools/BQMatlab

Here you will find test_*.m files that test and demonstrate various uses of the API, and the +bq contains the API's source code.

The API itself is written in Matlab using inline Java and should be easy to understand. This way it's easy for developers to add functionality and/or fix behavior directly in Matlab without recompiling anything.

A few more examples

PHFDetector uses points produced by the NuclearDetector3D and posts back classified points: source:bisque/modules/PHFDetector/PHFDetector.m

BotanicamTrainer takes query as input, trains a classifier model and posts model file back to Bisque: source:bisque/modules/BotanicamTrainer/BotanicamTrainer.m

Botanicam uses latest model produced by BotanicamTrainer to recognize a new incoming image: source:bisquik/browser/bisque/modules/Botanicam

ImageMatting takes graphical strokes as input, segments an image and posts back final segmentation: source:bisque/modules/ImageMatting/ImageMatting.m

Command line

Here we'll look at the example produced with compiled Matlab code that is run as a stand alone executable. RootTip module is doing that in the python code: source:bisque/modules/RootTip/RootTip.py

This python module will be called 3 times during its asynchronous execution with command line parameters as: setup, start, teardown. During setup the image is downloaded locally. During start the command line executable is called and during teardown results are posted back to Bisque.

Python

This example is a complete module reading data, performing its work and posting in one python file: source:bisque/modules/MetaData/MetaData.py

Describing execution

Ok, we just looked at 3 different codes, now we just need to describe how to run them. For that Engine Service is using two files: "runtime-module.cfg" and "setup.py".

setup.py

setup.py is needed to make the code executable, for example compilation steps for either c or matlab. Also some dependency checking can come in right here like checking the existence of some required libraries or executables. For example NuclearDetector3D will compile matlab script: source:bisque/modules/NuclearDetector3D/setup.py

RootTip will prepare python and compile matlab script: source:bisque/modules/RootTip/setup.py. Botanicam will ensure libsvm is available: source:bisque/modules/Botanicam/setup.py

runtime-module.cfg

This config file describes files that constitute the module's code and the way to run them. For example looking at the source:bisque/modules/RootTip/runtime-module.cfg you will see that the module consists of RootTip, pydist, araGT, araGT.ctf files. Here RootTip is actually a python file packaged by the bbfreeze and araGT is the compiled matlab code.

In the case of NuclearDetector3D the configuration is a bit different: source:bisque/modules/NuclearDetector3D/runtime-module.cfg defining:

files
files = NuclearDetector3D, NuclearDetector3D.ctf, +bq, java.opts

where !NuclearDetector3D and !NuclearDetector3D.ctf are produced by the matlab compiler and java.opts is needed to !Matlab !JVM running parameters.

runtime.platforms

Here you can also define platforms that the module prefers and can run on. Currently we support two: condor and command. The order defines preference.

runtime.platforms=condor, command

condor would allow this module to be submitted as a condor job whereas command instructs Engine Service to execute the module locally using thread pool. For example MetaData module only uses command.

environments

Environments allows an advanced definition of all variables required to run your code, for example, "Matlab" will ensure all path and environment variables required by Matlab. Staged indicates Bisque to copy all the module files into a staging directory prior to execution, usually because a module will be producing some local files either for temporary purposes or for upload.

environments=Staged,Matlab

Currently we have the following environments:

  • Staged: Creates a temporary directory where all executable files would be placed for running (in condor it would be a shared directory) and storing temporary files
  • Matlab: Sets necessary environment variables to run Matlab or compiled matlab scripts
  • MatlabDebug: Sets necessary environment variables to run Matlab and runs interpreter with the desired function

Registering

Once all is done we need to restart the engine service that will be serving your module. On most simple test installs your engine service is running on the same machine with the main Bisque install but on a different port, the default engine port is configured on port 27000. You can find that in the site.cfg config file under e1.url (more about site.cfg here). Use the standard while in the virtual environment within your server:

bq-admin server stop
bq-admin server start

Once restarted, your new module should appear in the engine service listing:

http://localhost:27000/engine_service/

Notice that in default installs engine is only listening on localhost. If you see your module in the listing it was successfully picked up by the engine and is available for registration with your Bisque site. Remember that the default configuration will only listen on localhost and will only be able to be registered on locally run Bisque servers. In order to expose your engine for a remote Bisque server you need to modify the host configuration(more about site.cfg here).

At this point we only need to register the module with the Bisque system. Currently it is done using the admin interface on the Bisque server:

bq-admin module -u username:password register http://localhost:27000/engine_service/Botanicam

or if you want to make your module public:

bq-admin module -u username:password -p register http://localhost:27000/engine_service/Botanicam

Accessing your module

Now you should be able to access your module at the main URL of your Bisque install:

http://BISQUE_URL/module_service/MyModule/
Last modified 6 years ago Last modified on 08/11/2013 11:12:06 AM