wiki:Developer

Developer Resources

Bisque is an image database and analysis system specifically for working with 5D biologically related scientific imaging.

A fundamental goal of the Bisque system is to provide an extensible platform for users and labs to create and integrate novel image analysis methods. We review the tools made available for developers to create and integrate new analysis methods. As mentioned before, there are several styles of interaction available from external applications to fully integrated web-based mini-apps. In all cases, the modules interact with the core modules using similar methods and a common API.

All access is through web protocols such as HTTP. Any programming environment that can make an HTTP request can reach all Bisque services. Data is exchanged using XML, which can be easily generated and parsed. The Bisque development suite provides convenience libraries for several programming environments that hide the web access and XML details from the programmer. This allows the programmer to ignore the particulars of the transport mechanisms and provides an environment similar to common single machine programming.

The development suite also contains libraries for creating external analysis and visualization tools that can interact with any Bisque system. Toolkits are available for popular image processing environments such as ImageJ (Java), Matlab, and C++. These toolkits provide commonly used user interface features such as finding and selecting images, image and annotation viewer, and tag editing.

Internally it is built a collection of web-accessible resources and services.

Bisquik Components

See our paper for an overview of the architecture.

The core BisquikWebServices are as follows:

  1. DataServer : Flexible schema database allows the user or module add/query meta-data freely.
  2. ImageServer : Store, manage and process 5D (x,y,z, time, channels) images.
  3. ClientServer : Handles user request and presents results in a standard web browser.
  4. BlobServer : Store blobs (binaries) and utilize storage services.
  5. Export Service: Provides facilities for exporting datasets, images, files and related metadata as compressed archives.
  6. Import Service: Ingests images and metadata from external locations
    1. Multi-file-multi-image: details of support for complex image arrangements
  7. Statistics Service: Summarizes tags and gobjects found in specified resources
  8. Usage Statistics Service: Creates statistics of the system usage: uploads, module runs,
  9. Feature Service: Compute numerical feature descriptors for images, gobjects and other resources
    1. HTTP API description
    2. Python API description
    3. Matlab API description

Proposed servers

  1. GOgjects indexing server: find images with objects of a given type or name, find all objects in specific location of a given image

Module System

  1. Module System - Overview, Module definitions, Module UIs, Cluster executions
    1. ModuleServer: Manager of module interactions
    2. EngineServer: Create web-accessible modules out of your own analysis
    3. Specification for module definition
    4. Specification for the MEX request
    5. Configuration of run-time modules
    6. Tutorial
    7. Troubleshooting
    8. Proposals

Analysis modules

Bisquik UI

  1. BrowserInterface?
  2. ImageViewer
  3. ResourceTagger?

Bisquik Extensions

  1. Client Side Interface Libraries(Java, matlab, pyton, ...)
  2. Service Extensions Adding a new service (indexing, generalized services)
  3. Toolkits for external tools

Metadata Storage

Datasets

Image Manipulation

Images are manipulated using the Image Service. The full list of available operation are here

Analysis

Development workflow

Configuration and Deployment

Bisque takes advantage of distributed computer resources by placing various services on separate ports or even on separate servers. For example, a typical system might have a single machine for database and another for web-services while reserving 1 - 128 machines for (image) analysis.

The only requirement is that each service node be able to make and receive web-style requests over an some port. Each service is written in Python/C, database services can be sqlite, mysql, or postgres

The system can be configured to run a variety of machine configurations. A few examples are listed below:

  • SingleConfig? : Web, Database, Analysis on a single machine.
  • ClusterConfig?: Web on public network, Database and Analysis workers on private internal set of nodes .
  • CloudConfig? : Similar to cluster config, with services distributed on sets of virtual machines.

Coding

Binary files as part of distribution

Binary files are distributed from our depot:

http://biodev.ece.ucsb.edu/binaries/depot/

To upload a file use the following script while in the Bisque environment:

bqdev-upload-binary -u tracusername -p tracuserpasswd path/to/binary.file

Each version is given a specific hash and can be referred exactly in the binary distribution file: source:bisque/config/EXTERNAL_FILES

Testing

Developers can test their work before checking in using NoseTests. A typical test run is accomplished (will be run from bqcore)

nosetests -s --logging-filter=bq.test -v [ name of test]

Writing Tests

There are many example tests in bqcore/bq/core/tests and bqcore/bq/data_service/tests. Please check these

as the framwork provides many setup/teardown functions.

Resources:

Some point to remember:

  • Be careful with state 1. Tests start with an empty database and it must be created
  • Be careful with state 2. Just because your test runs alone, it may not run in group (bad cleanup)
  • Tests are ordered Alphabetically in a class/module: try names like test_a_new, test_b_delete
  • A test must have a doc string. Otherwise nothing is noted on the error report.
  • Useful option:
    • -s : Capture and show standard output
    • -w <dir>: Look for tests only in <dir>

Example Test:

import logging
from bq.core.tests import TestController
log = logging.getLogger('bq.test.doc_resource')
xml1 = '<image x="1" y="2"><tag name="foo" value="bar"/></image>'
DS = "/data_service/doc"
class TestDocController(TestController):
    application_under_test = 'main'  # section to loaded from test.ini
    def test_a_new(self):
        "new --> create a new document"
        response = self.app.post (DS, params=xml1, content_type="text/xml")
        assert response.status == '200 OK'
        assert 'image' in response.lxml.get('uri')

Benchmarking

  • install repoze.profile and configure
    easy_install repoze.profile
    
    Add the following to you middleware.py
    app = AccumulatingProfileMiddleware(app,
            log_filename = base_config.get('web.profile.log', 'profile.prof'),
            discard_first_request=asbool(base_config.get('web.profile.discard', 'true')),
            flush_at_shutdown = asbool(base_config.get('web.profile.flush', 'true')),
            path = base_config.get('web.profile.path', '/__profile__')
            ) 
        
    
  • Use Apache benchmarking tools
     ab -n100 -c4 http://localhost:9090/data_service
    
  • Access profiling data at http://localhost:9090/__profile_

Last modified 3 years ago Last modified on Nov 14, 2014 10:07:48 AM