[mdlug] OpenGroupware Coils [Was: my own cloud]

[Adam Tauno Williams]

On Sat, 2011-07-02 at 07:31 -0400, Carl T. Miller wrote:
> Adam Tauno Williams wrote:
> > In the vein of shameless self-promotion - the project I work on aims to
> > be a full-function WebDAV/CalDAV/GroupDAV server [as well as integrating
> > workflow].
> > <https://sourceforge.net/projects/coils/>
> Adam, that's more like teasing than shameless self-promotion.
> Tell us a bit more about this project.  I looked at the page
> and now I have more questions than answers.
> For example, is this a plugin for OpenGroupware, or more of
> a separate service that integrates with OG?  Do you need OG
> to use it?  If not, what clients/services/packages would be
> able to use coils?

Since you asked...

OpenGroupware Coils is a ground-up reimplementation of the OpenGroupware
project's Logic and ZideStore bundles;  it can exist in parallel with an
OpenGroupware "Legacy" installation or stand-alone. It is also a fusion
of OpenGroupware's functionality and another moribund project named
"BIER" [Business Integration Engine] which is/was a BPML based workflow
engine with very strong ETL capabilities.   

Using OpenGroupware Coils standalone is currently a bit awkward since
Legacy provides a fair amount of basic functionality like managing users
and teams; but I'm in the process of committing the last bits to have
user and team management in the Coils logic.  But other than those
issues it works stand-alone just fine.  It was designed to work in
parallel so sites could easily transition from Legacy to Coils with
little [and often no] modification to local / custom applications.

External dependencies are:

   AMQP message broker - this is typically RabbitMQ.  Coils is
implemented as a set of 
   "components" which communicate over an AMQ message bus;  this allows
components
   to be spread over separate hosts or individually restarted [or auto
restart is performed if
   a component fails.  Developing as discrete components also simplifies
development and
   testing.  RabbitMQ is simple to deploy and manage.  I'd recommend
anyone doing any 
   non-trivial development to look into RabbitMQ / AMQP for how to make
building networked
   applications *much* easier.

   PostgreSQL RDBMS - Coils uses an extended version of the same schema
as 
   OpenGroupware Legacy.  PostgreSQL is fast, scalable, and has lots of
related tools.

   An SMTP server, local or remote.  - For obvious reasons.

Optional dependencies:

  Authentication can be performed against an LDAP Dit (OpenLDAP, Active
Directory, etc...)
 
  OpenGroupware Coils' workflow engine can utilize external databases;
currently
  PostgreSQL and Informix [via informixdb] are supporting.  Adding
additional databases
  is very trivial.

Features:

  * WebDAV server - This works in Windows XP, Vista, V as well as via
GNOME's VFS [nautillus]. We are trying hard to be a full-function WebDAV
server rather than one of the many bare-bones WebDAV implementations
that exist [and once you try to use them the limitations become
obvious].  This includes LOCK/UNLOCK support and LOCK-NULL resources [so
applications like Microsoft Office work as expected].  For example you
can "net use" the Coils WebDAV server mounted as a drive-letter.   This
isn't complete but the implementation is very close and most things
just-work.  We really need more testing.  To facilitate access to PDF
images of invoices, etc... this implementation is used in-production
every day.
  * CalDAV server - Testing is primarily performed against GNOME's
Evolution [which is easily the most complete CalDAV client I'm aware
of].  I've also tested with Mozilla's Sunbird/Lightning.  This is still
being debugged - primarily time-zone parsing, just because I broke it
recently.  Read access works reliably.  Using CalDAV you can manager
calender, journal/memo, and todo items.   
    - Sadly Evolution is the only client I'm aware of that supports
CalDAV journals [memos] but they are extremely handy;  otherwise
journal/memos are available via WebDAV for editing/viewing with any text
editor.  
    - Management of tasks via CalDAV works very well; one feature I'm
particularly proud of is support for attachments - files can be
attached/removed to/from tasks.  At least for me this is a very useful
way to store documents  [in OpenGroupware documents can also be 'linked'
to pretty much any object;  but CalDAV clients are oblivious to such
things].  You can also view workflow processes as a calendar.  This
allows you to easily visuallize the files that were uploaded and
processed by the workflow engine the previous evening, etc... The
calendar event start and end time is that start and completion time of
the workflow and the comment contains the process log indicating the
actions that were performed and the duration of each atction, etc...
  * On top of the WebDAV implementation is a REST API allowing access
and navigation using any HTTP capable client or platform.    For example
a list of processes for a user can be retrieved [as JSON] with a simple
GET to "/dav/Workflow/.ps"  You can list the workflows available to a
user via a GET to "/var/Workflow/Routes/.ls" ... or your can
navigate /dav/Workflow/Routes with a WebDAV client [like Nautilus or
Windows Explorer] and see all the Routes as subfolders....  Entities
[Contacts, Appointments, Processes, Documents, etc...] can be retrieved
as JSON, XML, iCalendar [if appropriate], or YAML - whatever you prefer
to parse.
  * Server functionality is available via a port of the zOGI API via
either XML-RPC or JSON-RPC.  So you can use whatever language bindings
you like [ CookComputing's XML-RPC bundle  for .NET works very well, as
does Python's standard-lib xmlrpclib module,  PHP's XML-RPC PEAR module
reasonably well (PHP has issues handing some character encoding, not but
a server can do about that) ].  The zOGI API is just as described for
OpenGroupware Legacy <http://code.google.com/p/zogi/> with a few
extensions.
  * The server provides some RSS feeds includes actions performed on
tasks related to projects the user is associated with.
  * GroupDAV support is implemented to provide access to Contacts from
clients like Evolution that support it.  Rudimentary CardDAV support is
also implemented.  CardDAV clients are only now becoming more common.
Contacts can also be accessed via the REST API.
  * Documents and attachments can be retrieved from the server using
simple HTTP, and depending upon MIME type, filters can be applied to
these documents via URL parameters [ these are called OpenGroupware
Server Side Filters, or OSSF].  Filter are provided for thumbnailing
images, retrieving just a section of a JSON file, and the XPATH
selection of an XML document.  JSON data is parsed as a stream to allow
for processing very large JSON documents.
  * Plugins can be installed in the system to insert addition
information into the JSON, YAML, or XML representation of an entity.
For example, retrieving an Enterprise entity, from my employers Coils
instance adds most recent invoice amount / date, sales in last 6 months
and previous six months, etc...

The workflow engine, which allowed me to port many corporate workflows
via drag-n-drop [awesome!] due to the use of BPML, is referred to as OIE
[OpenGroupware Integration Engine].  Workflows are described in the BPML
XML dialect as "Routes",  and a "Process" is created from the Route when
data is submitted.  Actions supported in routes include, but is not
limited to:

  * abending the process
  * creating, accept, complete, or comment on a task
  * retrieving an entity from the groupware database [as XML]
  * LDAP Search [result as the DSML XML dialect]
  * sending a mail message, or mail "blast" [mail merge]
  * SQL DELETE, INSERT, UPDATE, and UPSERT
  * read or write from a specified "format"
  * XPATH & XML document merge
  * Convert text to PDF
  * Watermark a PDF
  * submit data to an LPR printer queue [no local print service
required]
  * create and queue another process
  * convert JSON to XML

For-each and switch [if/else if/else] flow control is supported in route
defintions.

Exceptions in a route can be driven into a specified path of execution.

Creating custom actions is trivial.

Format classes are provided for line-oriented delimited [CSV, TSV,
etc...] and fixed-recond length data.  There is also a format for
generating [outputing too] XLS documents including formats, page styles,
and headers / footers.These classes are very well tested.  

Format classes for reading tabular XLS sheet is under active
development.  As well as reading / writing DIF documents.

There is overall documentation at
<http://sourceforge.net/projects/coils/files/WMOGAG-Coils.pdf/download>

Mailing list at
<https://lists.sourceforge.net/lists/listinfo/coils-project>

All levels of contribution are appreciated.  It is a really useful
product, it shovels lots of data every day; it is also fun to work on.