Thursday 27 July 2017

iDempiere and Buckminster: building a web-based OSGi application

iDempiere and Buckminster

iDempiere is an open source, web-based Enterprise Resource Planning (ERP) system that has been built with tools from the Java world, but instead of using a familiar build automation tool like Maven, or a newer tool like Gradle, the iDempiere team used Buckminster to manage the build.

We tackled OSGi in an earlier post, now Buckminster as well? OK, roll up your sleeves...

This post looks at the use of Buckminster in the iDempiere project so that we can get a better understanding of the build process, and why Buckminster was used.

What is Buckminster?

Buckminster is a set of frameworks and tools for automating build, assemble and deploy processes. The project is named after the architect R. Buckminster Fuller who said about problem-solving "when I have finished, if the solution is not beautiful, I know it is wrong."

R. Buckminster Fuller
The problem that Buckminster is trying to solve is how to build and assemble multiple software components from multiple sources in combinations that are specific to a given target platform.

Buckminster uses a Component Query (or CQUERY) to identify the components and their associated component information and create a bill of materials (BOM). It then locates all the components in the BOM using a Resource Map (or RMAP) and materializes the project by copying them into the target platform, resolving the dependencies according to the work context, system setup or operating system.

Components

In the context of iDempiere, a component is a collection of files that make up an OSGi bundle, or plug-in, and in the context of the Eclipse IDE, a component may be an Eclipse project. Components can also exist outside the workspace, and they may be dependent on other components. 

RMAP component location mapping
Buckminster can store component information as a Component Specification (CSPEC) in an XML file inside the component, but it is also able to read meta-data from pre-existing sources such as plugin.xml, feature.xml, POM files, etc, and translate such information on-the-fly into CSPEC, which it does in the case of iDempiere.

Buckminster allows you to retrieve components from existing repositories and component formats, such as CVS, SVN, git, Maven, or p2 repositories. This enables the iDempiere team to integrate existing Adempiere code with bespoke iDempiere plug-ins and other third-party modules, without reconfiguring them.

Note that Buckminster can get the same component from different repositories, depending on the circumstances (different geographic locations, for example, or different software development environments like DEV, TEST, QA, PROD, and so on), making it easy to build your own on-site repositories for iDempiere.

Materializing iDempiere in Eclipse

Materialization Specification (MSPEC)

When you install iDempiere in an Eclipse development environment, you must first materialize the project by using a wizard to populate the workspace from the Buckminster Materialization Specification (MSPEC):

File > Import > Buckminster > Materialize from Buckminster MSPEC, CQUERY, or BOM > 
URL: $IDEMPIERE_HOME/org.adempiere.sdk-feature/adempiere.mspec

where $IDEMPIERE_HOME is the root directory of your iDempiere download.

The iDempiere MSPEC instructs Buckminster to install the materialized components in ${workspace.root}/targetPlatform/, using the Component Query in adempiere.cquery, and gives the MSPEC a human-readable name:
org.adempiere.sdk:eclipse.feature:1.0.0.qualifier

Component Query (CQUERY)

Buckminster then reads the Component Query from the CQUERY file named in the MSPEC, triggering its resolution and provisioning process. The CQUERY tells Buckminster where to find the Resource Map, adempiere.rmap, and also defines a number of advisorNodes which specify how groups of components should be handled.

The CQUERY resolution runs for a few minutes to build the Bill of Materials. The wizard gives you the option of saving the BOM to a file if you are interested in reading 20,097 lines of XML.

Resource map (RMAP)

Finally, Buckminster materializes the BOM by looking in the Resource Map (RMAP) for the location of the components and copying the JARs to the targetPlatform.

Note: materialization takes a very looooooong time, so put the kettle on and read the BuckyBook, the definitive guide to Buckminster (perhaps not: see below).

Update

After this post was written, Low Heng Sin, one of the core architects of iDempiere, explained to me that they used Buckminster because Maven was not capable of building Eclipse plug-ins when the project started.

Since then the Tycho set of plug-ins and extensions has been developed to build OSGi bundles using Maven.  Tycho uses OSGi metadata and OSGi rules to deduce project dependencies dynamically and inject them into the Maven project model at build time.

The development of Buckminster appears to have come to a halt so Tycho may be a better bet for new OSGi projects.

Tuesday 25 July 2017

iDempiere and OSGi: a modular Java architecture

iDempiere and Modularity

iDempiere is an open source Enterprise Resource Planning (ERP) system that describes itself as "OSGi + ADempiere", but what does that mean in practice? 

This post describes OSGi and the technology stack that was used to build iDempiere so that developers and solution architects can see how the system hangs together.

History of iDempiere

In the beginning, there was Compiere ("to accomplish, complete, fulfill" in Italian), an open source ERP system that was started in 1999. Initially it was very successful but after a long-standing disagreement between the developer community and the management company behind Compiere about its monetization strategy, the project was forked by a number of developers to create ADempiere (a word like 'compiere', but with the additional sense of "to fulfil obligations, or discharge duties", and also "to honor and respect").

ADempiere was also very successful, but its monolithic design had a number of limitations so some of the core architects forked the codebase in 2011 to create a more modular system called iDempiere, based on the OSGi specification.

OSGi and modularity

The OSGi specification defines a dynamic component system for Java. It is a specification that describes how individual bundles of code, also known as plug-ins, can be connected together in a dynamic way by a services layer that offers a publish-find-bind model for plain old Java objects (POJOs).

In standard Java everything in a JAR may be visible to all other JARs, but with OSGi, everything in that JAR is hidden unless it is explicitly exported in a predetermined manner. There is no sharing by default, so a bundle that wants to use another JAR must explicitly import the parts it needs. Conversely, a bundle must explicitly export functionality that is to be used by other bundles.

OSGi layered architecture
OSGi does this by specifying a layered model to manage plug-ins:
  • Bundles: these are the OSGi plug-in components that deliver the application functionality, consisting of POJOs with additional meta-data defined in the MANIFEST.MF file. Bundles hide their internals from other bundles and communicate through well defined services. Bundles are self-contained, reusable units of deployable code, usually in the form of a JAR;
  • Services: this layer connects bundles in a dynamic way by offering a publish-find-bind model for the POJOs registered in the service registry. Without OSGi, the standard Java solution is to use static factory methods and Java Reflection to load classes dynamically, resulting is a variety of passive, non-standard APIs and configuration mechanisms;
  • Life-Cycle: the API that allows you to list, install, start, stop, update, and uninstall bundles on the fly without bringing down the whole system. Bundles can be managed directly from the OSGi console command line interface and they can be managed remotely;
  • Modules: the layer that defines how a bundle's dependencies are managed by importing and exporting code according to the bundle's manifest. Modules are discrete, coarse-grained deployable units;
  • Security: the layer that handles  security by separating bundles from each other and enforcing permissions to allow or deny bundles access to each other;
  • Execution Environment: the underpinning OSGi platform that manages the containers. Like the Eclipse IDE, iDempiere is built on Equinox, which is the reference implementation of the OSGi specification (two other notable implementations of OSGi are Knopflerfish OSGi and Apache Felix).

Modular structure of iDempiere

The team that forked the code partitioned the monolithic Adempiere application into a large number of plug-ins, each with its own classpath and set of dependencies. If you download the iDempiere codebase and open it with Eclipse, you will see that it consists of dozens of projects: one for each OSGi bundle, or plug-in. 

This allows developers to create custom-built plug-ins without touching the upstream codebase. The Equinox OSGi container manages these plug-ins as free-standing, deployable Java objects that interact with the rest of the core iDempiere bundles by means of APIs.

Bundles are managed in a service registry

The advantages are:
  • explicit dependency management: iDempiere plug-ins declare what they need and what functionality they provide, whilst otherwise remaining isolated from each other;
  • versioned dependency: you can have different versions of the same Java class in different bundles if necessary because your plug-in's dependencies are managed by the OSGi container, not the JVM;
  • small footprint: plug-ins are not packaged with all their dependencies, and they follow a lazy activation policy so they are only loaded as and when required;
  • easy release: plug-ins can be developed and released independently of upstream developments;
  • hot redeploy: individual plug-ins can be redeployed on the fly without affecting the others.

Core iDempiere plug-ins

org.adempiere.server

Now that we have looked at OSGi, let's dig deeper into the structure of iDempiere and look at the core plug-ins that make up the application.

When iDempiere starts, a Bash shell script idempiere-server.sh executes the Equinox OSGi launcher org.eclipse.equinox.launcher, passing it the name of the initial executable entry point as a runtime option, namely org.adempiere.server.application. This Java class implements the Equinox OSGi IApplication interface and starts the application.

The manifest of org.adempiere.server directs Equinox to load these bundles as dependencies:
  • org.eclipse.equinox.app: implementation of the Equinox OSGi Application Container service;
  • org.eclipse.jetty.apache-jsp: support for JSP in the embedded Jetty HTTP container;
  • org.adempiere.base: the core functionality of iDempiere.

Equinox OSGi Application Container

The Equinox OSGi framework implementation is probably better known as the foundation of the Eclipse IDE platform, but it is actually a standalone OSGi implementation. It is launched by the org.adempiere.server plug-in when iDempiere is started and is responsible for loading all the other iDempiere plug-ins and managing the relationships and dependencies between them.

Read this tutorial on OSGi modularity with Eclipse by the ever-dependable Lars Vogel, and watch this video by Kirk Knoernschild, the author of "Java Application Architecture: Modularity Patterns with Examples Using OSGi", for a very useful overview of the current state of modularity on the Java platform.

Jetty HTTP Server

iDempiere uses the Jetty OSGi infrastructure to embed an HTTP server inside an OSGi container. This hosts the static content, servlets, OSGi web bundles (the OSGi equivalent of WAR files), and provides support for the Java ServerPages (JSPs) used by the original Adempiere code.

In production, it would be wise to put a reverse proxy in front of Jetty, running web application security software such as mod_security. Jetty can expose JMX services which should be monitored in a production environment. The Jetty configuration files are in $IDEMPIERE_HOME/jettyhome/etc/.


org.adempiere.base

This is the base plug-in that provides all the core functionality needed to build iDempiere plug-ins and includes interfaces such as:
  • IResourceFinder: finds resources in each plugin;
  • IColumnCallout: called whenever a column on a table changes its value;
  • IModelValidator: called whenever a record is saved;
  • IMenuAction: called whenever a window menu is created;
  • IModelFactory: called to create new class models;
  • and many others.

Wait, there's more...

These core plug-ins are at the heart of iDempiere, but there are many others that provide the application functionality, including the persistence layer, the report writer and the user interface, so follow the tutorial by Carlos Ruiz from GlobalQSS and download the code to explore further.

Here is a follow-up post that describes how Buckminster is used to build iDempiere on OSGi.