Mule Mixed JVM Languages HOWTO and HOWNOT
Published on 20130323.
Third-party languages running on the JVM promote opportunities to write software that's better than traditional Java implementations by some metric:
- Time to market
- Clarity of purpose
- Better fit to a specific problem domain
Mule Integration Platform enterprise and casual users are discovering this power and are experimenting or rolling out applications based on other languages like Scala, Python, Ruby, JavaScript, or even Lisp dialects.
This series of articles will focus on the things to know, to do, to learn, and to avoid when doing one of these implementations. The whole series will cover:
- HOWTO and HOWNOT to mix languages in your apps - general overview, case study, and recommendations
- JVM scripting for Mule/Spring 3 applications HOWTO - when, why, and how to integrate robust code into enterprise and other high performance, high availability applications (plus shout back at the original Mule Punching article in Python and how things have changed since it was published)
- Mule Applications and Scala - No JSR-223 Required! - learn to integrate next generation software written in the next generation language with the next generation Mule -- Java programming is optional
What You'll Need
- Working knowledge of the Mule ESB Integration Platform
- No fear! We'll dive into some Mule and MuleStudio internals, and change some of them, to get things working
- Some knowledge of Spring 3, dependency injection, and Mule configuration kung f00
- An open mind -- some things will be done very differently from what Java programmers were conditioned to expect
Although this is a Mule-centric series, most of what's discussed here applies to Spring as well. Review the Mule config files to see why.
The Basics
JSR-223 describes the integration points between third-party languages and the JVM. Keep in mind that app server or even JVM support isn't automatic just because a language appears on the list. Mule and MuleStudio are built atop Spring 3, and support for JSR-223 languages is dependent on two factors:
- Whether Spring 3 supports it out-of-the-box -- only Python, Ruby, JavaScript, and Groovy are supported
- Whether the language run-time authors (or someone else) have provided working JSR-223 ScriptEngine and ScriptEngineFactory libraries
The JSR-223 list of supported languages is misleading because though all of the languages listed may theoretically work well with Java, in practice many languages lack or have insufficient the javax.script integration layer. Many developers feel frustrated when they don't understand this fact and decide to use "scripting" in an app context without understanding this requirement.
Remember: there are many JVM languages, interpreted and compiled. Only a subset can be combined with apps running in a JVM using JSR-223.
Multi-language Mule/Spring Application Architecture
Only four scripting languages are available out-of-the-box in Mule and Spring. It's a good design and support decision: all four have large user bases, good documentation, and popularity across many problem domains. There may be cases, however, where a non-supported language is desired because:
- The implementers want to leverage an existing code base; or
- Problem domain expertise is available in a different language (e.g. Scheme) that the app server developers lack; or
- There is a definite time-to-market advantage when using a language that isn't available in Mule/Spring
This need to deploy a new, different scripting language in the app server requires modifications to MuleStudio for the development and testing phase, and for Mule when deployed to dev, QA, staging, and production servers. The modifications are simple:
- Find a current version of the stand alone, stable scripting run-time
- Find a current/viable version of the script engine interface compliant with JSR-223
- Put both in the app server's global class path ($MULE_BASE/lib/user for Mule servers)
Hands On: Enabling Scheme
The devil is in the details. Just saying "Scheme" opens a can of worms because:
- There is SISC Scheme, considered to be the purest and more standards compliant of the JVM Scheme dialects
- And there is jScheme, for those who want a less Lispy interface between the Java and Scheme worlds
- Last, there's Clojure and its excellent concurrency support and unique syntax
All of them are incompatible with one another. JSR-223 support for them is all over the map, and varies from the good (SISC), the bad (Clojure), to the non-existent (jScheme doesn't seem to have been updated in a long while).
SISC Scheme was chosen because it was the easiest illustrate and to implement in a Mule/Spring world. Any other dialect (or language) could've been implemented just as well. It's all a matter of how much coding is necessary to integrate/update the scripting engines to work in a modern app server.
So, what are the necessary steps to implement Scheme on Mule?
First Step: Find the Appropriate Script Engine
A search for JSR-223 Script Engines will eventually lead to the dev.java.net project that has all the scripting engines that were introduced with Java 6. It's not maintained often, and it's a bit hard to find. Direct searches for a script engine specific to SISC Scheme were fruitless (searches for more mainstream languages returned more accurate results). After some work (about 90 minutes), a viable script engine for SISC was found.
Since SISC Scheme is the topic of this implementation, it'll be referred to only as "Scheme" in the rest of this document.
The difficulty in finding a viable script engine, by the way, is another reason for trying to stick to more mainstream languages: their implementations are easier to integrate because they get more love from developers and vendors.
Download all the JSR-223 script engines available at java.net to figure out how they work. They're all in one place.
Second Step: Get a Working .jar File with the Appropriate Engine
- Pull the code off the java.net site:
svn checkout https://svn.java.net/svn/scripting~svn
- Add the source to a MuleStudio project or Maven project
- Add the javax.script.ScriptEngineFactory service descriptor to the project's manifest in the services directory
- Create a .jar with this code
Watch out! This may be easier said than done. For example, there is an abandoned JSR-223 scripting engine for Clojure in Google Code. It doesn't work because it relies on run-time communications between the app server and the Clojure run-time, and it uses a library that was deprecated a long time ago (clojure-common) that hasn't been supported in the Clojure run-time in years. Just adding the code to a project isn't a guarantee that it will even build.
There is also a matter of licensing - importing the code into another project is OK but this notice must appear in the Scheme engine's code and any documentation at the request of the licensor:
SISC Scheme script engine - Copyright © 2006, Sun Microsystems, Inc.
All rights reserved.
The Scheme run-time must be in the build path or it will be impossible to build the Scheme script engine library.
Beware that the code may also need some work -- most of these engines haven't been updated since 2007 and Java 1.4 was the JSR's target version. The engine for SISC Scheme did not even compile at first and required manual editing. These are the changes made to the SchemeScriptEngine class so it would compile without warnings or errors:
- Removed an import sisc.io.*; because it uses no classes from that package
- Annotated methods with @SuppressWarnings("all") because of weird type casting applied; the code works, so this isn't a huge issue
- Annotated a method with @SuppressWarnings("deprecation")
The main goal of this exercise was to get SISC Scheme working. The warnings and errors need fixing and must be updated to current Java and SISC libraries by a competent programmer who'll adhere to best practices. The JSR-223 sample code is amateurish (or downright horrible) in many places.
Build and export the .jar file. The project should look like this:
Notice that the latest release of Scheme (sisc*jar files) is among the referenced libraries, the script engine and factory show no errors, and there is a brand spanking new Scheme script engine .jar file ready for use in any JVM and app container that support JSR-223.
Third Step: Deploy the Script Engine and Language Run-Time to Mule
Start by defining a simple workflow that unleashes the power of Scheme scripting in a modest workflow:
The GreetingEndpoint will process asynchronous requests posted to http://server.name:8090/greeting by submitting them to the Scheme component.
It isn't necessary to use the dialogs to manage the code. In fact, it's always better to use a good programming editor like Vim or Emacs instead of the half-assed Eclipse built-in editors for producing scripts. MuleStudio users should feel equally comfortable editing within the IDE dialog like in this example, by editing the .flow file in MuleStudio's XML tab, or by editing a stand-alone script/class/module/program and then referring to it in the .flow or Mule .xml configuration.
Bundling the required third-party library .jar files for a web app is standard operating procedure for all app server frameworks, including Mule. The first developer reaction will be to refer to the .jars in build path so they become available to the application. That, however, doesn't work in Mule or Spring 3.
Trying to load a simple greeting component written in Scheme will throw an error every time the server tries to instantiate it:
org.mule.api.lifecycle.InitialisationException: Scripting engine 'scheme' not found.
Available engines are: [com.sun.script.jruby.JRubyScriptEngineFactory@2f600492,
org.codehaus.groovy.jsr223.GroovyScriptEngineFactory@6ed3199e,
com.sun.script.juel.JuelScriptEngineFactory@69127c4d,
com.sun.phobos.script.javascript.RhinoScriptEngineFactory@9bc850f,
apple.applescript.AppleScriptEngineFactory@7d880b02,
com.sun.phobos.script.javascript.EmbeddedRhinoScriptEngineFactory@c70856b,
com.sun.script.jython.JythonScriptEngineFactory@70b125bd,
com.sun.script.javascript.RhinoScriptEngineFactory@18d210ab]
This isn't very encouraging... what happened?
Looking at the multi-language application architecture diagram earlier in this document explains the issue: the script engine and run-time can't be instantiated at the application level. They must be available to the app server context before allocating any components. The Mule configuration file can't find the Scheme engine in the class path and grinds to a halt.
How to overcome this?
Simple: make the script engine and the language run-time part of the app server's class path. Best practices is to place them in the $MULE_BASE/lib/user directory:
Restart the app server and it will load its configuration file without further incident to process requests. The run-time and script engines are reachable.
Note: this example shows where to put the files in the MuleStudio path. Stand alone Mule just requires the files to be in $MULE_BASE/lib/user.
Restart the server and make a request to the GreetingEndpoint and the Mule server will respond with:
Hello, world!
Time for a well-earned Monster Energy or Red Bull break!
Using Scheme Components for Real Work
3rd-party languages running in the JVM provide all the functionality of a Java component, and more if the language supports abstractions or libraries unavailable in Java. This isn't wishful thinking. There are many Mule production systems in the real world in which the mission-critical components are written in Python or Ruby instead of Java. The trick is to understand how to organize the code and bind the scripts to Mule for best results; the Mule Punching article also describes when, why, and how.
Mule, Spring 3, and any other app container provide standard functionality to the apps running in it by binding the scripts and the framework with standard objects. Some of the Mule context binding components available message processing scripts include:
- The payload
- MessageContext
- Access to the flow's Mule log handler
The MuleSoft documentation has an excellent list of all the scripting context bindings.
These context bindings appear to the hosted language as global variables or pre-allocated, "ready to use" objects. The syntax and rules for accessing them vary between JVM languages. There is wide variation even among languages that are supposed to be "similar" like Clojure, JScheme, and Scheme. Refer to the scripting language and run-time documentation to figure out how to call Java objects from it and vice versa.
The SISC documentation tells how to make calls from Java to Scheme in rigorous detail. There is a significant challenge to overcome, however: the Scheme types don't correspond to Java's. Other run-times like Jython or JRuby have smarter, automagic mapping. Scheme programming in Mule/Spring will require manual conversions for every object and primitive. Very annoying.
The SchemeScriptEngine includes the Mule context bindings in a GLOBAL_SCOPE procedure named context that behaves as a dictionary. The context bindings provide access as shown:
Unlike almost in all other hosted languages, log is a reserved word for the Scheme natural logarithm function, so a new name was mapped. It was necessary to import the OO package and a custom mapper (not shown - the code is kludgy because it wasn't developed by a Schemer) to generate a useful Scheme object. This Java-to-Scheme mapping using Scheme's object system is a huge undertaking (2 days to figure that out vs. about 4 hours to write the rest of this piece).
The output from the program is straight forward:
- The native procedures legends correspond to context and to the converted logger
- The Java Mule object, log, in its raw state; remember that it's impossible to use it in Scheme without explicit type conversion!
The scripted component will generate the greeting to the log every time the endpoint is hit. Implementing any of the other Mule context binding objects would follow the same steps:
- Define the Scheme class
- Map the existing Java object to the Scheme class
- Use it following the Scheme syntax
Conclusions
Scripting in the JVM is a rewarding investment when done well. It's no accident that Mule/Spring only support a handful of languages: exotic scripting like Scheme requires too much work to implement, and it needs too much developer participation to leverage the app server's resources. It was a thankless, bug ridden, kludgy job that in itself suggests that Scheme isn't a good Mule/Spring scripting language. Every business object would require manual mapping, creating more work instead of simplifying things. Compare the Python version:
log.info("greetings, Python dudes!")
against the Scheme version
(import s2j) (import oo) (import mule-object-mapper) (define info (generic-java-method '|info|)) ; logger symbol - log is the Scheme function for natural logarithm (define logger (context 'log)) (define logger-proc (mule-log-mapper logger)) (info logger-proc "greetings, Schemers!")
Adding debugging time for the JSR-223 engines and the common development everyday snafus on top, implementing an exotic language may not be worth it.
When possible, stick to Python, Ruby, JavaScript, or Groovy. All have rich libraries, functional and OO features, acceptable performance, and large talent pools . Implement the exotic scripts only if there is an overwhelming technology or business requirement, and if your lifecycle and budget accommodate maintaining the engines, scripting interfaces, and the application itself.
Thanks!
In alphabetical order, I want to thank Riastradh, Chris K. Jester-Young, taylanub, and the rest of the #scheme channel on Freenode for their help, along with Devin Jeanpierre and Phillip Ross for their excellent troubleshooting and functional programming advise.
Scalable Systems Newsletter
Subscribe to the newsletter and get every issue mailed free - with access to the latest system scalability, high availability, and performance news.
