The Tesla Testament
Amazon best-seller December 2007!

Non-stop action. A vulnerable hero. A quest to save the world. The Tesla Testament is the most exciting novel of the decade.

Developing With Google App Engine

This book introduces development with Google App Engine, a platform that provides developers and users with infrastructure that Google itself uses for deploying their massively scalable applications.

Member of The Internet Defense League

Dynamic App Runtime Updates Using Mule Punching

Mule punching is a technique for extending or modifying the run-time code in Mule without stopping the server. The term is based on the Ruby programming jargon duck punching, where if duck typing doesn't work, you punch the duck until it does. If the services app doesn't do what you need, you can punch the mule until it does by changing the methods and attributes of service components and transformers without stopping the server, and even reconfigure the behavior of third-party packages without access to their code... all while the Mule server runs.

My team is building an enterprise-class system that has components that run on Mule and in a distributed environment across hundreds (or even thousands) of systems like in diagram 1. We selected a Java/ESB for the data center portions of the project, and Python for the distributed parts, both backed by mongoDB, arguably the most powerful and flexible NoSQL database today. As we got down to business, we realized that several data access and business logic functions had to be replicated in Java and Python.


Diagram 1 - System Architecture

This doubled the development and testing effort and we began exploring how to leverage JSR-223. We chose the Mule Integration Platform as our core application container but, powerful as it is, development and testing are very time consuming because of the compilation/testing/deployment/testing cycles in Mule. What if we could write only a single set of components in Python, and run them across all environments? What if we could reduce how much time we screw with Maven, Eclipse, Hudson, and Mule during each development integration cycle?

Motivation

What We Used

Installation

Install Mule as recommended, set the the $MULE_HOME and $MULE_LIB environment as indicated. This will be important later. The rest of the components' installation isn't complicated but does require that you follow these instructions exactly or you'll spend a lot of time troubleshooting why things don't work as expected.

$MULE_HOME points at /home/appserver/mule in our environment.

Installing Jython

Download Jython 2.5.2 when it becomes available; we used Jython 2.5.1 trunk because the stable release has a bug in the Scripting Engine that prevents Mule from binding its environment to the Python components. You need the sources for building the Jython 2.5.1 environment and its installer: 


svn co https://jython.svn.sourceforge.net/svnroot/jython/trunk/jython

svn co https://jython.svn.sourceforge.net/svnroot/jython/trunk/installer

cd jython

ant clean ; ant install

Run the installer (see figure 1); it's very important that you generate the standalone jython.jar file because it contains every Python and Java class for your Mule environment; if you end up building any of the other runtimes you'll be haunted by frustration and misery every time that you try to run your scripts or import a module, whether from Java or from Python. You've been warned! So, do this: 


java -jar jython_installer-2.5.1.jar

Figure 1 - Jython Installer - Select 'Standalone'!

Ensure that everything is working by writing a simple greeting program for Jython to run: 


echo ‘print "Hello Pythonic world!"' | java -jar $MULE_HOME/lib/user/jython.jar

If all goes well, your program will greet you back after initializing its own environment. You're ready to rock now!

Running Python Code in Mule

MuleSoft documents how to run scripts within Mule and uses Groovy for its examples. Here is the Python equivalent to their sample script with some logging thrown in: 


  <script:component>

    <script:script engine='jython'>

response = 'Payload = '+payload

log.info(response)

result = response

    </script:script>

  </script:component>

Yes, Python's indentation rules apply even if the script is embedded in a Mule configuration file. It's a better idea to build the script and either put it in a .jar file or in the file system (more on that later), and just define a script component in the config file: 


<script:script name='Greeting'

    engine='jython'

    file='/home/appserver/mule/lib/user/greeting.py' />

There are two possible ways of referencing the script in the file attribute:

  1. Provide the full path to the script, or its path relative to Mule's home directory
  2. Provide its relative path from the .jar file's root containing the script; this .jar will life in $MULE_HOME/lib/user like any other end-user, application package

Mule provides result and log objects via context binding, along with other useful references to things like the MuleContext, raw message, and any message properties.

Defining Robust Python Services for Mule

Though scripting can be used for quick development of simple components, we have a requirement to make robust Python components that will work in the Mule container as well as in a distributed environment that includes Windows, AIX, Linux, OS X, and Solaris. Thus we laid the code out in the same way that we'd arrange our Java code: service components have an entry point that takes the message request and returns some object as a response. One other requirement for us is to keep the Mule and any other Java code isolated from the rest of the Python code because of reusability. As such, the code is laid out the three distinct layers shown in diagram 2:


Diagram 2 - Dynamic Components Organization in Mule Server
  1. A simple script file that provides the entry point for Mule to call the business logic
  2. A Mule component that combines any calls to Java and Python code that are specific to the Mule environment
  3. Any number of business logic components, written in 100% Pure Python for maximum portability

In addition to this, the Mule component should get references to objects like the loggers or Mule entities only through the Mule script bindings to avoid or minimize class loader conflicts in the scripted components.

A Sample Application

Download the sample code and configuration file - .zip file, 4.1 KB

Listing 1 - mulescript.py shows a basic script's entry point. This file is loaded when the script is first invoked via an endpoint. All it needs to do is import the Mule component that will perform the actual work, create an instance, and execute a service request, like in mule-punch-config.xml.

The Mule component in listing 2 - the muleservice.py module is similar to a restlet (see the Mule RESTPack Restlet interface for details). This component handles only GET requests and creates an instance of BusinessObject from listing 3 - businessmodule.py to service the request. This module is the "glue" that combines Java and Python code calls specific to the Mule environment. By confining them to this module we get the best of both worlds: a single place where portability issues may arise and where we can control the interface with the Java environment, and the ability to change the code dynamically that affects this and the business objects layer.

The component needs to behave like a full-blown Mule entity, which normally responds with an instance of a MuleMessage. The code in lines 44 and 45 shows how a new DefaultMuleMessage is created and set with the object's response and optional HTTP code (200 or 400, depending on how the request was handled). The Python and Java code are interacting without issues in this layer, even though the original response is a string generated from Python. Mixing and matching is easy.

The classes and objects, Java or otherwise, are imported only once when the script first loads, unless the scripting engine is told to reload. Just add a call to reload(module) for every module or script that you want to be able to change/update dynamically without having to stop the Mule server.

Mule Punching Your Code

One of the most annoying things about Mule development is dealing with build/test/package/stopMule/deploy/startMule/test cycle. The ability to deploy code dynamically may be valuable in a production server as well. Imagine, for example, that the code in the previous example needs to return a JSON object instead of a plain string. In following SOA/Mule best practices, the business objects should deal with native objects, whether Java or Python. Either the service component or a transformer should modify those business objects for presentment to conform to a given transport or protocol.

Don't stop your Mule server. Start your favorite programming editor and open the file $MULE_HOME/lib/user/article/muleservice.py you're about to mule punch JSON support into the service's response code without stopping Mule!

JSON support is native to Python version 2.6 - unfortunately, Jython is based on 2.5. Fortunately, support is based on a de facto standard defined in the simplejson module. As a developer you have the option of making up for this shortcoming in Jython in either of these two ways:

Let's add simplejson support first. Assuming that you downloaded and installed the module and it's in your class path/sys.path, make these changes to muleservice.py:

Save the file, and hit the service endpoint again - this time you will get either of these responses: 


{"nStatus": 200, "response": "2010-02-22"}

{"nStatus": 400, "response": "Invalid HTTP method called!"}

Pretty nifty, isn't it?

Using Mule/Spring Components

Imagine that you don't want to add more code to the server since other components in your infrastructure already use the Gson Java package. Ensure that gson-1.3.jar (or a more current version) exists in $MULE_HOME/lib/user so that Mule can find it, then make these changes WITHOUT STOPPING MULE:

Try testing your endpoint. You will get this error: 


ImportError: No module named gson

Even though the Gson module is available in your server it may not have been used yet by any Java or Python component; the Spring container hasn't loaded the .jar, and thus it's invisible to your script. Let's make it available to Spring by adding this line to the mule-punch-config.xml file: 


<spring:bean id='GsonCache' class='com.google.gson.Gson'/>

Stop Mule to reload the configuration file. Ensure that any Mule/Spring/application components are loaded by Mule's Spring container during server initialization so that they are visible in the scripts. Also, ensure that any .jar files required in your script are stored in $MULE_HOME/lib/user, not in $MULE_HOME/lib/opt, or a different class loader will handle them and they won't be available to the script.

Start Mule, and test the endpoint. This time you'll get the same response, without errors. We still have simplejson and now Gson in the same script, though, so let's remove simplejson from muleservice.py without stopping the server:

Test your endpoint again. This time you'll get the response: 


{"nStatus":200,"response":"2010-02-22","encoder":"Gson"}

The toJson() Java method is catching a Python dictionary and operating on it without issues. Jython will map the objects to the appropriate Java counterparts without user intervention. APIs on both Python and Java become available to the other without fuss.

You can continue to modify the code as described in the previous sections. You can now perform dynamic Mule component updates!

File System And Editor Lazy Commits

Some editors and some file systems don't commit the file instantly to the file system. OS X HFS+, for example, has a "lazy fsync()" that results in a slight delay when saving a file on a busy system. Some editors make a backup before committing the file to disk. Either way you may see an occasional one- to two-second delay between saving the file and being able to see the update on the Mule server. Check your editor's and file system's tech notes to see how to get around this. On Linux, using ext3 and Vim with full sync() the delay is almost non-existent. On Samba-mounted drives it can take several seconds to update the file.

Conclusion

Mule punching is a great technique for dynamic Mule code updates at runtime that can help expedite the development process while leveraging the combined APIs from a scripting language and Java. More research is needed but a refinement to this technique could be used for dynamic updates in production servers as a stopgap measure until a clean dynamic loading mechanism like OSGi is available for Mule servers. Faster development, seamless updates, and portable integration... What's not to like?

Acknowledgements

Special thanks to Alex Grönholm, sabi, pjenvey, and the rest of the team in the #jython IRC channel on freenode for their help with solving problems during our R&D phase.

(Leave a comment)

Scalable Systems Newsletter

Subscribe to the newsletter and get every issue mailed free - with access to the latest system scalability, high availability, and performance news.