Java Reload (experimental) simple Support

The "jreload" module is not the definitive word about java classes reloading under jython. It is still experimental and its interface may improve or change to become more pythonic.

"jreload" cannot cover all the possible reload-flavors, and its goal is to offer a simple interface for the most common cases, e.g. quick trial-and-error experimenting.

Java classes reloading in jython is not enabled by "jreload", some of the modifications occurred to jython run-time have made it possible. Now jython can deal with java classes with the same name (without clashes) and run-time supports unloading of java classes by different ways, which is a must feature for some uses of reloading

[The expert user can now play directly with class-loaders and reloading as he would from java.]

The main idea of the "jreload" interface is that of a load-set. A load-set is a package-like object that can host a complete hierarchy of java packages, which can be reloaded as a whole.

Why there is no support for reloading a single class? Java classes are loaded through class-loaders, actually there is no support in java to redefine an already loaded class through its class-loader. In order to reload a class one should create a new class-loader, but classes from different class-loaders cannot interoperate, so we need to reload all related classes through the new class-loader. Note: The normal python reload built-in does nothing for java classes and simply returns the old version.

The "jreload" module exports the following functions:

makeLoadSet(name, path)
reload(loadSet)

makeLoadSet creates and returns a new load-set with the given name. The created load-set will behave like a package and import statement related to it can be issued. name should be a valid python identifier like any python module name and should not clash with any module that has been or will be imported. Internally the created load-set will be added to sys.modules, the same way it happens to modules on import. You can issue the same makeLoadSet from many places, it is idempotent like modules imports are.

path should be a list of directory or jar paths. The created load-set will enable importing the classes present there. path should be disjoint from both sys.path and java classpath, otherwise one can get very confusing results.

For example: if a load-set 'X' is created and its hierarchy contains java packages 'p' and 'p.q', then the following references can be used in import statements: 'X', 'X.p', 'X.p.q'.

reload(loadSet) reloads all the classes in the package hierarchy hosted by loadSet and returns loadSet.

Note: The current version of "jreload" (jreload.__version__=='0.3') does not support removing or substituting jars on the fly.

Example

The following example should help make things clearer: (its files should be present in the jython Demo dir)

Demo/jreload/example.jar contains example.Version (source) and example.PrintVer (source)

Demo/jreload/_xample contains a slightly modified version of example.Version (source)

>>> import sys
>>> import os
>>> import jreload
>>> def xp(name): return os.path.join(sys.prefix,'Demo/jreload/'+name) # builds a path under 'Demo/jreload' 
...
>>> X=jreload.makeLoadSet('X',[xp('.'),xp('example.jar')])
>>> from X import example
>>> dir(example)
['PrintVer', 'Version', '__name__']
>>> X.example.Version
<jclass example.Version at 6781345>
>>> from X.example import * # works but in general import * from java pkgs is not for production code
>>> v=Version(1)
>>> PrintVer.print(v)
version 1
>>> os.rename(xp('_xample'),xp('example')) # _xample becomes example, hiding and "patching" jar contents 
>>> jreload.reload(X) # (re)loads example dir example.Version and jar example.PrintVer
<java load-set X>
>>> nv2=example.Version(2)
>>> example.PrintVer.print(nv2)
new version 2
>>> PrintVer.print(nv2)
Traceback (innermost last):
  File "<console>", line 1, in ?
TypeError: print(): 1st arg can't be coerced to example.Version
>>> example.PrintVer.print(v)
Traceback (innermost last):
  File "<console>", line 1, in ?
TypeError: print(): 1st arg can't be coerced to example.Version
>>> os.rename(xp('example'),xp('_xample'))

Note: Differently from python packages reload, load-sets reload the complete hosted hierarchy.
Note: Class versions across reloads are not interoperable.

Like for python classes and python reload, old versions are kept around, if there are still references to them. But what happens if they are no longer used?

Java Classes Unloading

One would expect that no longer referenced java classes would be unloaded, but the situation is not that simple.

In order to give a python-class-like view on python side and for implementation reasons jython wraps java classes (in instances of org.python.core.PyJavaClass). Clearly the mapping from java classes to their wrapped version should be unique (e.g. to guarantee == and 'is' semantic). So jython keeps this mapping in an internal table. This is also good because building the wrappers is expensive.

Note: Typically one should care about java classes unloading only for very dynamic applications, like IDEs or long-running apps, that would go out memory if old versions of reloaded classes would not be collected.

Clearly the entries somehow block unloading. On the other hand java classes unloading is just a memory consumption optimization (and as such is it presented in Java Language Specification). Actual jvms clearly support this. JPython simply kept the entries in the table forever but Jython and "jreload" try to make unloading possible.

Note: java never unloads system classes (java.* etc) nor classes from classpath. Further Jython cannot unload sys.path java classes. So the whole unload issue makes sense only with "jreload" or custom class-loaders.

Java 2 and jython internalTablesImpl option

Under java2 jython offers table implementations that exploit soft/weak references in order to discard entries (when this is OK) for unloading.

A possible policy would be to keep an entry as long as the corresponding java class is still referenced outside the table (both by java or jython code). But this one cannot be implemented. [Tech.: One cannot add fields to final java class java.lang.Class!] So entries are kept as long as the wrapped version is still in use.

These implementations can be chosen trough python.options.internalTablesImpl registry option. Note: they only influence classes unloading, there is no need and reason to use them, unless one depends on class unloading to avoid memory leakage.

internalTablesImpl = weak -- Sets implementation using weak-refs. Table entries for not referenced (outside the table) wrapped versions are "discarded" at garbage collection points. If a class or some of its instances are continuously passed from java to jython side, but no long-living reference to it is kept from jython side, this can imply a performance penalty (rebuilding the wrapped version is expensive). On the other hand this is a good setting for testing if unloading actually happens or some references hunting around prevent it.

[Note: With jdk 1.3 java -verbose:class can help tracking class unloads, and System.gc forces class unloading. With jdk 1.2 java -verbose:gc should give some information on class unloading, but unloading of classes happen at unpredictable points and System.gc does not trigger it. Also weak-refs allow testing for unloading and gc.]

internalTablesImpl = soft --Sets implementation using soft-refs. Table entries for not referenced (outside the table) wrapped versions are "discarded" on memory shortage, given soft-reference definition. Soft-references behavior is not specified in full details, so the actual behavior will depend on the concrete jvm. But if actual (jvm) implementations are not too bad, this should be a good setting for production code, which relies on unloading to avoid out of memory failures.

Java 1.1

To be honest the unloading support that jython can offer under java 1.1 (given the absence of weak/soft-refs) is error-prone and anything serious would require "too much" caution, but this should not be a real issue. Support is offered only for "jreload" needs, in these forms:

Before reload(X) one can issue X.unload(). X.unload() discards all the entries for the old versions of the classes in X. This is safe only if all python subclasses and all instances of them have been destroyed.

One can "extract" the information needed in order to discard the entries for the versions actually present in X at a later point (after a whole series of reloads):

u_t1=X.unload # extract unload info for time t1 versions
... reloads ...
u_t1() # discard entries for time t1 versions

u_t1() is safe only if at that point all subclasses/instances of the involved versions have been destroyed.

Note: these idioms work also with the standard internal tables implementation under java2, and for compatibility even with the weak/soft implementations.

JReload Example Source Files

Jar example.Version

package example;

public class Version {

 private int ver;

 public Version(int ver) {
  this.ver = ver;
 }

 public String toString() {
   return "version "+ver;
 }

}

Back to example transcript

example.PrintVer

package example;

public class PrintVer {

 static public void print(Version ver) {
  System.out.println(ver);
 }

}

Back to example transcript

New example.Version

package example;

public class Version {

 private int ver;

 public Version(int ver) {
  this.ver = ver;
 }

 public String toString() {
   return "new version "+ver;
 }

}

Back to example transcript