Since JDK 6 that Java provides the Java Compiler API. It allows one to compile Java source from Java applications. By default, it generates a set of class files with the generated bytecode. This post shows how to build the necessary support to generate that bytecode in-memory using the Java Compiler API.

Motivation

Why would I need this? A while ago I had to build an expression evaluation engine that works in a way pretty much similar to Apache JXPath. JXPath allows you to navigate in a Java objects graph using XPath-like expressions through the use of reflection. If you are building a library like this, it should have a structure similar to the following:

  • A lexer, that translates the input text into input tokens
  • A parser, that takes the list of input tokens and generates a semantically valid AST (Abstract Syntax Tree).

Once we add the ability to navigate in a Java objects graph dynamically, we need to transverse the tree and generate something. We have two options:

  1. We use reflection to understand the graph anatomy and navigate through it to generate the calculated value. That forces you to go through reflection to get the expression result
  2. We use reflection to understand the graph anatomy and navigate through it to generate the java source code out of it, so it can be compiled to be evaluated later as if it was part of the application.

So remaining of this post shows how to get rid of this class files and use the Java Compiler API to generate bytecode into memory so it can be executed later as part of the application.

The usage

Let’s say we already built it. Here’s how to use it.

String source =
    "import java.util.Collections;\n"+
    "public class StringSorterByText {\n"+
        "public void sort(List<Strings> strings) {\n"+
            "Collections.sort(strings);\n"+
        "}\n"+
    "}";

// **** COMPILE
InMemoryCompiler compiler = new InMemoryCompiler();
CompilationPackage pkg = compiler.singleCompile("HelloWorld", source);

// **** LOAD
CompilationPackageLoader loader = new CompilationPackageLoader();
Map<String, Class<?>> classes = loader.loadAsMap(pkg);

// **** EXECUTE (using reflection)
Object instance = classes.get("StringSorterByText").newInstance();
classes.get("StringSorterByText").getMethod("sort", List.class)
      .invoke(instance, newArrayList("c","a","b"));

The CompilationPackage assembles a list of CompilationUnit instances, each one representing a class that we are compiling. Bear in mind that (anonymous) inner classes inside a class generate isolated classes by themselves, meaning that if we added an inline comparator implementation to our Collections.sort call, that anonymous inner class would generate a separate compilation unit for the resulting compilation package.

Yep, we still execute the main method using reflection this way. But that’s one global call instead of one call for each step in the object graph.

If we can tell the Java Compiler the application’s classpath, we can drive our generated code by interfaces and use them instead.

StringSorter sorter = (StringSorter)classes.get("StringSorterByText").newInstance();
sorter.sort(newArrayList("c","a","b"));

How to build it?

What we will need:

  • An in-memory compiler that instructs the Java Compiler API to generate bytecode in-memory instead of class files
  • A class loading mechanism that include those classes into the execution’s classpath.

The in-memory compiler

public CompilationPackage compile(String name, String source) {
    JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();

    DiagnosticCollector<JavaFileObject> collector = new DiagnosticCollector<>();
    InMemoryClassManager manager = new InMemoryClassManager(compiler.getStandardFileManager(null, null, null))
    
    // defining classpath
    String classpath = loadClasspath();

    // add classpath to options
    List<String> options = Arrays.asList("-classpath", classpath);

    // java source from string
    List<JavaSourceFromString> strFiles = newArrayList(
        new JavaSourceFromString(className, classCode)
    );

    // compile
    CompilationTask task = compiler.getTask(null, manager, collector, options, null, strFiles);
    boolean status = task.call();

    // check for compilation errors
    if (status) {
        List<CompilationUnit> compilationUnits = manager.getAllClasses();
        return new CompilationPackage(compilationUnits);
    } else {
        // something's really wrong
        LOGGER.error(buildCompilationReport(collector, options));
    }
}

private String loadClasspath() {
    StringBuilder sb = new StringBuilder();
    URLClassLoader urlClassLoader = (URLClassLoader) Thread
            .currentThread().getContextClassLoader();
    for (URL url : urlClassLoader.getURLs()) {
        sb.append(url.getFile()).append(
                System.getProperty("path.separator"));
    }

    return sb.toString();
}

The default JavaFileManager implementation class uses a simple implementation of type JavaFileObject to read/write bytecode into class files. This InMemoryClassManager extends the standard JavaFileManager to read/write bytecode into memory using a custom implementation of the JavaFileObject

public class InMemoryClassManager extends
        ForwardingJavaFileManager<JavaFileManager> {

    private List<CompilationUnit> memory = newArrayList();

    public InMemoryClassManager(JavaFileManager fileManager) {
        super(fileManager);
    }
    
    @Override
    public FileObject getFileForInput(JavaFileManager.Location location, String packageName, String relativeName)  throws IOException {
        throw new UnsupportedOperationException();
    }

    @Override
    public JavaFileObject getJavaFileForInput(Location location, String className, JavaFileObject.Kind kind) throws IOException {
        throw new UnsupportedOperationException();
    }

    @Override
    public JavaFileObject getJavaFileForOutput(Location location,
            String name, Kind kind, FileObject sibling) throws IOException {
        JavaMemoryObject co = new JavaMemoryObject(name, kind);
        CompilationUnit cf = new CompilationUnit(name, co);
        memory.add(cf);
        return co;
    }

    @Override
    public boolean isSameFile(FileObject a, FileObject b) {
        return false;
    }

    public List<CompilationUnit> getAllClasses() {
        return memory;
    }
}

The JavaSourceFromString class stores Java source code from a String into a JavaFileObject and can be built as follows:

public class JavaSourceFromString extends SimpleJavaFileObject {
    private final String code;
    private final String className;

    public JavaSourceFromString(String className, String javaSourceCode) {
        super(URI.create("string:///" + className.replace('.', '/')
                + Kind.SOURCE.extension), Kind.SOURCE);
        this.className = className;
        this.code = javaSourceCode;
    }

    @Override
    public final CharSequence getCharContent(boolean ignoreEncodingErrors) {
        return code;
    }

    @Override
    public final String getName() {
        return className;
    }
}

The in-memory class loader

Now that we have the ability to compile our source into memory in the form of a CompilationPackage, we need a way to load it in the form of executable classes into our application’s classpath. For that CompilationPackageLoader was built, as below:

public class CompilationPackageLoader {
    public List<Class<?>> load(CompilationPackage pkg) throws ClassNotFoundException {
        ByteArrayClassLoader bacl = ByteArrayClassLoader.newInstance();
        List<Class<?>> loadedClasses = newArrayList();

        for (CompilationUnit unit : pkg.getUnits()) {
            Class<?> cls = bacl.loadClass(unit.getName(), unit.getBytecode());
            loadedClasses.add(cls);
        }

        return loadedClasses;
    }

    public Map<String, Class<?>> loadAsMap(CompilationPackage pkg) throws ClassNotFoundException {
        List<Class<?>> classes = load(pkg);
        return Maps.uniqueIndex(classes, BY_CLASS_NAME);
    }

    private static final Function<Class, String> BY_CLASS_NAME = new Function<Class, String>() {
        @Override
        public String apply(Class aClass) {
            return aClass.getName();
        }
    };
}

The real trick is at the ByteArrayClassLoader, that is able to build a Class<?> instance out of an array of bytes.

public class ByteArrayClassLoader extends ClassLoader {

    private byte[] bytecode;

    private ByteArrayClassLoader() {
        super(ByteArrayClassLoader.class.getClassLoader());
    }

    public Class loadClass(String fqdn, byte[] byteCode)
            throws ClassNotFoundException {
        this.bytecode = byteCode.clone();;
        return loadClass(fqdn);
    }

    @Override
    protected Class findClass(String name) throws ClassNotFoundException {
        if (bytecode != null) {
            try {
                return defineClass(name, bytecode, 0, bytecode.length);
            } catch (ClassFormatError e) {
                throw new ClassNotFoundException(name, e);
            }
        }
        return null;
    }

    private static class BaclPrevilegedAction implements PrivilegedAction<ByteArrayClassLoader> {
        @Override
        public ByteArrayClassLoader run() {
            return new ByteArrayClassLoader();
        }
    }
}       

The bytecode wrappers

Now we are just missing the implementation of CompilationPackage and CompilationUnit, which are simple immutable objects holding the bytecode.

 
public class CompilationUnit {
    private final String name;
    private final JavaMemoryObject memoryObject;

    public CompilationUnit(String unitName, JavaMemoryObject memoryObject) {
        this.name = unitName;
        this.memoryObject = memoryObject;
    }
    
    public String getName() {
        return name;
    }

    public byte[] getBytecode() {
        return memoryObject != null ? memoryObject.getClassBytes() : null;
    }
}

public class CompilationPackage {
    private final List<CompilationUnit> units;

    public CompilationPackage(List<CompilationUnit> units) {
        this.units = newArrayList(units);
    }

    public List<CompilationUnit> getUnits() {
        return units;
    }
}

Right now we have everything we need to compile Java source into memory and execute it.

Conclusion

It was seen what it takes to implement an In-memory Java Compiler using the Java Compiler API.

In the case you need to do some kind of trick like compiling source code into memory in a long running application, feel free to give it a try. The full source code can be found here.

Have fun.