Docker and JAVA_OPTS

While adjusting some environment variables recently, I came across an odd issue with Docker, Spring Boot and JAVA_OPTS. JAVA_OPTS comes from the Tomcat/Catalina world and when searching for "Docker and javaopts" on Google you'll find many references to just adding JAVA_OPTS to the Docker environment. After some testing, I found this to be incorrect when running a Spring Boot jar in a Docker container, I'll explain why and give a solution in this post.

Before I start, let's setup a basic test environment that prints out the current memory setup so we can test in various situations. I've created this repo as a test case and you can refer back to it when needed.

Test Application Setup

Looking at the sample code, we have a basic pom file that imports Spring Boot dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>vc.c4.javaopts</groupId>
  <artifactId>example-javaopts</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <name>Spring Boot Tomcat JAVA_OPTS</name>
  <description>Spring Boot Tomcat JAVA_OPTS Example</description>
  <url>https://github.com/cl4r1ty/spring-boot-javaopts/</url>

  <properties>
    <java.version>1.7</java.version>
    <main.basedir>${basedir}/../..</main.basedir>
    <spring-boot.version>1.3.1.RELEASE</spring-boot.version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-dependencies</artifactId>
        <version>${spring-boot.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

    <build>
     <plugins>
       <plugin>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-maven-plugin</artifactId>
         <version>${spring-boot.version}</version>
         <executions>
           <execution>
             <goals>
               <goal>repackage</goal>
             </goals>
           </execution>
         </executions>
       </plugin>
     </plugins>
   </build>

</project>

The main application is a single Java file that prints out memory details:

package vc.c4.javaopts;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import java.text.NumberFormat;

@SpringBootApplication
public class JavaOptsApp {

  private static Log logger = LogFactory.getLog(JavaOptsApp.class);

  public static void main(String[] args) throws Exception {
    SpringApplication.run(JavaOptsApp.class, args);
    Runtime runtime = Runtime.getRuntime();

    final NumberFormat format = NumberFormat.getInstance();

    final long maxMemory = runtime.maxMemory();
    final long allocatedMemory = runtime.totalMemory();
    final long freeMemory = runtime.freeMemory();
    final long mb = 1024 * 1024;
    final String mega = " MB";

    logger.info("========================== Memory Info ==========================");
    logger.info("Free memory: " + format.format(freeMemory / mb) + mega);
    logger.info("Allocated memory: " + format.format(allocatedMemory / mb) + mega);
    logger.info("Max memory: " + format.format(maxMemory / mb) + mega);
    logger.info("Total free memory: " + format.format((freeMemory + (maxMemory - allocatedMemory)) / mb) + mega);
    logger.info("=================================================================\n");
  }

}

Running this app in IntelliJ with no memory settings will show some basic output with memory settings:

========================== Memory Info ==========================
Free memory: 221 MB
Allocated memory: 245 MB
Max memory: 3,641 MB
Total free memory: 3,617 MB
=================================================================

Changing the memory values to -Xmx3g -Xms3g we get a different output:

========================== Memory Info ==========================
Free memory: 2,713 MB
Allocated memory: 2,944 MB
Max memory: 2,944 MB
Total free memory: 2,713 MB
=================================================================

Now that we have a baseline, let's move onto the Docker stage.

Docker

The first thing we'll do is setup a basic Dockerfile similar to the one in the Spring Boot guide. Since the jar will be built in the target folder we grab it from there and place a renamed copy into the container.

# Base java:8
FROM java:8

# Add jar to container
ADD /target/example*.jar javaopts.jar

# Entry in json format
ENTRYPOINT ["java", "-jar", "/javaopts.jar"]

To generate the jar, just run maven real quick.

mvn clean package

The jar will then be in the target folder ready for building the Docker container.

$ ls -ahl target
total 11760
drwxr-xr-x   7 clarity  clarity   238B Dec 22 12:42 .
drwxr-xr-x  13 clarity  clarity   442B Dec 22 12:42 ..
drwxr-xr-x   3 clarity  clarity   102B Dec 22 12:42 classes
-rw-r--r--   1 clarity  clarity   5.7M Dec 22 12:42 example-javaopts-0.0.1-SNAPSHOT.jar
-rw-r--r--   1 clarity  clarity   3.3K Dec 22 12:42 example-javaopts-0.0.1-SNAPSHOT.jar.original
drwxr-xr-x   3 clarity  clarity   102B Dec 22 12:42 maven-archiver
drwxr-xr-x   3 clarity  clarity   102B Dec 22 12:42 maven-status

With the jar and the Dockerfile ready we can build the container from the base folder of the repository.

$ docker build -t spring-boot-javaopts .
Sending build context to Docker daemon 6.254 MB
Step 1 : FROM java:8
 ---> d4849089125b
Step 2 : ADD /target/example*.jar javaopts.jar
 ---> 518b9e05c3a9
Removing intermediate container cf4ff4db572a
Step 3 : ENTRYPOINT java -jar /javaopts.jar
 ---> Running in 15f032c1cc5c
 ---> 29a4656cdd5c
Removing intermediate container 15f032c1cc5c
Successfully built 29a4656cdd5c

We can see the newly built image by listing our Docker images.

spring-boot-javaopts$  docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
spring-boot-javaopts                            latest              29a4656cdd5c        8 seconds ago       648 MB

With our newly built container that will run Spring boot application let's run it for the first time! Note: I'll be cutting out some of the output lines to save space since we only care about the memory detail output.

docker run spring-boot-javaopts

2015-12-22 20:44:12.516  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : ========================== Memory Info ==========================
2015-12-22 20:44:12.516  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Free memory: 17 MB
2015-12-22 20:44:12.517  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Allocated memory: 30 MB
2015-12-22 20:44:12.517  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Max memory: 485 MB
2015-12-22 20:44:12.518  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Total free memory: 471 MB
2015-12-22 20:44:12.518  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : =================================================================

First run looks good. Let's try setting the JAVA_OPTS as we're used to!

docker run -e JAVA_OPTS='-Xmx3g -Xms3g' spring-boot-javaopts

2015-12-22 20:45:40.030  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : ========================== Memory Info ==========================
2015-12-22 20:45:40.030  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Free memory: 16 MB
2015-12-22 20:45:40.031  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Allocated memory: 30 MB
2015-12-22 20:45:40.031  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Max memory: 485 MB
2015-12-22 20:45:40.032  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Total free memory: 471 MB
2015-12-22 20:45:40.034  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : =================================================================

Woah! The Xmx and Xms settings didn't have any affect on the Java settings in the container! Something must be wrong here and it's not immediately obvious if all you search for is for "Docker and javaopts" on Google. Spring Boot is smart enough to handle many passed in environment variables but those are application specific. For example, when you look for a value x.y.z, Spring Boot will look in application.properties by default for x.y.z and in the environment variables for X_Y_Z. Due to this, passing in the environment variable -e X_Y_Z=1234 can be used in your application when using a Docker container. However, since JAVA_OPTS are used by Java and not in the application we run into this problem.

After some searching, I came across the fact that JAVA_OPTS are very specific to Catalina (Tomcat). Looking in the bin folder of a tomcat install you'll find a shell script that handles passing JAVA_OPTS into the exec lines. With this info, we can now look to change the Dockerfile to adjust for handling passed in environment variables. I named this new Dockerfile ExecDockerfile so I can have both in the same repo.

The change needed to enable JAVA_OPTS is to execute the Java line with an ENTRYPOINT exec command and place the environment variable into the line.

# Base java:8
FROM java:8

# Add jar to container
ADD /target/example*.jar javaopts.jar

# Entry with exec
ENTRYPOINT exec java $JAVA_OPTS  -jar /javaopts.jar

Let's rebuild the container but specify the new ExecDockerfile specifically using the -f flag.

$ docker build -f ExecDockerfile -t spring-boot-javaopts .
Sending build context to Docker daemon 6.254 MB
Step 1 : FROM java:8
 ---> d4849089125b
Step 2 : ADD /target/example*.jar javaopts.jar
 ---> Using cache
 ---> 518b9e05c3a9
Step 3 : ENTRYPOINT exec java $JAVA_OPTS  -jar /javaopts.jar
 ---> Running in 87f0d69e6171
 ---> a5622e7d302e
Removing intermediate container 87f0d69e6171
Successfully built a5622e7d302e

Let's run the container again.

$ docker run -e JAVA_OPTS='-Xmx3g -Xms3g' spring-boot-javaopts

2015-12-22 21:04:03.662  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : ========================== Memory Info ==========================
2015-12-22 21:04:03.662  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Free memory: 2,740 MB
2015-12-22 21:04:03.663  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Allocated memory: 2,969 MB
2015-12-22 21:04:03.663  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Max memory: 2,969 MB
2015-12-22 21:04:03.663  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : Total free memory: 2,740 MB
2015-12-22 21:04:03.664  INFO 1 --- [           main] vc.c4.javaopts.JavaOptsApp               : =================================================================

Conclusion

By starting the jar with ENTRYPOINT exec java $JAVA_OPTS, we've successfully enabled changing memory settings using Docker environment variables. If you think you're changing the memory settings currently but are not passing in $JAVA_OPTS into your start line in some fashion, I suggest you look at your Dockerfile config.