Remote JVM profiling on Amazon EC2

Using SSH Port Forwarding (SSH tunneling) establish a secure connection with an Amazon EC2 instance, run Java VisualVM or JConsole locally, and monitor a remote Tomcat process over this connection.

This approach requires Tomcat 6.0.24 or latter as this was the version that the JmxRemoteLifecycleListener was introduced. I'll look at how to monitor older versions of Tomcat using X11 Forwarding in an upcomming post.

Setup security

First we need to open a port on the EC2 firewall, for JMX/RMI communication, and limit access to the public IP address of our client computer.

To find the client's IP address enter

curl jsonip.com

Using a security group of trader, a port of 10001 for JMX/RMI, and a client IP address of 111.40.27.251, we'll open the port using the EC2 command line tool

ec2-authorize trader -p 10001 -s <your public IP address>

Configure Tomcat

Determine the host you want to monitor. 16 means filter only running instances.

ec2-describe-instances --filter instance-state-code=16

Now SSH to the instance

ssh -i $EC2_AWS/ec2.pem neil@ec2-123-12-1-123.compute-1.amazonaws.com

To check what version of Tomcat is running

$CATALINA_HOME/bin/version.sh

To set the JMX/RMI port number we need to enable and configure the JMX Remote Lifecycle Listener for Tomcat. First we need to download the required jar.

sudo wget -P $CATALINA_HOME/lib http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.28/bin/extras/catalina-jmx-remote.jar

Install into Tomcat

cp catalina-jmx-remote.jar $CATALINA_HOME/lib

Next edit /etc/default/tomcat6

sudo vim /etc/tomcat6/server.xml

And add the following listener

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10001"/>

Now edit the /etc/default/tomcat6 file

sudo vim /etc/default/tomcat6

And add the following system properties. Note that the java.rmi.server.hostname is the public hostname of the EC2 instance.

JAVA_OPTS="${JAVA_OPTS} -Djava.rmi.server.hostname=ec2-123-12-1-123.compute-1.amazonaws.com -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Restart Tomcat for the changes to take affect

sudo /etc/init.d/tomcat6 restart

Monitor Tomcat

Now, on the client computer create the SSH tunnel that will forward port 10001 on our local client computer to port 10001 on the remote EC2 instance.

ssh -N -v -L 10001:localhost:10001 ubuntu@ec2-123-12-1-123.compute-1.amazonaws.com

Open a new terminal and enter either

$JAVA_HOME/bin/jvisualvm -J-Djava.io.tmpdir=/tmp/tomcat6-tmp

Or

$JAVA_HOME/bin/jconsole -J-Djava.io.tmpdir=/tmp/tomcat6-tmp localhost:10001

Sometimes takes a few seconds for the localhost:10001 connection to show in the UI.

Done.

The above was tested using:

Server
- Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
- Tomcat 6.0.28
- Ubuntu 10.04 Lucid Lynx

Client
- Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
- Ubuntu 11.04 Natty Narwhal

Why are we using -J-Djava.io.tmpdir=/tmp/tomcat6-tmp?

See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7009828

References:
- http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/intro.html
- http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html
- http://gabenell.blogspot.com/2010/04/connecting-to-jmx-on-tomcat-6-through.html