This post was created by the VMware Tanzu Vanguard community. (https://tanzu.vmware.com/vanguard)
This community is a small group of VMware Tanzu users, who came over from the Pivotal community, and represent some of VMware's largest and coolest TAS (Tanzu Application Services) and TKGI (Tanzu Kubernetes Grid Integrated) customers and partners. This community is lead by the incomparable Brian Chang --> https://twitter.com/techadvoguy
While I am mentioned in the credits.. I really only contributed the xkcd image :p

------------

Log4j Security Vulnerabilities CVE-2021-44228 - Mitigation Strategies for TAS Operators

By the Tanzu Vanguard community - key contributors: Simmy Xavier, Charles Lester, Juergen Sussner, Jonathan Regehr & Robert Kloosterhuis

Summary Brief:

Apache Log4j is a very widely used and popular logging library within the Java logging framework. There is a vulnerability named as Log4Shell identified and is being tracked officially under CVE-2021-44228 (and a second one under CVE-2021-45046). The vulnerability allows for RCE (Remote Code Execution) attacks which significantly increase the risk of exploitation. Hackers could use this to post malicious code which can be used for crypto mining or information extraction. There are reports of increased scanning happening across the Internet to identify vulnerable systems and infect them with malware or ransomware. This issue was discovered as early as Dec 1st by Chen Zhaojun of Alibaba Cloud Security Team and impacted across log4j-core v2.0 to v2.14.1. Apache had released the version v2.15.0 as of Dec 5. Apache has released version v2.16.0 as of Dec 14. This vulnerability has a severity rating of 10 out of 10 and is treated as a Zero-day vulnerability as of Dec 10 when this became a public disclosure.

Description:

If you have a system that uses log4j and you can get that system to log a JNDI URL or a shell command, log4j will actually execute the shell command.

The simplest one is a Minecraft server - they log any chat messages that are sent, so if you put something malicious in the chat message, log4j will execute it as it logs the message.
The TAS platform itself uses a vulnerable Log4j library in some of the tiles and the TAS tile. Apps built with the Java buildpack may also pull in vulnerable libraries

CISA is not able to confirm that merely using a newer JRE is sufficient for protection. See the discussion at this blog http://www.openwall.com/lists/oss-security/2021/12/10/3 and in the meantime, it is also confirmed that a modified version of this exploit is not restricted to a specific JVM version

Full remediation requires the use of log4j >= 2.16.0. The mitigation strategies merely reduce the attack surface area but do not fully protect against the threat.

Mitigation strategies for TAS

For systems running log4j >= 2.10.0 (thanks, Simmy Xavier!):

One approach is to set the LOG4J_FORMAT_MSG_NO_LOOKUPS variable in the running environment variable group, reflected in example (1) below. The limitation to this approach is that “Any user-defined variable takes precedence over environment variables provided by these groups.”
Another approach is to place the variable in the environment of every app, reflected in example (2) below. The limitation to this approach is that a subsequent restage of the application will cause the variable to be lost.
Instead of setting the env variable LOG4J_FORMAT_MSG_NO_LOOKUPS you can also add -Dlog4j2.formatMsgNoLookups=true to the JAVA_OPTS variable

For any systems running log4j 2.*

The more comprehensive mitigation strategy is to remove the JndiLookup class from the classpath (example: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class)

For older 1.x versions:

Although 1.x seems to be not affected by this, it is an old version which is out of support for a really long time and may be vulnerable to various other problems. Therefore Log4J 1.x should also be considered for updates. Depending on the Apps this could be achieved fairly simply by using the API bridge as described here: https://logging.apache.org/log4j/2.x/manual/migration.html

Example mitigation strategies for TAS running log4j >= 2.10.0 (thanks to Simmy Xavier):

Set the running environmental variable group (see https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#evgroups) (restart requires CLI >= 7) (note: if you have any existing running environmental variables, then you’ll need to add those into the srevg command, as the command expects to receive all the variables for the group, i.e., the command will set the revg to only what you specify in the command):

cf srevg '{"LOG4J_FORMAT_MSG_NO_LOOKUPS":"true"}'

cf restart <app-name> --strategy rolling

2. Set the environment variable for a particular app (restart requires CLI >= 7):


cf set-env <app-name> LOG4J_FORMAT_MSG_NO_LOOKUPS true

cf restart <app-name> --strategy rolling

3. Script to loop through all apps in a space, apply the change in (2) and restart the app (restart requires CLI >= 7):

cf apps | sed -n '4,$p'| awk '{print $1}' | while read appName

do cf set-env $appName LOG4J_FORMAT_MSG_NO_LOOKUPS true; 

cf restart $appName --strategy rolling

done

4. Script to apply the change in (1), then loop through every app in every space in every org, and restart the app (restart requires CLI >= 7) (you may want to edit the loops to exclude certain orgs, spaces, or apps):

'''

Applies the fix, then runs through every org in every space, and restarts every app

Note that “Any user-defined variable takes precedence over environment variables provided by these groups.” 

'''

cf srevg '{"LOG4J_FORMAT_MSG_NO_LOOKUPS":"true"}'

for org in $(cf orgs | sed -n '4,$p' | awk '{print $1}')

  do 

    cf t -o $org 1>/dev/null 2>&1

    for space in $(cf spaces | sed -n '4,$p' | awk '{print $1}')

      do cf t -o $org -s $space 1>/dev/null 2>&1

        rc=$?

        if [[ $rc -eq 0 ]]

          then

               apps=$(cf apps | sed -n '4,$p' | awk '{print $1}')

               for app in $apps

                  do cf restart $app –-strategy rolling

                  done

          else echo "cf t -o $org -s $space failed"

        fi

      done

  done

5. Script to loop through every app in every space in every org, apply the change in (2) and restart the app (restart requires CLI >= 7) (you may want to edit the loops to exclude certain orgs, spaces, or apps):

'''

Runs through every org in every space, applies the (temporary) fix, and restarts the app 


'''

for org in $(cf orgs | sed -n '4,$p' | awk '{print $1}')

  do 

    cf t -o $org 1>/dev/null 2>&1

    for space in $(cf spaces | sed -n '4,$p' | awk '{print $1}')

      do cf t -o $org -s $space 1>/dev/null 2>&1

        rc=$?

        if [[ $rc -eq 0 ]]

          then

               apps=$(cf apps | sed -n '4,$p' | awk '{print $1}')

               for app in $apps

                  do cf set-env $app LOG4J_FORMAT_MSG_NO_LOOKUPS true

                     cf restart $app --strategy rolling

                  done

          else echo "cf t -o $org -s $space failed"

        fi

      done

  done

Hint:

When using cf restart app –strategy rolling, the process of the rolling restart utilizes TAS features called deployments and this requires new apps to be started while the old ones are still running. This requires some additional ORG Quota or in other words, a rolling restart will fail in an ORG with no Quota left.

Apache mitigation recommendations

Apache's recommendations, located at https://logging.apache.org/log4j/2.x/security.html, depending on the version of log4j2, are:

Log4j version	Mitigation Plan
2.16.0	Nothing
2.15.0	Upgrade to 2.16.0
2.10.0 - 2.14.1	Upgrade to 2.16.0 OR Add Environment Variable "LOG4J_FORMAT_MSG_NO_LOOKUPS" to "true" OR Add system property "log4j2.formatMsgNoLookup" to "true"
2.0 - 2.9.1	Upgrade to 2.16.0 OR Remove JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
1.x	Log4j 1.x is no longer supported at all, and a bug related to Log4Shell, dubbed CVE-2021-4104, exists in this version

Several of the VMware products along with other vendors are using this popular framework and actively working in releasing a workaround and or a patch. VMware products impacted and the status of patch and workaround is posted under the Security Advisory located at https://www.vmware.com/security/advisories/VMSA-2021-0028.html

Based on the blog on Spring.io (https://spring.io/blog/2021/12/10/log4j2-vulnerability-and-spring-boot), Spring Boot users are only impacted if they have switched the default logging system to log4j2.

Workarounds until a patch can be applied across the TAS foundation would be to set the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS as true This could be done at a Global Level or at an App level but in either case require a restart for it to take effect.

Setting at Global level - cf srevg '{"LOG4J_FORMAT_MSG_NO_LOOKUPS":"true"}'

Setting at an App Level - cf set-env <app-name> LOG4J_FORMAT_MSG_NO_LOOKUPS true

Restart the app instances in a rolling fashion (require cf cli v7+) - cf restart <app-name> --strategy rolling

Validating the change - cf env <app-name> | grep LOG4J

Other useful scripts

Script to set environment variable (all apps in a space)

cf apps | sed -n '4,$p'| awk '{print $1}' | while read appName; do cf set-env $appName LOG4J_FORMAT_MSG_NO_LOOKUPS true; done

Script to perform Rolling restart (all apps in a space)

cf apps | sed -n '4,$p'| awk '{print $1}' | while read appName; do cf restart $appName --strategy rolling; done

Script to validate (all apps in a space)

cf apps | sed -n '4,$p'| awk '{print $1}' | while read appName; do cf env $appName | grep LOG4J ; done

How TAS as immutable infrastructure helps

Despite all the mitigation strategies, there is still a risk that some remote Code got dropped in a running Container. To proactively cope with that you can use the features of TAS where containers get recreated from an immutable source Image (the droplet). So why not run the restart scripts above on a regular basis, to constantly wipe out all that got into a container.

Monitoring app changes on TAS

Pathing TAS is essential and having Apps secured should be the first priority. But you may also want to know how your apps are behaving and if they have any vulnerable version within their containers. To get to know this you can set up a search job to investigate all running containers.

The Following Script can be run as a task in TAS

API=`echo $VCAP_APPLICATION | jq -r ".cf_api"`




cf login -a $API -u $USER -p $PASSWD -o dummyorg -s dummyspace

for org in $(cf orgs | sed -n '4,$p' | awk '{print $1}')

  do

    cf t -o $org 1>/dev/null 2>&1

    for space in $(cf spaces | sed -n '4,$p' | awk '{print $1}')

      do cf t -o $org -s $space 1>/dev/null 2>&1

        rc=$?

        if [ $rc -eq 0 ]

          then

               apps=$(cf apps | sed -n '4,$p' | awk '{print $1}')

               for app in $apps

                  do 

                     log4jversion=`cf ssh "$app" -c "cd /app; find -iname '*$PATTERN*'" |tr 'n' ' '` 

                     rc=$?

                     if [ $rc -eq 0 ]

                     then

                        if [ -z "$log4jversion" ]

                            then

                                echo "ORG=$org   SPACE=$space   APP=$app LOG4JVERSION=not found"

                            else

                                echo "ORG=$org   SPACE=$space   APP=$app LOG4JVERSION=$log4jversion"

                        fi 

                     else 

                        echo "ORG=$org   SPACE=$space   APP=$app LOG4JVERSION=not-ssh-able"

                     fi

                  done

          else echo "cf t -o $org -s $space failed"

        fi

      done

  done

This script will ssh into every container running on TAS and search its filesystem for log4j versions. You can utilize the TAS scheduler (https://network.pivotal.io/products/p-scheduler/) to run this TASK once a day

The Log output can be forwarded to any Log Management System which allows you to create a “real time” dashboard.

If you use Splunk, the query would be:

index=<TAS_FOUNDATION> cf_app_name=AdminScripts event_type=LogMessage 

| rex field=msg  "ORG=(?<orgname>.*)   SPACE=(?<spacename>.*)   APP=(?<appname>.*) LOG4JVERSION=(?<testresult>.*)" 

| eval files=split(testresult," ")

| rex field=files "log4j-core-(?<log4jversion>.*).jar"

| eval log4jversion=mvjoin (mvsort(mvdedup(log4jversion)), ",")

| table orgname, spacename, appname, log4jversion, files

| sort log4jversion desc

This creates a nice visualization like this

With this, you can get an Overview of which app uses which version and as you can see in this example there are sometimes “hidden” Versions or more than one Version within the app as agents like the AppDynamics Agent have their own version in place.

Patching apps the hard way…

You may also face apps that refuse to be patched because there is no source code available or no pipeline or for whatever reason. In this case, you can try the following approach to patch such apps.

```
cf app  vulnerableApp –guid
```
```
cf curl /v3/apps/<guid>/packages
```

Copy the download link from the links section

```
cf oauth-token 
```

curl -L <downloadLink> –header “Authorization: <oauthToken>” -o app.zip

Now path whatever needs to be patched in the app.zip

```
cf create-app-manifest vulnerableApp
```

cf push fixedApp -f vulnerableApp_manifest.yml -p app.zip

```
cf stop vulnerableApp
```

This will deploy a second, patched version alongside the vulnerable app with the same settings, effectively having a blue-green deployment of a patched and a vulnerable app.