Log4j vulnerability: What the FAQ happened?

On Thursday, December 9th 2021, a far-reaching zero-day vulnerability in the Apache Log4j library, was announced to the public as CVE-2021-44228 and given the moniker Log4Shell. It was initially unearthed by script kiddies playing in a sandbox game called Minecraft.

A few days have passed, and if you have any involvement in running Java-based services, and haven’t already done so, please stop reading now and follow mitigation guides to ensure you aren’t getting pwned any harder than you already were a minute ago.

Even if you already did that, please double check, as mitigation advice has been evolving. At the time of writing, the main recommendation is to upgrade any log4j-core library deployments to version 2.16.0. If a more recent version is available, I’d say consider using that instead.

Still with me? Great! Please grab a drink, and sit down.

Here we’re going to do some digging into the Log4j functionality that was exploited, what it does, how it was coded, how it was documented, and how it evolved during development of Log4j to get to where it is today: on the front page of every tech and mainstream news outlet in the known universe.

Does the vulnerability affect Log4j, or just Log4j?

First, a note on the two major versions of Log4j:

• 1.x: Apache Log4j, aka Log4j 1
• 2.x: Apache Log4j 2, aka Log4j

The vulnerability raised only affects Log4j 2, and this major version is mostly referred to in this article simply as Log4j. The 1.x series is distinguished here by calling it Log4j 1.

Log4j 1 reached end-of-life on 2015-08-05 and hasn’t been maintained since. And isn’t it ironic, don’t you think, that those who held on for dear life to Log4j 1 (there are many of us) are unaffected by the vulnerability. Some mitigation guides are recommending for Log4j 1 users to upgrade to Log4j 2, on the grounds that Log4j 1 has known security issues (notably CVE-2019-17571) and of course may have more lurking.

But that’s like trying to upsell punters from a 737-800 to a 737 MAX – the trust is gone and can’t easily be re-earned. A less brazen move would be to upgrade to slf4j with a logback backend.

What was the exploited feature?

Although the reported exploit hinges on specific use of JNDI, the fact that logging a message can do anything remotely interesting or surprising at all – at least, anything more interesting and surprising than appending a line to a log file – depends on a special feature of Log4j referred to here as Message Pattern Lookups.

This is essentially an extended use of the Property Substitution feature available for Log4j configuration files, documented as:

Log4j 2 supports the ability to specify tokens in the configuration as references to properties defined elsewhere. (...)

In a manner similar to Ant or Maven, this allows variables declared as ${name} to be resolved using properties declared in the configuration itself. (...)

While this is useful, there are many more places properties can originate from. To accommodate this, Log4j also supports the syntax ${prefix:name} where the prefix identifies tells Log4j that variable name should be evaluated in a specific context.

One of these prefixes is jndi, which is used in reported exploits, but there are lots of other options available, with varying and non-obvious potential for malfunction or abuse.

These options are known in Log4j as Lookups, documented as:

Lookups provide a way to add values to the Log4j configuration at arbitrary places.

Notice we’re still only talking about configuration here. The Log4j configuration files are under the control of developers and system administrators, and kept carefully isolated from any users or would-be haxx0rs.

So far, what we can see is a powerful feature that provides application developers who adopt Log4j with a way to customize their logging to include useful context, without needing to write custom code.

It’s also fairly well-documented, although it’s difficult to understand exactly how the configuration mechanism works, not least in the section Lookup Variables with Multiple Leading '$' Characters. Say for example, you actually want to include a literal ${somethingOrOther} in your log file format. You’d need to escape it somehow. The documentation doesn’t mention how. Then again, as a well-meaning commenter on Stack Overflow might reply: But why would you want to do that?

And there we have it. It's a feature for interpolating special variables in configuration files. Maybe I mentioned that already.

But wait, there’s more?

But wait, there’s more! The property substitution / Lookup feature actually doesn’t just apply to the configuration file after all. It also applies to the logged messages!

That is, if you have code in your application that logs something like this:

logger.info(someUntrustedValue);

then whoever controls the untrusted value can exploit the vulnerability.

Of course, we all know this by now, but prior to sometime in early 2021 (around 10 months prior to ‘day zero’), your best bet to ingest this juicy tidbit of information would’ve been to thoroughly read the source code. That’s because the feature was barely documented, receiving at best oblique mentions in the official Log4J manual.

There was never any mention of this feature in documentation relating to the Log4j API calls themselves. In particular, it wasn’t mentioned alongside the Substituting Parameters and Formatting Parameters features on the Log4j 2 API page. It also wasn’t mentioned in the Logger JavaDocs, which is the documentation that application developers using Log4j might be exposed to via their IDE.

What if I’m a security-conscious developer who reads all official documentation thoroughly? Am I still vulnerable?

Suppose you are a security-conscious developer, who may or may not have read the documentation on Lookups in configuration, or noticed the hints that they apply to message content too. Regardless, you have the notion that you don’t want any non-trivial parsing or processing to be applied to values from an untrusted source being passed in for logging. Probably you’re familiar with SQL injection attacks and know to use placeholder parameters, rather than string concatenation. You also know that Log4j 2 supports parameter substitution, which is one feature than Log4j 1 lacked.

You’d likely expect that by writing log entries as follows:

logger.info("Log message with some untrusted value: {}", someUntrustedValue);

instead of as:

logger.info("Log message with some untrusted value: " + someUntrustedValue);

… that you’ve done your job well and that your logging is idiomatic, performant (a real word since 2018) and most importantly, secure.

Well, you’d be wrong. Unlike the formatting parameter features that are documented (including the {} placeholder used above), the Lookup processing takes place after the parameters have been formatted into the message. Using parameter arguments is just as vulnerable as logging a concatenated string.

It’s subtle, but parameter arguments might even be more vulnerable in this case, since there’s no avenue for application code to apply even paranoid security sanitization to the logged message as a whole. It’s conceivable a malicious payload could be split across multiple parameters. Consider this:

logger.info("Just a harmless old log statement, nothing to see here: {}{}{}{}",
    sanitize(value1), sanitize(value2), sanitize(value3), sanitize(value4));

If supplied with values of: “$”, “{jnd” and “i:lda”, “p:​//example​.com}”, this has a fighting chance of getting past whatever checks are in the hypothetical ‘sanitize’ method.

No, really? Please, give a blow-by-blow account of how this came to be.

One nice thing about open source software is that the code is available on public record. Not only that, the complete version history of the source code is available from GitHub to snoop around in, along with issue ticket tracking and discussion around proposed changes on pull requests.

So even though I have no affiliation with the project, and while I’ve used Log4j (at least, Log4j 1) as a developer, I’ve never dug into the code before now, but it’s within my reach to establish a timeline of development on this feature. Here’s what I found:

2011-05-13 – Initial code commit

Let’s get this party started!

2011-10-19 – Message Pattern Lookup functionality committed

The commit message reads:

Add environment lookup and regex replacement for pattern layout

At this stage, the crucial piece of code looks like this:

/**
 * Formats a logging event to a writer.
 *
 * @param event logging event to be formatted.
 * @return The event formatted as a String.
 */
public String formatAs(final LogEvent event) {
    StringBuilder buf = new StringBuilder();
    for (PatternFormatter formatter : formatters) {
        formatter.format(event, buf);
    }
    String str = buf.toString();
    if (replace != null) {
        str = replace.format(str);
    }
    return config == null ? str : config.getSubst().replace(event, str);
}

(This was in the PatternLayout class, utilizing functionality in the StrSubstitutor class. It’s been refactored since. In recent versions, this logic is implemented in class MessagePatternConverter, particularly in its inner class LookupMessagePatternConverter.)

This method does the following:

1. Format the log message layout from configuration, to include the logged message (formatted as a single string) and other fields such as the log timestamp
2. Post-process the formatted string to apply any configured regex replacements (e.g. replace all tabs with spaces)
3. Post-process further to apply Lookup substitutions

Steps 2 and 3 were added in this commit, and it’s step 3 that lays the foundation for the vulnerability.

There are several questionable design choices being made right here, questions including:

• Why not apply Lookup substitutions just to the log format string in the configuration file, not to the message?
• Even if we really want Lookup substitutions on the message, why not apply them just to the message format, and not the message arguments (which could well be untrusted data)?
• Why do regex replacements apply to strings possibly containing Lookup placeholders, and not apply to the result of Lookup substitutions? (There are some tricky edge cases here.)

The first question is critical to the vulnerability, and from just the code above it looks like a bug. However, in the same commit there’s a unit test added that checks that Lookup substitutions do indeed apply to the message:

ThreadContext.put("MyKey", "Apache");
logger.error("This is a test for ${ctx:MyKey}");
…
assertEquals("LoggerTest This is a test for Apache\n", msgs.get(0));

This code comes from test method RegexReplacementTest.testReplacement.

This breaks from the convention of testing a class’s behavior in its corresponding unit test class. The code change is in the PatternLayout class, and while a test class PatternLayoutTest already exists, it wasn’t updated to cover the Lookup or regular expression substitutions.

This may all seem nit-picky, but the lack of an obvious unit test doesn’t help the feature’s visibility, and increases the chances of the feature breaking in the course of future code changes. It also signals that just maybe the feature wasn’t considered to be critically important.

There’s no test to cover ${…} placeholders in message arguments, or to check any cases that stray from the ‘happy path’, such as if the variable lookup fails, or if the opening ${ lacks a closing }.

2012-07-29 – log4j-core version 2.0-alpha1 released

First public release.

2013-07-18 – A Log4j user raises a feature request

LOG4J2-313: JNDI Lookup plugin support

This is where the vulnerability gets buffed from pet rock to weapons-grade.

The request was to add support for Lookups in JNDI in Log4j configuration files. This introduced the ${jndi:…} syntax used in published Log4Shell exploits.

The requester also contributed a patch to implement this, which was accepted by the Log4j team.

The use case, as discussed in the ticket, only related to configuration files. There’s no indication that the requester intended for the Lookups to apply in message text – or was even aware of this possibility.

2013-09-14 – log4j-core 2.0-beta9 released

First release with Log4Shell vulnerability.

2014-07-12 – log4j-core 2.0 released

First release out of beta.

2014-11-25 – A Log4j user encounters a problem

An early adopter using Log4j in conjunction with Apache Camel ‘discovered’ the Message Pattern Lookups feature when Camel’s logging started raising exceptions.

Camel also makes use of string interpolations such as ${some.thing}, and sometimes includes the uninterpolated string in log output. Under Log4j 1, this worked fine, but raised exceptions on Log4j 2.

This discovery is captured in the Stack Overflow question log4j2 how to disable "date:" lookup - log4j throws exception. Key quotes from the discussion:

it should be possible to log all i want - there should be no limitations, allowed/disallowed strings! Many applications use slf4j/decouple from logging implementation - but this problem is log4j2 specific i think - big problem?!

But why do you want to log the ${} pattern anyway (and not the actual time)?

there are libraries that use ${} syntax and those libraries do log such stuff. one example apache camel

Please raise this issue on the log4j2 Jira issue tracker.

And so, the very same day, an issue ticket was raised:

LOG4J2-905: Ability to disable (date) lookup completely, compatibility issues with other libraries like Camel

… and it languished awaiting a response from the Log4j team for a year and a half.

2016-05-14: In the meantime...

In the meantime, the Log4j team were working on performance optimizations, including wrapping up this ticket:

LOG4J2-1297: Document "gc-free" configuration and performance

This ticket doesn’t directly impact functionality, but it’s a milestone in the timeline because it introduces the very first hint to the documentation that Message Pattern Lookups are occurring.

And a little too ironically, it only makes an appearance to confess a downside of the feature: it breaks Log4j’s new hot #3 top selling point: Garbage-free logging

That first hint is here:

PatternLayout with the following limited set of conversion patterns is garbage-free. (...)

%m, %msg, %message - Log message (unless message text contains '${')

Got it, there’s something special about ${ in a message, that may produce garbage objects. Who’d have thunked it?

Reading on gives some clarification. In section Impact on Application Code: Autoboxing, there’s a note:

Note: not all logging is garbage free. Specifically: (...)

Logging messages containing '${': substituting a ${variable} creates temporary objects.

Aha, so that ${ is special because it invokes a variable substitution! What sort of variable would that be now? I read the whole page. The author does not say.

2016-08-22 – Discussion concludes on LOG4J2-905

... and a fix (my pinkie is itching) has been committed.

The ‘fix’ (ahh, that’s better) is to add a ‘nolookups’ option, so that where in your Log4j configuration you have a message placeholder such as %m, you can replace that with %m{nolookups}, and then Message Pattern Lookups no longer apply.

Applying this option would’ve guarded against the vulnerability – if you were fortunate enough to (for no readily apparent reason, mind you) make this configuration change to each and every log destination, for every deployment, of every application using Log4j.

This change also documented the ‘nolookups’ option, introducing the strongest hint yet that the Message Pattern Lookup feature existed – albeit obscurely, in the documentation for the Pattern Layout message placeholder:

Use {nolookups} to log messages like "${date:YYYY-MM-dd}" without using any lookups. Normally calling logger.​info("Try ${date:YYYY-MM-dd}") would replace the date template ${date:YYYY-MM-dd} with an actual date. Using nolookups disables this feature and logs the message string untouched.

The discussion on the ticket is sobering to read (you may have that drink now), and I take some liberty in paraphrasing it as:

Log4j users: This behavior is undocumented, surprising and nonstandard. It broke our logging. It could be bad and should be disabled by default.

Log4j maintainers: No, it will remain on by default. You can now disable it for each log format if you need to. That is the solution to the problem.

2016-09-29 – log4j-core 2.7 released

New release, including changes for LOG4J2-905.

2017-11-09 – Over a year and 1000 issue tickets later...

... and another crack was made at the global switch idea:

LOG4J2-2109: Add property to disable message pattern converter lookups

This change added and documented the log4j.formatMsgNoLookups property switch, which early Log4Shell advisories advised to enable, for those unable to quickly upgrade their log4j-core library version.

It also added support for the LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable, which is an alternative to the log4j.formatMsgNoLookups property. Unfortunately, due to an incomplete edit that slipped through code review, it was mis-documented as FORMAT_MESSAGES_PATTERN_DISABLE_LOOKUPS.

This documentation bug was never corrected. Fortunately, it seems no prominent mitigation advisories were later misled into recommending setting the (nonexistent and ineffective) FORMAT_MESSAGES_PATTERN_DISABLE_LOOKUPS variable.

The documentation introduced for this parameter is yet another hint that the Message Pattern Lookup feature exists, effectively naming it as ‘message pattern lookups’ but falling short of outright stating what it does. It read:

Disables message pattern lookups globally when set to true. This is equivalent to defining all message patterns using %m{nolookups}.

This ticket was apparently initiated by the Log4j team, and there’s no context recorded for what precipitated the about-face from the earlier stance on LOG4J2-905.

2017-11-19 – log4j-core 2.10.0 released

New release, including changes for LOG4J2-2109.

 

… this space left blank, to signify the passing of time …

 

2019-10-02 – Another Log4j user discovery

Another Log4j user ‘discovered’ the Message Pattern Lookup feature by being inconvenienced by it, and published a blog post on their experience:

Log4j 2 - The Ghost in the logging framework

A pertinent excerpt from the post:

Huh. Could it be that the responses from the server may actually contain something that log4j2 would recognize as a variable? A quick look at the response gives us the shocking answer. Not only once or twice but some of those SOAP responses contain more than 3000 occurrences of ${Project} or ${WorkItem}.

<ns8:queryWorkItemsReturn uri="subterra:data-service:objects:/default/ABC${WorkItem}ABC-22904" unresolvable="false">

This provoked log4j2 to parse the 2MB responses and replace the “variables” which in turn led to the server closing the connection on us in between.

To be honest, while you can argue about the format and use case, this API decided to use this format represent certain URIs. That Log4j2 is trying to actually interpret the placeholders in values passed in is the (at least to us) the surprising part.

Good for us, log4j2 already provides a flag on the %m placeholder to disable recursive variable substitution.

In other words, the behavior caused surprise, and impacted performance to the extent that it broke something.

2020-24-12 – Log4j user contributes documentation

A helpful Log4j user (whose randomly-generated GitHub avatar happens to resemble a mustachioed scorpion) contributed this pull request:

Document how to disable message pattern lookups #450

This introduced the following one-paragraph section to the documentation:

Disables Message Pattern Lookups

A message is processed (by default) by lookups, for example if you defined <Property name="foo.​bar">FOO_BAR </Property>, then logger.​info("${foo.​bar}") will output FOO_BAR instead of ${foo.​bar}. You could disable message pattern lookups globally by setting system property log4j2.formatMsgNoLookups to true, or defining message pattern using %m{nolookups}.

FOO_BAR indeed. With this change, the official documentation finally mentions that there is a Message Pattern Lookups feature, which applies the Lookups feature (already documented as applying to configuration, remember) to logged messages themselves.

Curiously, it’s a short section buried right in the middle of the page on Configuration, while the feature discussed affects calls to the logging API. There’s an excuse for that: the purpose of the documentation isn’t to explain how to use the feature, it’s to warn you it exists and let you know how to turn it off.

All up, it’s not a great piece of documentation. It doesn’t clearly reference any Log4j classes, methods or features. It’s not accurate regarding whether it applies to all logging via Log4j, or just logging via Pattern Layout. It’s a reasonable starting point, but the Log4j maintainers could’ve improved it before accepting it into their mainline.

2021-03-12 – log4j-core 2.14.1 released

This release includes the above documentation addition.

… and that was the final release prior to Log4Shell disclosure.

 

I’m going to pause the timeline here to highlight what I consider to be the most disturbing aspect of this whole story. The feature in question has been in place for a decade, and those that have commented on it have pointed out that it’s surprising, non-standard, undocumented, bad-for-performance and not useful.

But in all of the official documentation, blog posts, Stack Overflow Q&A, Jira issue ticket discussions, pull request comments, code comments, unit tests and code commit comments that I reviewed, there was not one suggestion – none whatsoever – of a security concern.

Developers – Log4j maintainers and users alike – apparently just didn’t have security on their radar. Or perhaps (and I’m speculating to be generous) it crossed someone’s mind, but they considered discussing potential security vulnerabilities to be taboo, absent a proof-of-concept exploit and early private disclosure.

Back now to the downhill stretch…

 

2021-11-24 – Apache informed of vulnerability

Presumably this includes the Log4j team specifically.

2021-11-29 – A Log4j maintainer raises a ticket

LOG4J2-3198: Message lookups should be disabled by default

The description from this ticket is worth quoting in full, because it serves as a well-written summary and endorsement, from the Log4j team themselves, of most of the issues pointed out by Log4j users in the preceding years:

Lookups in messages are confusing, and muddy the line between logging APIs and implementation. Given a particular API, there's an expectation that a particular shape of call will result in specific results. However, lookups in messages can be passed into JUL and will result in resolved output in log4j formatted output, but not any other implementations despite no direct dependency on those implementations.

There's also a cost to searching formatted message strings for particular escape sequences which define lookups. This feature is not used as far as we've been able to tell searching github and stackoverflow, so it's unnecessary for every log event in every application to burn several cpu cycles searching for the value.

Once again, there is no mention of security concerns. However, given the timing, one could reasonably infer that Log4j team is already aware of Log4Shell, and is scrambling to fix it before day zero. All the good reasons given for making the change, at this point in time, merely serve as a pretext for addressing the security issue.

This entails winding back many of the earlier decisions. Changes here include:

• Message Pattern Lookup is disabled by default.
• The ‘nolookups’ in %m{nolookups} remains valid, but has no effect
• Support for log4j2.formatMsgNoLookups is removed along with its documentation (it’s implicitly allowed, but now has no effect)
• Likewise, the never-properly-documented LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable no longer has any effect

And finally, just to keep our heads spinning:

• A new option, ‘lookups’ is introduced instead to explicitly enable the lookups, as in %m{lookups}
• The Disables Message Pattern Lookups documentation section is reworked as Enabling Message Pattern Lookups

I’d like to point out, in my best 70’s phaser voice, that for any hypothetical Log4j user who is actually depending on the Message Pattern Lookup feature, the above is already a breaking change. Why not break it good and proper, by removing the feature entirely? In a week from now, not even our hypothetical user, with their hypothetically questionable sanity, will want to enable it.

2021-12-05: Another ticket is raised by the Log4j team

LOG4J2-3201: Limit the protocols jNDI can use and restrict LDAP.

Its description, in full:

LDAP needs to be limited in the servers and classes it can access. JNDI should only support the java, ldap, and ldaps protocols by default.

That's all. It isn’t explained why this ‘needs’ to happen. In most contexts, this terse wording would suggest to me that the Log4j maintainers simply don’t feel the need to explain on public record what their motivations are for changing their software. But given the nature of the change, particularly with hindsight, it’s clear enough that there is a security concern at play.

LOG4J2-313, which introduced JNDI Lookup support, wasn’t explicit about which protocols were needed, but presumably the java, ldap and ldaps protocols were considered to be essential to ‘legitimate’ use of the feature, and the intention here is to reduce the attack surface while preserving useful functionality.

The feature is also locked down by default in terms of:

• which servers can be referenced via ldap: the local host only by default, and
• which classes can are allowed to be loaded: just Java ‘primative classes’ (sic) such as String

Typical legitimate production use is unlikely to reference an LDAP server co-hosted with the application, and may run afoul of the other restrictions, so there is an escape hatch added in the form of configuration parameters:

• allowedLdapHosts
• allowedLdapClasses
• allowedJndiProtocols

In yet another documentation SNAFU, and mounting inconvenience to anyone still intent on making use of the JNDI Lookup feature in their Log4j configuration files, these are mis-documented as allowdLdapHosts, allowdLdapClasses and allowdJndiProtocols (each is missing the ‘e’ in ‘allowed’).

Note that ldap (or ldaps) is also the main attack vector of Log4Shell – at least, the sample exploits published so far. Guidance for detecting intrusion attempts suggests looking for the following signatures:

• ${jndi:ldap:/}
• ${jndi:ldaps:/}
• ${jndi:rmi:/}
• ${jndi:dns:/}
• ${jndi:iiop:/}

Of these, the ticket by default blocks the rmi, dns and iiop protocols, which are considered possibly exploitable.

2021-12-09 – Log4Shell is publicly disclosed

The scatological hits the metaphorical.

And in the wake of the disclosure, the following ticket is raised by a Log4j user:

LOG4J2-3202: Only allow lookups in message, not in parameters.

This ticket puts a spotlight on a significant aspect of the Message Pattern Lookup feature that I’ve already mentioned: it applies to the whole formatted message, including arguments to that message. That fact isn’t critical to the vulnerability, but makes it far more widely exploitable in practice.

The Log4j team declined to address this ticket, promptly closing it as a duplicate of LOG4J-3198 with the following remark:

This is resolved in the pending 2.15.0 release. Note that we aren't supporting lookups in the message or parameters, as it's less obvious than doing the lookup in code.

We should have some sympathy for the Log4j team at this point. They may have had some sleepless nights. But neither the LOG4J-3198 ticket nor anything else in the 2.15.0 release addressed the issue raised here. The Message Pattern Lookup feature still existed, it still applied to message parameters, and this fact was still undocumented and untested.

2021-12-10 – log4j-core 2.15.0 released

Eagerly-awaited new release, including changes for LOG4J2-3198 and LOG4J2-3201.

This release provided the preferred means for mitigating the vulnerability at the time: upgrading existing deployments of the log4j-core library to version 2.15.0 or above.

2021-12-11 – With Log4j security under intense scrutiny...

... the Log4j team raises the following ticket:

LOG4J2-3208: Disable JNDI by default

The description starts out:

Dealing with CVE-2021-44228 has shown the JNDI has significant security issues. While we have mitigated what we are aware of it would be safer for users to completely disable it by default, especially since the large majority are unlikely to be using it.

This shows a willingness to be proactive and transparent in guarding against potential security concerns, which is heartening, if belated. It also acknowledges the interests of the users of the software (en masse) in deciding which features to implement and support.

The few are also looked after, with the introduction of the optional log4j2.enableJndi switch to keep the functionality enabled.

2021-12-13 – Finally, the following ticket is raised...

LOG4J2-3211: Remove support for Lookups in messages

Log a message if either the lookup or nolookup option is specified on %m. Remove calls to StrSubstitutor.replace().

And so, the software lifecycle comes full circle. The Message Pattern Lookup feature is retired, by the same hand that willed it into existence some ten years prior.

Any users who had previously applied the %m{nolookup} flag will now be shown the confusing but harmless info log message:

The nolookups option will be ignored. Message Lookups are no longer supported.

The same day, log4j-core 2.16.0 was released, including changes for LOG4J2-3208 and LOG4J-3211.

Upgrading to this release is, at time of writing, the preferred mitigation for the vulnerability.

And that brings us up to the present day!

What now?

Well, that’s anyone’s guess. My layperson’s advice is that after upgrading log4j-core to a (fingers crossed) invulnerable version, it would be a good idea to leave some time on your calendar for:

• Checking logs for intrusion attempts
• Checking application code for previously-vulnerable log calls
• Weeding hackers and malware out of compromised servers and networks
• Paying out (or not paying out) on any extortion threats from hackers who may (or may not) have infiltrated your organization
• Publicly downplaying any sensitive information leaks that said hackers may perpetrate

... and, yeah, I really do think.