Patching 11.2.8-11.2.16 EPM with Multiple Foundation Server Errors

Many of our clients are upgrading to Oracle EPM 11.2.15+ these days, and some of these installations are from pre-11.2.14 environments. As part of the upgrade-in-place process, we will patch from 11.2.8, for instance, to 11.2.14, then run the upgrade-in-place from 11.2.14 to 11.2.15+. It is a fairly easy process, but has one annoying issue – the patching process from 11.2.8 to 11.2.14 will break Shared Services under certain conditions, as will the patching from 11.2.15 to 11.2.16. Let me explain.

In a distributed environment, with two FoundationServices deployments on separate servers, each is responsible for running Workspace and Shared Services. Generally, the first, FoundationServices0, is also the host for the WebLogic Admin instance. The 11.2.14 instructions direct us to run the ConfigTool utility to re-deploy FoundationServices. As the documentation does not state which server to perform this activity on, we do so for both deployments. The deployment appears to go as planned, all services restart normally. We clear our browser cache to ensure no old code is being kept, and navigate in our browser to Workspace, then to Navigate>Administer>Shared Services. OHS will randomly direct us to either FoundationServices0 or FoundationServices1. If we end up on FoundationServices0, everything looks normal. If we are directed to the second server’s FoundationServices1 JVM, we encounter errors like this:

Clicking “OK” shows another error:

Clicking “OK” shows another error:

Clicking “OK” again returns us to our Workspace HomePage.

To test for this in your environment, stop the FoundationServices0 service on your “Server1”, and ensure FoundationServices1 is running on your “Server2”. You will likely see these errors. Stopping FoundationServices1 and starting FoundationServices0 again, after waiting for the JVM to fully start, will behave as normal when you try to access Shared Services.

Running the validate.bat script on “Server2” shows errors for the CES, LCM, Registry and Web Application tests in my case because the round-robin of OHS pointed the validation URLs to FoundationServices1’s server. If it did not, it would only have showed failure for the Web Application test, the only one of the 4 that goes directly against the local server AND local port by default:

The fix for the issue is simple: copy the WebLogic Admin server’s “config.xml” from the EPMSystem domain’s config directory, in my case it is located at Oracle\Middleware\user_projects\domains\EPMSystem\config . Copy this, in my case, from Server1 to Server2, renaming the original in case you need to refer to it.

Now, make sure FoundationServices0 is stopped on Server1, and start FoundationServices1 on Server2. After it fully starts, clear your browser cache for good measure, and try to get to Shared Services again. No errors!

Now, the cause of the error is interesting! There is a reference to something called “Struts”, that is present in config.xml on Server2, but is slightly different on Server1, our WebLogic Admin server. The entry on Server2 is:

  <library>

    <name>struts#2.5@2.5.22</name>

    <target>CalcMgr,FoundationServices</target>

    <module-type>war</module-type>

    <source-path>D:/Oracle/Middleware/EPMSystem11R1/common/misc/11.1.2.0/struts.war</source-path>

    <security-dd-model>DDOnly</security-dd-model>

    <staging-mode>nostage</staging-mode>

  </library>

The entry in our WebLogic Admin server’s config.xml is:

   <library>

    <name>struts#2.5@2.5.30</name>

    <target>CalcMgr,FoundationServices</target>

    <source-path>D:/Oracle/Middleware/EPMSystem11R1/common/misc/11.1.2.0/struts.war</source-path>

    <security-dd-model>DDOnly</security-dd-model>

    <staging-mode>nostage</staging-mode>

As you can see, the version number changed but did not update on the second server’s config.xml file. I was able to validate that the config.xml file was modified on Server2 by comparing the pre-configuration and post-configuration files I’d saved at each step while researching this issue. Pre-config and post-config, the struts versions remain identical, while other changes to the file were observed. Per Oracle Support Doc 2925903.1 (Error Accessing Shared Services Console: An Error occurred processing the result from the server. Description: Invalid or could not find the module configuration "cas.containers.tadpole”) this issue is due to WL Admin Server not running while FoundationServices1 is being deployed. I disagree, WL Admin Server was in a “RUNNING” state on Server1 when Server2’s deployment was started. There are no firewalls or other blocking software in place between these two servers in my lab environment. We have observed the behavior occurring at two separate client engagements in multiple environments as well.

During my research, I came across a document from Oracle Support that includes links to updating EPM versions 11.2.12-11.2.15 to a newer version of Struts due to a reported vulnerability (CVE-2023-50164). The document is at https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=314082797801137&id=2997871.1 though I caution anyone trying to apply it, that 11.2.16 patch bundle will require the patch be reversed before applying 11.2.16.

Applying the 11.2.15 upgrade-in-place doesn’t change the Struts version in config.xml after deployment. Applying the 11.2.16 patch does change the version to 2.5.33 on Server1, but the instructions for patching say to only re-deploy Foundation Services on the primary (WL Admin) server, not to both. What happens if I follow those instructions, will config.xml on Server2 update when I start FoundationServices1? I started WL Admin on Server1, then started FoundationServices1, and checked the value for Struts in config.xml:

Turning off FoundationServices0 and validating:

Again, FoundationServices1 is not picking up changes to Server1’s config.xml during startup. Once again, the solution was to replace the config.xml file on Server2 with the updated version from Server1. The last permutation to test: reverting the Server2 config.xml, then running configtool to re-deploy FoundationServices1, and see if it updates the file.

Unfortunately, it does not. Same issues, same resolution – copy the file from Server1 to Server2 and restart the service.

It seems this issue is not as straight-forward as Oracle has communicated, as WL Admin server was running during the deployment steps.