WebLogic Admin Server Woes
Recently, I was working on a DB password update for EPM, in a multi-domain environment, and things did not go as planned. What follows is the abbreviated version of an all-day adventure, with some guidelines and steps to help save you time if you are ever in a similar situation.
To set the stage, it is important to understand that in an EPM 11.2.x mixed OS environment, you will have at least 2 WebLogic domains, and therefore two WebLogic Admin servers.
In the scenario I encountered, the first environment was Linux, where the non-HFM components were managed. The products in that domain were Foundation Services, Calculation Manager, Financial Reporting and Financial Data Quality Management Enterprise Edition (FDMEE). Because the environment also requires Hyperion Financial Management (HFM) and that is not available on Linux in the current version, there was also an HFM specific WebLogic domain and Admin Server on Windows.
So, with that said, the process for a DB password update is to update the WebLogic Admin console data sources, and then to run the EPM Configuration Utility to run the DB Configuration for each product (keeping the current DBs). One item to note is the configuration should be run on each server, since if the Foundation Services password is changing, each server needs the connection to the EPM Registry updated.
The issue that arose was around the Windows domain for WebLogic. Interestingly, the process worked as designed on the Linux WebLogic domain, and Admin Server started and stopped without issue. The Windows WebLogic Admin server failed to start completely, and would crash each time with errors similar to: Com.rsa.jsafe.JSAFE_PaddingException: Invalid Padding .
Doing a good deal of research, it was determined that WebLogic Admin server had corrupted values in the Config.xml, and was thus out of synch with the Boot.properties file. The fix to resolve this requires you to replace the values of the following passwords in the config.xml:
<credential-encrypted><PLAIN TEXT PASSWORD></credential-encrypted>
The config.xml in my example had 2 values that were encrypted, one for ‘myrealm’ and one for ‘EPMSystem,’ which represents the embedded LDAP for that Admin server.
Now, the challenge here is ‘what’ value goes in that field. Well, in this case it was two different passwords which needs to be placed in these fields. The first instance of this is for the ‘myrealm’ and this represents the same password as you are using for the WebLogic Admin console user (in our case epm_admin’s password). The second instance is the ‘EPMSystem’ value, and that password is the same as the Foundation Services DB password.
Once that is done and the Config.xml is updated, you still must set the WebLogic Admin server to run in Development and not Production mode. This is because you cannot start a WebLogic Admin server in Production mode with clear-text passwords in the Config.xml file. The question is where does one make that change? There is one setting in the Config.xml, but that was not sufficient, and it turns out it is also found in the SetDomainEnv.bat file. If you search those files for ‘Production Mode’ you can set both to False.
Once those steps are all complete, you are able to now start your Admin Server in Development mode, and login to the WebLogic admin console. Once there, you can update the same fields through the console, and this will save the values back to the Config.xml file, in the proper encrypted state. Then you stop WebLogic Admin server, and put the files updated above back to Production mode, and restart WebLogic Admin server again….and this will fix your broken WebLogic Admin Server.
This was a fun issue to solve. As always it is a pleasure to share these tips and tricks with you. If you have any EPM, Cloud or general IT infrastructure assistance, reach out to iArch Solutions today.