Correcting WLS Console Hang with SSH Node Manager Environment

I was having a frustrating situation yesterday where the WLS console would just hang after clicking the Servers link. Naturally I go to the Unix console to tail the logs and see what’s what, and these prompts for a password keep popping up. Entering the admin password is no help.

So I went to the next natural step of rebuilding the domain (the build is scripted, and yes, I backed up the domain first) and the problem is still there!

An email to a colleague more steeped in the dark arts of server administration than I results in the suggestion that this is due to a problem with SSH (the domain uses SSH Node Manager) and tells me test it with:

ssh username@hostname

to confirm connectivity and permissions.  What I get back is:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Permissions 0755 for '/app/home/username/.ssh/id_rsa' are too open.

It is required that your private key files are NOT accessible by others.

This private key will be ignored.

bad permissions: ignore key: /app/home/username/.ssh/id_rsa

Password:

With this new detail in mind, I went to an environment that was working, found it had the file permission set at 600, and preforming

chmod 600 /app/home/username/.ssh/id_rsa

Got me up and running again

If you found this interesting, please share.

Fix for InvalidRegistrationException with a WLP Producer and WebCenter Portal Consumer

I have come across a case where I have deployed a WebCenter Portal EAR to a newly created domain that was copied in from another domain. When attempting to update the WLP producer registration with the new environment, I received a com.bea.wsrp.faults.InvalidRegistrationException.

I had done this sort of new domain configuration many times, with the only difference here being that the EAR I was deploying was copied from a running domain. While I don’t know the actual root cause of why it had a problem this time, I suspect that OEM is writing back to the EAR for some reason. Anyways…

After much head-banging, and successfully registering other domains to the same producer in a vain attempt to prove I was not insane, I re-read the error for the umpteenth time and finally realized what was causing the exception. Do you see it?:

<BEA-000000> <<Jun 16, 2014 3:08:28 PM PDT> <Error> <oracle.portlet.client.connection.wsrp.HTTPClientTransport> <WCS-40152> <A request to the producer URL "http://myhost5:8001/myAccount/producer/wsrp-2.0/markup" resulted in a status 500 response with fault string "The given registrationHandle [22001] is invalid.". The fault code given was "{urn:oasis:names:tc:wsrp:v2:types}InvalidRegistration". The producer generated a timestamp of 2014-06-16T15:08:28.584-07:00 and associated the following stack trace with the fault message: com.bea.wsrp.faults.InvalidRegistrationException: The given registrationHandle [22001] is invalid.

So, since it was trying to register for the first time with a handle that had not been created yet, it made sense that it would fail. The WLP schema is a bit too complex to allow just creating the one row, so instead I went back to JDeveloper, updated the registration handle there and built a new EAR. The new EAR had a new value for MDS so that when I tried for the last time to update the producer it decided to request a handle rather than insist it already had one.

Added May 20, 2015:

This still puzzled me, though. I ran across the same issue again and realized where the problem originates. The migration process being followed was to always deploy the same EAR successfully tested in a lower environment. When automated, this EAR comes from the build server. When done manually (and pressed for time) it is taken directly from the lower environment. For existing environments this has never been an issue. But it was an issue for new environments. So when it happened again, I undeployed the EAR copied from the lower environment and deployed the EAR from the build server. The error went away.

It seems that something is written back to the EAR after deployment. Not what I would expect, but at least I know that in the future it is best to always deploy the EAR from the build server rather than one that has been pushed from the build server to another environment.

If you found this interesting, please share.

Adding Problem 3 for “23.2.1 Portlet Displays a Portlet Consumer Error”

The Troubleshooting Oracle WebCenter Portlets documentation lists two problems under the Portlet Displays a Remote Portlet Communication Error section. Both seem legitimate causes and solutions, and I have run across both of them in the past. I hit a new one this week. Number three is running out of available threads in the Managed Server that is handling the request. This one was fun to find, because OOM errors that do not cause the server to fall over eventually clear up, and with an adequately sized cluster (the memory errors were caused by a dependent service failing) the issue is very random.

The symptom to look for is the OOM error in the log, which looks like this:

oracle.portlet.client.container.PortletException: java.lang.OutOfMemoryError: 
unable to create new native thread
	at oracle.portlet.client.tech.common.PortletFutureUtils.get(Portlet
FutureUtils.java:72)
	at oracle.portlet.client.tech.common.PortletSessionManager$InitSession
CompletionListener.conditionChange(PortletSessionManager.java:340)
	at oracle.portlet.client.service.pipeline.FutureConditionTask.done
(FutureConditionTask.java:168)
	at java.util.concurrent.FutureTask$Sync.innerSetException(Future
Task.java:273)
	at java.util.concurrent.FutureTask.setException(FutureTask.java:125)

Restarting the servers will give you temporary relief, though for the long-run you need to find the root cause. In this case, it was an undersized WebCenter Content cluster running on the same physical box sucking all the resources down. A larger cluster and load balancing corrected the issue for good (knock wood).

If you found this interesting, please share.

Leveling the Playing Field

(semi-continuation of A Tale of Two Migrations)

Before I get into the trials and tribulations of dealing with the inevitable custom implementations that are part of every WebLogic Portal application that is older than a year and has more than 5 pages (and there are newer, smaller apps with more customizations, trust me on that), it is best to look at some basics. In the path taken by this blog series we are using WSRP with the retiring WebLogic Portal (WLP) application as the producer and WebCenter Portal (WCP) as the migration target and WSRP consumer. It is definitely of value to review the standard documentation about how to configure this standards-based integration, which can be found for 10.3.6 at http://docs.oracle.com/cd/E41236_01/wlp.1036/e14235/chap_webcenter_interop.htm/ .

With all of the useful points provided in the documentation, the one point that I would like to add is to first start with upgrading your WebLogic Portal application and environment to the latest version. I am a strong proponent of not upgrading unless there is a clear value in doing so, and in this case there are several. One is that a migration from WLP to WCP is almost definitely going to require at least one (and most likely several) Oracle Support cases to complete. You will hear every time you open a case “please upgrade to the latest version”. I’m not a big fan of this approach, but since the goal of the project is to move everything to the WebCenter and all of the functionality will eventually need to be updated to the latest version as part of that process, it is worthwhile to get that out of the way now.

Another value of upgrading your WebLogic Portal before embarking on a staged migration is that with one of the release of WebLogic Portal 10.3.x from .2 or later (I didn’t know the specific version, am too lazy to go look it up, and don’t feel bad about that because it is moot at this point in time) and the corresponding WebCenter Portal release (see previous parenthetical comment) the WSRP engine was moved from the portal framework to the application server (WebLogic Server). This makes a great deal of sense since WebLogic Server has had a WSRP engine for quite some time (again, not doing the historical research for the curious). Since the underlying WSRP support is coming from the application server, it makes good sense to have the same engine on both sides of the tractor-pull known as WSRP. While it may not be entirely necessary between some versions (I strongly suspect –but have no proof- that this is the case between 10.3.5 and 10.3.6), it eliminates one point in the long chain that will eventually be examined in detail for the source of some unexpected result or other, so why not get it out of the way now?

Finally, WebLogic Server is still the gold standard of J2EE application servers and does continuously improve with each release (disclaimer: I have not worked with the 12c versions and am just hoping that they have not proven me wrong). So why not start off with the best foundation possible before dealing with the known risk of every migration which is that there will be unknown and unforeseen issues that arise.

In my case I pushed for the WebLogic Portal and Server upgrade to 10.3.6 because WebCenter 11.1.1.8 had just been released with its capability of including the MAR within the EAR, a major time and hassle saver. WebCenter 11.1.1.8 was release with no support for WebLogic Server 10.3.5, which explains the “coincidence” of WebLogic Portal 10.3.6 coming out two weeks before hand. Interestingly enough, development on WCP 11.1.1.8 is done with JDeveloper 11.1.1.7 which is only configured to work with WebLogic Server 10.3.5. Ah, if Fox Mulder were only a software engineer instead of an FBI agent 😉

If you found this interesting, please share.

A Tale of Two Migrations

My first portal migration project was not originally expected to be a migration; it was scoped and planned as an upgrade. The expectation of an upgrade was well-founded given the high-level scope of moving from one version to the next. Those involved with WebLogic Portal at the time know the move from 7.x to 8.x was in fact a migration rather than an upgrade. While there were some bridge APIs to allow the use of old code it was poorly suited to anything that was even slightly customized, which is to say every implementation in production. The project had been expected to take 4 – 6 weeks with one developer and actually took 10 weeks and 2 developers. It was the first of two migration/upgrades for the same customer and the lessons learned from the first move allowed the second move to be completed in 3 weeks with a single developer.

Fast-forward a decade and enterprises with WebLogic Portal are once again faced with a migration. This time we know up front that it will be a migration and IT managers along with their business customers are hesitant to move forward without a clear path to follow. While the last ten years have seen all J2EE-based portal products move to standard APIs for their foundation, they still have unique frameworks that serve as (from the vendors point-of-view) market differentiator and (from the customers perspective) a new form of vendor lock-in. In the case of WebLogic Portal, the vendor remains the same (albeit through acquisition) yet there is still no easy way to switch to the newer version, and a risk to staying with the old as it has been feature-frozen since 2010 and is slated for “sustaining-only support”.

The migration path from WebLogic Portal (WLP) to WebCenter Portal (WCP) is unclear for many reasons. Perhaps the most common and least-realized barrier to an “automated” migration is that WLP in his current form has 10 years of layered patches and deprecated-yet-still-in-production APIs supporting features that are a broad amalgam of proprietary solutions and customized integrations with standard APIs fueled by a demanding customer base for a (relatively) small vendor where WCP’s current form is currently a little over 3 years old and managed by a company that believes in setting the standards and letting the market follow.

Another source of great confusion is the mixed messages over time and between messengers as to whether a staged migration is practical or if it must be done in either a parallel phase out or big bang approach. To make that even more confusing, all of those options are equally the “best” approach, though the true determination of the correct approach is specific to a combination of the enterprise time-to-market needs, development skill sets, ability to maintain multiple environments and maturity of the enterprise architecture.

Given all these contributors to the fear, uncertainty and doubt surrounding a move off of a long-lived portal platform to a new, unfamiliar technology landscape I am currently in the process of acting as guide down this winding path for a company where all the various factors pointed mostly in the direction of a staged migration.

Like that first move to WLP 8.x, the approach from the 20,000 foot elevation looked very simple and straight-forward. Build the header, footer and navigation components in WebCenter and consume all of the legacy WebLogic Portal-produced portlets over WSRP. Where the WLP application had a backing file at the desktop-level that assembled user and account information at log-in, the backing file would be modified for use at the portlet level and the rest would be easy-peasy. This original understanding was entirely correct except the last phrase, i.e., no easy-peasy.

To be continued…

References for Historical Release Dates

http://en.wikipedia.org/wiki/Oracle_WebLogic_Server

http://en.wikipedia.org/wiki/Oracle_WebCenter

If you found this interesting, please share.