Customizing WebCenter Logging with WLST

Like many enterprises, the one I’m currently working with only provides developers with the Monitor role in production. This requires scripting tasks that are well-documented for manually performing but take some real thinking and testing to get right as a script.

One of my team members frequently says that code should be self-documenting, so rather a long blog entry I thought I would simply post the script along with some minimal comments and see if everyone finds that useful:

def configureAdfLogHandlers():
#The full script has multiple methods, so I print the currently running method to follow along
	print 'Start configureAdfLogHandlers()'
	managedServers=cmo.getServers()
	for managedServer in managedServers:
		sname=managedServer.getName()
		path = '/Servers/' + sname
		cd(path) 
		lh = listLogHandlers(returnMap='true')
		for l in lh:
			lname = l.get('name')
#Even with this script, the odl logs are quite verbose and we run an automated arciving process. 
#To simplify the process, we just put the logs in the archive path since the odl log handlers don't provide an archive location option
			odlfile = '${domain.home}/logs/LOGS/${weblogic.Name}-' + lname + '.log'
			print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
			print '   Setting log handler configuration for ' + lname
			print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
#The custom code rarely uses any levels between debug and warning, and the notice level for Oracle code is too verbose for most trouble-shooting needs
			configureLogHandler(target=sname,name=lname,path=odlfile,rotationFrequency='daily',retentionPeriod='day',maxLogSize='1g',maxFileSize='100m',level='WARNING:7')
		print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
		print '   Setting logger configuration for ' + sname
		print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
#The custom code used by the application sets of warnings in these packages that are not issues for the application, so we set the level to ERROR here
		setLogLevel(logger='oracle.webcenter.page',level='ERROR:1',target=sname,persist='1',addLogger='1')
		setLogLevel(logger='oracle.webcenter.framework.service.Utility',level='ERROR:1',target=sname,persist='1',addLogger='1')
		setLogLevel(logger='org.apache.myfaces.trinidadinternal.skin.SkinCSSDocumentHandler',level='ERROR:1',target=sname,persist='1',addLogger='1')

connect(os.getenv('WLS_USER'),os.getenv('WLS_PW'),os.getenv('ADMIN_T3'))
configureAdfLogHandlers()
disconnect()
exit()

 

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

Fixing Silent Domain Builds with WebCenter Portal

Automating domain creation is important for enterprises to minimize maintenance cost and be prepared for efficient disaster recovery. One enterprise I worked with was struggling with the domains they were building through scripts as the Enterprise Manager was not working properly in the completed domains and the managed servers would come up in Admin mode.

After comparing the scripted domain to a manually created domain that worked correctly it was found that the paths for the libraries did not match. The paths in config.xml were missing the beginning of the path where the libraries were actually located. While the installation paths chosen were not the default paths, this did not entirely explain the issue, and internal processes made the custom paths highly desirable.

The correction for this issue was to add a function to the offline WLST script run on completion of the domain creation to correct the library paths:

def fixApplicationSourcePaths():
  appPathPrefix = os.getenv('DOMAIN_HOME')+'/applications/'
  cd('/AppDeployments/')
  deployed_apps = ls(returnMap='true')
  for app in deployed_apps:
    cd('/AppDeployments/'+app)
    apppath = get('SourcePath')
    if apppath.find('/app/')!=0:
      apppath = apppath.split('/')
      apppath = apppath[-1]
      set('SourcePath',appPathPrefix+apppath)
  cd('/Libraries/')
  deployed_libs = ls(returnMap='true')
  for lib in deployed_libs:
    cd('/Libraries/'+lib)
    apppath = get('SourcePath')
    if apppath.find('/app/')!=0:
      set('SourcePath',appPathPrefix+apppath)

With this change, domains were created consistently as desired.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

Fix for InvalidRegistrationException with a WLP Producer and WebCenter Portal Consumer

I have come across a case where I have deployed a WebCenter Portal EAR to a newly created domain that was copied in from another domain. When attempting to update the WLP producer registration with the new environment, I received a com.bea.wsrp.faults.InvalidRegistrationException.

I had done this sort of new domain configuration many times, with the only difference here being that the EAR I was deploying was copied from a running domain. While I don’t know the actual root cause of why it had a problem this time, I suspect that OEM is writing back to the EAR for some reason. Anyways…

After much head-banging, and successfully registering other domains to the same producer in a vain attempt to prove I was not insane, I re-read the error for the umpteenth time and finally realized what was causing the exception. Do you see it?:

<BEA-000000> <<Jun 16, 2014 3:08:28 PM PDT> <Error> <oracle.portlet.client.connection.wsrp.HTTPClientTransport> <WCS-40152> <A request to the producer URL "http://myhost5:8001/myAccount/producer/wsrp-2.0/markup" resulted in a status 500 response with fault string "The given registrationHandle [22001] is invalid.". The fault code given was "{urn:oasis:names:tc:wsrp:v2:types}InvalidRegistration". The producer generated a timestamp of 2014-06-16T15:08:28.584-07:00 and associated the following stack trace with the fault message: com.bea.wsrp.faults.InvalidRegistrationException: The given registrationHandle [22001] is invalid.

So, since it was trying to register for the first time with a handle that had not been created yet, it made sense that it would fail. The WLP schema is a bit too complex to allow just creating the one row, so instead I went back to JDeveloper, updated the registration handle there and built a new EAR. The new EAR had a new value for MDS so that when I tried for the last time to update the producer it decided to request a handle rather than insist it already had one.

Added May 20, 2015:

This still puzzled me, though. I ran across the same issue again and realized where the problem originates. The migration process being followed was to always deploy the same EAR successfully tested in a lower environment. When automated, this EAR comes from the build server. When done manually (and pressed for time) it is taken directly from the lower environment. For existing environments this has never been an issue. But it was an issue for new environments. So when it happened again, I undeployed the EAR copied from the lower environment and deployed the EAR from the build server. The error went away.

It seems that something is written back to the EAR after deployment. Not what I would expect, but at least I know that in the future it is best to always deploy the EAR from the build server rather than one that has been pushed from the build server to another environment.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

Adding Problem 3 for “23.2.1 Portlet Displays a Portlet Consumer Error”

The Troubleshooting Oracle WebCenter Portlets documentation lists two problems under the Portlet Displays a Remote Portlet Communication Error section. Both seem legitimate causes and solutions, and I have run across both of them in the past. I hit a new one this week. Number three is running out of available threads in the Managed Server that is handling the request. This one was fun to find, because OOM errors that do not cause the server to fall over eventually clear up, and with an adequately sized cluster (the memory errors were caused by a dependent service failing) the issue is very random.

The symptom to look for is the OOM error in the log, which looks like this:

oracle.portlet.client.container.PortletException: java.lang.OutOfMemoryError: 
unable to create new native thread
	at oracle.portlet.client.tech.common.PortletFutureUtils.get(Portlet
FutureUtils.java:72)
	at oracle.portlet.client.tech.common.PortletSessionManager$InitSession
CompletionListener.conditionChange(PortletSessionManager.java:340)
	at oracle.portlet.client.service.pipeline.FutureConditionTask.done
(FutureConditionTask.java:168)
	at java.util.concurrent.FutureTask$Sync.innerSetException(Future
Task.java:273)
	at java.util.concurrent.FutureTask.setException(FutureTask.java:125)

Restarting the servers will give you temporary relief, though for the long-run you need to find the root cause. In this case, it was an undersized WebCenter Content cluster running on the same physical box sucking all the resources down. A larger cluster and load balancing corrected the issue for good (knock wood).

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

Leveling the Playing Field

(semi-continuation of A Tale of Two Migrations)

Before I get into the trials and tribulations of dealing with the inevitable custom implementations that are part of every WebLogic Portal application that is older than a year and has more than 5 pages (and there are newer, smaller apps with more customizations, trust me on that), it is best to look at some basics. In the path taken by this blog series we are using WSRP with the retiring WebLogic Portal (WLP) application as the producer and WebCenter Portal (WCP) as the migration target and WSRP consumer. It is definitely of value to review the standard documentation about how to configure this standards-based integration, which can be found for 10.3.6 at http://docs.oracle.com/cd/E41236_01/wlp.1036/e14235/chap_webcenter_interop.htm/ .

With all of the useful points provided in the documentation, the one point that I would like to add is to first start with upgrading your WebLogic Portal application and environment to the latest version. I am a strong proponent of not upgrading unless there is a clear value in doing so, and in this case there are several. One is that a migration from WLP to WCP is almost definitely going to require at least one (and most likely several) Oracle Support cases to complete. You will hear every time you open a case “please upgrade to the latest version”. I’m not a big fan of this approach, but since the goal of the project is to move everything to the WebCenter and all of the functionality will eventually need to be updated to the latest version as part of that process, it is worthwhile to get that out of the way now.

Another value of upgrading your WebLogic Portal before embarking on a staged migration is that with one of the release of WebLogic Portal 10.3.x from .2 or later (I didn’t know the specific version, am too lazy to go look it up, and don’t feel bad about that because it is moot at this point in time) and the corresponding WebCenter Portal release (see previous parenthetical comment) the WSRP engine was moved from the portal framework to the application server (WebLogic Server). This makes a great deal of sense since WebLogic Server has had a WSRP engine for quite some time (again, not doing the historical research for the curious). Since the underlying WSRP support is coming from the application server, it makes good sense to have the same engine on both sides of the tractor-pull known as WSRP. While it may not be entirely necessary between some versions (I strongly suspect –but have no proof- that this is the case between 10.3.5 and 10.3.6), it eliminates one point in the long chain that will eventually be examined in detail for the source of some unexpected result or other, so why not get it out of the way now?

Finally, WebLogic Server is still the gold standard of J2EE application servers and does continuously improve with each release (disclaimer: I have not worked with the 12c versions and am just hoping that they have not proven me wrong). So why not start off with the best foundation possible before dealing with the known risk of every migration which is that there will be unknown and unforeseen issues that arise.

In my case I pushed for the WebLogic Portal and Server upgrade to 10.3.6 because WebCenter 11.1.1.8 had just been released with its capability of including the MAR within the EAR, a major time and hassle saver. WebCenter 11.1.1.8 was release with no support for WebLogic Server 10.3.5, which explains the “coincidence” of WebLogic Portal 10.3.6 coming out two weeks before hand. Interestingly enough, development on WCP 11.1.1.8 is done with JDeveloper 11.1.1.7 which is only configured to work with WebLogic Server 10.3.5. Ah, if Fox Mulder were only a software engineer instead of an FBI agent 😉

Facebooktwitterredditlinkedinmail
© Scott S. Nelson