Problem:

The WebLogic Administration Server does not start with the following exception in stdout logs:

<Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: There are 1 nested errors: weblogic.management.ManagementException: [Management:141266]Parsing Failure in config.xml: javax.xml.namespace.QName; local class incompatible: stream classdesc serialVersionUID = -9120448469326609940, local class serialVersionUID = 4418622981286495151

Background & Analysis:

This is a known problem for Oracle. The problem is actually with the JRockit 1.5.0_07 version (and later) used with WebLogic Server 9.1 or 9.2, in which there is an incompatible change of the serialVersionUID of the javax.xml.namespace.QName class. Refer Known Problems.

Solution:

There are two solutions to eliminate the error. You may use any one of the following:

SOLUTION 1: Set a system property as follows:

com.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0

SOLUTION 2: Add the following option to the java command in the script you use to start WebLogic Server:

-Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0

Root Cause:

Incompatibility of JRockit 1.5.0_07 (or later) with WebLogic Server 9.1/9.2

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

Several occurrences of the following seen in the WebLogic Server 9.2 MP2 access logs:

“HEAD /bea_wls_internal/WLDummyInitJVMIDs HTTP/1.0" 404 1214

Background & Analysis:

If the WebLogic proxy plugin is used in a Web Server to load balance requests across WebLogic servers in a  cluster, then the plugin sends a dummy request to the cluster in order to retrieve the cluster server list. The cluster server list is present in the HTTP Response Header and not in the body. So, it does not matter if a non-existent web application is requested. That’s why a request is made to a non-existent web application/context called /bea_wls_internal/WLDummyInitJVMIDs, thereby resulting in a HTTP 404 error, even though the objective of obtaining the cluster server list is achieved. Well, Oracle could have handled this functionality better by internally deploying a dummy application to prevent these HTTP 404 errors

Solution:

There are two solutions to get rid of the HTTP 404 errors:

SOLUTION 1: Configure a default web application which does nothing (dummy).

SOLUTION 2: Obtain patch CR271671 (for 9.2 MP2) from Oracle to suppress HTTP 404 errors in access logs if you do not wish to configure a default web application. If you’re using another version of WebLogic Server, request Oracle for a suitable patch similar to CR271671.

Root Cause:

/bea_wls_internal/WLDummyInitJVMIDs is a non-existent web application since the WebLogic proxy plugin obtains required information from HTTP Response Headers alone.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

WLjvm will display details comprising the JVM version and options for a specified WebLogic Server. The displayed details are well-formatted and easy to read. The only parameter that needs to be passed to WLjvm is the name of the WebLogic Server whose JVM details you wish to obtain.

Why WLjvm?: At times, it’s required to know a few details about the WebLogic Server JVMs, like their versions and startup options. Well, you could use "ps" on Linux or "/usr/ucb/ps" or "pargs" on Solaris to give you this information. However, these UNIX utilities display information in a format which isn’t too pleasing to the eye (although "pargs" displays well-formatted details, you still need to use "ps" to determine the PID to use "pargs"). My intention is not to make WLjvm an overkill or make people lazy. It’s just that I observed many folks wanting to access JVM details quite regularly and so used a very simple script (and an alias to call it) to automate this task.

System Requirements:  Solaris/Linux, Korn Shell (/bin/ksh), WebLogic Server (all versions to date)

Download WLjvm v1.0     Download WLjvm v1.0 ReadMe

WLfdmon is a simple korn shell script that will enable you monitor open file descriptors used by WebLogic Servers. The salient features of WLfdmon are given below:

• Can be run in interactive and non-interactive modes
• Configuration-driven
• Generates statistics on open fd usage of WebLogic Servers. These Statistics can be used for trend analysis.
• Logs alarms when open fd threshold is breached. Alarms can notify Support Staff of abnormal application/server behaviour.
• Logs lsof output when open fd threshold is breached. This output will be useful for root cause analysis of excessive open fd usage.
• Housekeeping for data and lsof output files

System Requirements:  Solaris/Linux, Korn Shell (/bin/ksh), lsof

Download WLfdmon v1.0     Download WLfdmon v1.0 ReadMe

NOTE:

(1) WLfdmon has been scanned and is safe to download.

(2) WLfdmon has been tested. If you wish to use WLfdmon, I recommend you also test it before deploying it on a Production system.

(3) Your rating of this post/software will be greatly appreciated. Also, feel free to leave comments.

Problem:

The WebLogic Server does not start. Following errors seen in stdout/stderr/server logs:

***************************************************************************
The WebLogic Server did not start up properly.
Exception raised: ‘weblogic.server.ServerLifecycleException: Server failed to bind to the requested port. See preceeding log message for details.’
Reason: Server failed to bind to the requested port. See preceeding log message for details.
***************************************************************************

Background & Analysis:

In order for a software process to listen at a port on a specific IP address, there should not be any other process already listening to that same port and IP address.

Solution:

 (1) Obtain details of the IP address (ListenAddress) and port(ListenPort) at which your WebLogic Server is supposed to listen.

(2) If the host bearing the IP address is unreachable, then contact your Network/System Administrators to ensure that the IP address is associated with a network interface card on the appropriate host.

(3) If the IP address is reachable, use the following commands on the host bearing the IP address (obtained in above step) to determine whether a process is already bound (listening) at the same port and IP address:

On Windows Systems (DOS prompt):

netstat –an | findstr “<IP address>:<port>.*LISTEN”

On UNIX Systems:

netstat –an | grep “<IP address>[\.:]<port>.*LISTEN”

If no software process is listening at the specified IP address and port, then the above commands should not display any output.

Examples:

(a) IP:Port not bound: For a Windows host on which no process listens at port 777 on 110.120.130.140,

C:\netstat -an | findstr "110.120.130.140:777.*LISTEN"

C:\

(b) IP:Port already bound: For a Solaris host on which a WebLogic Server is listening at port 777 on 110.120.130.140,

$ netstat -an | grep "110.120.130.140[\.:]777.*LISTEN"
110.120.130.140.777        *.*               0      0 131072      0 LISTEN

(4) If step (3) indicates that a process is already listening at the port and IP address at which you want your WebLogic Server to listen, then that process needs to be killed (if it’s not meant to be listening there).

(5) After ensuring that no other process is listening at the WebLogic Server ListenAddress and ListenPort, start the WebLogic Server.

NOTE:

• To obtain the IP address for a domain name, use nslookup (other utilities also available).
• If your WebLogic Servers are running within a Veritas Cluster Server (VCS) High Availability framework, then the IP address for your servers will be floating and you must execute the above commands on all hosts within the Veritas cluster.
• This problem and suggested commands are not specific to WebLogic Server processes, but are applicable to all software processes.

Root Cause:

For the problem I experienced, the IP address of the host on which a WebLogic Server was supposed to listen, went missing (error in VCS config caused the IP address to drop off the network interfaces on all the hosts within the VCS cluster) and consequently, was not reachable.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.