Problem:

When trying to install the WebLogic 8.1 SP6 binary (platform816_linux32.bin) on 64-bit Ubuntu 9.04 Desktop, the following errors were observed:

oracle@mrkips-laptop: ./platform816_linux32.bin
oracle@mrkips-laptop: ./platform816_linux32.bin: No such file or directory

Background & Analysis:

The first couple of checks (obvious) for such an error are:

(1) Check if the file exists in the appropriate location

(2) Check if the file has the required permissions (read/execute)

The above checks were successful and so the error was misleading.

(3) Using the file command, the following was observed:

oracle@mrkips-laptop:/mrkips/oracle$ file platform816_linux32.bin 
platform816_linux32.bin: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 
dynamically linked (uses shared libs), for GNU/Linux 2.0.0, stripped

The above output indicates that the WebLogic installer binary is 32-bit and uses 32-bit shared libraries.

(4) The host OS was 64-bit Ubuntu and this could be confirmed using the uname command:

oracle@mrkips-laptop:/mrkips/oracle$ uname -a
Linux mrkips-laptop 2.6.28-13-generic #45-Ubuntu SMP Tue
 Jun 30 22:12:12 UTC 2009 x86_64 GNU/Linux

32-bit shared libraries are not available by default on 64-bit Ubuntu.

Solution:

Get the 32-bit shared library package for use on amd64 and ia64 systems, ia32-libs, using any one of the following methods:

sudo apt-get install ia32-libs

               OR

Launch the Synaptic Package Manager from the Ubuntu Desktop and install the ia32-libs package along with any required dependencies.

Root Cause:

The 32-bit shared libraries required for the installation of the 32-bit WebLogic binary installer were not available by default on 64-bit Ubuntu 9.04 Desktop.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Background:

When developing scripts, it is sometimes necessary to determine the absolute location of the script within the script, so that relative paths may be defined. For example, if a script writes logs and data files to different directories relative to the location of the script, you need to determine the location of the script, irrespective of where the script is installed or from where it’s executed. Of course, you can hard code this value in the script, but that’s a dirty solution because you’ll need to change the variable whenever you change the location of the script.

Implementation:

(1) UNIX Shell:

1
2
3
4
5
6
7
8
9

if [ -n "`dirname $0 | grep '^/'`" ]; then
   SCRIPT_LOCATION=`dirname $0`
elif [ -n "`dirname $0 | grep '^..'`" ]; then
     cd `dirname $0`
     SCRIPT_LOCATION=$PWD
     cd – > /dev/null
else
     SCRIPT_LOCATION=`echo ${PWD}/\`dirname $0\` | sed ’s#\/\.$##g’`
fi

(2) Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import sys
import os.path
from os import *
dirloc = os.path.dirname(sys.argv[0])
curdir = os.getcwd()
if dirloc.startswith("/"):
   SCRIPT_LOCATION = dirloc
elif dirloc.startswith(".."):
     os.chdir(dirloc)
     SCRIPT_LOCATION =  os.getcwd()
     os.chdir(curdir)
else:
     _SCRIPT_LOCATION = curdir + "/" + dirloc
     SCRIPT_LOCATION = _SCRIPT_LOCATION.rstrip("/.")
print SCRIPT_LOCATION

Both implementations above are based on the assumption that a script may be executed in any of the following 3 ways only:

(1) From anywhere within the directory hierarchy , using the absolute path (beginning with ‘/’)

(2) From anywhere below the directory containing the script, within the directory hierarchy (beginning with ‘../’)

(3) From anywhere above the directory containing the script, within the directory hierarchy (beginning with ‘./’)

NOTE:

(2) Your rating of this post will be much appreciated. 

Problem:

A Java application running on a WebLogic Server uses Messaging Bridges to send messages to JMS destinations on a remote server. The target destinations were configured with the domain name of the remote server. When the remote server changed its IP address and modified DNS entries to map its domain name to the new IP address, the Messaging Bridges could no longer connect to the Remote Server.

Background & Analysis:

When a Java application looks up a domain name, the result of the domain name resolution is cached in the java.net.InetAddress class. By default, for Sun Hotspot JVM 1.5.x and earlier, a successful domain name resolution is cached forever (yikes!) and an unsuccessful domain name resolution is cached for 10 seconds. By default, for Sun Hotspot JVM 1.6.x, a successful domain name resolution is cached for 30 seconds (huh, why even bother caching positives only for a few seconds?) and an unsuccessful domain name resolution is cached for 10 seconds.

Which parameters control domain name resolution caching?

Either of the following sets of parameters control caching for successful (+ve) and unsuccessful (-ve) domain name resolutions are:

1. networkaddress.cache.ttl and networkaddress.cache.negative.ttl

OR

2. sun.net.inetaddr.ttl and sun.net.inetaddr.negative.ttl

The above parameters represent the time-to-live (expiry period) for java.net.InetAddress cache entries, in seconds. Parameters in (1) are recommended, as parameters in (2) are Sun proprietary parameters and may be removed in future releases.

How to configure the domain name resolution caching parameters?

The networkaddress.cache.ttl and networkaddress.cache.negative.ttl parameters can be configured by setting their values in the ${JAVA_HOME}/jre/lib/security/java.security file.

The sun.net.inetaddr.ttl and sun.net.inetaddr.negative.ttl can be configured by passing them as command-line options to the startup command for the JVM (-Dsun.net.inetaddr.ttl=30)

Sample Code to determine values (especially useful for determining default values):

1
2
3
4
5
6
7
8
9
10

public class DNScacheTTL
{
  public static void main(String[] args)
   {
     System.out.println(”networkaddress.cache.ttl = 
sun.net.inetaddr.ttl = “+sun.net.InetAddressCachePolicy.get());
     System.out.println(”networkaddress.cache.negative.ttl 
= sun.net.inetaddr.negative.ttl = “+sun.net.InetAddressCachePolicy.getNegative());
   }
}

The above code will throw warnings when compiled with certain JVMs because it uses Sun’s proprietary methods. You may ignore these warnings.

Examples: Code above inserted in DNSCacheTTL.java, compiled and then executed as follows on Sun JVM 1.6:

java DNScacheTTL

networkaddress.cache.ttl = sun.net.inetaddr.ttl = 30

networkaddress.cache.negative.ttl = sun.net.inetaddr.negative.ttl = 10

java -Dsun.net.inetaddr.ttl=100 DNScacheTTL

networkaddress.cache.ttl = sun.net.inetaddr.ttl = 100

networkaddress.cache.negative.ttl = sun.net.inetaddr.negative.ttl = 10

Solution:

The WebLogic Servers were restarted to clear the java.net.InetAddress caches. Consequently, the messaging bridges looked up the new domain name mapping for the remote server and connected to its target destination successfully.

Root Cause:

Domain Name lookup caching in the JVM was configured with default parameters. Hence, old domain name < –- > IP address mapping was cached forever.

Tips for setting caching parameters:

It’s easy to be tempted to set the networkaddress.cache.ttl value to 0 to turn off caching. However, note that Sun wants you to cache for two reasons – (1) Prevent DNS spoofing and (2) Improve performance. DNS Spoofing will not be an issue if the systems involved communicate over a secure network. So, assuming your systems are part of a secure network, trade-offs must be made between improved performance and service impact (when remote interfaces change – should be a rare occurrence) with caching. If your application is time-critical and cannot even withstand a latency/unavailability of a couple of minutes (e.g. synchronous call with user waiting for response), then turn off caching. However, if your application can tolerate delays by a few minutes or more (e.g. asynchronous message transfers), then you may set the value of networkaddress.cache.ttl to something like 300 (5 minutes). In such a scenario, if a remote system changes its domain name < – > IP address mapping, then your application will connect to the new IP address after 5 minutes, in the worst case. This also ensures that under normal circumstances, if your application makes several connections to the remote server within a 5 minute period, it will lookup the domain only once (assuming successful lookup) in 5 minutes, thereby improving performance. Refer to the table below for some example scenarios. So, hope you get the idea and set networkaddress.cache.ttl suitably. For almost all purposes, the default value of 10 seconds for networkaddress.cache.negative.ttl will suffice.

While considering the above guidelines for setting JVM caching parameters, you must also take into account DNS caching by the Operating System and network devices.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

The WebLogic Administration Server does not start with the following exception in stdout/stderr logs:

The WebLogic Server did not start up properly.
java.io.InvalidClassException: javax.management.MBeanAttributeInfo;

local class incompatible: stream classdesc serialVersionUID =7043852349133450673, local class serialVersionUID = 3875704819898565848
at java.io.ObjectStreamClass.initNonProxy
(Ljava.io.ObjectStreamClass;Ljava.lang.Class;Ljava.lang.ClassNotFoundException;

Ljava.io.ObjectStreamClass;)V(Unknown Source)
at java.io.ObjectInputStream.readNonProxyDesc(Z)
.
.
.
.

Background & Analysis:

This problem occurred on an environment where WebLogic Server 8.1 was used with JRockit 1.5.

Solution:

Use JRockit/Hotspot 1.4.x with WebLogic 8.1

Root Cause:

WebLogic 8.1 is incompatible with JRockit/Hotspot JVM 1.5 and later versions.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

The WebLogic Administration Server does not start with the following exception in stdout logs:

<Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: There are 1 nested errors: weblogic.management.ManagementException: [Management:141266]Parsing Failure in config.xml: javax.xml.namespace.QName; local class incompatible: stream classdesc serialVersionUID = -9120448469326609940, local class serialVersionUID = 4418622981286495151

Background & Analysis:

This is a known problem for Oracle. The problem is actually with the JRockit 1.5.0_07 version (and later) used with WebLogic Server 9.1 or 9.2, in which there is an incompatible change of the serialVersionUID of the javax.xml.namespace.QName class. Refer Known Problems.

Solution:

There are two solutions to eliminate the error. You may use any one of the following:

SOLUTION 1: Set a system property as follows:

com.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0

SOLUTION 2: Add the following option to the java command in the script you use to start WebLogic Server:

-Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0

Root Cause:

Incompatibility of JRockit 1.5.0_07 (or later) with WebLogic Server 9.1/9.2

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

Several occurrences of the following seen in the WebLogic Server 9.2 MP2 access logs:

“HEAD /bea_wls_internal/WLDummyInitJVMIDs HTTP/1.0" 404 1214

Background & Analysis:

If the WebLogic proxy plugin is used in a Web Server to load balance requests across WebLogic servers in a  cluster, then the plugin sends a dummy request to the cluster in order to retrieve the cluster server list. The cluster server list is present in the HTTP Response Header and not in the body. So, it does not matter if a non-existent web application is requested. That’s why a request is made to a non-existent web application/context called /bea_wls_internal/WLDummyInitJVMIDs, thereby resulting in a HTTP 404 error, even though the objective of obtaining the cluster server list is achieved. Well, Oracle could have handled this functionality better by internally deploying a dummy application to prevent these HTTP 404 errors

Solution:

There are two solutions to get rid of the HTTP 404 errors:

SOLUTION 1: Configure a default web application which does nothing (dummy).

SOLUTION 2: Obtain patch CR271671 (for 9.2 MP2) from Oracle to suppress HTTP 404 errors in access logs if you do not wish to configure a default web application. If you’re using another version of WebLogic Server, request Oracle for a suitable patch similar to CR271671.

Root Cause:

/bea_wls_internal/WLDummyInitJVMIDs is a non-existent web application since the WebLogic proxy plugin obtains required information from HTTP Response Headers alone.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

WLjvm will display details comprising the JVM version and options for a specified WebLogic Server. The displayed details are well-formatted and easy to read. The only parameter that needs to be passed to WLjvm is the name of the WebLogic Server whose JVM details you wish to obtain.

Why WLjvm?: At times, it’s required to know a few details about the WebLogic Server JVMs, like their versions and startup options. Well, you could use "ps" on Linux or "/usr/ucb/ps" or "pargs" on Solaris to give you this information. However, these UNIX utilities display information in a format which isn’t too pleasing to the eye (although "pargs" displays well-formatted details, you still need to use "ps" to determine the PID to use "pargs"). My intention is not to make WLjvm an overkill or make people lazy. It’s just that I observed many folks wanting to access JVM details quite regularly and so used a very simple script (and an alias to call it) to automate this task.

System Requirements:  Solaris/Linux, Korn Shell (/bin/ksh), WebLogic Server (all versions to date)

Download WLjvm v1.0     Download WLjvm v1.0 ReadMe

WLfdmon is a simple korn shell script that will enable you monitor open file descriptors used by WebLogic Servers. The salient features of WLfdmon are given below:

• Can be run in interactive and non-interactive modes
• Configuration-driven
• Generates statistics on open fd usage of WebLogic Servers. These Statistics can be used for trend analysis.
• Logs alarms when open fd threshold is breached. Alarms can notify Support Staff of abnormal application/server behaviour.
• Logs lsof output when open fd threshold is breached. This output will be useful for root cause analysis of excessive open fd usage.
• Housekeeping for data and lsof output files

System Requirements:  Solaris/Linux, Korn Shell (/bin/ksh), lsof

Download WLfdmon v1.0     Download WLfdmon v1.0 ReadMe

NOTE:

(1) WLfdmon has been scanned and is safe to download.

(2) WLfdmon has been tested. If you wish to use WLfdmon, I recommend you also test it before deploying it on a Production system.

(3) Your rating of this post/software will be greatly appreciated. Also, feel free to leave comments.