Problem:

When activating changes (clicking the button "Activate Changes" as shown in the image on the left) on the Administration console of a WebLogic 10.0 MP1 domain comprising an admin server and two managed servers (each managed server on a different host), it took around 5 minutes for the activation to complete.

Background:

From WebLogic Server versions 9.x and later, any changes performed on the Administration console must go through a three-step process – (1) Lock and Edit (2) Edit config (3) Activate Changes. It’s the third step in this process that took about 5 minutes to complete. The changes were successfully made, albeit after 5 minutes. Interestingly, when we located all the managed servers in the domain on the same host, this problem disappeared and the activation of changes took less than 10 seconds. However, locating all managed servers on one host cannot be a solution. We enabled debug for Deployment on all servers. Given below is the output of the debug during occurrence of the problem:

####<Sep 29, 2009 10:56:45 AM BST> <Debug> <Deployment> <myhost> <myadmin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1254218205661> <BEA-000000> <Experienced Exception while c.tryLock() and it is ignored :: java.nio.channels.OverlappingFileLockException
at sun.nio.ch.FileChannelImpl.checkList(FileChannelImpl.java:853)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:820)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:967)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.getFileLock(ConfigDataUpdate.java:374)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.getFileLock(ConfigDataUpdate.java:357)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.acquireFileLock(ConfigDataUpdate.java:338)
.
.
.

Solution:

After liasing with Oracle Support, we upgraded our JVM and the upgrade resolved the problem. After the upgrade, the activation of changes took less than 10 seconds irrespective of whether the managed servers were located on the same host or not. Details of the upgrade are given below:

Old JVM:

java version “1.5.0_14″
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
BEA JRockit(R) (build R27.5.0-110_o-99226-1.5.0_14-20080528-1505-linux-x86_64, compiled mode)

New JVM:
java version “1.5.0_17″
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_17-b04)
BEA JRockit(R) (build R27.6.3-40_BR8141840-120019-1.5.0_17-20090828-1133-linux-x86_64, compiled mode)

Root Cause:

Bug in JVM 1.5.0_14 (JRockit R27.5.0)

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

At times, you may want to determine the elapsed wall-clock time or uptime for a running process. While you can determine the process’ elapsed time using certain versions of the ps utility, it can also be determined using a simple shell script with a bit of perl. Both these methods are described below with the init process (PID = 1):

SOLUTION 1: The ps utility on some variants of UNIX (e.g. Linux, but not Solaris)

ps -oetime 1
    ELAPSED
      39:45

SOLUTION 2: A shell script with a bit of perl (should work on all variants of UNIX)

Download the shell script (puptime.ksh) or copy and paste from below:

#!/bin/ksh
# puptime.ksh - Gavin Satur - http://it.cybergav.in/2009/11/09/puptime
# Simple script to calculate the uptime (elapsed wall clock time) of a process
##############################################################################
#
# Accept PID and do some basic validation
#
proc_pid=$1
if [ -z "$proc_pid" ]; then
   print "\nERROR : Missing input argument. SYNTAX: ksh puptime.ksh <pid>\n"
   exit 999
fi
if [ ! -d /proc/$proc_pid ]; then
   print "\nERROR : No directory for PID $proc_pid in /proc. Check if process is running!\n"
   exit 999
fi
#
# Calculate start time of process and current time in epoch time using a bit of perl
#
proc_stime=`perl -e 'use File::stat;  my $filename = "$ARGV[0]"; $sb = stat($filename); printf "%s", $sb->mtime;' /proc/$proc_pid`
currtime=`perl -e 'print time;'`
#
# Calculate process uptime in seconds and then slice'n'dice for human-friendly output
#
proc_time=$(( currtime - proc_stime ))
proc_time_days=$(( proc_time / 86400 ))
proc_time_secs=$(( proc_time % 86400 ))
proc_time_hours=$(( proc_time_secs / 3600 ))
proc_time_secs=$(( proc_time_secs % 3600 ))
proc_time_minutes=$((proc_time_secs / 60 ))
proc_time_secs=$(( proc_time_secs % 60 ))
print "\nUPTIME FOR PROCESS WITH PID $proc_pid = $proc_time_days day(s) $proc_time_hours hour(s) $proc_time_minutes minute(s) $proc_time_secs second(s) \n"

./puptime.ksh 1
UPTIME FOR PROCESS WITH PID 1 = 0 day(s) 0 hour(s) 39 minute(s) 37 second(s)

Problem:

The Administration server of a WebLogic domain comprising WebLogic Server 10.0 and WebLogic Integration 10.2, consumes high CPU and throws java.lang.OutOfMemory errors.

Background:

The WebLogic Domain’s admin server had only two web applications deployed on it – the WebLogic Administration console and WebLogic Integration console. After start-up, its CPU utilization gradually increased and reached around 80% within a couple of days. Also, java.lang.OutOfMemory errors were observed in the server logs. This behaviour was observed even when there was no load on the managed servers and the web applications on the admin server were not accessed (all servers idle from a user perspective).

WebLogic Domain details:

Version: WebLogic Server 10.0 MP1, WebLogic Integration 10.2
JVM: JRockit R27.5.0-110 (JRE Standard Edition build 1.5.0_14-b03)
Admin Server JVM Heap: Minimum (Xms) = Maximum (Xmx) = 2 GB
Number of managed servers: 2
Operating System: 64-bit Red Hat Enterprise Linux 5.1
CPU Architecture: AMD64

Solution:

The following patches were applied and the problem was resolved. Contact Oracle support or use their Smart Update procedure to obtain the patches.

Root Cause:

Known issues with WebLogic Integration 10.2

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

A software’s log4j framework could not find the log4j configuration file which I specified. Details with debug below:

Command:

java -Dlog4j.debug=true -Dlog4j.configuration=/mysoftware/config/mylog4j.properties ...

Debug Output:

log4j: Trying to find [mysoftware/config/mylog4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@164b95.
log4j: Trying to find [mysoftware/config/mylog4j.properties] using sun.misc.Launcher$AppClassLoader@164b95 class loader.
log4j: Trying to find [mysoftware/config/mylog4j.properties] using ClassLoader.getSystemResource().
log4j: Could not find resource: [mysoftware/config/mylog4j.properties].

Background:

I did not want to use the default location for the log4j configuration file and wanted to be able to specify to a file with a name and location of my choice.

Solution:

The debug output indicates that the absolute path of the resource (log4j.properties) is missing a beginning “/”. That’s because the file:// protocol was not specified. So, the following worked and enabled log4j locate the properties file:

java -Dlog4j.debug=true -Dlog4j.configuration=file:///mysoftware/config/mylog4j.properties ...

NOTE: If your log4j configuration file is called log4j.xml or log4j.properties, then placing it in the application classpath will suffice and you will not need the -Dlog4j.configuration option.

Root Cause:

The file:// protocol was not used to specify the log4j configuration file.

Reference:

Short Introduction to log4j – Ceki Gülcü

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

When using WLShell 2.1.0.1 to connect to a WebLogic server, the connection fails and the following error is displayed:

couldn’t find or load connector class: wlshell.connect.jmx.weblogic.Connector for protocol: t3 check the libraries required by this connector are in the classpath. use the "ver" and "info -v" commands to display the current classpath.

Background:

In order for WLShell 2.1.0.1 to start properly and connect to a WebLogic server, the following must be present in the CLASSPATH: weblogic.jar, wlshell-2.1.0.1.jar, wlshell-2.1.0.jar and log4j-1.2.8.jar

My CLASSPATH had all the above jars, but I still received the error.

WebLogic was installed as a user called "bea" and I was running WLShell as my user (saturg). Both users were not part of the same group, but I ensured that weblogic.jar was accessible and readable by user saturg.

Solution:

As I did not have privileges to make both users bea and saturg part of the same group, I executed the following command as the bea user in the WebLogic installation root directory:

find . -type f | xargs -i chmod 744 {}

So, basically, I ensured that all files in the WebLogic installation were accessible and readable by all users on the host.

Note:The recommended method is to make both the weblogic installation user and WLShell user part of the same group, thereby restricting access to the WebLogic installation

Root Cause:

The error message is misleading as the class wlshell.connect.jmx.weblogic.Connector is available in wlshell-2.1.0.jar. The solution above indicates that WLShell requires to access other jars in the WebLogic installation (apart from weblogic.jar) or rather classes in weblogic.jar require to access other jars in the WebLogic installation.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.