Ahoy there! This is my blog in which I jot down some of my experiences in IT (stuff related to my job and other random IT stuff). Hope you find something useful. My primary fields of interest in IT are Korn/Bash Shell Scripting, web/middleware/database technologies (ZXTM, Apache, WebLogic Server, Oracle, etc.), IT Operations Management, ITIL and UNIX (any flavour/distribution).

Archive for January, 2010

Problem:

Messaging Bridge on WebLogic Server 8.1 does not start. Following errors seen in server log:

####<Jan 27, 2010 10:10:13 AM GMT> <Info> <MessagingBridge> <myhost> <managed1> <ExecuteThread: ‘4′ for queue: ‘MessagingBridge’> <<WLS Kernel>>
<> <BEA-200032> <Bridge "MyBridge" is configured to disallow degradation of its quality of service in cases where the configured quality of service is unreachable.>
####<Jan 27, 2010 10:10:13 AM GMT> <Error> <MessagingBridge> <myhost> <managed1> <ExecuteThread: ‘4′ for queue: ‘MessagingBridge’> <<WLS Kernel>> <> <BEA-200025> <Bridge "MyBridge" failed to start, because the quality of service configured (Exactly-once) is unreachable. This is likely due to an invalid configuration or adapter limitations.>
####<Jan 27, 2010 10:10:13 AM GMT> <Info> <MessagingBridge> <myhost> <managed1> <ExecuteThread: ‘4′ for queue: ‘MessagingBridge’> <<WLS Kernel>> <> <BEA-200034> <Bridge "MyBridge" is shut down.>

           

NOTE: MyBridge connects a WebLogic JMS destination (source) to an MQ destination (target).

Background & Analysis:

In order for Messaging Bridges on WebLogic 8.1 to use Exactly-Once QOS, the following requirements must be met:

  • Messaging Bridge adapter must be jms-xa-adp.rar and its JNDI name is eis.jms.WLSConnectionFactoryJNDIXA.
  • Connection Factories for source Bridge destinations must be XA-enabled.
  • Connection Factories used for target Bridge Destinations must be XA-enabled.
  • Messaging Bridges must be configured with Exactly-Once QOS.
  • The “QOS Degradation Allowed” checkbox must be unchecked.

With the above, it is recommended that the Messaging Bridges be Synchronous for better performance (fewer transaction commits).

From the log snippet above, you can see that the Messaging Bridge MyBridge could not start because the QOS (Exactly-Once) was unreachable and the Bridge was not allowed to degrade its QOS.

The QOS will typically be unreachable due to adapter, bridge configuration or bridge destination configuration issues as referred to in the log snippet.

Solution:

Enabled XA on the ConnectionFactory for the target Bridge Destination on MQ.

In order to satisfy the Exactly-Once QOS, both source and target destination connection factories must be XA-enabled.

Root Cause:

The ConnectionFactory for the target Bridge Destination on MQ was configured as non-XA, thereby preventing the Messaging Bridge from initiating an XA connection from the WebLogic Bridge destination to the MQ Bridge destination. Since the Messaging Bridge was not allowed to lower its QOS to make the connection, it failed to start properly.

 NOTE: By allowing QOS degradation, the MessagingBridge will connect to the MQ destination even if the ConnectionFactory for the Bridge Destination on MQ were non-XA. However, the choice of QOS must be driven by business requirements and not by technical workarounds.

 Reference: Oracle Documentation

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.6.5_908]
Rating: +4 (from 4 votes)

Problem:

WebLogic Administration Server (with WebLogic Integration) 8.1 does not start. Following errors seen in stdout/stderr/server logs:

####<Jan 25, 2010 4:30:26 PM GMT> <Error> <JDBC> <myhost> <myadmin> <main> <<WLS Kernel>> <> <BEA-001151> <Data Source "cgDataSource" deployment failed with the following error: null.>

####<Jan 25, 2010 4:30:26 PM GMT> <Info> <JDBC> <myhost> <myadmin> <main> <<WLS Kernel>> <> <BEA-001156> <Stack trace associated with message 001151 follows:

weblogic.common.ResourceException

        at weblogic.jdbc.common.internal.DataSourceManager.createDataSource(DataSourceManager.java:264)

####<Jan 25, 2010 4:30:30 PM GMT> <Error> <WLW> <myhost> <myadmin> <main> <<WLS Kernel>> <> <000000> <Failed to obtain connection to datasource=cgDataSource, using generic DB properties>

####<Jan 25, 2010 4:30:31 PM GMT> <Error> <WLW> <myhost> <myadmin> <main> <<WLS Kernel>> <> <000000> <Error in startup class com.bea.wli.store.DocumentStoreSetup Method: init:

java.lang.IllegalStateException: Unable to start DocumentStore:  com.bea.wli.store.DocumentStoreException: Could not find SQL Document Store cgDataSource

            .

            .

Background & Analysis:

WebLogic Integration (WLI) is a software Business Process Integration framework that runs on WebLogic Server. WLI also includes a console application (wliconsole) to manage WLI configuration. This console application is deployed on the WebLogic Administration Server. Since the console application interacts with the database, it uses default data sources and connection pools (e.g. cgDataSource and cgConnectionPool) for database connectivity.

The errors above indicate that the cgDataSource failed to deploy and consequently, a startup class could not obtain connections to the database, thereby failing deployment and preventing the Administration Server from starting.

Data Sources use Connection Pools to obtain database connections.

Solution:

Ensure that the Connection Pool for the cgDataSource is configured properly (correct JDBC driver, URL, credentials, etc.) and targeted/deployed on the Administration Server (not just the cluster).

Root Cause:

The connection pool (cgConnectionPool) for the data source cgDataSource was not deployed on the Administration Server.

 

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.6.5_908]
Rating: -1 (from 1 vote)

How to encrypt passwords for WebLogic 9.x+?

In WebLogic versions prior to 9, JSAFE decryption exceptions and password encryption could be resolved/performed by simply setting plain-text passwords in config.xml as described here.

However, in WebLogic versions later than 9.0, setting plain-text passwords in config.xml (Production mode environments) will throw the following error:

<Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: [Management:141266]Parsing Failure in config.xml: java.lang.IllegalArgumentException: In production mode, it’s not allowed to set a clear text value to the property: PasswordEncrypted of ServerStartMBean>

So, for WebLogic 9.x+ versions, if you doubt passwords in config.xml, you will need to encrypt plain-text passwords and configure them in config.xml as follows:

STEP 1: Encrypt the password you wish to change in config.xml.

Example: You experience JSAFE exceptions with the password of one of your connection pools and you doubt the encrypted password in config.xml.

  • Source the WebLogic environment (to set CLASSPATH and other variables) as follows:
cd <domain-dir>/bin
. ./setDomainEnv.sh

  • Encrypt the plain-text password as follows:
cd <domain-dir>
java weblogic.security.Encrypt <password>

STEP 2: Update the config files (config.xml, jdbc, etc.) with the encrypted password obtained above.

NOTE: The above encryption procedure will work even with earlier versions of WebLogic (e.g. 8.1).

VN:F [1.6.5_908]
Rating: +12 (from 12 votes)

Disposable email – Use and forget

Have you provided your personal email account when registering on websites using email confirmation? Have you hesitated to provide comments on blogs, forums, etc. because of fear of having to provide your personal email address? Of course, you always have the option of registering a free email account just for this purpose, but this means you have yet another username/password pair to remember or require some time to create a fictional identity. Instead, how about using a temporary email address that requires no registration? That’s exactly what 10 Minute Mail offers!! It provides you with a temporary email address that is valid for 10 minutes (more than enough time to respond to email registrations). Beat the spam caused by various online registrations!

Well, I’ve now found that Disposable Email Addressing (DEA) has been around for some time and there are several DEA services on the web. Refer this blog post for 20 such disposable email services (Mailinator, MyTrashMail, etc.). Website admins are nowadays blocking registrations with temporary email addresses. Well, you can always try!

VN:F [1.6.5_908]
Rating: +1 (from 1 vote)

First and foremost, if feasible, please consider a Hotspot JVM upgrade. The 1.4.2 JVM is outdated and there have been several significant improvements to GC ergonomics in later Hotspot JVMs. The JDK 1.4.2 Standard edition reached its EOL in late 2008. However, you can purchase support for later updates of JDK 1.4.2 (later than 1.4.2_19) that are branded under Java For Business (JFB) v1.4.2, to receive security updates and critical fixes until 2013. With regards to CMS, Sun engineers were finding their feet with CMS in JVM 1.4.2, enhanced CMS and made it the default collector in JVMs 5 and 6 and are about to replace CMS with G1 (Generation First) in JVM 7.

For those of you still using the Concurrent Mark Sweep (CMS) garbage collector with Hotspot JVM 1.4.2, here’s some information which I hope you will find useful.

What is CMS garbage collector?

The Concurrent (actually mostly concurrent) Mark Sweep (CMS) garbage collector is a non-default old (tenured) generation garbage collector introduced in JVM 1.4 that manages a JVM’s heap by collecting garbage in the old (tenured) generation in a few phases, some of which are concurrent and the others which are stop-the-world. As a consequence of the concurrent phases, this collector minimizes GC pauses and hence is also referred to as the low pause collector.

When would you use the CMS Collector?

When you want to minimize GC pauses (frequent requirement for websites and interactive applications)

Does CMS require more cpu and memory than the default collectors?

Yes. You need adequate CPU resources (multiple processors) as CMS adds CPU overhead by using a background thread for the concurrent phases. You need adequate memory to allocate slightly larger heaps than you would for the default collectors, as objects will continue to enter the old generation during the mostly concurrent phases.

How do you enable the CMS Collector?

Use the JVM flag -XX:+UseConcMarkSweepGC. When you do this, -XX:+UseParNewGC is implicitly used. However, you may explicitly specify -XX:+UseParNewGC if you wish (but it’s just redundant).

Note: You cannot use the default collector or the parallel scavenge collector (-XX:+UseParallelGC) in the Young Generation when using CMS in the old generation. CMS is tightly coupled with ParNewGC in the young generation. ParNewGC is an enhanced version of the parallel scavenge collector that enables GC to be done in the young generation while CMS is in progress in the old generation.

 Does the CMS Collector require tuning?

You’re lucky if the CMS collector gives you optimal performance by just enabling it with -XX:+UseConcMarkSweepGC. Every Hotspot 1.4.2 JVM I’ve come across (enterprise systems), has required some CMS tuning for optimal performance. Of course, this requirement entirely depends on your application’s object profile. But, if tuning your Hotspot 1.4.2 is required, then tuning CMS requires more effort than tuning the default collectors and CMS has a bunch of JVM flags to play around with. Some important JVM flags to consider when tuning CMS are given below:

What do you need to watch out for when using CMS (problems/gotchas)?

  • The dreaded concurrent mode failure – a failure which occurs when CMS’ concurrent GC phases are interrupted for certain reasons and a Serial, Mark-Sweep-Compact GC is required.
  • Default SurvivorRatio=1024 and Default MaxTenuringThreshold=0. Note that these are default values only when using CMS with JVM 1.4.2 and can trouble you if you’re tuning your JVM for short-lived objects. If your application creates mostly short-lived objects and you wish to use the Young Generation as a filter to retain these objects as long as possible and clean them up with minor GCs (parallel scavenges) to reduce the pressure on the CMS collector, then you must change these default values as these default values ensure that the survivor spaces are not used. Refer this article to understand the peculiarities of MaxTenuringThreshold.
  • The value set by –XX:CMSInitiatingOccupancyFraction is used as same threshold for both old and permanent generation occupancies. i.e. CMS GC will be initiated in the old and permanent generations when either one of or both the old and permanent generation occupancy exceeds the value of CMSInitiatingOccupancyFraction. This is inconvenient and it implies that you must pay close attention to permanent generation occupancy also and size the permanent generation appropriately.
  • The -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled JVM flags could increase remark pauses, but can contain permanent generation growth (prevent OutOfMemory errors caused by a full permanent generation) and protect against poor GC in old generation (objects in the old generation that are referenced by classes in the permanent generation will not be collected until the classes in the permanent generation are collected).

What tools are available to assist you with JVM tuning?

First and foremost, before tuning your 1.4.2 JVM, ensure you profile your application and resolve application issues (e.g. memory leak). To tune your JVM, you have tools which broadly fall under two categories:

Runtime Monitoring: These tools attach to your JVM to provide loads of runtime data. These tools are useful, when you’re monitoring your JVM in runtime and using these tools interactively. An excellent tool is VisualVM and its VisualGC plugin. A screenshot of the VisualGC plugin is given below:

VisualVM_VisualGC

GC Log File Analysis: These tools enable you to do offline analysis of GC log files and prepare reports for trend analysis. If you wish to run a load test for a couple of hours and measure performance after the test, then you will need to capture GC logfiles and analyze them with such tools. Now, in this area, Sun has been lacking good tools. HP chose a wise strategy of defining a specific format for GC logfiles generated by HP JDKs and developing the excellent HPjmeter to parse the logfiles and create fancy charts along with several metrics. Well, the Sun forums indicate that a GCHisto plugin is being developed to analyze Hotspot GC log files. I have tried out this plugin (beta) and found it to be nowhere as comprehensive and sophisticated as HPjmeter. Well, will wait for GCHisto to be completed and plugged into VisualVM before trying it out again.

In order to assist my colleagues and me with Hotspot JVM 1.4.2 GC log file analysis when using CMS, I’ve developed a quick-and-dirty korn shell script to provide a summary of some key GC metrics for a specific GC logfile. You may download the script CMSGCStats.ksh and execute it without arguments (ksh CMSGCStats.ksh) for usage tips. Refer a sample screenshot of the script’s output below:

 

CMSGCStats

References:

(1) JVM 1.4.2 Garbage Collectors

 

COLLECTOR GENERATION JVM FLAGS TO TURN ON DESCRIPTION
Serial Young None. Default. Single-threaded, stop-the-world, copying collector
Parallel Scavenge Young -XX:+UseParallelGC Multi-threaded, stop-the-world, copying collector (not to be used with CMS)
ParNew Young -XX:+UseParNewGC

Multi-threaded, stop-the-world, copying collector to be used along with CMS. This option is automatically turned on when using CMS and doesn’t have to be explicitly specified.

Serial Old Old None. Default.

Single-threaded, stop-the-world, mark-sweep-compact collector

CMS Old -XX:+UseConcMarkSweepGC

Mostly concurrent low-pause collector that uses a background thread for the concurrent phases.

VN:F [1.6.5_908]
Rating: 0 (from 0 votes)