First and foremost, if feasible, please consider a Hotspot JVM upgrade. The 1.4.2 JVM is outdated and there have been several significant improvements to GC ergonomics in later Hotspot JVMs. The JDK 1.4.2 Standard edition reached its EOL in late 2008. However, you can purchase support for later updates of JDK 1.4.2 (later than 1.4.2_19) that are branded under Java For Business (JFB) v1.4.2, to receive security updates and critical fixes until 2013. With regards to CMS, Sun engineers were finding their feet with CMS in JVM 1.4.2, enhanced CMS and made it the default collector in JVMs 5 and 6 and are about to replace CMS with G1 (Generation First) in JVM 7.

For those of you still using the Concurrent Mark Sweep (CMS) garbage collector with Hotspot JVM 1.4.2, here’s some information which I hope you will find useful.

What is CMS garbage collector?

The Concurrent (actually mostly concurrent) Mark Sweep (CMS) garbage collector is a non-default old (tenured) generation garbage collector introduced in JVM 1.4 that manages a JVM’s heap by collecting garbage in the old (tenured) generation in a few phases, some of which are concurrent and the others which are stop-the-world. As a consequence of the concurrent phases, this collector minimizes GC pauses and hence is also referred to as the low pause collector.

When would you use the CMS Collector?

When you want to minimize GC pauses (frequent requirement for websites and interactive applications)

Does CMS require more cpu and memory than the default collectors?

Yes. You need adequate CPU resources (multiple processors) as CMS adds CPU overhead by using a background thread for the concurrent phases. You need adequate memory to allocate slightly larger heaps than you would for the default collectors, as objects will continue to enter the old generation during the mostly concurrent phases.

How do you enable the CMS Collector?

Use the JVM flag -XX:+UseConcMarkSweepGC. When you do this, -XX:+UseParNewGC is implicitly used. However, you may explicitly specify -XX:+UseParNewGC if you wish (but it’s just redundant).

Note: You cannot use the default collector or the parallel scavenge collector (-XX:+UseParallelGC) in the Young Generation when using CMS in the old generation. CMS is tightly coupled with ParNewGC in the young generation. ParNewGC is an enhanced version of the parallel scavenge collector that enables GC to be done in the young generation while CMS is in progress in the old generation.

 Does the CMS Collector require tuning?

You’re lucky if the CMS collector gives you optimal performance by just enabling it with -XX:+UseConcMarkSweepGC. Every Hotspot 1.4.2 JVM I’ve come across (enterprise systems), has required some CMS tuning for optimal performance. Of course, this requirement entirely depends on your application’s object profile. But, if tuning your Hotspot 1.4.2 is required, then tuning CMS requires more effort than tuning the default collectors and CMS has a bunch of JVM flags to play around with. Some important JVM flags to consider when tuning CMS are given below:

What do you need to watch out for when using CMS (problems/gotchas)?

• The dreaded concurrent mode failure – a failure which occurs when CMS’ concurrent GC phases are interrupted for certain reasons and a Serial, Mark-Sweep-Compact GC is required.
• Default SurvivorRatio=1024 and Default MaxTenuringThreshold=0. Note that these are default values only when using CMS with JVM 1.4.2 and can trouble you if you’re tuning your JVM for short-lived objects. If your application creates mostly short-lived objects and you wish to use the Young Generation as a filter to retain these objects as long as possible and clean them up with minor GCs (parallel scavenges) to reduce the pressure on the CMS collector, then you must change these default values as these default values ensure that the survivor spaces are not used. Refer this article to understand the peculiarities of MaxTenuringThreshold.
• The value set by –XX:CMSInitiatingOccupancyFraction is used as same threshold for both old and permanent generation occupancies. i.e. CMS GC will be initiated in the old and permanent generations when either one of or both the old and permanent generation occupancy exceeds the value of CMSInitiatingOccupancyFraction. This is inconvenient and it implies that you must pay close attention to permanent generation occupancy also and size the permanent generation appropriately.
• The -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled JVM flags could increase remark pauses, but can contain permanent generation growth (prevent OutOfMemory errors caused by a full permanent generation) and protect against poor GC in old generation (objects in the old generation that are referenced by classes in the permanent generation will not be collected until the classes in the permanent generation are collected).

What tools are available to assist you with JVM tuning?

First and foremost, before tuning your 1.4.2 JVM, ensure you profile your application and resolve application issues (e.g. memory leak). To tune your JVM, you have tools which broadly fall under two categories:

Runtime Monitoring: These tools attach to your JVM to provide loads of runtime data. These tools are useful, when you’re monitoring your JVM in runtime and using these tools interactively. An excellent tool is VisualVM and its VisualGC plugin. A screenshot of the VisualGC plugin is given below:

GC Log File Analysis: These tools enable you to do offline analysis of GC log files and prepare reports for trend analysis. If you wish to run a load test for a couple of hours and measure performance after the test, then you will need to capture GC logfiles and analyze them with such tools. Now, in this area, Sun has been lacking good tools. HP chose a wise strategy of defining a specific format for GC logfiles generated by HP JDKs and developing the excellent HPjmeter to parse the logfiles and create fancy charts along with several metrics. Well, the Sun forums indicate that a GCHisto plugin is being developed to analyze Hotspot GC log files. I have tried out this plugin (beta) and found it to be nowhere as comprehensive and sophisticated as HPjmeter. Well, will wait for GCHisto to be completed and plugged into VisualVM before trying it out again.

In order to assist my colleagues and me with Hotspot JVM 1.4.2 GC log file analysis when using CMS, I’ve developed a quick-and-dirty korn shell script to provide a summary of some key GC metrics for a specific GC logfile. You may download the script CMSGCStats.ksh and execute it without arguments (ksh CMSGCStats.ksh) for usage tips. Refer a sample screenshot of the script’s output below:

References:

(1) JVM 1.4.2 Garbage Collectors

Some movies may be distributed in more than one file (e.g. more than one CD). Recently, I purchased a movie in Xvid format which consisted of two parts (two Xvid files). So, here’s how I joined the files to generate one Xvid file and viewed the entire movie in one go:

STEP 1: Download and install an Xvid codec

Xvid is a video codec library for the MPEG-4 standard. I downloaded an Xvid codec here.

STEP 2: Download and install (extract) VirtualDub

VirtualDub is a free video capture/processing software for Microsoft Windows. You may download VirtualDub here.

STEP 3: Start VirtualDub and load the video files

Start VirtualDub (   ) and load the video files as follows:

To load the first video file, click File –> Open video file on the Menu bar or type CTRL + O.

To load the remaining files, add them one at a time by clicking File –> Append AVI segment on the Menu bar.

STEP 4: Configure Audio and Video settings

For video, click Video –> Direct stream copy on the Menu bar.

For audio, click Audio –> Direct stream copy on the Menu bar.

STEP 5: Save the joined video file

To save all video files as one joined file, click File –> Save as AVI on the Menu bar, choose a filename and save.

Problem:

HPjmeter 4.0 console does not launch after installation on Windows. When trying to launch it, the Command window briefly appears and then disappears without launching the console.

Background:

HPjmeter 4.0 console, a component of HPjmeter 4.0, is used to view performance data and analyze profiling and GC log files for a JVM. HPjmeter 4.0 console, being developed in java, can be executed on any platform supporting Java. However, the console is currently compatible with only JDK 5 and JDK 6. When you launch the installer (hpjmeter_console_4.0.00_windows_setup.exe), it will first scan your HDDs for installed JDKs and I guess it will pick up the first JDK it finds (see screenshot below).

I have both JDK 1.4.2_11 and JDK 6u17 on my HDD. Although only JDK 6u17 meets HPjmeter 4.0 console’s requirements, the installer selected JDK 1.4.2_11 (must have found this JDK first!) and completed the installation. Well, the installer will not even tell you which JDK it has found and used for the installation. You can determine this information only by checking the hpjmeter.bat file in the bin directory within the installation directory. The JDK used by the installer will be the value of the variable JM_JAVA_HOME in the hpjmeter.bat file. So, for my installation, JM_JAVA_HOME indicated that HPjmeter 4.0 console was using JDK 1.4.2_11, thereby not meeting the software requirements.

The HPjmeter 4.0 console installer should either search and select only a compatible JDK or prompt you for the location of a compatible JDK if it cannot find one. Although the installer isn’t too smart, the product (HPjmeter 4.0) is a very good and robust performance analysis tool for JVMs.

 Solution:

Ensure you have a working JDK 5 or JDK 6 installation. Ensure that the value of JM_JAVA_HOME in the hpjmeter.bat file is the location of the JDK 5 or JDK 6 installation.

Root Cause:

HPjmeter 4.0 console installer detected and used an incompatible JDK during installation. Currently, the HPjmeter 4.0 console is compatible with only JDK 5 and JDK 6.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

Solaris hosts indicate high cpu utilization caused by Sun’s CST (cstd.agt) and Net Connect (srsproxy) processes.

Background:

Configuration and Service Tracker (CST) and Net Connect are tools provided by Sun Microsystems for proactive system management at a customer site. The software processes launched by these tools run on SOlaris hosts and regularly send data to Sun Microsystems to enable Sun to track system availability and performance and continually improve Sun’s products and services. cstd.agt and srsproxy are processes belonging to CST and Net Connect respectively. While there have been problems/patches for CST and  problems/patches for Net Connect related to high cpu utilization, note that both these tools have reached their EOL and Sun no longer supports them. Instead, Sun has replaced these tools with the Services Tools Bundle (STB).

 Solution:

Remove CST and Net Connect software from your Solaris host(s) or replace CST and Net Connect with STB.

Root Cause:

CST and Net Connect have reached their EOL and are no longer supported. Hence, using them could lead to high CPU utilization problems.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

What is MaxTenuringThreshold?:

In a Sun Hotspot JVM, objects that survive Garbage Collection in the Young Generation are copied multiple times between Survivor Spaces before being moved into the Tenured (Old) Generation. The JVM flag that governs how many times the objects are copied between the Survivor Spaces is MaxTenuringThreshold (MTT) and is passed to a JVM as –XX:MaxTenuringThreshold=n , where n is the number of times the objects are copied. The default value of ‘n’ is (or actually was) 31.

Setting the MaxTenuringThreshold:

A few years ago, while my colleagues and I were tuning a 1.4.2_11 Hotspot JVM using flags like PrintTenuringDistribution and tools like  visualgc, we found that setting MTT=10 along with other flags gave us the best results (JVM throughput, pause time, footprint). However, recently when tuning a 1.4.2_11 Hotspot JVM for another application that had mostly short-lived objects, I suggested testing a value of MTT=80 (I still have no idea how the value 80 came to my mind) which is ridiculous as you’ll soon know. My objective was to retain the short-lived objects for as long as possible in the Young Generation to allow them to be collected by Minor GCs as opposed to the Full GCs in the tenured generation. Anyway, all our performance tests of the application on JVM 1.4.2_11 with MTT=80 and other JVM flags showed significant improvement in JVM performance than before (when it was untuned).

Last week, I came across some interesting proposals discussed among Sun engineers last year, regarding modifying the way MTT is handled by the JVM. I don’t know whether the proposals have been implemented, but they give some good insight into how MTT works. To quote those discussions,

Each object has an "age" field in its header which is incremented every time an object is copied within the young generation. When the age field reaches the value of MTT, the object is promoted to the old generation (I’ve left out some detail here…). The parameter -XX:+NeverTenure tells the GC never to tenure objects willingly (they will be promoted only when the target survivor space is full). (out of curiosity: does anyone actually use -XX:+NeverTenure?) Originally, in the HotSpot JVM, we had 5 bits per object for the age field (for a max value of 31, so values of MTT would make sense if they were <= 31). A couple of years ago (since 5u6 IIRC), the age field "lost" one bit and it now only has 4 (for a max value of 15).

Refer http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-May/000309.html for more details. So, basically , when I set MTT=80, the JVM would have actually “never tenured” the objects in the Young Generation until the Survivor Spaces were full. Hope Sun have fixed that problem as per the proposal or at least provide proper documentation (similar to the referenced article) for their JVMs which explains how MTT works. Well, MTT=80 did not have an adverse impact on our application, but we eventually switched to MTT=8 (the value 8 was a guess and didn’t provide very different results). I suggest that MTT is not set at first and be used only if your analysis of GC logs and your requirements indicate that you need to retain short-lived objects in the Young Generation for longer. As a matter of fact, when tuning a JVM, always start with basic flags for Heap Size and nothing else. Then, based on load tests, customer experience metrics (e.g. response time, response errors) and analysis of GC logs, set JVM flags and retest. Tuning is iterative and apart from all the tools available, a must-have quality (especially for complex applications) is patience.