Ahoy there! This is my blog in which I jot down some of my experiences in IT (stuff related to my job and other random IT stuff). Hope you find something useful. My primary fields of interest in IT are Korn/Bash Shell Scripting, web/middleware/database technologies (ZXTM, Apache, WebLogic Server, Oracle, etc.), ICT Operations Management, ITIL and UNIX (any flavour/distribution).

Archive for June, 2009

Problem:

The WebLogic Server does not start. Following errors seen in stdout/stderr/server logs:

***************************************************************************
The WebLogic Server did not start up properly.
Exception raised: ‘weblogic.server.ServerLifecycleException: Server failed to bind to the requested port. See preceeding log message for details.’
Reason: Server failed to bind to the requested port. See preceeding log message for details.
***************************************************************************

Background & Analysis:

In order for a software process to listen at a port on a specific IP address, there should not be any other process already listening to that same port and IP address.

 

Solution:

 (1) Obtain details of the IP address (ListenAddress) and port(ListenPort) at which your WebLogic Server is supposed to listen.

(2) If the host bearing the IP address is unreachable, then contact your Network/System Administrators to ensure that the IP address is associated with a network interface card on the appropriate host.

(3) If the IP address is reachable, use the following commands on the host bearing the IP address (obtained in above step) to determine whether a process is already bound (listening) at the same port and IP address:

On Windows Systems (DOS prompt):

netstat –an | findstr “<IP address>:<port>.*LISTEN”

 

On UNIX Systems:

netstat –an | grep “<IP address>[\.:]<port>.*LISTEN”

       

If no software process is listening at the specified IP address and port, then the above commands should not display any output.

Examples:

(a) IP:Port not bound: For a Windows host on which no process listens at port 777 on 110.120.130.140,

C:\netstat -an | findstr "110.120.130.140:777.*LISTEN"

C:\

(b) IP:Port already bound: For a Solaris host on which a WebLogic Server is listening at port 777 on 110.120.130.140,

$ netstat -an | grep "110.120.130.140[\.:]777.*LISTEN"
110.120.130.140.777        *.*               0      0 131072      0 LISTEN

(4) If step (3) indicates that a process is already listening at the port and IP address at which you want your WebLogic Server to listen, then that process needs to be killed (if it’s not meant to be listening there).

(5) After ensuring that no other process is listening at the WebLogic Server ListenAddress and ListenPort, start the WebLogic Server.

 

NOTE:

  • To obtain the IP address for a domain name, use nslookup (other utilities also available).
  • If your WebLogic Servers are running within a Veritas Cluster Server (VCS) High Availability framework, then the IP address for your servers will be floating and you must execute the above commands on all hosts within the Veritas cluster.
  • This problem and suggested commands are not specific to WebLogic Server processes, but are applicable to all software processes.

Root Cause:

For the problem I experienced, the IP address of the host on which a WebLogic Server was supposed to listen, went missing (error in VCS config caused the IP address to drop off the network interfaces on all the hosts within the VCS cluster) and consequently, was not reachable.

 

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.6.5_908]
Rating: +1 (from 1 vote)

WebLogic: SerializedSystemIniException

Problem:

The WebLogic Server does not start. Following errors seen in stdout/stderr/server logs:

 <21-Jun-2009 08:18:13 o’clock BST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: weblogic.security.internal.SerializedSystemIniException: [Security:090207]Version mismatch. have 117, expected 1>

            .

            .

Background & Analysis:

Read this tutorial to understand how a WebLogic server uses SerializedSystemIni.dat and how this file may become corrupted.

 

Solution:

Restore a backup of SerializedSystemIni.dat and start the WebLogic servers.

If you do not have a backup, then you will need to reconfigure the domain.

 

Root Cause:

The WebLogic domain’s SerializedSystemIni.dat file is corrupted.

 

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.6.5_908]
Rating: 0 (from 0 votes)

WebLogic: JSAFE_PaddingException

Problem:

WebLogic Server 8.1 does not start. Following errors seen in stdout/stderr/server logs:

 ####<Jun 18, 2009 11:44:13 AM BST> <Error> <Management> <> <admin_mkdom> <main> <<WLS Kernel>> <> <BEA-140001> <An error occurred while getting attribute Credential on MBean mkdom:Location=admin_mkdom,Name=mkdom,Type=EmbeddedLDAPConfig. Method: null. Exception: com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte..

com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte.

            at com.rsa.jsafe.JA_PKCS5Padding.a(Unknown Source)

            at com.rsa.jsafe.JG_BlockCipher.decryptFinal(Unknown Source)

            .

            .

Background & Analysis:

JSAFE is an encryption/decryption engine for Java, developed by RSA. So, you can be sure that you have a problem with encryption/decryption whenever you encounter exceptions related to JSAFE.

All passwords in a WebLogic domain are encrypted by hashes in a file called SerializedSystemIni.dat. Read this tutorial to understand how WebLogic uses SerializedSystemIni.dat. If passwords that have been encrypted in one WebLogic domain are used in another WebLogic domain, then all the encrypted passwords will not be decrypted and the WebLogic servers will not start.

 

Solution:

As the passwords need to be encrypted using the current domain’s SerializedSystemIni.dat file, do the following to configure valid encrypted passwords and start the WebLogic servers:

(1) In all WebLogic configuration files contain encrypted passwords, find and replace as follows:

ALL occurrences of  PasswordEncrypted=”{3DES}…….” 

                                              with

                                Password=”plain-text password”

 

where, plain-text password  =  corresponding plain-text password for encrypted password

Example: If the password for a user used by a connection pool is “olympics” and its encrypted value is “{3DES}hKjgbvd==”, then replace PasswordEncrypted=“{3DES}hKjgbvd==” with Password=”olympics”

(2) Start the WebLogic servers.

 

Root Cause:

The encrypted passwords used in configuration files within a WebLogic domain were encrypted using SerializedSystemIni.dat in another WebLogic domain.

UPDATE (24th January 2010): For WebLogic Server 9.x+, the solution above will not work. Refer this article for tips.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.6.5_908]
Rating: +4 (from 4 votes)

The SerializedSystemIni.dat file is a critical part of every WebLogic domain and if not administered properly, could result in your servers not starting and you having to recreate the entire domain. So, a good understanding of this file is important.

 

What is it?

SerializedSystemIni.dat is a WebLogic domain file which contains hashes.  Currently, these hashes use Triple-DES block ciphers (that’s why the encrypted passwords begin with “{3DES}”). SerializedSystemIni.dat is located in the domain directory (WebLogic Server 8.1 and earlier) or in domain/security directory (WebLogic Server 9.x and later).

 

When and how is it created?

The SerializedSystemIni.dat is created during the creation of a WebLogic domain. The hashes in the file are created using an algorithm that binds the file to the domain in which it has been created. So, a SerializedSystemIni.dat file can be used only within the domain in which it has been created (cannot be used in other WebLogic domains). Also, in WebLogic Server versions 8.1 SP6 and earlier, the SerializedSystemIni.dat file (along with msi-config.xml and fileRealm.properties files)  is replicated into a managed server’s root directory every 5 minutes for managed servers which have MSI File Replication enabled. These files are replicated even if the admin and managed servers share the same server root directory (doesn’t make sense to me).

 

What is it used for?

SerializedSystemIni.dat is used for encryption/decryption of plain-text/ciphertext within a WebLogic domain.

Gotchas!

  • If SerializedSystemIni.dat is corrupted (modified) or missing, then the WebLogic Servers in your domain will not start and you will have to reconfigure the domain.
  • If MSI File Replication is enabled for your managed servers in domains running on WebLogic server versions 8.1 SP6 or earlier, then a bug in the 5-minute replication could cause corruption of SerializedSystemIni.dat occasionally during server restart and will make it a zero-byte file if the disk drive or mountpoint in which the WebLogic domain is located is 100% full.
  • If SerializedSystemIni.dat is transferred to its domain in ascii mode (for example, from a configuration management system) , the file could become corrupted.

 

Best Practices

  • Ensure that you have a working backup of SerializedSystemIni.dat. i.e. test the recovery of SerializedSystemIni.dat and server start-up using the backup copy.
  • If you’re using WebLogic Server versions 8.1 SP6 or earlier and have MSI File Replication enabled, then ensure you contact Oracle, obtain and deploy patch CR260218 (Guardian Signature Patterns Release 1.1.34 and Signature ID 000168) to fix the bug in replication that corrupts SerializedSystemIni.dat or makes it a zero-byte file.
  • If you require to transfer SerializedSystemIni.dat to its domain via FTP, always use binary mode for the file transfer.
VN:F [1.6.5_908]
Rating: +6 (from 6 votes)

Design for Supportability

By Design for Supportability (DFS), I mean designing a product with adequate features  to facilitate administration, support, maintenance  and problem diagnosis. DFS is not just a methodology, it’s a way of thinking for design and development.  This post will pertain to Application Software Design. The basic principle of DFS should be applicable to all product designs.

Benefits of DFS:

  • Increases probability of error detection in the testing phase, thereby leading to a better quality product.
  • Quicker resolution times for incidents and root cause analysis, thereby leading to an increase in customer satisfaction.
  • Smaller support teams, thereby leading to a reduction in the total cost of ownership (TCO).

    Given below are some typical problems experienced by Application Support teams and how they may be addressed/avoided by DFS:

SL# Problem experienced by Support Team How it may be addressed by DFS
1. Inadequate logging. No errors in logs.
  • Use a good logging framework and a well-structured log syntax (eg. log4j) for application logs.
  • Segregate application and server logs.
  • Handle all application exceptions and log descriptive errors, rather than allow errors to be directed to standard error/output.
  • Add debug to critical paths/exit statuses within the application to facilitate troubleshooting problems when debug is enabled.
2.

Logging severity levels cannot be changed at runtime. e.g. to enable debug for the application, the application needs to be redeployed or the underlying server restarted.

  • Use a good logging framework which permits runtime changes (eg. log4j)
3.

Redundant Logging. Rapid utilization of free space on disk.

  • Log adequate details, as required by the severity level. Nothing more, nothing less.
  • Data logged must be in accordance width data privacy/confidentiality laws.
4. Configuration changes are not dynamic.
  • If the application is customizable and driven by configuration files, then as far as possible, dynamic changes to configuration at runtime must be allowed. The more changes to an application that can be made without redeployment/server restart, the better.
5.

Lack of Administration tools. Support teams either perform application administration manually or spend time and money on developing tools.

  • Some applications require ad hoc administrative actions to be performed (eg. process a backlog of messages, resubmit messages, track messages, restart data feeds, etc.) or regular maintenance (eg. housekeeping).
  • Application administration tools must be part of the application deliverable. Spending time and effort on these tools during the design and development stages will pay rich dividends for the in-life support of the application. Also, the designers and developers will be best placed to build such tools.
  • As far as possible, application administration tools must be built to be customizable and automated (eg. as scheduled jobs). The lesser manual intervention by Support teams, the better.

 

How to implement DFS?: If you understand the concept of DFS, you’ll know what’s required to implement it in your project. Here are some guidelines to assist with implementation of DFS.

  • Design for Supportability requires dedicated resources across all teams (design, development, build, testing and support) to work together with utmost cooperation, bearing in mind the ultimate objective – a superb application.
  • Design and Development teams must involve Application Support teams in the early stages of application design (a key requirement for DFS). This will enable the Support team to get an early and good insight into the application and provide feedback with respect to Support requirements. Typically, design and development teams feel it unnecessary to involve Support/Testing teams during the design phase and Support/Testing teams feel it unnecessary to spare resources for the design phase. For DFS to work, this mindset must change.
  • Application Support teams must be involved in the testing phase wherein apart from normal use cases, as many abnormal use case scenarios as possible must be tested. This testing must also cover application administration tools. The Support team must follow a systematic approach to simulate real-world scenarios and test various aspects of the application from a Support perspective and provide feedback to the Development team.

    To all you Application Support folks out there: Spread the word on DFS among your design and development communities. It will make your job easier. DFS should help achieve economies of scale in your Support projects and open the door to new opportunities. Well, there will always be some human presence required (customers won’t want to interact with machines).

VN:F [1.6.5_908]
Rating: 0 (from 0 votes)

When I started this blog in March 2009, I used the editor on the Wordpress Admin console to create and publish posts. Soon, I heard about Windows Live Writer, tried it out and never looked at another blogging tool since then. Windows Live Writer (WLW) is an offline desktop application with rich editing features, which you can use to create posts and publish them to your blog (all popular blogs are supported). A screenshot of WLW taken while creating this post is shown below:

 

WindowsLiveWriter

 

WLW provides the standard features available with good text editors along with blog-specific features like managing different blog accounts, opening and retrieving recently published posts, scheduling publishing of posts and extension via plug-ins. For example, I’ve used a text template plug-in to create a template for certain types of posts, so that whenever I wish to create such a post, I simply use the template and fill in the blanks.

So, WLW makes blogging easy, but I still had the problem of blogging being tied to the laptop on which I had WLW installed. I use 2 laptops, one at office and one at home. Sometimes, I get some ideas for a post at office and wish to just make some quick notes for continuation later. How nice it would be if I can carry the same WLW installation with me wherever I go. i.e. use a portable WLW. Searched online and found a Portable launcher for WLW here. Thanks to Scott Kingery, blogging has been made even easier by becoming portable. I use portable WLW on my USB drive so that all my blog work is saved to the USB drive. Instructions for installing WLW and making it portable are given below:

Installation Of Windows Live Writer:

(1) If you do not already have Windows Live Writer on your computer, download Windows Live Writer from here and install it.

     NOTE: Windows Live is a software suite provided by Microsoft. So, when you launch the installer downloaded in step (1), you will be provide with options to install one or more Windows Live applications. If you’re only interested in WLW, then select only WLW and proceed with the installation.

 

Making Windows Live Writer Portable:

(2) downloadWLWPortablev3.png  or visit TechLifeWeb for the latest update on this software.

 

(3) Extract the downloaded ZIP file for WLW Portable into your USB drive. For example, my USB drive is assigned drive letter P: and I extracted WLWPortablev3.0 into P:, thereby creating a directory P:\WLWPortable3 with sub-directories and files within.

 

(4) Assuming you installed WLW in step (1) into the default location, copy all the files from C:\Program Files\Windows Live\Writer to the following location within the extracted WLWPortable3 on your USB drive:  WLWPortable3\WindowsLiveWriterPortable\App\WindowsLiveWriter

(5) You can now launch WLW Portable v3.0, by clicking on WLWPortable3\WindowsLiveWriterPortable\WindowsLiveWriterPortable.exe. I dragged this WLWPortable executable file onto my RocketDock so that I can easily launch WLWPortable as soon as my USB drive is inserted into my laptop. A screenshot of my set up (WLWPortable Launcher in RocketDock) is shown below:

 

WLWPortable_RocketDock

 

As the above steps make WLW portable, you may even uninstall WLW from your computer and you will still be able to use WLW from your USB drive (remember you copied a bunch of installation files to your USB drive in step(4)).

VN:F [1.6.5_908]
Rating: 0 (from 0 votes)

I required to create some “download buttons” for this blog. I didn’t have any appropriate Image creation/editing tool to create buttons and when searching the www, came across a good website called Da Button Factory which allows you to create buttons online. Salient features of Da Button Factory are:

  • The website is very clean, intuitive and simple to use.
  • Allows creation of 3 types of button : Rectangular, Rounded, Round
  • Allows creation of single colour and dual colour buttons.
  • Allows button to be integrated into your website as CSS code or an image (gif/png/jpg).
  • Allows customization of text, font, shadow, size, border, text padding and colour.

Using two colour gradient, you could create a button that is quite close to, if not like, the Web 2.0 buttons out there (colourful, glossy, shaded buttons).

A screenshot of Da Button Factory, taken while creating my download button is given below. It shows you the various options mentioned above. I opted for a PNG image, as it’s the optimal choice for small images with text (like buttons).

 

dabuttonfactory

VN:F [1.6.5_908]
Rating: +1 (from 1 vote)

WLtd – WebLogic Thread Dumper

WLtd is a simple korn shell script that will enable you manage thread dump operations on WebLogic Servers efficiently and effectively. The salient features of WLtd are given below:

  • Can be run in interactive and non-interactive modes (so can be run as a cron job)
  • Configuration-driven
  • For more than one specified WebLogic Server, thread dumps are taken parallelly (as coprocesses – beneficial on multi-cpu hosts).
  • Thread dumps are extracted from the WebLogic Server stderr logfiles and stored in text files.
  • Option to email thread dumps, as an attachment (text files will be archived and compressed).
  • Housekeeping of thread dump text files

 

System Requirements:  Solaris/Linux, Korn Shell (/bin/ksh)

 

Download WLtd v1.0                 Download WLtd v1.0 ReadMe

 

VN:F [1.6.5_908]
Rating: 0 (from 0 votes)

On an earlier post, I mentioned the convenience of using portable applications on a USB drive. Typically, if you require to use a portable application on your USB drive, you will require to navigate to that application’s installation directory on your USB drive and then launch the application. Creating shortcuts to the applications on your desktop is pointless and will defeat the purpose of portability (tying you to the desktop). Fortunately, some software vendors have developed application launchers or docking software (similar to Windows QuickLaunch bar) that can be completely installed on a USB drive. By using one such dock called RocketDock (similar to the early Mac dock), I have quick access to my portable applications.

 

Download RocketDock v1.3.5

In order to launch RocketDock automatically on a Windows computer as soon as a USB drive containing it is inserted, I did the following:

(1) Downloaded the RocketDock executable. You may use the download button above or click here for the latest version.

NOTE: RocketDock is licensed under a Creative Commons Public License and by downloading RocketDock using the download button above, you are agreeing to use it as per the terms of the Creative Commons Public License. Although the RocketDock website says that RocketDock isn’t supported on 64-bit versions of Windows Operating Systems, I have been using RocketDock with 64-bit Windows Vista Ultimate with no issues experienced so far.

(2) Installed the RocketDock software in a directory RocketDock in the  root of my USB Drive. For example, if my USB drive letter is P:, then the location of the RocketDock directory will be P:\RocketDock.

(3) Added a file called Autorun.inf to the root of my USB drive (P:\Autorun.inf) with the following contents:

[autorun]

open = RocketDock\RocketDock.exe

icon = RocketDock\RocketDock.exe

action = Gimme RocketDock mate!

label = Mr.KIPS

 

The above steps will display a pop-up window similar to the following when the USB drive is inserted:

 

rocketdock_autoplay

 

(4) When RocketDock launched, I then dragged the executables of my most frequently accessed portable applications onto the RocketDock, resulting in a dock similar to the screenshot below:

 

rocketdock

 

So, RocketDock has enabled me access my portable applications quickly.

I have not had a perfect user experience with RocketDock. Sometimes, RocketDock just does not launch when my USB drive is inserted and rarely, it just disappears off the screen. However, both these glitches can be addressed by simply launching the RocketDock manually (RocketDock.exe).

VN:F [1.6.5_908]
Rating: 0 (from 0 votes)

When I started working as an independent contractor, I looked for ways to carry my work with me with minimum fuss and break free from the office PC/laptop. I was looking to install my favourite software completely on a USB drive without being tied to a PC/laptop (eg. by using Windows registry). I had been using a SanDisk Titanium U3 USB drive for a while, but was frustrated with the slow U3 Launchpad and unavailability of U3 versions of popular software. A little research led me to PortableApps.com which provides both individual portable applications and a portable suite of applications that can be run on a variety of hardware (USB drives, iPods, portable hard drives, etc.). I opted to use individual portable applications so that I could install only what I required. So, I formatted my SanDisk U3 USB Drive and installed portable versions of some of the software I use frequently (PuTTY, Mozilla Firefox, Notepad++, etc.) from PortableApps.com. The launch screen of Portable PuTTY is shown below.

 

putty_portable_launch

Although using some portable applications on my USB drive is a bit slower than their non-portable counterparts installed on my laptop, I don’t mind trading this minor performance impact (noticeable only for certain applications like Portable Firefox) for the significant convenience that portable applications bring me. With portable applications on my USB drive, I carry all my work in my pocket and easily switch working between my office and home laptops. For example, I can use all my bookmarks and plugins in Firefox or all my saved sessions in PuTTY (Portable PuTTY can save sessions to a file, thereby removing dependency on the windows registry) on both my office and home laptops (and any other computer) by simply switching the USB drive from one to the other.

Portable Applications give you the advantages of carrying your applications (and obviously data) with you, accessing them from any computer with a USB port and leaving no data behind on the host computer.

VN:F [1.6.5_908]
Rating: 0 (from 0 votes)