Problem:

A Solaris 9 mountpoint was 100% utilized (as per “df”) and no new files could be added.

df output:

cybergavin@myhost:/dashboard> df –h /dashboard
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/A19278-S01-7uitx-dg/dashboard
                        16G    16G   2.1M   100%    /dashboard

du output:

cybergavin@myhost:/dashboard> du –sk /dashboard
1789259 /dashboard

Background & Analysis:

As you can see above, both “du” and “df” provide significantly different metrics for the utilization of /dashboard. The “df” output tells me that I have very little free space (~ 2.1 MB) whereas the “du” output indicates that I have around 14 GB free space.

Well, first and foremost, df and du intend to give you disk usage stats, but they do not work in the same way. Refer this article to understand the differences between df and du.

Secondly, the mountpoint /dashboard was mounted on a VxFS. The dmesg output showed the following:

Feb  1 09:29:00 myhost vxfs: [ID 702911 kern.notice] NOTICE: msgcnt 112748 mesg 001: V-2-1: vx_nospace -  /dev/vx/dsk/A19278-S01-7uitx-dg/dashboard file system full (1 block extent)

An explanation for the above (quite obvious) message is given in this Symantec article.

I found a runaway background process (iostat –x 2) running for the past 2 months. It was a process launched by a shell script. The shell script exited, but the process wasn’t killed. The process was redirecting its output to a file and that file was also deleted. Consequently, the process’ stdout file descriptor (1) was not closed and the process was still writing to stdout. This caused the space occupied by the stdout to be hidden. To determine how much space is actually being used by the process when writing to stdout, try the following command (<pid> = process id):

ls -l /proc/<pid>/fd/1

Solution:

Killed the runaway process and the mountpoint utilization dropped significantly to 14%. Further, df and du outputs correlated.

Root Cause:

A runaway process was consuming most of the disk space and this disk space consumption was “hidden” because the file to which the process’ stdout was being redirected, was deleted.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

Messaging Bridge on WebLogic Server 8.1 does not start. Following errors seen in server log:

####<Jan 27, 2010 10:10:13 AM GMT> <Info> <MessagingBridge> <myhost> <managed1> <ExecuteThread: ‘4′ for queue: ‘MessagingBridge’> <<WLS Kernel>>
<> <BEA-200032> <Bridge "MyBridge" is configured to disallow degradation of its quality of service in cases where the configured quality of service is unreachable.>
####<Jan 27, 2010 10:10:13 AM GMT> <Error> <MessagingBridge> <myhost> <managed1> <ExecuteThread: ‘4′ for queue: ‘MessagingBridge’> <<WLS Kernel>> <> <BEA-200025> <Bridge "MyBridge" failed to start, because the quality of service configured (Exactly-once) is unreachable. This is likely due to an invalid configuration or adapter limitations.>
####<Jan 27, 2010 10:10:13 AM GMT> <Info> <MessagingBridge> <myhost> <managed1> <ExecuteThread: ‘4′ for queue: ‘MessagingBridge’> <<WLS Kernel>> <> <BEA-200034> <Bridge "MyBridge" is shut down.>

NOTE: MyBridge connects a WebLogic JMS destination (source) to an MQ destination (target).

Background & Analysis:

In order for Messaging Bridges on WebLogic 8.1 to use Exactly-Once QOS, the following requirements must be met:

• Messaging Bridge adapter must be jms-xa-adp.rar and its JNDI name is eis.jms.WLSConnectionFactoryJNDIXA.
• Connection Factories for source Bridge destinations must be XA-enabled.
• Connection Factories used for target Bridge Destinations must be XA-enabled.
• Messaging Bridges must be configured with Exactly-Once QOS.
• The “QOS Degradation Allowed” checkbox must be unchecked.

With the above, it is recommended that the Messaging Bridges be Synchronous for better performance (fewer transaction commits).

From the log snippet above, you can see that the Messaging Bridge MyBridge could not start because the QOS (Exactly-Once) was unreachable and the Bridge was not allowed to degrade its QOS.

The QOS will typically be unreachable due to adapter, bridge configuration or bridge destination configuration issues as referred to in the log snippet.

Solution:

Enabled XA on the ConnectionFactory for the target Bridge Destination on MQ.

In order to satisfy the Exactly-Once QOS, both source and target destination connection factories must be XA-enabled.

Root Cause:

The ConnectionFactory for the target Bridge Destination on MQ was configured as non-XA, thereby preventing the Messaging Bridge from initiating an XA connection from the WebLogic Bridge destination to the MQ Bridge destination. Since the Messaging Bridge was not allowed to lower its QOS to make the connection, it failed to start properly.

 NOTE: By allowing QOS degradation, the MessagingBridge will connect to the MQ destination even if the ConnectionFactory for the Bridge Destination on MQ were non-XA. However, the choice of QOS must be driven by business requirements and not by technical workarounds.

 Reference: Oracle Documentation

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

WebLogic Administration Server (with WebLogic Integration) 8.1 does not start. Following errors seen in stdout/stderr/server logs:

####<Jan 25, 2010 4:30:26 PM GMT> <Error> <JDBC> <myhost> <myadmin> <main> <<WLS Kernel>> <> <BEA-001151> <Data Source "cgDataSource" deployment failed with the following error: null.>

####<Jan 25, 2010 4:30:26 PM GMT> <Info> <JDBC> <myhost> <myadmin> <main> <<WLS Kernel>> <> <BEA-001156> <Stack trace associated with message 001151 follows:

weblogic.common.ResourceException

        at weblogic.jdbc.common.internal.DataSourceManager.createDataSource(DataSourceManager.java:264)

####<Jan 25, 2010 4:30:30 PM GMT> <Error> <WLW> <myhost> <myadmin> <main> <<WLS Kernel>> <> <000000> <Failed to obtain connection to datasource=cgDataSource, using generic DB properties>

####<Jan 25, 2010 4:30:31 PM GMT> <Error> <WLW> <myhost> <myadmin> <main> <<WLS Kernel>> <> <000000> <Error in startup class com.bea.wli.store.DocumentStoreSetup Method: init:

java.lang.IllegalStateException: Unable to start DocumentStore:  com.bea.wli.store.DocumentStoreException: Could not find SQL Document Store cgDataSource

            .

            .

Background & Analysis:

WebLogic Integration (WLI) is a software Business Process Integration framework that runs on WebLogic Server. WLI also includes a console application (wliconsole) to manage WLI configuration. This console application is deployed on the WebLogic Administration Server. Since the console application interacts with the database, it uses default data sources and connection pools (e.g. cgDataSource and cgConnectionPool) for database connectivity.

The errors above indicate that the cgDataSource failed to deploy and consequently, a startup class could not obtain connections to the database, thereby failing deployment and preventing the Administration Server from starting.

Data Sources use Connection Pools to obtain database connections.

Solution:

Ensure that the Connection Pool for the cgDataSource is configured properly (correct JDBC driver, URL, credentials, etc.) and targeted/deployed on the Administration Server (not just the cluster).

Root Cause:

The connection pool (cgConnectionPool) for the data source cgDataSource was not deployed on the Administration Server.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

In WebLogic versions prior to 9, JSAFE decryption exceptions and password encryption could be resolved/performed by simply setting plain-text passwords in config.xml as described here.

However, in WebLogic versions later than 9.0, setting plain-text passwords in config.xml (Production mode environments) will throw the following error:

<Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: [Management:141266]Parsing Failure in config.xml: java.lang.IllegalArgumentException: In production mode, it’s not allowed to set a clear text value to the property: PasswordEncrypted of ServerStartMBean>

So, for WebLogic 9.x+ versions, if you doubt passwords in config.xml, you will need to encrypt plain-text passwords and configure them in config.xml as follows:

STEP 1: Encrypt the password you wish to change in config.xml.

Example: You experience JSAFE exceptions with the password of one of your connection pools and you doubt the encrypted password in config.xml.

• Source the WebLogic environment (to set CLASSPATH and other variables) as follows:

cd <domain-dir>/bin
. ./setDomainEnv.sh

• Encrypt the plain-text password as follows:

cd <domain-dir>
java weblogic.security.Encrypt <password>

STEP 2: Update the config files (config.xml, jdbc, etc.) with the encrypted password obtained above.

NOTE: The above encryption procedure will work even with earlier versions of WebLogic (e.g. 8.1).

Have you provided your personal email account when registering on websites using email confirmation? Have you hesitated to provide comments on blogs, forums, etc. because of fear of having to provide your personal email address? Of course, you always have the option of registering a free email account just for this purpose, but this means you have yet another username/password pair to remember or require some time to create a fictional identity. Instead, how about using a temporary email address that requires no registration? That’s exactly what 10 Minute Mail offers!! It provides you with a temporary email address that is valid for 10 minutes (more than enough time to respond to email registrations). Beat the spam caused by various online registrations!

Well, I’ve now found that Disposable Email Addressing (DEA) has been around for some time and there are several DEA services on the web. Refer this blog post for 20 such disposable email services (Mailinator, MyTrashMail, etc.). Website admins are nowadays blocking registrations with temporary email addresses. Well, you can always try!