A weird Spring JDBC issue

Recently, I resolved a weird Spring JDBC problem related to stored procedure compilation on multi-threading environment.

On production server, we see SP error when server starts up as below:

2015-01-23 07:40:13,454 INFO   [TokenStore] Failed to find access token for token bbd333e9-f02a-4b89-99a7-75929a99c4d4
org.springframework.jdbc.BadSqlGrammarException: CallableStatementCallback; bad SQL grammar [{call AUTH_PKG.GET_TOKEN(?, ?, ?, ?, )}, ?]; nested exception is java.sql.SQLException: ORA-06550: line 1, column 64:
PLS-00103: Encountered the symbol ")" when expecting one of the following:

 It is very strange because the Spring generated SQL is wrongly formatted!  When look at the log files for the bad and good starts of the server, I found the problem is the proc is not compiled properly when it is called when the server starts up.

Usually we put compile() method in the SP constructor to allow Spring to compile the proc before it is referred/executed. But we didn’t do that for the proc classes in this package. This will delay the proc compilation until its execution time. When it is done correctly, we should see this log:

2015-01-26 14:42:04,147 DEBUG () [RefreshToken] [[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] SQL operation not compiled before execution - invoking compile
2015-01-26 14:42:04,148 DEBUG () [RefreshToken] [[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] Compiled stored procedure. Call string is [{call AUTH_PKG.GET_TOKEN(?, ?, ?, ?, ?)}]
2015-01-26 14:42:04,148 DEBUG () [RefreshToken] [[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] RdbmsOperation with SQL [AUTH_PKG.GET_TOKEN] compiled

But at times when we have trouble we don’t see the above. We see the proc execute time exception immediately without the “proc compiled” log line. Seems like Spring executed the proc before it was compiled. 

 Looking at Spring source code of StoredProcedure class, we found the “isCompiled()” method is not synchronized/thread safe.  When the same proc being called by multiple threads, the proc compilation is messed up. This causes failure for all the future calls to the proc.


Because this proc is a frequent call when the server starts up, we always see trouble in this proc but not in the others. Also because weblogic will queue up the requests during server starts up, we always end up with multiple threads calling to the same proc when the class initialization is done. 

The fix would be always put “compile()” statement in the SP class constructor to allow Spring initialize/compile it before it can be referred/used. 

JRebel rocks

Recently I used JRebel to do hot deployment of a complicated Web application on BEA WebLogic 12. I found it is very easy to setup and give me a great improvement in web development. I can see the result of my Java or xhtml code changes on the website in a flash. Here is a summary of what I did to enable JRebel with WebLogic and Eclipse:
1. Download JRebel plugin for Eclipse, validate with a license.
2. Enable the Eclipse projects with JRebel by "Add JRebel Nature".
3. Check the auto generated rebel.xml file for each java project to make sure the monitored target directories are correct. Only thing I needed to change is the Web directory since we didn't put our web stuff under default WebContent directory.
4. Build the war file as usual to the deployment directory.
5. Add jrebel.jar to the weblogic domain home directory.  Add this to the startWebLogic.cmd file:
Add -javaagent:%DOMAIN_HOME%\jrebel.jar -Drebel.remoting_plugin=false to the `set SAVE_JAVA_OPTIONS=` line.
6.  Start the Weblogic server and you should see the projects' target directories are monitored by JRebel for changes now.

Here is how it works behind the scene:
JRebel will monitor and detect changes in the target directories for compiled Java files or copied over web based files. Eclipse is moving the changes from source to target directories (compile or copy). Weblogic will allow JRebel to hot swap the files whether it is a Java class, or Xhtml or a properties file.

Combining JReble with Eclipse remote debugging, the Web development and debugging just becomes really easy!

Q&A for Activemq JMX connection issue (permission exception) using AnyConnect VPN

After connected to VPN using Cisco Any connect client pgm, I found Java program will have connection issue when accessing the JMX service hosted by Activemq on the same machine(localhost). Permission exception is thrown when trying to connect to the JMX rmi service. The IP address bound to the JMX is an IP address assigned by VPN.

I found the root cause is the JMX stub can not be reached when the service is bound to the IP address assigned by VPN. This IP is behind the NAT and permission exception is thrown when trying to be reached. The workaround is to force bind the JMX server to the public/accessible IP address. When running services locally, adding JVM option ”-Djava.rmi.server.hostname=127.0.0.1” to the startup scripts will solve the issue.

Conflicting classes in multiple dependent jars

Working in Teradata, I discovered an issue related to multiple versions of Spring dependent jars. The problem is that we use Ivy to add project jar dependence. Each jar/project can have different versions of Spring jars in its library. For a master project which depends on several other base projects and having its own Ivy library, as a whole, its library will have different version of Spring jar files. These jars can contain same class with different contents. Some earlier versions might not have the implementation for certain methods added later. This problem is mostly applicable in Eclipse when all the library files are loaded in CLASSPATH, including the library files for dependent projects. A standalone jar invocation can avoid this issue by providing a precise Manifest file containing only needed jars!

One error I got when running the project in Eclipse is as below:

Invalid property 'maxConcurrentConsumers' of bean class [org.springframework.jms.listener.DefaultMessageListenerContainer]

Notice due to the usage of Spring injection, the above error is not discovered at compilation time. In the example DefaultMessageListenerContainer.setMaxConcurrentConsumers is provided in Spring 2.0.6 but not in Spring 2.0. Other issues are also found including Spring's support for Circular references.

I found it is hard to remove the dependence of the old versions of Spring jars from dependent projects. Even using "transitive = 'false'" tag on the Ivy configuration does not work. I have to live with multiple versions of jars containing the same class. To make sure using the latest version of the jar, I have to depend on the JVM class loader order. Fortunately I found this is doable within eclipse. Eclipse's JVM will load the jars in the order of your specification in the "Java Build Path" --> "Order and Export". As long as the correct version of the jar (or ivy.xml) is before the obsolete jar (or dependent project), the issue can be avoided.

Configure MQ on iSeries

To run MQ on iSeries, one must start the MQ subsystem with command:

strsbs sbsd(qmqm/qmqm)

Then we can access the queue manager with "wrkmqm" command.

Europa catch

Europa is the name for Eclipse 3.3 EE version. I recently began to Europa for JEE5 development. Found a big catch. I used Europa with JBoss4.2.2. For some reason I found I can not deploy a class named "TestServlet" as a web service class. No matter what the package name I gave. It always result in error "Allocate exception for servlet TestServlet". Very strange.

import signature files from iSeries

Here is how I imported the company's digital certificate (key pair) from iSeries machine:

1. get the KDB file from the iSeries file structure and move it to a windows box
2. Use a java-based tool named iKeyMan.bat to open the key db file. The tool is shipped with Websphere product.
3. export the key pair to an existing "jks" format file which is java keystore. The file will server as the keystore, and can be used to sign jar files.

DB connection error in websphere + iSeries

Recently we have a stale connections problems in a WebSphere App server running on iSeries database. I realized the problem might be the getConnection() method is not synchronized(serialized) at the JDBC driver implementation. So two threads can potentially get the same connection and step on each other. It sounds like a bug to me. The solution is easy, just synchronize the code that call getConnection() method, of course, I am assuming the calling class is already in a Singleton mode.