Verse Of The Day

Tuesday, July 31, 2007

Using JConsole To Connect To Remote JVM

We have been having issues with a certain Java web application we inherited. Actually, we've had a few problems with it, but that's another story.

One issue was that, even though there is nothing running on this Linux box except a single instance of Tomcat (5.5) with this single web application deployed to it, we will get into frequent issues where we get paged for "high CPU utilization". Log in to the machine, and the Java process is taking 100% CPU. It might stay like that for a few hours, or even a few days. Occasionally it clears up on it's own, but usually the server stops responding and we have to restart Tomcat.

We have had another issue where we would get Out of Memory exceptions (Heap space), and the JVM would stop running. The two issues weren't necessarily related, but we couldn't rule out the possibility either.

We don't have much visibility into the application and what's going on since it is a vendor built "black box". We have some customized source code, but most of it is off limits to us. They rolled their own database connection pool, MVC framework, persistence framework, etc.

In comes Jconsole. JConsole is an awesome tool that comes bundled with JDK 1.5 and above. It connects to the JVM and gives you all the info you could want on the various JVM memory pools, garbage collection, threads, classloading, and lets you manage anything exposed via JMX. It also has lots of pretty graphs, such as memory usage and garbage collections over time, for any or all of the memory pools (heap, non-heap, or individual pools, like permgen and eden space). Same goes for threads and loaded classes --current number, peak, total created.

The best thing about JConsole is the ability to connect to remote JVM's so you don't add too much overhead on the box being monitored. I have JConsole hooked up to my test and production servers, and it helped me prove that the two issues above were connected. There is some condition in the application (yet to be found, actually, first step was proof of what's really happening) that causes a substantial memory leak, and once the memory usage gets at it's ceiling, the garbage collection thread basically runs constantly, trying to regain some trivial amount of memory, then filling it up, and running GC again.

With an average of 100 active sessions at a time plus full garbage collection running non-stop, the CPU gets consumed quickly. I monitored the app for about a week and a half, and memory and CPU looked great -- there were a bit over 200 full GC's in that time period. Then this past weekend, we had to restart because of a DNS issue. That was Friday evening, and by Monday morning there were over 2000 full GC's performed, and I was restarting a non-responsive server by lunch time.

Below are two screenshots, one is of several days of "normal" memory usage, notice gradual rise and then sharp decrease at full GC, all the while keeping well below the JVM's allotted memory ceiling. Second is this past weekend's issue, where memory is hovering at ceiling and a full GC doesn't do much. Also notice the old generation memory pool is quite full.


To set up JConsole to run on a local JVM, you only need ot pass one extra argument to the JVM:
To set up remote (with no security), it is a matter of adding a few more parameters to the remote JVM at startup:
Then when you start up JConsole, go to the remote tab, enter the server name and the port that you specified (in this case, 8004).

Simple as that. It starts collecting stats immediately and the graphs appear. As you explore the JMX tree, you will notice yo ucan click on some of the stats and the simple integer displays "opens up" into a full graph display. I'm doing this to watch the active sessions patterns through the day and week.

Monday, July 09, 2007

I Feel So Naughty

Most of my development experience has been with structured languages such as C and Java.

Given that, I feel almost naughty working with Ruby's loose typing. It's like I am getting away with something, doing something oh so wrong. For some reason it never seemed odd with shell scripts, but does here. It's both liberating and a little scary at the same time.

No compiler errors, no warnings, no runtime exceptions, no type mismatches, nothing. You can make x = 3 then turn around and make x = "dude", and then set x = with out any issues.

Sunday, July 08, 2007

And The Tree Came Crashing Down

We went to church this morning, and then off to pick up the kids. When we got home, everything was wet, so we knew it had rained. Then for some reason, I felt the need to go over and check on the chickens. What I saw amazed me. 3 big branches of the maple tree had come crashing down on the coop and the fenced in run. I went back to check it out, and the chicken run was completely full of branches and leaves -- I couldn't see the chickens, but I could hear them clucking and running around through the leaves. The back part of the fence was completely down to the ground.

Luckily, no damage was done to the roof of the coop, no chickens were hurt, and once I cleared out all the branches, I was able to repair the fence with minimal effort. Now it's mostly a cleanup effort.

Another Picture of Lexi

Here's an updated picture of Lexi the Wheaten Terrier. She's growing up and getting a lot lighter now. You can see a lot more of the white hair around her face and her belly is almost completely white now.

Friday, July 06, 2007

Black Raspberry Time

It's that time of year again. The black raspberries are starting to ripen in the back yard and in the woods behind the house. The berries seem to be quite a bit smaller than normal, and a little less abundant than normal, probably because we've had a lack of rain so far this summer. But the berries that are out there are quite yummy. We picked a small bowl on the 4th, and I picked a handful today after work. In the morning, I plan on taking my big bowl out again for round two. Normally I can fill a large mixing bowl in one pass, and this time I just got manybe 1/5 of a bowl the first time through.

Tuesday, July 03, 2007

Office Art, Part 1

I drew this on the back of one of my project folders and snapped the picture with my ViewSonic v36 Pocket PC. This is what happens when I get bored in meetings.