Verse Of The Day

Wednesday, January 18, 2006

Log Aggregation And Searching

Ever have one of your customers call you about a transaction that failed in some way, or a message that never reached it's destination, or some other problem? Of course you have. And that usually means tracking down the issue through log files, and lots of them. You have a cluster of application servers, sitting behind load balancers, talking to queue managers and legacy systems. Each of which has their own log files to pour over. Did this particular transaction hit node 1 of the cluster, or node 2, 3, 4, ..., X? Did it get to the cluster or fail on the load balancer?

The old way would be to login to each server and grep through logfiles. The new way is Splunk ( It's a log aggregation and indexing tool with a clever front end to search. Imagine having a Google-like search engine for ALL of your logs, all in one place.

I am playing with the free server version (1.1) right now, and it is slick. There is also a pay version that includes more features and support. The front end does a lot with AJAX principles, so you get full features like a popup-menu as you type with possible search terms and the number of occurrences of each, boolean searches for more complex queries, and search results where every term is clickable. For example, hover on a word or phrase in your search results, and every occurrence in the result set highlights. Clicking searches on that term, control-clicking AND's that term to the current search criteria, and control-alt-clicking NOT's that word from the criteria.

The backend is highly configurable. For example, you can set up directory monitors to watch a directory for new files, or set up tailing processors to index a live file in real-time. There are other options as well, but I'm still new at Splunking. I can see already that this can be an invaluable tool for troubleshooting complex systems, though. I definitely would recommend checking it out soon.

Tuesday, January 10, 2006

Resetting Google Desktop Search Index

Resetting Google Desktop Index

I ran into an issue where Google Desktop told me my index was at its max size. Crappers, I'm not done doing stuff on this PC, so I'm screwed from this point on. Of course, GDS docs tell you to uninstall and reinstall -- what a pain.

A quick Google search brings up Scott Hanselman's blog entry ( ) that tells you how to reset without uninstalling. Basically, just shutdown GDS, and then whack the files under your C:\Documents and Settings\ <username> \Local Settings\application data\Google\Google Desktop Search directory. He mentions "some long number" and files that begin with "sidebar_*.*", but my version of GDS didn't have any of that.

Once you restart GDS it will start from the beginning. This is great because I was getting search hits for stuff I deleted long ago.