Why do I keep doing Google searches to find this site, when I can just post a link here? I'm an idiot.
More importantly, why do I really keep needing to look this up? I'm an idiot.
Thursday, November 29, 2007
Wednesday, November 28, 2007
Splunkin' Like A Madman And Log Management
Splunkin'
After playing with Splunk over a year ago (and liking what I saw) , we finally got the green light to go forward with an implementation at work. Splunk is a log aggregation and search tool. Basically it's a really fancy way of grep'ing through log files without logging in to each server and grep'ing through logs. One search will search logs across all the servers.
Right now I am in the process of estimating our peak daily volume, which more of less means I am ssh'ing in to each app server and Apache server and looking at the file sizes for the various logfiles. The licensing for Splunk is tiered based on peak daily volume, with a free version that will index up to 500MB a day. Their pricing model seems like a real bargain when you compare it to other enterprise software vendors out there, especially when you see the license fee is a "perpetual license" -- you pay once and it is good forever. If your logging volume increases, you can upgrade the license without screwing around with the software installation, and the cost is prorated as the difference between the old and new.
Everyone I have dealt with over at Splunk has been really cool to work with so far as well. Not a bunch of pushy sales-droid types, and they seem more than willing to work with you for unique configurations.
This should make it a lot easier to find and pinpoint issues across our 3 Jboss clusters and the handful of stand-alone Jboss and Tomcat servers and multiple Apache servers.
Apache Log File Management
And speaking of logs, we had an issue where the mod_jk.log file hit the 2 gig limit (32 bit Linux) on one of our Apache servers the other day. That was fun to track down, since some stuff seemed to work and other stuff did not. Our web applications were working fine, but web services calls into our boxes were failing. According to our Jboss logs, we were processing the WS calls and returning valid responses, but the clients were getting errors about the connection being closed while reading the response, or null objects being returned, depending on which platform they were connecting from.
So now I am working on getting logrotate setup for our Apache installations. While I am in this mode, I think I'll whip up some Nagios checks to alert if the filesizes grow to say 1.5G (which shouldn't happen with daily log rotation and compression).
Setting it up is pretty easy. I edited the /etc/logrotate/httpd file to add the following:
After playing with Splunk over a year ago (and liking what I saw) , we finally got the green light to go forward with an implementation at work. Splunk is a log aggregation and search tool. Basically it's a really fancy way of grep'ing through log files without logging in to each server and grep'ing through logs. One search will search logs across all the servers.
Right now I am in the process of estimating our peak daily volume, which more of less means I am ssh'ing in to each app server and Apache server and looking at the file sizes for the various logfiles. The licensing for Splunk is tiered based on peak daily volume, with a free version that will index up to 500MB a day. Their pricing model seems like a real bargain when you compare it to other enterprise software vendors out there, especially when you see the license fee is a "perpetual license" -- you pay once and it is good forever. If your logging volume increases, you can upgrade the license without screwing around with the software installation, and the cost is prorated as the difference between the old and new.
Everyone I have dealt with over at Splunk has been really cool to work with so far as well. Not a bunch of pushy sales-droid types, and they seem more than willing to work with you for unique configurations.
This should make it a lot easier to find and pinpoint issues across our 3 Jboss clusters and the handful of stand-alone Jboss and Tomcat servers and multiple Apache servers.
Apache Log File Management
And speaking of logs, we had an issue where the mod_jk.log file hit the 2 gig limit (32 bit Linux) on one of our Apache servers the other day. That was fun to track down, since some stuff seemed to work and other stuff did not. Our web applications were working fine, but web services calls into our boxes were failing. According to our Jboss logs, we were processing the WS calls and returning valid responses, but the clients were getting errors about the connection being closed while reading the response, or null objects being returned, depending on which platform they were connecting from.
So now I am working on getting logrotate setup for our Apache installations. While I am in this mode, I think I'll whip up some Nagios checks to alert if the filesizes grow to say 1.5G (which shouldn't happen with daily log rotation and compression).
Setting it up is pretty easy. I edited the /etc/logrotate/httpd file to add the following:
/app/j2http/apache-2.0.55/logs/*log {
daily
rotate 10
copytruncate
notifempty
compress
}
Tuesday, November 27, 2007
Chicken Farming Comes To An End
Our first experiment in raising chickens for food ended this past weekend. We had 5 Rhode Island Red hens left, and I really didn't want to take care of them through our harsh upstate winter. Since I was already spending my weekend butchering and packing my deer, making venison jerky, and grinding venison into burger, I figured I would go ahead and take care of the last 5. So now, with 5 chickens in the freezer and none in the coop, our first year as chicken farmers has officially ended.
We learned a lot and had some fun raising chickens, and the neighbors didn't seem to mind too much. The meat and extra eggs were nice as well. This is definitely something we will do again in the spring, but there are a few things I will do differently.
We learned a lot and had some fun raising chickens, and the neighbors didn't seem to mind too much. The meat and extra eggs were nice as well. This is definitely something we will do again in the spring, but there are a few things I will do differently.
- Make the chicken run around the coop bigger. It was originally supposed to be bigger but I ran out of time and ran into big rocks on the one corner. So I let them free range a lot, which wasn't terrible, but led to some minor annoyances here and there.
- Make a nest box. I meant to do this but again ran out of time. Then I read about chickens not laying until they are 6-8 months old, so figured they would all be gone by then anyways. Then my neighbor Dave started finding eggs in his flower garden, and we started finding eggs in our barn. If we start them with a nest box and minimize the free ranging, we should have to go on Easter egg hunts all the time.
- Make the chicken run higher or put a top on it. Even when I closed the chickens in, they literally flew the coop and got out anyways.
- Get some bigger meat chickens. The Buff Orpington's and Rhode Island Reds we got are known as dual purpose birds, good eating size plus good layers. It seemed to take forever for these birds to reach a good eating size, so next time I will get some Cornish X Rocks or other big meaty type birds as well as the Buff's and Red's.
- Get an open watering trough. The watering jug I got is a pain, since you have to take it apart and turn it over to fill, then put it together and turn it back over. It would be so much easier (and drier) to just pour water into a trough instead of messing with that thing.
Subscribe to:
Posts (Atom)