Ubuntu perfect setup for Java Developers

MySQL General Query Log Filtering using grep and regular expressions

MYSQL has a very useful feature called Query Log which allows you among other things to debug your application code by viewing DB DML's submitted live from the application straight into DB back-end (Low level, the best!).

To configure this little joy you will need to play a bit with your mysql server settings.

Edit your my.cnf file (Under debian that is /etc/mysql/my.cnf) in the section of

[mysqld]

add

log=/var/log/mysql/query.log

This is explained in greater details at this post.


Now the thing is query log as the name suggests logs everything that has SQL smell on it and I do mean EVERYTHING (besides CRUD operations you will see: commit/rollbacks, show tables, use DB-XYZ and co.) - Which is mostly junk when the only thing you really need is to trace your hibernate based Console save to get the back-end integration going... To summarize in a catch phrase: You wan't to filter out the log spam when debugging your application (vs. your replication ;).

To be more efficient and be able to look at the logs in real time I've created a fast and dirty regex. The following chain of piped grep's will leave you only with INSERT, UPDATE, DELETE statements which is exactly what I was after (Your mileage might vary).

One would use something like this:
tailf /var/log/mysql/query.log | grep -E '[[:space:]]+[[:digit:]]+[[:space:]]Query' | grep -ivE 'Query([[:space:]])+(/\*.*\*/)?(select|set|show|commit|rollback|use)'

The command works by constantly (tailf) pushing events from mysql query.log throw the 2 greps which do some smarty ass filtering to leave you with the interesting part. If you need other command (SELECT for example) you can remove them from the second grep section. If you want to see just a single table add another 3'rd grep to the chain and filter just by table name.

Comments are welcome.

Enjoy,
Maxim.

Eclipse 3.6 Helios - Final version available for download

I don't know what's so special about this build from eclipse.org foundation but for some reason the excitement is high.

Well friends, without further adieu I give you the path to the final release download location 4 days before the official release schedule

Check it out from here http://mirrors.xmission.com/eclipse/technology/epp/downloads/release/helios/R/

Enjoy.

In the working for automatic User Agent Detecting algorithem

I'm thinking about writing a tool that will extract information from User-Agent strings.

In the mean time, here is a quite comprehensive list of tools I've been able to find around the net that can assist me in this task


Tone of links for known User Agent strings

DB:
http://www.useragentstring.com/
http://www.user-agents.org/index.shtml?moz
http://www.icehousedesigns.com/useragents/spiderlist.php
http://www.botsvsbrowsers.com/
http://useragentstring.com/pages/useragentstring.php

Auto detection tools:
http://nerds.palmdrive.net/useragent/code.html
http://stackoverflow.com/questions/1005153/auto-detect-mobile-browser-via-user-agent
http://mdbf.codeplex.com/
http://detectmobilebrowser.com/
http://stackoverflow.com/questions/2508214/regexp-that-matches-user-agents-of-end-user-browsers-but-not-crawlers-with-90
http://stackoverflow.com/questions/927552/parsing-http-user-agent-string
http://www.djangosnippets.org/snippets/267/
http://browsers.garykeith.com/downloads.asp
http://user-agent-string.info/
http://pypi.python.org/pypi/httpagentparser
http://www.handsetdetection.com/
http://www.tnl.net/ua/OS/Cross-Platform
http://stackoverflow.com/questions/1005153/auto-detect-mobile-browser-via-user-agent
http://api.jquery.com/jQuery.browser/#jQuery.browser.version2
http://www.quirksmode.org/js/detect.html

INFO:
http://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/


If you know any other resources that can help please post in comments.
I will update this post as soon as I have some news on the subject.

Thank you,
Maxim.

Resetting cached password of Subclipse

I was asked to document a FAQ about Subclipse and cached passwords (my turn to play with the IT hat!). Here is the email (kept here for future reference).



Hello Friends,

Eclipse subversion plugin (Subclipse) will cache the username & password of the last user that did a successful commit with your eclipse installation.
This can be problematic if this user is not you, it turns out that until further notice all your svn commits will be under different username.

To resolve this issue, delete the following 2 folders:
rm -rf /opt/eclipse3.3/configuration/org.eclipse.core.runtime/.keyring
rm -rf ~/.subversion/
Then restart eclipse.

An alternative solution would be to kindly ask the busy IT department to change the svn password of the user under which your commits are being executed, then on first commit attempt Eclipse should prompt your for username & password, which will allow you to input your information.

Reference: http://nlp.cs.byu.edu/~rah67/wordpress/?p=4

Maxim.

Running Average calculation implementation in Java

Quick blog and store note:

Here is a code to calculate average series of length N without the problem of overflow that naive average calculation could face.

/**
* Copyright (c) 2010, Maxim Veksler 
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without modification, 
* are permitted provided that the following conditions are met: 
* 
* Redistributions of source code must retain the above copyright notice, this list
* of conditions and the following disclaimer.
* 
* Redistributions in binary form must reproduce the above copyright notice, 
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution. Neither the name
* of the http://bytecoded.blogspot.com/ nor the names of its contributors
* may be used to endorse or promote products derived from this software 
* without specific prior written permission.
* 
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
* THE POSSIBILITY OF SUCH DAMAGE.
*/

/**
* Calculate average of arbitrary length series of doubles, running average ensures 
* result won't overflow.
* 
* Algorithm: CELLCOUNT represents number of cells calculated so far. 
* For each : cell
*  avg =  ((CELLCOUNT-1) * avg) + ((1/CELLCOUNT) * avg)
* 
* @param series vararg list of parameters (array of doubles).
* @return Average calculated from supplied series.
*/
public static double movingAverage(double... series) {
if(series.length < 1) {
   return 0;
  } 

  double avg = series[0];
  for(int i = 1; i < series.length; i++) {
   avg = (avg * ((double)i/(i+1))) + (series[i] * ((double)1/(i+1)));
  }
  
  return avg;
 }

Jasper Reports - Subreport usage demonstration

This post is part of Jasper Reports demo documentation project, please read that post for information on downloading the discussed demo files

jasperreports-3.7.0/demo/samples/subreport

This demonstration will teach you how to reuse sub-reports for enhancing report visual layout. Subreports among other things allow you to achieve high level of data presentation complexity with relatively small overhead.

a Subreport in Jasper terminology is single jrxml report file which is designed to be included in a master report, the master report should pass parameters to the subreport (For ex. rowId or street address) which are accessed by the subreport using $P{"parameterName"} notation. Optional feature for subreport is return parameters which allow passing information from calculations done within the subreport back to the master report, under the hood these are variables which are defined in the master report and passed to the subreport element (we will see an example of this later in the post).


Lets start with the basics setup required to get the report demo running:

First you will need to start the hsqldb database.

Next open all 3 reports in iReports.

• MasterReport.jrxml
• ProductReport.jrxml
• AddressReport.jrxml

Test the report is working:

1. In iReports, assuming you followed the hsqlsdb setup guide you should select "jdbc:hsqldb:hsql://localhost" as your datasource.
2. Make sure you have focus on the MasterReport tab
3. Now click "Preview"
1. In the popup windows select "Use Default"

This should bring a semi-working report that looks something like this:

To configure a fully functional we will need to do some adjustment steps, as we are operating the report from iReport instead from Java code.

MasterReport has has a parameter called ProductsSubreport of type net.sf.jasperreports.engine.JasperReport this parameter can be used in production environment to dynamically supply the subreport Class Instance that will be used by one of this subreport elements (more on this later). For now, lets change this to supply a fixed sub report element so that we can preview the full report in iReports as Jasper team designed it.

Locate in iReports the element's:

1. MasterReport -> Details 1 -> $P{ProductsSubreport} then set the following fields:
   
   
   
   
   • Subreport Expression = "ProductReport.jasper"
   • Expression Class = java.lang.String
   
   
   
   
   
2. MasterReport -> Parameters -> ProductsSubreport then set the following fields:
   
   
   
   • Use as a prompt = [ ] (unselected)
   
   
   
   
   

Now if you try to preview the report again the following will be shown:

Good! The report is fully operation. Lets start analyzing.

Understanding Report logical Structure

This report prints product orders grouped by city. The report layout is as following:

== Report Title =
 Heading + Total number of orders
== Page Title ==
Page Heading
== Details ==
= First page =
City name     COUNTER: Number of address ordered in this city
= Next n pages =
City name     cont


Then 2 lists order side by side displaying:
First list =  Products ordered by people in current city
Second list = Address that made orders in current city

== Page Footer ==
Page X out of XXX

Understanding Report JRXML structure

MasterReport.jrxml

Master Report :: Query
The main report is based on the following SQL query: SELECT City FROM Address GROUP BY City ORDER BY City which should self explentionary.


Master Report -> Title
This section contains static text and a graphical horizontal bar.

Pay attention to the Text Field that as part of his expression uses the dynamically calculated variable $V{ProductTotalPrice} this variable is returned from the sub reports called during report filling phase and the results are summarized by the engine. More on this later.

Master Report -> Page Header 
Nothing special here. Static text that will be printed at the start of each page.

Master Report -> Details 1

The heart of the report.

It is important to understand that a details band can grow beyond single page size, Jasper has special treatment for these situation via settings which can be set on each element as we shall see next:

On the first page we want to print "CityName"  COUNTER Addresses, on the following pages for the same city OTOH all we want to print is "CityName"  cont

The idea is that if a city band grows to several pages we will reprint the "CityName" and the text "Cont"

TextField $F{City} has the setting Print Repeated Values and Print When Details Overflows which causes it repeat on each page, even when the band for current city grows beyond 1 page size.


TextField $V{CityAddressCount} + " addresses" has the setting Print Repeated Values which will allow this field to be printed even if 2 cities have the same number of addresses but note this Print When Details Overflows is turned off which causes this text to be printed only on the first page of the band !

Label (continued) has the setting Print Repeated Values turned off and Print When Details Overflows enabled. This cause this label to be printed only on overflowed bands (meaning only on 2nd pages and following...).

Subreport $P{ProductsSubreport} is the embedding of the ProductReport subreport. Note this report gets the Database connection from the MasterReport, we also pass a parameter to this report City = $F{City}. This report returns a variable, the way this is implement in Jasper Reports is as following: We define a variable in the master report ProductTotalPrice, then we pass this variable to the subreport which sets values into this variable. Then, after the subreport evaluation has ended in the MasterReport we can use this variables value. In the present context we use this method to calculate a summary (TOTAL) of the price of all the products being evaluated by the subreport. Note that as part of the definition of the variable on the subreport element we configure Calculation Type = Sum this allows the engine to do the summary of this variable (which is passed to many subreport's - A subreport is evaluated for each city). See Variables section for another note on the ProductTotalPrice variable.


Master Report -> Variables


ProductTotalPrice this variable is used by the MasterReport to display the sum of the price of all products that have been evaluated by the subreport ProductReport note this variable has the property Calculation = System meaning that it is being manipulated separately (not by the engine). This is actually a little hack (IMHO) because we do sum it using the engine only we do it on the Subreport $P{ProductsSubreport} variable configuration properties. Just in case you wondered: Yes! it works.

ProductReport.jrxml

To be continued...

AddressReport.jrxml

To be continued...