The Boggling of the Mind

Scott Adams sent me on a Wikipedia journey today with his article Monte Hall Problem. I started reading about the double-slit experiment, its extension in the quantum-eraser experiment, and finally quantum entanglement.

I suggest you read them, but only if you are prepared on looking at the world around you a little differently. If you're happy with the things the way they are I suggest you read something else instead.

history meme

history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head

Home:

155 cd
97 sudo
79 ls
69 svn
51 heyu
47 mvn
38 grep
37 less
28 vi
23 screen

Work:

252 mvn
119 cd
86 svn
50 ls
40 rm
32 less
22 ssh
22 man
22 cp
21 psql

Open Source in the Enterprise

I guess many like me working in a large enterprise can relate to a story like this one: Jonathan Schwartz's Blog: Freedom's Choice:

The CIO responded categorically with 'we don't run MySQL, we run [name withheld to protect the proprietary].' The CISO said, 'We can't just let developers download software off the net, you know, we've got regulation and security to worry about.' The CTO smiled. Everyone else appeared to be sitting on their hands. I was going to leave it at that. Thanks for the business.

Until a (diplomatically) assertive Sun sales rep piped up, 'Um... no, I connected with a buddy of mine over at MySQL, and had him check - you've downloaded MySQL more than 1,300 times in the last twelve months.'

After a profoundly awkward silence, one of the individuals from their internal development team piped up, 'Actually, everybody uses it. Why bother hassling with license agreements when MySQL's got you covered. We're stoked you bought them.'

How does your enterprise deal with this? Is freely available software treated as an opportunity, or a threat?

Unit testing the database

A recurring challenge in any application employing a database is how to unit test the database access layer.

Quite often one of two suggestions seems to appear: Either using DbUnit together with an in-memory database, or by setting up separate database instances for unit testing purposes. Jakub Korab suggests that you set up two database schema per developer.

But is this a unit test? Try asking yourself the following question: What am I trying to assert here?

• That the database engine works?
  
• That the DB driver works?
• That your object mapping library is functioning?

The correct answer is of course your code. You really should trust the developers of the database engine, DB driver, and object mapping library to test their code themselves.

So the thing you should assert is correct usage of the API.

For instance, I use Spring JDBC to talk to a PostgreSQL database, more specifically using Spring's SimpleJdbcTemplate. This class implements a handy interface, SimpleJdbcOperations, making it very easy to set up a mock for the entire database stack.

Your entire testing setup then collapses into adding behavior to this SimpleJdbcOperations mock, verifying that the correct invocations appear, optionally also verifying the ordering of the invocations.

This scheme does however have limitations. One is that you don't get the benefit of actually verifying that the SQL syntax is correct. The other is that any errors due to table column names and types go undetected by this unit test.

Asserting correct SQL syntax
The first problem is quite easy to solve. SQL syntax is not that hard, especially if you are strict on what syntax standards you want to hold for your statements. For example, I have written this EasyMock argument matcher to validate an SQL insert statement:

import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.easymock.EasyMock;
import org.easymock.IArgumentMatcher;

public class SqlStatementMatcher {

public static String insertStatement(String tableName, int argumentCount) {
EasyMock.reportMatcher(new SqlStatementMatcher.InsertStatementMatcher(tableName, argumentCount));
return "Insert Statement Matcher";
}

static class InsertStatementMatcher implements IArgumentMatcher {

private final String tableName;
private final int arguments;
private final Pattern pattern;

public InsertStatementMatcher(String tableName, int arguments) {
   this.arguments = arguments;
   this.tableName = tableName;
   this.pattern = Pattern.compile(
"^ *INSERT +INTO +([ a-z_]+) +\\(([ a-z_,]+)\\) +VALUES +\\(([ ?,]+)\\)( +RETURNING [a-z_]+)?$");
}

public void appendTo(StringBuffer buffer) {
   buffer.append("a valid SQL INSERT statement into table ").append(tableName)
.append(" with ").append(arguments).append(" arguments");
}

public boolean matches(Object argument) {
   if (!(argument instanceof String)) {
       return false;
   }
   String input = (String)argument;
   Matcher matcher = pattern.matcher(input);
   if (!matcher.matches()) {
       return false;
   }
   if (matcher.groupCount() != 4) {
       return false;
   }
   String actualTable = matcher.group(1);
   if (! (tableName.equals(actualTable))) {
       return false;
   }
   String[] argumentNames = matcher.group(2).split(",");
   String[] placeholders = matcher.group(3).split(",");
   if (argumentNames.length != placeholders.length) {
       return false;
   }
   if (argumentNames.length != arguments) {
       return false;
   }
   return true;
 }
}
}

This matcher verifies that the SQL syntax, database table name, and number of columns is correct. It also verifies that the number of placeholder arguments match the number of column names.

As you probably can imagine this matcher is very easily extended to matching select statements and even stronger syntax checking.

Asserting correct table column names and types
This problem is in part tested by the strong typing, both of the parameters for update calls and return type for query calls. But the aforementioned matcher could also be extended to check column names if so desired.

The final acid test for any database access layer is the integration test
This is where you put your application through a proper testing scenario. It could for instance be run as part of an automatic nightly build and test, or routinely as part of the release process. Don't mistake your unit test for integration tests! Unit testing has its purpose, integration test quite another.

Når er "Når er du ferdig" ferdig?

Den er iallefall ferdig filmet. Dersom du ikke fikk med deg min lightning talk på Smidig 2007, eller har lyst til å se den igjen, så kan du se den hos Confreaks.

Jeg skal snakke på Smidig 2007

Jeg fikk nettopp beskjed om at det forslaget jeg hadde til en Lightning Talk på Smidig 2007 ble akseptert.

Den heter Når er du ferdig? og går av stabelen tirsdag kl 08:54 i Mezzaninen. Utgangspunktet er det jeg skrev i When is a feature done? tidligere i år. Dersom du har innspill du har lyst til å få med i det jeg sier, kan du jo sende meg det på forhånd.

Better exception handling

I read Karsten Wagner's Blog: Better exception handling today.

The post in essence touches three key elements of exception handling and design.

Exceptions are an equally important part of API design
Equally important as classes, method, and arguments. What exceptions are thrown where and why, and whether they should be checked, has to be carefully thought of. One should also be careful when the class hierarchy of exceptions, if one should be made at all.

For instance it isn't always the case that all exceptions should be checked (or unchecked for that matter). Be very specific on which cases which could benefit from a checked exception and thus explicit error handling on the client.

The post uses classes from java.io as an example of how not to do it. java.sql and java.remote could also be mentioned with their checked exceptions causing boilerplate code time and time again. An essential problem of their checked IOException, SQLException and RemoteException is that the exception has no way of informing the executing code whether the failure is transient or permanent, and what the recovery or restart actions could be.

For SQLException in JDK 6 this is not longer the cases, as there are now several subclasses of SQLException providing this information. See SQLNonTransientException, SQLRecoverableException, SQLTransientException, and SQLWarning for details.

Nulls considered harmful
I have written about this topic before, as others have. With the ability to declare whether a method accepts or returns nulls, static analysis tools, or even the compiler, could do checks both at compile and execution time giving better error reporting on null references. Static analysis could produce compiler errors when an argument declared not to accept nulls are served a value known to be null.

At run time checks could be added like the one added for arrays indexing, for instance in form of assertions:
assert someVarName != null : "Invalid null dereference of declared nonnull variable 'someVarName'";
These assertions could potentially be disabled by the -da command line option to the JVM.

Unchecked exceptions are better than their reputation
They are far better than many other alternatives, like SIGSEGV and core dumps, and are actually quite robust and helpful in diagnosing an error.

For most cases, a RuntimeException means you have error situations you cannot or will not handle. The default result in Java is that the stack trace is written to the console, and the executing thread stops. Whether or not the execution is restartable depends on your application. If it's a background thread responsible for some kind of periodic check, the thread will need to be rescheduled. If it's a thread responsible for serving some external request, the client could just retry the request.

Aspects to the assistance
One possible way of working around some of these issues is to put AspectJ to work.

Save state to add a restartable checkpoint
Use this to add a checkpoint before attempting an operation known to be erratic and cause inconsistent state:

@Pointcut("call(* dangerousOperation(..))")
public void dangerousOperation() {}

@Before("dangerousOperation() && this(myObject)")
public void saveState(MyClass myObject) {
  myObject.saveState();
}

@AfterThrowing(pointcut="dangerousOperation() && this(myObject)", throwing="e")
public void restoreState(MyClass myObject, SomeDangerousCheckedException e) {
  myObject.recoverState();
  log.warning("Recovered " + myObject + " from " + e);
}

Handling a certain checked exception in a common way
When using an API throwing checked exceptions you have to add boilerplate try / catch blocks around each and every call to this API, something that could be very tedious.

One solution is to use a wrapper API which catches these exceptions and provides error handling and / or unchecked exceptions. Spring JDBC does this for JDBC access.

But you can also use an aspect for this, like this one from ibm.com/developerworks/ :

public class SqlAccess {

  private Connection conn;
  private Statement stmt;

  public void doUpdate(){
    conn = DriverManager.getConnection("url for testing purposes");
    stmt = conn.createStatement();
    stmt.execute("UPDATE ACCOUNTS SET BALANCE = 0");
  }

  public static void main(String[] args)throws Exception{
    new SqlAccess().doUpdate();
  }
}

private static aspect exceptionHandling{
  declare soft : SQLException : within(SqlAccess);

  pointcut methodCall(SqlAccess accessor) : this(accessor) && call(* * SqlAccess.*(..));

  after(SqlAccess accessor) : methodCall (accessor){
    System.out.println("Closing connections.");
    if(accessor.stmt != null){
      accessor.stmt.close();
    }
    if(accessor.conn != null){
      accessor.conn.close();
    }
  }
}

How do you achieve 100% test coverage?

When doing Test-Driven Development, one critical metric is test coverage. And it is often quite difficult to achieve 100% test coverage on a given method using its "normal" unit tests.

This can often lead to very strange looking test code written specifically to test all possible branches and every corner case. Sometimes this is useful, but more often than not its testing for testings sake.

But if your test suite is good enough and tests all the designed behavior, uncovered code is a good indication of unused code. Try to delete it. If it still compiles and all tests pass, the code was dead weight code you can safely remove.

This is specifically often true for getters and setters for each and every attribute on a particular class. Put YAGNI (You Ain't Gonna Need It) to use and remove all setters and getters not in active use by your code. If you need one later, you could always add it. This makes the code much more compact and easier to read.

But what if the getters and setters are part of an API? Well, then the API should be an interface, shouldn't it?