The Boggling of the Mind

Scott Adams sent me on a Wikipedia journey today with his article Monte Hall Problem. I started reading about the double-slit experiment, its extension in the quantum-eraser experiment, and finally quantum entanglement.

I suggest you read them, but only if you are prepared on looking at the world around you a little differently. If you're happy with the things the way they are I suggest you read something else instead.

history meme

history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head

Home:

155 cd
97 sudo
79 ls
69 svn
51 heyu
47 mvn
38 grep
37 less
28 vi
23 screen

Work:

252 mvn
119 cd
86 svn
50 ls
40 rm
32 less
22 ssh
22 man
22 cp
21 psql

Open Source in the Enterprise

I guess many like me working in a large enterprise can relate to a story like this one: Jonathan Schwartz's Blog: Freedom's Choice:

The CIO responded categorically with 'we don't run MySQL, we run [name withheld to protect the proprietary].' The CISO said, 'We can't just let developers download software off the net, you know, we've got regulation and security to worry about.' The CTO smiled. Everyone else appeared to be sitting on their hands. I was going to leave it at that. Thanks for the business.

Until a (diplomatically) assertive Sun sales rep piped up, 'Um... no, I connected with a buddy of mine over at MySQL, and had him check - you've downloaded MySQL more than 1,300 times in the last twelve months.'

After a profoundly awkward silence, one of the individuals from their internal development team piped up, 'Actually, everybody uses it. Why bother hassling with license agreements when MySQL's got you covered. We're stoked you bought them.'

How does your enterprise deal with this? Is freely available software treated as an opportunity, or a threat?

Unit testing the database

A recurring challenge in any application employing a database is how to unit test the database access layer.

Quite often one of two suggestions seems to appear: Either using DbUnit together with an in-memory database, or by setting up separate database instances for unit testing purposes. Jakub Korab suggests that you set up two database schema per developer.

But is this a unit test? Try asking yourself the following question: What am I trying to assert here?

• That the database engine works?
  
• That the DB driver works?
• That your object mapping library is functioning?

The correct answer is of course your code. You really should trust the developers of the database engine, DB driver, and object mapping library to test their code themselves.

So the thing you should assert is correct usage of the API.

For instance, I use Spring JDBC to talk to a PostgreSQL database, more specifically using Spring's SimpleJdbcTemplate. This class implements a handy interface, SimpleJdbcOperations, making it very easy to set up a mock for the entire database stack.

Your entire testing setup then collapses into adding behavior to this SimpleJdbcOperations mock, verifying that the correct invocations appear, optionally also verifying the ordering of the invocations.

This scheme does however have limitations. One is that you don't get the benefit of actually verifying that the SQL syntax is correct. The other is that any errors due to table column names and types go undetected by this unit test.

Asserting correct SQL syntax
The first problem is quite easy to solve. SQL syntax is not that hard, especially if you are strict on what syntax standards you want to hold for your statements. For example, I have written this EasyMock argument matcher to validate an SQL insert statement:

import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.easymock.EasyMock;
import org.easymock.IArgumentMatcher;

public class SqlStatementMatcher {

public static String insertStatement(String tableName, int argumentCount) {
EasyMock.reportMatcher(new SqlStatementMatcher.InsertStatementMatcher(tableName, argumentCount));
return "Insert Statement Matcher";
}

static class InsertStatementMatcher implements IArgumentMatcher {

private final String tableName;
private final int arguments;
private final Pattern pattern;

public InsertStatementMatcher(String tableName, int arguments) {
   this.arguments = arguments;
   this.tableName = tableName;
   this.pattern = Pattern.compile(
"^ *INSERT +INTO +([ a-z_]+) +\\(([ a-z_,]+)\\) +VALUES +\\(([ ?,]+)\\)( +RETURNING [a-z_]+)?$");
}

public void appendTo(StringBuffer buffer) {
   buffer.append("a valid SQL INSERT statement into table ").append(tableName)
.append(" with ").append(arguments).append(" arguments");
}

public boolean matches(Object argument) {
   if (!(argument instanceof String)) {
       return false;
   }
   String input = (String)argument;
   Matcher matcher = pattern.matcher(input);
   if (!matcher.matches()) {
       return false;
   }
   if (matcher.groupCount() != 4) {
       return false;
   }
   String actualTable = matcher.group(1);
   if (! (tableName.equals(actualTable))) {
       return false;
   }
   String[] argumentNames = matcher.group(2).split(",");
   String[] placeholders = matcher.group(3).split(",");
   if (argumentNames.length != placeholders.length) {
       return false;
   }
   if (argumentNames.length != arguments) {
       return false;
   }
   return true;
 }
}
}

This matcher verifies that the SQL syntax, database table name, and number of columns is correct. It also verifies that the number of placeholder arguments match the number of column names.

As you probably can imagine this matcher is very easily extended to matching select statements and even stronger syntax checking.

Asserting correct table column names and types
This problem is in part tested by the strong typing, both of the parameters for update calls and return type for query calls. But the aforementioned matcher could also be extended to check column names if so desired.

The final acid test for any database access layer is the integration test
This is where you put your application through a proper testing scenario. It could for instance be run as part of an automatic nightly build and test, or routinely as part of the release process. Don't mistake your unit test for integration tests! Unit testing has its purpose, integration test quite another.

Better exception handling

I read Karsten Wagner's Blog: Better exception handling today.

The post in essence touches three key elements of exception handling and design.

Exceptions are an equally important part of API design
Equally important as classes, method, and arguments. What exceptions are thrown where and why, and whether they should be checked, has to be carefully thought of. One should also be careful when the class hierarchy of exceptions, if one should be made at all.

For instance it isn't always the case that all exceptions should be checked (or unchecked for that matter). Be very specific on which cases which could benefit from a checked exception and thus explicit error handling on the client.

The post uses classes from java.io as an example of how not to do it. java.sql and java.remote could also be mentioned with their checked exceptions causing boilerplate code time and time again. An essential problem of their checked IOException, SQLException and RemoteException is that the exception has no way of informing the executing code whether the failure is transient or permanent, and what the recovery or restart actions could be.

For SQLException in JDK 6 this is not longer the cases, as there are now several subclasses of SQLException providing this information. See SQLNonTransientException, SQLRecoverableException, SQLTransientException, and SQLWarning for details.

Nulls considered harmful
I have written about this topic before, as others have. With the ability to declare whether a method accepts or returns nulls, static analysis tools, or even the compiler, could do checks both at compile and execution time giving better error reporting on null references. Static analysis could produce compiler errors when an argument declared not to accept nulls are served a value known to be null.

At run time checks could be added like the one added for arrays indexing, for instance in form of assertions:
assert someVarName != null : "Invalid null dereference of declared nonnull variable 'someVarName'";
These assertions could potentially be disabled by the -da command line option to the JVM.

Unchecked exceptions are better than their reputation
They are far better than many other alternatives, like SIGSEGV and core dumps, and are actually quite robust and helpful in diagnosing an error.

For most cases, a RuntimeException means you have error situations you cannot or will not handle. The default result in Java is that the stack trace is written to the console, and the executing thread stops. Whether or not the execution is restartable depends on your application. If it's a background thread responsible for some kind of periodic check, the thread will need to be rescheduled. If it's a thread responsible for serving some external request, the client could just retry the request.

Aspects to the assistance
One possible way of working around some of these issues is to put AspectJ to work.

Save state to add a restartable checkpoint
Use this to add a checkpoint before attempting an operation known to be erratic and cause inconsistent state:

@Pointcut("call(* dangerousOperation(..))")
public void dangerousOperation() {}

@Before("dangerousOperation() && this(myObject)")
public void saveState(MyClass myObject) {
  myObject.saveState();
}

@AfterThrowing(pointcut="dangerousOperation() && this(myObject)", throwing="e")
public void restoreState(MyClass myObject, SomeDangerousCheckedException e) {
  myObject.recoverState();
  log.warning("Recovered " + myObject + " from " + e);
}

Handling a certain checked exception in a common way
When using an API throwing checked exceptions you have to add boilerplate try / catch blocks around each and every call to this API, something that could be very tedious.

One solution is to use a wrapper API which catches these exceptions and provides error handling and / or unchecked exceptions. Spring JDBC does this for JDBC access.

But you can also use an aspect for this, like this one from ibm.com/developerworks/ :

public class SqlAccess {

  private Connection conn;
  private Statement stmt;

  public void doUpdate(){
    conn = DriverManager.getConnection("url for testing purposes");
    stmt = conn.createStatement();
    stmt.execute("UPDATE ACCOUNTS SET BALANCE = 0");
  }

  public static void main(String[] args)throws Exception{
    new SqlAccess().doUpdate();
  }
}

private static aspect exceptionHandling{
  declare soft : SQLException : within(SqlAccess);

  pointcut methodCall(SqlAccess accessor) : this(accessor) && call(* * SqlAccess.*(..));

  after(SqlAccess accessor) : methodCall (accessor){
    System.out.println("Closing connections.");
    if(accessor.stmt != null){
      accessor.stmt.close();
    }
    if(accessor.conn != null){
      accessor.conn.close();
    }
  }
}

How do you achieve 100% test coverage?

When doing Test-Driven Development, one critical metric is test coverage. And it is often quite difficult to achieve 100% test coverage on a given method using its "normal" unit tests.

This can often lead to very strange looking test code written specifically to test all possible branches and every corner case. Sometimes this is useful, but more often than not its testing for testings sake.

But if your test suite is good enough and tests all the designed behavior, uncovered code is a good indication of unused code. Try to delete it. If it still compiles and all tests pass, the code was dead weight code you can safely remove.

This is specifically often true for getters and setters for each and every attribute on a particular class. Put YAGNI (You Ain't Gonna Need It) to use and remove all setters and getters not in active use by your code. If you need one later, you could always add it. This makes the code much more compact and easier to read.

But what if the getters and setters are part of an API? Well, then the API should be an interface, shouldn't it?

When is a feature done?

Far too often a new feature is considered done at an incomplete state.

This is my suggested checklist for when to consider a feature DONE:

1. It's checked in to source control
2. It compiles
3. It passes its unit tests (no, not having unit tests does not count)
4. It doesn't break any other unit tests
5. It increases the overall test coverage for the module
6. The continuous integration server passes the build
7. It's integration tested with other features and / or modules it pertains to
8. It's documented in the release notes and / or product documentation
9. The feature does what it's supposed to do (as defined by the specifications for the feature)
   

If you answer no to any of these questions, you're not done. Furthermore, if you answer N/A to any of them you should ask yourself whether the feature should be implemented at all.

The last point is perhaps the most difficult to assess, unless you have executable specifications in form of FIT tests or something similar. Furthermore, if you're using Scrum, the feature must be demonstrated on the end-of-iteration demo to be considered done.

Useful Maven goals

• dependency:analyze
• project-info-reports:dependencies
  

Null safety in Java revisited

In a previous post, I ranted on the problem of null in Java leading to NullPointerExceptions appearing when you least expect it.

Well, years have passed, and JDK versions also. JDK 7 is in the works, and null safety is one of the issues in question. As far as I can tell, three suggestions have surfaced:

1. Symbols in relation to the type or dereference, like ? in the Nice Option Types, or # instead of .


2. @NotNull and @Nullable annotations to method parameters, return values and instance variables. This exists in IDEA and FindBugs, and possibly Eclipse


3. A new keyword, nonnull

In my view, the following requirements for compile-time nullable tests are most essential to focus on:

1. The ability to declare whether a method's parameters or return value could be null and issue a compiler error if a known null is passed, and a warning if a possible null is passed


2. Compile-time warnings for dereferencing where there's risk of null variables.

Research done on the ratio of parameters in a number of Java projects indicate that 80% of all parameters are non-nullable by design, i.e. the method intends for the parameter to be non-null. The paper continues in arguing that due to this, the default rule for references should change from nullable to non-nullable.

My suggestion is to reuse the null keyword for this in method declarations. For instance, the following method does not accept nulls in any argument, and never returns one either. If the implementation does not assert this, the compiler must issue an error:

public Object get(String query, Object value);

Conversely, the following implementation accepts null for its second argument, and might also return a null:

public null Object get(String query, null Object value);

But what's the alternative to returning a null? The NullObject design pattern has addressed this : return a special case instance which is a valid return type, but identifiable as meaning "nothing".

Pet Projects 2006

Paul René challenge me on pet projects I've been doing in 2006. Well, here goes:

Linux Desktop

My employer is a Microsoft customer, and the laptop issued to me had Windows 2000 Pro, and later XP pro, as standard software.

Being a long-time Linux advocate and addict, I opted to liberate myself from the Microsoft monopoly and attempt to use a Linux desktop for all my daily work.

The results? I do have a VMware Workstation installation with XP Pro, Office and Groove Virtual Office. This is used for in-house Internet Explorer-only applications, really troubling spreadsheets or documents, and PDA / mobile synchronisation.

But email, development, almost all web-based applications, most documents,presentations, and spreadsheets, and printing, work like a charm. The Linux installation started as Fedora Core 4 on a Compaq Evo N600c, and is currently a Fedora Core 5 on a Thinkpad T60, on its third disk drive. All upgrades have been very smooth -- disk switches were performed as LVM volgroup migrations, often on line.

My current plan is to migrate it to Fedora Core 6 on a Dell laptop. More on that later.

Home Automation

Clas Ohlson had a closing sale on X10 modules spring 2006, and I took the opportunity to buy the start kit with PC interface, RF interface, lamp and appliance modules, and an infrared motion sensor. Via heyu and xtend, my PC controls lighting and heating, ensuring that the heaters are switched off when nobody's home and switched on the second we enter the door.

I also considered using it as an alarm system, but realised that getting an SMS with what might be a false positive or a burglar when many miles from home would be more traumatizing than ignorance..

SMS server

I use an old mobile phone connected to the computer via SMS Server Tools to send alerts and receive commands. It's also connected to the Jabber server monitor described below, so short messages sent to this phone is distributed via Jabber. I can also send replies via a special command to the Jabber agent.

Temperature monitoring

I've also purchased a one-wire temperature controller kit from Quasar Electronics (designed by Quozl), and hooked up two DS18S20 sensors, one in the "server room" and one outside. RRDtool now ensures that I have access to historical temperature data from outside my living room for years to come. I also plan placing a sensor in the living room, and perhaps controlling the heater there based on both outside and inside temperature.

I had plans on monitoring the temperature of my freezer also, but this stopped when I realised my laptop only had a 5V serial port, and not 12V as the Link45 one-wire controller demands. Perhaps I should have gone for the USB version..

Jabber server monitoring

My main server runs a Perl Net::Jabber client with a command handling API and monitoring capability. It features a monitoring plugin interface, capable of testing temperatures, available memory and disk, system load, etc. against threshold values and sending me an IM when the thresholds are exceeded.

Computer fan control

My computer sits in an outside shed on the south side of the building, and experiences vast temperature variations. Due to this, air throughput has to be varied significantly between sun, shade, and winter. The CPU and exhaust fans are thermistor controlled, but the intake fans are controlled by computer. I found some fans with resistor control of fan speed, and connected the resistor wires to a parallel port relay board. With the relay open, the fan stays at its idle speed, and when the relay closes it runs at full speed. A small cron job checks system and disk temperatures and toggles between idle speed and full speed.

YaGoohoo!gle and Twingine

Twingine was a 2005 project, but was alive and kicking also in 2006. It had a total of 1,539,760 hits throughout the year.