Bad math in BBC

In the BBC Magazine story Amanda Knox and bad math in court, there is an explanation of probability relating to evidence in a legal trial.

The article quotes mathematician Coralie Colmez, co-author of "Math on Trial: How numbers get used and abused in the courtroom". Well, I hope they misquoted her. The math presented in the article does not appear to add up.

The example given is:

"You do a first test and obtain nine heads and one tail... The probability that the coin is fair given this outcome is about 8%, [and the probability] that it is biased, about 92%. Pretty convincing, but not enough to convict your coin of being biased beyond a reasonable doubt," Colmez says.

Technically in this situation we talk about the probability that a fair coin will give a particular outcome. We can calculate the chance of a fair coin giving nine or more heads (or zero or one tails), using the binomial distribution.

In Excel we can use =BINOMDIST(1,10,0.5,TRUE). In the statistical program R the formula is pbinom(1,10,0.5).

The answer is 0.01074219, or about 1%. I don't know where the 8% quoted in the article comes from.

The more subtle problem is that the chance of a fair coin producing a result is not the same as the chance of the coin being fair given a result. This is what the article seems to imply, and is a common mistake.

The theory we need for this kind of statement is Bayes' Theorem, which deals with conditional probabilities:

P(A|B) = P(B|A) P(A) / P(B)

Where P(A|B) means "Probability of A, given B".

To know the probability of this particular coin being fair after an experiment we'd need to know more information about the whole population of coins.

Ant Profiling

In my new job at Assurity Consulting I've been looking at some large builds. Some that take 6 hours or more to run in the Continuous Integration Server. There's a fairly random mix of Ant, Maven and good old shell. The last few days I spent some time diving into the details of a fairly large Ant build.

I've been using a combination of three techniques to analyze where the time is being spent in this build:

1. Analysis by inspection.
2. Analysis of "ant -d" output.
3. Ant Profiler.

There are a number of files and many tasks and targets in this build, and I haven't been able to walk through the entire thing yet. But by browsing through "interesting" parts, I have been able to find out some significant things about the build.

For example, the build makes extensive use of <antcall>. Sometimes people use <antcall> when they don't understand the correct usage of "depends" on Ant targets. In this build, many of the <antcall> usages would be more appropriately done with <macrodef>. <antcall> is inefficient to use on targets within the same build file, because it reparses and copies properties. Please learn to use <macrodef> with Ant. It is your friend.

To extract some data about the Ant build as a whole, I ran it with "ant -d". On this particular build, it generates 600,000 lines of output. Again, a lot of detail.

With some shell piping we can see potential areas of duplication, or work being repeated:

ant -d >ant.debug.out 
sort ant.debug.out | uniq -c | sort -nr | head

For example, we can see whether certain Java source files are compiled more than once during the build:

egrep "^ *\\[javac\\]" ant.debug.out | sort | uniq -c | sort -nr | less

In this case I found that some classes are compiled twice, three, four or five times. Unfortunately, not enough classes to make a large difference to the build time, but still ...

Finally, I found a great Ant profiler called antro. This profiler is really neat. It is completely non-invasive. It uses the Ant listeners feature, and can be run like this to analyze a build:

ant -listener ru.jkff.antro.ProfileListener -lib ~/antro/antro.jar build.xml

This generates a JSON file, which can then be loaded into the profiler's GUI. The GUI is run simply with:

java -jar antro.jar

What could be simpler?

It provides a tree view of the build times, where you can drill down into any node and explore the detail. Using this tool it's easy to get a bird's eye view of the build time breakdown, as well as drill into areas of interest. For example, I was able to assess the overhead of <antcall> by looking at some of those nodes. Unfortunately, there seems to be no way to see the total number or overhead for a single task type such as <antcall> across the whole build. I was able to find the total number of <antcall>s (not the times though) from the "ant -d" output. This allowed me to estimate the total time overhead of <antcall> for this build.

Git and SVN on OSX 10.8 - is this the best way?

Don't get me started on Apple. I bought my first Mac a few weeks ago, and I'm still trying to figure out why. They are pretty, but they are also pretty frustrating. Other than the hardware compatability problems with Linux, which may actually have driven me crazy, Ubuntu is a much better-designed system.

For one thing, Apple seems determined to gradually eliminate standard UNIX software from this platform. Which kind of defeats the whole point I bought the thing -- I wanted a reliable UNIX-based system. I realize that Apple could not give a toss about UNIX geeks, having a very lucrative mass market of schmucks to serve, but still that doesn't console me much. One day I will post a long list of frustrations I have found since starting with OSX.

Today I just want to post how I got something working, in case I need it later. In Ubuntu, if I want to use Git and SVN, I have to type something like this:

sudo apt-get install subversion
sudo apt-get install git-core

Something like that. Also maybe

sudo apt-get install git-svn

I don't remember. Actually, I don't have to remember, or type it, because bash on Ubuntu has completions. There's nothing to it.

In OSX, there is no standard package management system. (Let alone bash completions.) There are a variety of independent efforts (Mac Ports, Homebrew, Fink), and they all hate each other. You pick one of those and hope for the best.

Except Apple keeps trying to break them.

The worst thing I have done with OSX so far is upgrading to 10.8. This was a stupid mistake, because 10.8 does not contain any useful features, but instead breaks things that I had come to depend on. I did it because I thought it was supposed to fix fullscreen mode with multiple monitors, which was already broken. It is still broken.

Prior to 10.8 I was able to get Git and SVN working using Homebrew. After 10.8 I get compilation errors.

So instead I installed SVN using the brain-dead Windows method of click-till-you-die installation from a binary DMG file from git-scm.com. And I installed Git similarly by clicking and clicking and clicking with a binary PKG file from WANDisco. This was fine, for independent Git and SVN operation. But Git-SVN did not work. I got

Can't locate SVN/Core.pm in @INC

because the Git Perl code is in one place, and the SVN Perl code is somewhere completely different.

Only the wise and benevolent creator can tell why these things need to talk to each other using Perl. It probably comes from Git being implemented using every UNIX hack tool under the sun.

I got them to work, apparently, by doing this:

sudo ln -s /opt/subversion/lib/svn-perl/auto /usr/local/git/lib/perl5/site_perl/auto
sudo ln -s /opt/subversion/lib/svn-perl/SVN /usr/local/git/lib/perl5/site_perl/SVN

What a hack! And only works because there was no auto/ directory there already! This is gonna break, I know it! Hopefully the Homebrew version will be working again by then.

Give me Ubuntu for this sort of thing any day. Shit, even give me an RPM-based system, I'll be happy with that.

Groovy AST transform gotcha

I'm a big fan of Groovy's AST transforms. Lately I've been using @Lazy in a lot of my code, because I love a declarative or functional style of programming, and @Lazy lets me do that really efficiently and concisely with Groovy. I hope that my colleagues agree!

We got caught by an interesting trap using a couple of other transforms recently: @EqualsAndHashCode and @Immutable. We like to use @EqualsAndHashCode to generate equals() and hashCode() for entity classes in our domain model. We use the 'includes' attribute to include only the primary key properties in the equals() comparison, like this:

@EqualsAndHashCode(includes = "id")
class Foo {
  String id
  String description
}

For this class, and its corresponding database table, the 'id' property is the key. Equality is not done by Java object equality, or by full value equality, but by comparing the entities' primary keys:

assert new Foo(id: "1", description: "cat") == 
       new Foo(id: "1", description: "dog")

This has important effects when storing these kinds of objects in collections.

Be careful mixing AST transform annotations though! We added @Immutable to some of our domain classes, like this:

@Immutable
@EqualsAndHashCode(includes = "id")
class Foo {
  String id
  String description
}

and got a surprising result:

assert new Foo(id: "1", description: "cat") != 
       new Foo(id: "1", description: "dog")

The reason is the order of the annotations. If we reverse them, it works correctly:

@EqualsAndHashCode(includes = "id")
@Immutable
class Foo {
  String id
  String description
}

assert new Foo(id: "1", description: "cat") == 
       new Foo(id: "1", description: "dog")

So what happens? The doc for @Immutable says:

The @Immutable annotation instructs the compiler to execute an AST transformation which adds the necessary getters, constructors, equals, hashCode and other helper methods that are typically written when creating immutable classes with the defined properties.

Later on, in more detail:

Default equals, hashCode and toString methods are provided based on the property values. Though not normally required, you may write your own implementations of these methods. For equals and hashCode, if you do write your own method, it is up to you to obey the general contract for equals methods and supply a corresponding matching hashCode method. [...]

So, I'm guessing that if we put @Immutable first, then @EqualsAndHashCode hasn't yet had a chance to do its magic, and @Immutable adds its default equals() etc, not the ones we want. But if we put @EqualsAndHashCode first, then its equals() etc methods are there for @Immutable to see, and we get the behavior we want.

Thus, problem solved, for now. But it does make one wonder a little about the interactions of all of these transforms and other annotations. We've been using Groovy AST transform annotations together with JPA annotations with no known problems to date, and I hope it continues that way.

Checking Foreign Key attribute consistency

While evolving a schema during development, it's sometimes hard to make sure that column attributes remain consistent across different tables.

Some databases provide domain types. So for example, with Firebird/InterBase you can define a domain type or alias for your invoice_number column as, say, VARCHAR(10). Then you can use the domain type in defining any tables containing invoice_number, and the column attributes will always be consistent.

Oracle doesn't have a natural way to do this. One trick I've used is preprocessing SQL DDL files with Ant and using "macros" for column types. So I might have a types.properties file like this:

...
INVOICE_NUMBER_TYPE = VARCHAR2(10)
...

Then in create_table_invoice.sql I might have:

CREATE TABLE invoice (
  invoice_number @INVOICE_NUMBER_TYPE@ NOT NULL,
  ...
);

And also use the macro elsewhere for any foreign key column.

But, this turned out to be a bit of a pain. For one thing, it makes the build more convoluted because of the preprocessing required. More importantly, it made the SQL DDL files "invalid". We couldn't just run one in sqlplus, without the preprocessing step first. We couldn't send one to a DBA. We didn't get good IDE support, because the IDE doesn't understand the type macros.

As a result, I've reverted to putting hardcoded column attributes in the DDL files. But this takes me back to the problem of keeping the foreign keys parent/child attributes consistent.

Another approach to that problem is to use a view like this:

CREATE OR REPLACE VIEW chk_foreign_key_type AS
SELECT
  ac.table_name child_table,
  acc.column_name child_column,
  atc.data_type child_data_type,
  atc.data_length child_data_length,
  atc.data_scale child_data_scale,
  ac2.table_name parent_table,
  acc2.column_name parent_column,
  atc2.data_type parent_data_type,
  atc2.data_length parent_data_length,
  atc2.data_scale parent_data_scale
FROM   all_constraints ac
       JOIN all_cons_columns acc ON acc.owner = ac.owner 
            AND acc.constraint_name = ac.constraint_name
       JOIN all_tab_columns atc ON atc.owner = ac.owner 
            AND atc.table_name = acc.table_name 
            AND atc.column_name = acc.column_name
       JOIN all_constraints ac2 ON ac2.owner = ac.owner 
            AND ac2.constraint_name = ac.r_constraint_name
       JOIN all_cons_columns acc2 ON acc2.owner = ac2.owner 
            AND acc2.constraint_name = ac2.constraint_name 
            AND acc2.position = acc.position
       JOIN all_tab_columns atc2 ON atc2.owner = acc2.owner 
            AND atc2.table_name = acc2.table_name 
            AND atc2.column_name = acc2.column_name
WHERE  ac.owner = 'your_schema'
  AND  ac.constraint_type = 'R'
  AND  (atc2.data_type <> atc.data_type
        OR atc2.data_length <> atc.data_length
        OR NVL(atc2.data_scale, -1) <> NVL(atc2.data_scale, -1))
ORDER BY 1, 2
/

COMMENT ON TABLE chk_foreign_key_type IS
  'Foreign key column(s) different from parent type/length/scale'
/

This view will return any foreign key where a data type, length or scale of a column in the child table does not match the corresponding column in the parent table.

Ant Target Dependency Graph

We can get a pretty good diagram of our Ant target dependencies very easily using an embedded Groovy script and GraphViz.

Add this to build.xml:

    
    
      def u(x) {x.toString().replace("-", "_").replace(".", "_")}
      new File("build.dot").text = """
digraph ant {
  ${project.targets.values().collect {target ->
    target.dependencies.collect {dep ->
      u(dep) + " -> " + u(target)
    }.join("\n")
  }.join("\n")}
}
"""
    
  

You will need to have the embeddable groovy-all.jar in Ant's classpath, e.g. in ~/.ant/lib/.

Then run "ant target-graph". It writes a build.dot file in the current directory.

Convert this into a picture using the GraphViz dot command:

dot -Tsvg -O build.dot

The results are surprisingly good.

Here's an example from the Apache commons-dbcp project:



I'm not going to show you the diagram I got for our system at work, the reason I wrote this script! The diagram is so big and complex, I was shocked. After being happy with Ant all these years, I think it's time to be getting serious about Gradle.

SlickEdit 2011 Wish List

Well, we're nearly through January 2011, and the SlickEdit 2011 beta is due any day now.

I've been using SlickEdit since 1996, and I still get kind of excited around this time of year when a new version comes out. Sometimes the team surprises me.

Here is my wish list for features in SlickEdit 2011:

• Groovy Language Support


• Scala Language Support


• Clojure Language Support


• Git DVCS Support


• Mercurial DVCS Support


• Bazaar DVCS Support


• Useful Subversion VCS Support


• Useful "subword" features

The first three are hot JVM languages that I'd love to see support for. I don't really expect to see them, but you never know. Last year, SlickEdit added support for Erlang, Haskell and F#, so they aren't completely in the dark about hot languages.

The next three items are popular distributed version control systems. Again, I don't really expect much from SlickEdit on that, yet. Forum posts on the topic have met with disappointing reponses from SlickEdit staff -- doesn't look like they "clicked" on DVCS yet. It took them an awful long time to move on from CVS to Subversion themselves, and even now the Subversion support is miserable compared to any other tool I've used. And so, my wish, modernise the Subversion support.

SlickEdit has a lot of advanced features for C/C++ programmers. C/C++ programmers probably make up a very large chunk, if not the majority, of SlickEdit users. And as far as I can tell, SlickEdit is actually one of the best "IDE"s available for C/C++. I don't do C/C++ any more myself though, I do JVM-based languages mostly. And with Java, SlickEdit also tries to be an uber-IDE, with Project Types, JUnit, Ant support and more. But here it falls far, far short of industry standards. Java programmers are really spoiled by the superb IDEs aavailable for them, and two of the best ones are even free. Anyway, I don't wish for SlickEdit to improve its Java IDE features. I'm happy to use a Java IDE for that kind of work. The point I'd like to make is that supporting current VCS systems, and supporting them really well, would benefit all SlickEdit users. I can't imagine many SlickEdit users are not using VCS, and many of them probably use a modern VCS, such as Git. It's really about time SlickEdit caught up with the VCS game.

For a couple of examples of excellent VCS integration, look at:

• IntelliJ IDEA and related products from JetBrains


• SmartSVN, SmartGit and even SmartCVS from Syntevo

SlickEdit 2011 included some rather dubious new features. My favorite "non useful feature" was Subword Navigation. You can move the cursor through camel-cased words such as AbstractBeanFactory. But when would anyone want to do that? Far more useful would be file or class completion/loading using smart "camel typing", as introduced by IntelliJ IDEA and copied by other tools. With IDEA, I can press Ctrl+N to open a class, then type "ABF" or "AbBeFa" to open the AbstractBeanFactory class. This is really useful, and would be something SlickEdit could really benefit from.

Anyway, I'm sure SlickEdit 2011 will contain a few pleasant surprises, as well as a few new annoying bugs. As always, it will be interesting to figure whether the feature-to-bug ratio improves, or not. I'm looking forward to the beta.