Friday, January 7, 2011

Groovy DSL/Builders: ZIP Output Streams

Let's follow up last week's post with another example of a very similar, very simple builder.

This one is for outputting ZIPped data to a stream. Let's take the standard example of using Java's ZIP support to zip up a folder of files.

Because the JDK does not include methods to traverse the filesystem, we need to define a method to be called recursively for subdirectories:


private void zipDirectory(File dir, ZipOutputStream zos) throws IOException {
for (File file : dir.listFiles()) {
if (file.isDirectory()) {
zipDirectory(file, zos);
}
else {
ZipEntry entry = new ZipEntry(file.getPath());
entry.setSize(file.length());
entry.setTime(file.lastModified());
zos.putNextEntry(entry);
IOUtils.copy(new FileInputStream(file), zos);
}
}
}


We cheated a little here by using the Apache Commons IO IOUtils class to actually copy the file bytes to the ZIP file. Also, we don't do anything here with IOExceptions.

With this method in place, we can create a ZIP file from a folder using:


ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(zipFile));
zipDirectory(new File(dir), zos);
zos.close();


Groovy's JDK IO extensions, and filesystem traversal methods, make this job quite a bit easier. Here's the Groovy code to do the same thing:


new ZipOutputStream(new FileOutputStream(zipFile)).withStream {zos ->
new File(dir).traverse(type: FileType.FILES) {File file ->
def entry = new ZipEntry(file.path)
entry.size = file.length()
entry.time = file.lastModified()
zos.putNextEntry(entry)
zos << file.bytes
}


This code is still a bit awkward in how it interacts with Java's ZIP API, in particular the creation of the ZipEntry object.

Using a simple builder, we can rewrite this as follows:


new ZipBuilder(new FileOutputStream(zipFile)).zip {
new File(dir).traverse(type: FileType.FILES) {File file ->
entry(file.path, size: file.length(), time: file.lastModified()) {it << file.bytes}
}
}

The ZipBuilder provides two methods:

  • zip(): creates and manages the ZipOutputStream

  • entry() (nested): creates and adds a ZipEntry to the enclosing zip stream


As with other builders, this builder promotes readable code that reflects the structure of the object to be created.

Here's the code for the builder itself:


class ZipBuilder {

@InheritConstructors
static class NonClosingOutputStream extends FilterOutputStream {
void close() {
// do nothing
}
}

ZipOutputStream zos

ZipBuilder(OutputStream os) {
zos = new ZipOutputStream(os)
}

void zip(Closure closure) {
closure.delegate = this
closure.call()
zos.close()
}

void entry(Map props, String name, Closure closure) {
def entry = new ZipEntry(name)
props.each {k, v -> entry[k] = v}
zos.putNextEntry(entry)
NonClosingOutputStream ncos = new NonClosingOutputStream(zos)
closure.call(ncos)
}

void entry(String name, Closure closure) {
entry([:], name, closure)
}
}


This builder uses the same style with Closures as the HSSFWorkbookBuilder described earlier.

There are a few other Groovy (and Java) features to note:

  • Java's ZIP library requires clients to write to the ZipOutputStream for each entry created. We need to make sure that no entry closes the ZipOutputStream -- it must be closed only when the zip stream is finished. (Many of Groovy's output methods close streams automatically.) For this reason, we wrap the output stream in a NonClosingOutputStream before passing it to an entry. This class is simply defined as a FilterOutputStream (OutputStream decorator) with a no-op close() method.

  • We use Groovy's @InheritConstructors to save repeating the trivial constructor.

  • The entry() method creates a new ZipEntry with its mandatory name property. It then populates additional optional properties from a Map, using Groovy's support for setting Java Beans properties as Map keys. These properties are intended to be provided as named arguments to the method, as shown in the example earlier. This makes for a very concise and intuitive way to set the properties.

  • The main overload of entry() is declared to take its arguments in this order: Map props, String name, Closure closure. When called, entry() is (typically) given arguments in a different order: String name, Map props, Closure closure. This is due to Groovy's convention for passing named arguments to a method, described here, in the section "Named Arguments".


One final note about this builder -- it doesn't just work with files. Because the constructor takes an OutputStream, it can write to any stream. So it could be used to write directly to a servlet response, for example. Similarly, the entries are populated as streams, so they can be filled by anything that can write to a stream.

Wednesday, December 29, 2010

Groovy DSL/Builders: POI Spreadsheets

It's well-known that Groovy is very rich for creating DSLs and fluent builder APIs.

I work a lot with the Apache POI library to generate Excel workbooks from data. We can use Groovy very easily to support a fluent and readable API for creating workbooks.

Here's a very simple example. Suppose we want to populate a workbook with two sheets with some data. Using the raw POI API, we could code something like this:

def workbook = new HSSFWorkbook()
def sheet1 = workbook.createSheet("Data")
def row10 = sheet1.createRow(0)
row10.createCell(0).setCellValue(new HSSFRichTextString("Invoice Number"))
row10.createCell(1).setCellValue(new HSSFRichTextString("Invoice Date"))
row10.createCell(2).setCellValue(new HSSFRichTextString("Amount"))
def row11 = sheet1.createRow(1)
row11.createCell(0).setCellValue(new HSSFRichTextString("100"))
row11.createCell(1).setCellValue(Date.parse("yyyy-MM-dd", "2010-10-18"))
row11.createCell(2).setCellValue(123.45)
def row12 = sheet1.createRow(2)
row12.createCell(0).setCellValue(new HSSFRichTextString("600"))
row12.createCell(1).setCellValue(Date.parse("yyyy-MM-dd", "2010-11-17"))
row12.createCell(2).setCellValue(132.54)
def sheet2 = workbook.createSheet("Summary")
def row20 = sheet2.createRow(0)
row20.createCell(0).setCellValue(new HSSFRichTextString("Sheet: Summary"))
def row21 = sheet2.createRow(1)
row21.createCell(0).setCellValue(new HSSFRichTextString("Total"))
row21.createCell(1).setCellValue(123.45 + 132.54)


This is not very readable. Even if we extract routines such as a common method to generate the cells in a row, the structure of our code does not follow closely the structure of what we want to create.
One of the big advantages of builders is that the structure of the code can match closely the structure of the generated result.

Here's the same workbook, created with a simple builder API:


def workbook = new HSSFWorkbookBuilder().workbook {
sheet("Data") { // sheet1
row(["Invoice Number", "Invoice Date", "Amount"])
row(["100", Date.parse("yyyy-MM-dd", "2010-10-18"), 123.45])
row(["600", Date.parse("yyyy-MM-dd", "2010-11-17"), 132.54])
}
sheet("Summary") { // sheet2
row(["Sheet: Summary"])
row(["Total", 123.45 + 132.54])
}
}


The HSSFWorkbookBuilder class required to do this is very straightforward:


import org.apache.poi.hssf.usermodel.HSSFRichTextString
import org.apache.poi.hssf.usermodel.HSSFWorkbook
import org.apache.poi.ss.usermodel.Cell
import org.apache.poi.ss.usermodel.Row
import org.apache.poi.ss.usermodel.Sheet
import org.apache.poi.ss.usermodel.Workbook

class HSSFWorkbookBuilder {

private Workbook workbook = new HSSFWorkbook()
private Sheet sheet
private int rows

Workbook workbook(Closure closure) {
closure.delegate = this
closure.call()
workbook
}

void sheet(String name, Closure closure) {
sheet = workbook.createSheet(name)
rows = 0
closure.delegate = this
closure.call()
}

void row(values) {
Row row = sheet.createRow(rows++ as int)
values.eachWithIndex {value, col ->
Cell cell = row.createCell(col)
switch (value) {
case Date: cell.setCellValue((Date) value); break
case Double: cell.setCellValue((Double) value); break
case BigDecimal: cell.setCellValue(((BigDecimal) value).doubleValue()); break
default: cell.setCellValue(new HSSFRichTextString("" + value)); break
}
}
}

}


The magic is in the handling of the nested closures, and setting the delegate for each to the builder so that methods are resolved against the builder.

Here's another example of using such a builder. This one takes an SQL query and creates a workbook with two sheets. The first sheet contains the result of running the query, and the second sheet contains the query text.


def workbook = new HSSFWorkbookBuilder().workbook {
sheet("Data") {
db.eachRow(
sql,
{meta -> row(meta*.columnName)}, // header row with columns names from ResultSetMetaData
{rs -> row(rs.toRowResult().values())} // data row for each ResultSet row
)
}
sheet("SQL") {
sql.eachLine {line ->
row([line])
}
}
}

Friday, February 5, 2010

Release builds with TeamCity: Selecting the branch

We've long had TeamCity doing regular "CI style" checkin builds for our Java/Ant projects. We recently added nightly builds for extra reports, and for longer-running performance tests. This was straightforward.

We finally got TeamCity doing our release builds. There were a couple of tricky points, which I thought would be worth writing up:

  • Selecting the branch
  • Manipulating the repository
  • Ensuring correct (release) versions of dependencies


Selecting the branch


This was the trickiest thing. Checkin and nightly builds always run against trunk. Well, you could set up a checkin build for a long-running development branch too, but that's not difficult. The release build should be done from the release branch, and that can be different for each release. Or it can be the same, if you have to do a fix release on an existing one!

We found a pretty good way to do this with TeamCity, using Build Configuration Templates and Configuration Parameters.

The idea is that you set up a template that contains all of the settings for the release build, except for the branch name. The branch name is specified by a configuration parameter. Then, the template is instantiated for each branch as desired. Each time the template is instantiated, the branch name configuration parameter is given for that instance.

Here are some details, using Subversion VCS and a hypothetical project named "xxx":

Create the Release Build Template


  1. Edit project's existing checkin build config.
  2. Click "Extract Template" to create a new template.
  3. Enter "release-build-template" for Name.
  4. Back in the checkin build config, click "Detach from Template".
  5. Click OK.

We've created a template, with no configurations attached.

Set up the Release Build Template

Edit the template. Change settings as given in the following sections.

Version Control Settings


  1. Create a new VCS root named xxx-branches.
  2. Specify Subversion as the Type of VCS.
  3. Enter Svn repo URL + "/xxx/branches" as the URL.
  4. Test the Connection.
  5. Save the VCS root.
  6. Attach the template to the xxx-branches VCS root.
  7. Detach the template from the xxx-trunk VCS root.
  8. Add checkout rule for VCS root: "%release.branch%=>.". This tells TeamCity to checkout the specific release branch into the working directory.
  9. Save Version Control Settings.

Runner Settings


  1. Change Target to "release-build", or whatever you want to call your release build target.
  2. Save Runner Settings.

You must have a target in your build script called "release-build", or whatever you want to call your release build target. This target must build the release and publish it somewhere. For example, it could copy it to a staging area on a server. Or, it could publish it to your enterprise repository.

The Ant target might look something like this:



Build Triggering Settings


  1. Delete/disable all triggering (VCS and Dependencies).
  2. Save Build Triggering Settings.
We've modified the template so that it will checkout the source from a release branch, with the specific branch given by a configuration parameter ("release.branch"). It will then build and publish the release.

Create a Release Branch Build Config for Release xx.yy

This procedure creates a release build config for a particular branch.

  1. Edit the release-build template's build configuration.
  2. Click Create Build Configuration From Template.
  3. Enter "release-xx.yy" for Name, where "xx.yy" is the name of your release branch.
  4. Enter the name of the branch for the release.branch parameter. For example, "RB-01.05".
We've created a configuration for running a release build on the branch.

To create a release:

  1. Ensure all changes for the release are checked into trunk.
  2. Create the branch. For example:
    svn copy $SVN/xxx/trunk $SVN/xxx/branches/RB-xx.yy

  3. Click Run on the release build configuration in TeamCity.



You can keep the release build configuration around for a particular branch as long as you like. If you are finished with a branch, you can delete the build configuration. If you need it again, it's easy to recreate it from the template.

That's it. I hope to write up some notes on the other points (manipulating the repository and ensuring the correct release versions of dependencies) soon.

Thursday, October 29, 2009

Oracle SQL-Developer 2.1

I've been trying out Oracle SQL-Developer 2.1 early adopter for the last couple of days, because SQL-Developer is the standard query tool at the company where I'm working.

Personally, I mostly use Aqua Data Studio. Apart from its odious activation scheme, ADS is excellent in every respect. However, ADS is kind of expensive, while SQL-Developer is free.

Previous versions of SQL-Developer were promising, but not very polished. They would paint strange colors on the screen in my installation, and behave poorly with regard to threading and large queries. The new version seems like a big improvement so far. Although it doesn't have the same brilliant keyboard support as ADS, it does seem to work pretty solidly, and appears reasonably attractive. (Much prettier than TOAD at any rate.)

I did notice one amusing glitch so far:








In fact the query only returned about 20 rows.

Friday, September 11, 2009

Table size frequencies

Here's an interesting query to run on your Oracle database:


Here's what I get on the main application I've been working on the last several years:


ROW_COUNT NUM_TABLES
---------------------- ----------
1-10 36
10-100 22
100-1000 22
1000-10000 22
10000-100000 22
100000-1000000 33
1000000-10000000 20
10000000-100000000 2
100000000-1000000000 3
1000000000-10000000000 1
183

Friday, July 24, 2009

Inline data for DbUnit tests in Grails

There is a DbUnit plugin for Grails: http://docs.codehaus.org/display/GRAILS/DBUnit+Plugin.

However, like almost all tutorials for DbUnit, this plugin assumes that the best way to organize your test data is using DbUnit's FlatXmlDataSet. The example given on the Wiki link above has this FlatXmlDataSet:


During my 5 years of working with DbUnit, I have come to the conclusion that FlatXmlDataSet is not the best way to organize test data. Here are my reasons:

  • XML is not a good representation for tabular data. CSV is more readable.

  • For most of my tests, I would rather see the test data together with the test, rather than separate.


DbUnit tests often have 3 "phases":

  1. Prime the database with setup data.

  2. Run some code of the System Under Test.

  3. Verify changed data in the database.


For most of my DbUnit tests, I would like to see the data used for the first and last parts together with the test itself.

For this reason, we've gravitated towards inline CSV datasets in most of our DbUnit test cases. You can achieve this using Groovy in Grails very simply.

The Wiki example, used in a test with inline datasets, might look like this:

(I'm using pipe characters instead of commas in this case!)
If the same Person data were used in different test methods, we might factor it out into a common method or field.

What I tend to find is that I use a lot of common reference data in my tests, but test-specific detail data. I handle the common reference data in a setUp() method, or even in the base DbUnitTestCase class. I put the test-specific detail data in each test. This prevents duplication of the common stuff, and keeps each test clean and focused. By reading the test method, you can see everything involved in the test.

We define cleanInsert() and a few other conveniences in DbUnitTestCase:

Friday, April 3, 2009

Tomcat Expert Seminar

Yesterday I attended SpringSource's Tomcat Expert Seminar.

This session rocked! I have learned a lot of neat ways to enhance our usage and troubleshooting techniques. The presenter, Filip Hanik, was not only extremely knowledgeable, but he was also entertaining and engaging. He handled questions from the audience extremely well, being able to give a direct and informative answer to nearly every question.

The most interesting topics for me were:

  • Large scale deployments -- showed flexible ways to organise deployment to simplify instance configuration and management.

  • Troubleshooting -- great tips on Tomcat but even more on Java and server-based systems in general.


I look forward to attending similar sessions in the future.