Release builds with TeamCity: Selecting the branch

We've long had TeamCity doing regular "CI style" checkin builds for our Java/Ant projects. We recently added nightly builds for extra reports, and for longer-running performance tests. This was straightforward.

We finally got TeamCity doing our release builds. There were a couple of tricky points, which I thought would be worth writing up:

• Selecting the branch
• Manipulating the repository
• Ensuring correct (release) versions of dependencies

Selecting the branch

This was the trickiest thing. Checkin and nightly builds always run against trunk. Well, you could set up a checkin build for a long-running development branch too, but that's not difficult. The release build should be done from the release branch, and that can be different for each release. Or it can be the same, if you have to do a fix release on an existing one!

We found a pretty good way to do this with TeamCity, using Build Configuration Templates and Configuration Parameters.

The idea is that you set up a template that contains all of the settings for the release build, except for the branch name. The branch name is specified by a configuration parameter. Then, the template is instantiated for each branch as desired. Each time the template is instantiated, the branch name configuration parameter is given for that instance.

Here are some details, using Subversion VCS and a hypothetical project named "xxx":

Create the Release Build Template

1. Edit project's existing checkin build config.
2. Click "Extract Template" to create a new template.
3. Enter "release-build-template" for Name.
4. Back in the checkin build config, click "Detach from Template".
5. Click OK.

We've created a template, with no configurations attached.

Set up the Release Build Template

Edit the template. Change settings as given in the following sections.

Version Control Settings

1. Create a new VCS root named xxx-branches.
2. Specify Subversion as the Type of VCS.
3. Enter Svn repo URL + "/xxx/branches" as the URL.
4. Test the Connection.
5. Save the VCS root.
6. Attach the template to the xxx-branches VCS root.
7. Detach the template from the xxx-trunk VCS root.
8. Add checkout rule for VCS root: "%release.branch%=>.". This tells TeamCity to checkout the specific release branch into the working directory.
9. Save Version Control Settings.

Runner Settings

1. Change Target to "release-build", or whatever you want to call your release build target.
2. Save Runner Settings.

You must have a target in your build script called "release-build", or whatever you want to call your release build target. This target must build the release and publish it somewhere. For example, it could copy it to a staging area on a server. Or, it could publish it to your enterprise repository.

The Ant target might look something like this:

Build Triggering Settings

1. Delete/disable all triggering (VCS and Dependencies).
2. Save Build Triggering Settings.

We've modified the template so that it will checkout the source from a release branch, with the specific branch given by a configuration parameter ("release.branch"). It will then build and publish the release.

Create a Release Branch Build Config for Release xx.yy

This procedure creates a release build config for a particular branch.

1. Edit the release-build template's build configuration.
2. Click Create Build Configuration From Template.
3. Enter "release-xx.yy" for Name, where "xx.yy" is the name of your release branch.
4. Enter the name of the branch for the release.branch parameter. For example, "RB-01.05".

We've created a configuration for running a release build on the branch.

To create a release:

1. Ensure all changes for the release are checked into trunk.
2. Create the branch. For example:
   

   svn copy $SVN/xxx/trunk $SVN/xxx/branches/RB-xx.yy
   

   
   
3. Click Run on the release build configuration in TeamCity.

You can keep the release build configuration around for a particular branch as long as you like. If you are finished with a branch, you can delete the build configuration. If you need it again, it's easy to recreate it from the template.

That's it. I hope to write up some notes on the other points (manipulating the repository and ensuring the correct release versions of dependencies) soon.

Oracle SQL-Developer 2.1

I've been trying out Oracle SQL-Developer 2.1 early adopter for the last couple of days, because SQL-Developer is the standard query tool at the company where I'm working.

Personally, I mostly use Aqua Data Studio. Apart from its odious activation scheme, ADS is excellent in every respect. However, ADS is kind of expensive, while SQL-Developer is free.

Previous versions of SQL-Developer were promising, but not very polished. They would paint strange colors on the screen in my installation, and behave poorly with regard to threading and large queries. The new version seems like a big improvement so far. Although it doesn't have the same brilliant keyboard support as ADS, it does seem to work pretty solidly, and appears reasonably attractive. (Much prettier than TOAD at any rate.)

I did notice one amusing glitch so far:








In fact the query only returned about 20 rows.

Table size frequencies

Here's an interesting query to run on your Oracle database:


Here's what I get on the main application I've been working on the last several years:

ROW_COUNT               NUM_TABLES
----------------------  ----------
1-10                            36
10-100                          22
100-1000                        22
1000-10000                      22
10000-100000                    22
100000-1000000                  33
1000000-10000000                20
10000000-100000000               2
100000000-1000000000             3
1000000000-10000000000           1
                               183

Inline data for DbUnit tests in Grails

There is a DbUnit plugin for Grails: http://docs.codehaus.org/display/GRAILS/DBUnit+Plugin.

However, like almost all tutorials for DbUnit, this plugin assumes that the best way to organize your test data is using DbUnit's FlatXmlDataSet. The example given on the Wiki link above has this FlatXmlDataSet:


During my 5 years of working with DbUnit, I have come to the conclusion that FlatXmlDataSet is not the best way to organize test data. Here are my reasons:

• XML is not a good representation for tabular data. CSV is more readable.


• For most of my tests, I would rather see the test data together with the test, rather than separate.

DbUnit tests often have 3 "phases":

1. Prime the database with setup data.


2. Run some code of the System Under Test.


3. Verify changed data in the database.

For most of my DbUnit tests, I would like to see the data used for the first and last parts together with the test itself.

For this reason, we've gravitated towards inline CSV datasets in most of our DbUnit test cases. You can achieve this using Groovy in Grails very simply.

The Wiki example, used in a test with inline datasets, might look like this:

(I'm using pipe characters instead of commas in this case!)
If the same Person data were used in different test methods, we might factor it out into a common method or field.

What I tend to find is that I use a lot of common reference data in my tests, but test-specific detail data. I handle the common reference data in a setUp() method, or even in the base DbUnitTestCase class. I put the test-specific detail data in each test. This prevents duplication of the common stuff, and keeps each test clean and focused. By reading the test method, you can see everything involved in the test.

We define cleanInsert() and a few other conveniences in DbUnitTestCase:

Tomcat Expert Seminar

Yesterday I attended SpringSource's Tomcat Expert Seminar.

This session rocked! I have learned a lot of neat ways to enhance our usage and troubleshooting techniques. The presenter, Filip Hanik, was not only extremely knowledgeable, but he was also entertaining and engaging. He handled questions from the audience extremely well, being able to give a direct and informative answer to nearly every question.

The most interesting topics for me were:

• Large scale deployments -- showed flexible ways to organise deployment to simplify instance configuration and management.


• Troubleshooting -- great tips on Tomcat but even more on Java and server-based systems in general.

I look forward to attending similar sessions in the future.

Linux on new Laptop

I bought a new laptop a couple of weeks ago, because Zena's old hand-me-down IBM R50p's screen bit the dust. So she gets my IBM/Lenovo Z61p, and I bought a Sony VGN-Z17GN. Wow, Sony have long model numbers.

The thing that sold me on the Sony was the high spec in the small (and elegant) package. It has 4GB RAM, 320 SATA hard drive, dual core processor and a 1600x900 screen. But the machine is tiny, and weighs only 1.5kg. In a small laptop bag and with the power supply, it's lighter than my previous laptop's backpack, empty.

Like most people, I don't care for Vista. So I actually applied the XP Pro upgrade option before I even got the machine home. But the OEM XP Pro disk did not include the machine's specific drivers. So I had to download a ZIP bundle from Sony which was supposed to include all of the drivers. It did include 20 of them, which meant about 20 times of clicking through the installer, accepting whatever inane licence agreement, and rebooting. Sony didn't include the ethernet driver in their bundle, so I hunted that down on the Intel web site. It in turned required MSXML, so I had to find that and install it too. In the end, a fairly typical experience of installing Windows, mind-numbingly tedious.

After I finally got XP working, more or less, I popped in a freshly-burned Ubuntu disk and began the Linux install. Wow, what a difference. I had actually been a little worried about how hard it would be to get the video annd ethernet working with Linux, since the machine is quite new on the market. (I have the first one sold in New Zealand.) Well, in a few minutes, with perhaps one or maybe two reboots, I had Ubuntu installed, and everything just works. Everything I care about, anyway. I have no idea whether the fingerprint reader is supported in Linux, but I don't care. To be honest, the wireless network doesn't work yet, but apparently it is supported directly in the next Ubuntu version, due out next month, so I'll just wait for that.

Once I restored my home directory from a backup, all my desktop and configuration settings were ready to go on the new machine. No registry hacking, no special software for migrating settings. It's funny how with Windows, a lot of the "features" are workarounds for problems that don't exist in other operating systems.

Tests For Your Data 2: When to Use

My friend Nigel Charman commented on Tests For Your Data with some good questions.

First the short answers.

Where possible the constraints are matched in the application. So the "double check" idea holds true. Occasionally this turns up bugs in the application. But more often it turns up bugs in manual edits of data.

With referential constraints, despite what some people seem to think, you need to define them in the database, regardless of whether they are also enforced at the application level. Exactly the same here. Double checks are useful, and applications are not infallible.

In my experience I'm running these only in production, and yes I regularly get production failures for them. (That's why I do them. ;-)

I probably shouldn't have hijacked the Continuous Integration metaphor for this idea. Basically this is a data management practice, and doesn't have much to do with the development cycle. However, it is a practice I am very passionate about. Just as a good test suite keeps my code healthy and vigorous, I feel that these data checks help me keep my production data healthy and clean.

Now for the meaty question: For what types of data is this approach suited?

Data from external sources. For data entered interactively, it's usually best to reject invalid data immediately at point of entry. For external-sourced data that are loaded in batch, this is not always the best way. Sometimes data are correlated with data loaded via another batch stream, in a separate transaction. There isn't always a good place to validate and reject bad data. In such cases, check views help catch the bad data.

Data in 3rd-party systems. We have applications for which we are not the developer. So we have little or no control over the application logic or database constraints. But with check views, at least we can identify data problems and work to fix them.

Multi-row conditions. The classic case here is checking for gaps and overlaps between multiple rows containing date ranges. In my work at Red Energy, we have many tables with bitemporal data (two time dimenions). It's quite hard to visualize these data simply by looking at the tabular form, so it's useful to have check queries to verify that the "shape" of the data is valid.

Aggregations. We have cases where different tables aggregate the same basic information by completely different keys. After having the IT manager complain to me twice about two reports (driven from two tables) not balancing, I added a check query to verify that the aggregations balance. The next time the problem happened, I was the first to know.

Parent-to-child relations. Foreign keys can enforce that every child has a parent. Sometimes you want to enforce that every parent has at least one child.

"Future" conditions. Sometimes you have "static" data that cover a range of time, such as calendar data or pricing. You enter data for the next three years, and then start running your system. A carefully-written check view can remind you when it's time to update for the next three years.

Believe it or not, we also have some check views on the check views. There is one that warns if any view (or PL/SQL package) in the database contains errors. There is another that verifies that every check view has a comment.