Issues with JBCP and Sybase

Posted on February 9, 2009 by

BCP is a method for quickly copying data into SQL Server or Sybase databases. It’s much faster than running INSERT statements, so if you have a large amount of data, it’s a tempting option.

However, the trade-off for the speed is that when importing data, it bypasses a large amount of error checking and handling. As a result, any error messages you do get from the database server can often be cryptic or can even appear to be unrelated to what you’re trying to accomplish!

As well as the command-line BCP tool, JBCP is a pure Java library that reimplements the BCP protocol for Java developers. Unfortunately it’s still relatively immature, and you may find that Sybase gives the following error message in response to almost all JBCP issues:

Bad row data received from the client while bulk copying into object 1874356894 in database 4

This can be caused by several things. The most common is simply that the table definition has not been setup correctly to allow rows to be BCP’d into it. One way to resolve the issue is to ensure that the table you are copying into has ‘lock allpages’ as part of its definition, for example:

CREATE TABLE bcptest (some_key int not null, mydata varchar(50) null) lock allpages
go

Other issues can be caused by using univarchar columns that exceed 255 bytes. This means that all univarchar columns you attempt to copy into must be no larger than univarchar(127). A valid workaround for this issue is to split large column definitions into several smaller columns, and then use an UPDATE statement to reassemble the data from the split columns after it has been uploaded.

To help googlers, here is the complete stack trace I was typically seeing from JBCP before I created my table with the additional lock allpages clauses.

java.sql.SQLException: Bad row data received from the client while bulk
copying into object 1874356894 in database 4. Received a row of length
72 whilst maximum or expected row length is 15.
[19/12 10:46:37.160] [TID:lee2BCP1]:Upload of 1 result: FAILED
       at
net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
       at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:3149)
       at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2587)
       at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:734)
       at net.sourceforge.jtds.jdbc.TdsCore.bcpBatch(TdsCore.java:2394)
       at net.sourceforge.jtds.jdbc.BCP.bcpBatch(BCP.java:1134)
..snip..

One final warning for using JBCP – you don’t always even see the problem by default – some exceptions are supressed causing silent failure! There be dragons!

Posted in Java | 1 Comment

Getting Rails 2.2 to work on Mac OS X, despite the MySQL ruby “gem”

Posted on January 29, 2009 by

rubygems makes package management a nightmare. This is demonstrated every time you try and setup a new server using your operating system’s ruby/rails packages. It gets fully demonstrated when you try and upgrade to Rails 2.2 on a Mac. You can go read the full details of the nightmare if you’re so inclined.

The summary is that rails 2.2 removes the mysql driver and so you have to install a native driver using rubygems. Normally this would be a single command, but MySQL is a mess on Mac OS X. If you have a working MySQL installation on a Mac, chances are you’ve used MAMP and/or installed it yourself.

After about two hours of pain, I finally came across this forum post which has the horribly hacky but wonderfully simple idea of copying the mysql driver that worked in older versions of rails into your rails 2.2 app! This finally worked! On my Mac, this meant running the following command:

 cp /Library/Ruby/Gems/1.8/gems/activerecord-2.1.2/lib/active_record/vendor/mysql.rb ~/Software/myapp/lib/

When I restarted my rails app, it actually worked! Joy!

It’s worth noting that it’s probably worth pushing through the pain to install the native mysql gem on your production servers, but this technique is a winner to keep development on the go.

Posted in Rails | Leave a comment

Head Hunting; a brand new way to recruit top talent

Posted on January 23, 2009 by

I’m pleased to announce the first public beta release of an exciting new service from Crossbone Systems.

Head Hunting is a revolutionary new way to recruit employees at a fraction of the cost of other methods such as recruitment agencies and big job sites. As a Facebook application, Head Hunting allows you to hire new employees with a little help from your friends.

This week the service has launched into “public beta”, which means anyone can post and search job listings for free. Continue reading →

Posted in Announcements, Starting up | Tagged , , , | 1 Comment

Fun with FBJS: “Can’t find variable: a43014453287_slider”

Posted on January 14, 2009 by

I’ve been trying to get a slider bar working recently as part of a facebook application. To get the example on the Facebook wiki working out-of-the-box was no problem.

The problems came when I wanted to create the slider bar as the result of a user clicking on an input. For some reason, Facebook only loads 3rd party Javascript (ie. mine) in the body of the page. If a user clicks a link that calls a Javascript function before the page has finished loading, the function may fail if it depends on functions kept in external Javascript files (that have not yet loaded). 

Because of the complexity of Facebook pages, there is often between 1 and 5 seconds when a user may click one of my links even though the page hasn’t technically finished loading. If this happens, the user will either see a very cryptic error or nothing will happen. It’s a completely unacceptable user experience so I was forced to come up with the workaround below.  

var sliderCreated = false; // can only create one slider per-page with this
function createSlider() {
    if (!sliderCreated) {
        try {
            new slider(document.getElementById('output'), document.getElementById('count_slider'), 0, 100000, 250, document.getElementById('salary_min'), , 500);
            sliderCreated = true;
        } catch (error) {
            // Errors may get thrown if the fbslider.js wasn't loaded quickly enough, so we wait a second
            setTimeout(function() { createSlider() }, 1000);
        }
    }
}

This will detect any errors thrown from the creation of the slider (which relies on an external Javascript file). If an error is found, then the attempt to create the slider will occur again after a second. This has the advantage that even if the second attempt fails (if the network is very slow), it will continue to retry with a delay between each attempt.

A variable (sliderCreated) is used to detect when the slider has been successfully created. In case the link that creates the slider is still clickable after the slider is created, this variable guards against multiple sliders being spawned on the page.

It is left as an exercise to the reader to add an upper limit to the retries if required.

Posted in General | Tagged ,

Effortless online VAT returns with KashFlow

Posted on November 20, 2008 by

Once you are registered for VAT, sending in your returns every three months can be a bit of a daunting task, especially if you’re not keeping strict electronic records on a regular basis.

I’ve blogged about various online accounting packages before, but if you want to keep track of your company’s finances, and make your VAT returns effortless, KashFlow is the way to go. Continue reading →

Posted in HOW-TOs | Leave a comment

The social networks of tomorrow

Posted on November 16, 2008 by

Ning is pants. Google groups are pretty pants.

Each offer a useful set of limited features, but if you’re trying to create a dynamic community, your creativity will hit the limits of their capabilities fairly quickly.

What should a complete modern social network look like? A social network is a collection of people with something in common to bind the group together. This could be an interest, it could be their employer, etc.

But what features should the modern social network offer, and does it exist as an out-of-the-box software package or hosted solution? Continue reading →

Posted in General | Tagged , | Leave a comment

Explained: learning Ruby on Rails (Part 2)

Posted on November 12, 2008 by

If you followed along in , you will have downloaded and installed Netbeans 6 to get a complete Ruby development environment. You should also have prototyped a web design idea, added some dynamic code to it, and got cosy with Ruby by implementing a small standalone application.

In this article, we’ll learn a few more techniques to help you build the next big thing for the web. Continue reading →

Posted in Rails | Tagged , , | Leave a comment

Explained: how the hell do I *really* learn Ruby on Rails?

Posted on October 27, 2008 by

I own several books on Rails, all of them very well written and pleasant to read. But there’s a big difference between reading a book and knowing & learning a platform. You have to code in that platform, you have to first build something trivial, and then build at least one thing that’s non-trivial. Only then do you start to become intimately familiar with a platform’s conventions, the APIs and its general approach to problem solving.

With that in mind, I present a basic guide to how I learnt Ruby on Rails at a steady pace and without feeling overwhelmed by its enormity. Continue reading →

Posted in Rails | Tagged , | 1 Comment

Warning: iPhone 3G should only be purchased by idiots

Posted on July 24, 2008 by

The buzz around the iPhone 3G launch caused a lot of excitement, plenty of crashed servers, and some very unusual queuing. Despite that, iPhones were still fairly easy to buy within the first week of the phone’s launch. As a small business owner and avid technologist, I couldn’t resist. Perhaps I should have.

Here are my five killer reasons why the iPhone 3G sucks. Continue reading →

Posted in Miscellaneous | Tagged , | 2 Comments

Y Combinator: Startup Ideas We’d Like to Fund

Posted on July 20, 2008 by

Anyone struggling to come up with “an idea” for a new techie startup could do a lot worse than checking out this latest article by Paul Graham entitled “Y Combinator: Startup Ideas We’d Like to Fund“.

Paul lists thirty different ways of thinking about start-up opportunities. I was relieved (and spurred on by the fact) that my own in-development project fits into at least one of the categories!

I was also pleased to see that most of the focus isn’t on blue-sky wacky concepts. It’s a much better idea to build and sell things that attack existing well-known problems or solutions by doing something subtley different or simply better.

Posted in Miscellaneous | Leave a comment