Blah.Blah.Blog

It seems my friendship with StringTokenizer is one based on deceit as it singularly failed to do what I wanted (nay - expected) it to do. I'll use a simple test to demonstrate exactly where my frustrations lay:

public void testTokenizerNullTokenHandling() {
    StringTokenizer tokenizer = new StringTokenizer("a,b,,d,e,,", ",");
    assertEquals(7, tokenizer.countTokens()); // fails
    assertEquals(4, tokenizer.countTokens()); // passes
}

To perform my validation correctly, I needed to be able to tell when a field in my source spreadsheet had no value, thus producing two consecutive commas. However, my comrade StringTokenizer treats consecutive delimeters as a single one, thus removing all knowledge that an empty position existed.

This causes me all sorts of pain: As each column in the spreadsheet, and therefore field in the CSV, holds data of a different type, treating consecutive delimeters in this manner causes fields to ooze left into neighbouring positions when empty values are supplied.

Interestingly, my second solution returned results slightly more in alignment with what I needed, but still not sufficent. Using split from java.lang.String resulted in a structure that included embedded empty values, but not ones at the beginning or end of the string being processed.

public void testSplitNullTokenHandling() {
    String tokens[] = "a,b,,d,e,,".split(",");
    assertEquals(7, tokens.length); // fails
    assertEquals(5, tokens.length); // passes
}

So, at this point I turned to my own means for a solution, and came up with:

protected List tokenizeSpec(String spec) {
    ArrayList tokens = new ArrayList();
    int delimeterPos = spec.indexOf(ATTRIBUTE_DELIMITER);
    if (delimeterPos < 0) {
        tokens.add(spec);
    }

    int tokenStartPos = 0;

    while (delimeterPos >= 0) {

        String token = spec.substring(tokenStartPos, delimeterPos);

        tokens.add(token.trim().length() > 0 ? token : null);

        spec = spec.substring(delimeterPos + 1);

        delimeterPos = spec.indexOf(ATTRIBUTE_DELIMITER);

    }

        

    return tokens;

}

This way, I get a nice data structure out the other end that I can quickly query for information such as "is there an element in field 3?". Admittedly, I loose the ability to determine how many values have been supplied, but if needed I could abstract this list and method into a class and provide a method to return exactly that.

In summary, I'm not particularly miffed that my passing acquaintance StringTokenizer and it's partner in crime String.split tokenize the way they do by default, but I was a little surprised there isn't some way to override the default implementation to provide such functionality as I imagine it would be a fairly common need.

Footnote: I'm extremely open to comments along the lines of "You, sir, are a fool of the highest order, pray tell what thou didn'tst use the bletch method from the foobar class to achieve your aim?"