The first and more serious bug is that the driver always returns numbers with a decimal point as a zero when using French “regional settings” in Windows.

For example, when using these Regional Settings…

Zut alors, c'est un comma!

…the sample .Net application that IBM provides in its QuickStart tutorial displays this:

French toast

We ran into this bug because many of our product’s users have the “French (Canada)” setting on their PCs. I assume that the root cause of the bug is that a comma, rather than a period, is used as the decimal point, as marked in yellow above. IBM support has confirmed that this is a bug are internally testing a fix in the Runtime Client, so the fix will likely be available in Fix Pack 3c or the one after that.

I’m surprised that nobody (including us) ran into this bug earlier, but that’s probably because most developers (including us) have got into the habit of reading data from UniVerse as strings, then converting the data into its proper format in their code. As I’ve mentioned in past articles, UniVerse has a “multivalue” heritage where all data is stored internally as strings. Developers should go against this grain at their own peril.

For example, rather than defining a currency field in the Universe dictionary like this…

COMMISSION
001 A
002 34
005 S
007 MD3
009 R
010 7

… define it as….

COMMISSION.STR
001 A
002 34
005 S
006 CHAR,7
009 R
010 7

… and read the value into a String. Then, you can convert it to a Decimal variable using code like…

Decimal.TryParse(strValue, System.Globalization.NumberStyles.Number,
System.Globalization.CultureInfo.InvariantCulture, out decValue);

The “InvariantCulture” thing is needed because, when defined as a CHAR, UniVerse will return a decimal value using — surprise, surprise — a period as the decimal point.

The second bug that we tripped over was a little more obscure. The bug was found by one of our developers, who was double-checking the configuration of our clients’ databases to make sure that all of our latest patches had been applied. He found that one particular SQL statement used in our application was failing with the following error:

Error  occurred: ERROR [Command_ExecuteDirect]  [IBM  U2][UCINET][UNIVERSE]:UniVerse/SQL: syntax error.  Unexpected verb.  Token was  "LIST". Scanned command was FROM VENDORS SELECT LIST  :  IBM.Data.DB2: -2147467259

This type of error usually indicates a mismatch between our dictionary entries and our application, which was the type of configuration issue that he was checking for. It wasn’t a big surprise, since the problem was found in a database that is only used for in-house training. Surprisingly, though, after reapplying the latest dictionary entries the bug remained.

Like most database systems that I’ve worked with, UniVerse’s SQL parser tends to return some pretty vague error messages. The particular SQL statement that was failing was quite long, containing about 20 columns, none of which were named “LIST”. By trial-and-error, I found that the culprit was a field named DEMO.LIST. The DICT definition of this field was fine – it exactly matched the dictionary entry used in the other databases at this client site, none of which were experiencing this error. While scratching my head over this, our developer told me that he found the same error in the same SELECT statement in a database at a different client. This database was also one used for in-house training.

This made me suspect that the error was caused by bad data. The databases used for in-house training are more likely to contain weird characters like embedded tabs or carriage-returns.

The dictionary definition of DEMO.LIST was a 100-character string:

DEMO.LIST
001 A
002 5
005 S
006 CHAR,100
009 L
010 100

As mentioned earlier, strings are usually a safe choice for data in UniVerse. I’m not aware of any type of data that can cause a SELECT of a a string to fail in UniVerse’s ADO.NET driver, but there’s a first time for everything. However, a bit of trial-and-error testing made it clear that this SELECT statement was failing on the plainest of records: vanilla text that couldn’t possibly cause a problem. It was a real whodunnit.

Back to the not-very-helpful error message, It was complaining about a “token” named LIST. If I simplified the variable name to something like FIELD1, did that affect the error message? Well, yeah: it made the error go away completely. The SELECT statement worked fine if the column name was changed to FIELD1, or DEMO or even DEMOLIST. Weird. We have periods in lots of our column names, and a lot of them are named LIST. DEMO, not so much.

At this point, things finally fell into place. The name that we give our databases that are used for in-house training is DEMO. And, of course, in standard SQL syntax, a period in a column name is used to identify the table, such as “SELECT TABLE1.FIELD1″. Sure enough, I ran some tests in our development database, named DEV, and found that naming a column “DEV.LIST” or “DEV.FIELD” or “DEV.WHATEVER” caused SELECT statements that use that field to fail.

The moral of the story, I suppose, is don’t use periods in your column names. That was IBM support’s suggestion, though I’m hoping that they will fix this bug as well.

I suppose that both of these bugs are indicative of what happens when you drag a 40-something year-old non-relational database kicking and screaming into the .Net world. Oops, better make that “dotNet” to be safe!

select USER.ID from "CSUSER-FILE"

is valid, but not

select USER.ID from [CSUSER-FILE]

or

select user.id from "csuser-file"

Oddly, names that contain a period do not have to be quoted – unlike its SQL database brethren, UniVerse is smart enough to figure out when the period delimits the table ID from the column name.

3) A "SELECT *" query will return the key fields along with whatever fields are defined in a special dictionary entry named "@".  For example, if you want SELECT * FROM USERS to return the FIRST-NAME and LAST-NAME fields, you’ll need to edit the dictionary as follows:

ED DICT USERS
Record name = @
2 lines long.

----: 2
0002: 1 2
Bottom at line 2.
----: R FIRST-NAME LAST-NAME
0002: FIRST-NAME LAST-NAME
Bottom at line 2.
----: FI
"@" filed in file "DICT USER".

If you don’t have a record named "@" in your file’s dictionary, then "SELECT *" will return the column named "@ID" if you have one (which you probably do, since the primary key is automatically added to all dictionaries under this name when you first create them).  If you happen to have deleted the "@ID" column, then SELECT * will fail with a "No column rows" error.

What if you have a 200 column table and you want SELECT * to return all 200 columns?  Sorry, but you’ll have to enter all 200 column names in that "@" record.  And if you add, delete or rename a column, you’ll have to remember to edit that "@" record. I feel your pain!  This cumbersome approach dates back to UniVerse’s ODBC driver, and I suppose they can’t change it now without breaking a lot of applications.  You can find the details described in inscrutable IBM-ese in the UniVerse ODBC Guide.

4) Errors encountered when executing your SQL statement are reported in a pretty obscure manner.  You’ll see a message box with the "Error in SQL.  Check the SQL and try again.  For error detail information refer to the IBM Output Message Pane".  It took me awhile to figure out where the "IBM Output Message Pane" was hidden.  It’s in the Output window (Ctrl-W O), but you’ll need to change the "Show output from" combo to "IBM Output Message Pane" as shown below. 


5) If you have subvalued fields, only the first subvalue in the field will appear in the Results grid.  (Subvalues are ASCII 252 delimiters which are used to fit multiple fields into the same column).  This is presumably a side effect of a bug in IBM’s ADO.NET driver which replaces subvalue delimiters with CR/LF.

6) There are a couple of bugs in the ADO.NET driver for which, as far as I know, there is no good workaround.  The first is that the SQL processor can’t handle hard-coded values which contain the "at sign", such as:

SELECT * FROM MY.TABLE WHERE THE.KEY = "DAN@WORK"

Depending on the structure of your data you may be able to get at the data by using a wildcard instead

SELECT * FROM MY.TABLE WHERE THE.KEY LIKE "DAN%WORK"

The other bug is that zero-length numeric fields such as the one in attribute 5 below will result in an “Input string was not in a correct format” error (that funny little accented character is ASCII 253, the multivalue delimiter):

0001: SMITH
0002: 1414 W. 8TH
0003: DENVER
0004: CO
0005: V2001ýýC8181
0006: 200ý3599ý1050

This latter bug was fixed by IBM in the recently released Fix Pack 3, but if you have users on earlier releases there is a workaround you can implement in your code, as explained later in the article.

7) Last, but not least, multivalues.  As described in my previous article there are a few different ways to configure your dictionary to return multivalues, but the method that IBM recommends is to list the multivalued fields in a special "association" dictionary entry that looks like this:

    DETAILS
001 PH ASSOCIATION OF ORDER DETAIL FIELDS
002 ORDER.IDS ORDER.AMOUNTS

The advantage of this approach is that you can return the items within the multivalued field as if they were separate columns, which makes the data a lot easier to work with.  The disadvantage is that you’ll have to puzzle out the syntax for a join with a "virtual table".   If the above field DETAILS was defined in the dictionary of a table named CUSTOMERS, then a SQL statement to return a list of order IDs and amounts would be:

SELECT a.KEY.FULL, ORDER.IDS, ORDER.AMOUNTS FROM CUSTOMERS a JOIN CUSTOMERS_DETAILS b ON a.KEY.FULL = b.KEY.FULL

Since the dictionary field that is automatically added for the primary key, "@ID", can’t be used in SQL statements, you’ll have to add one yourself in order to use it in the JOIN — in the above example I added a field named KEY.FULL to CUSTOMERS.

If by this point you’re fully freaked out by the UniVerse way of implementing SQL, then I’ve got some good news for you: the code itself is pretty much plain vanilla and won’t require much explanation. 

Before you start coding, you’ll need to add a reference to the IBM.Data.DB2 DLL to your project References list, as shown at the right.  You’ll also need to add "IBM.Data.DB2" to your list of namespaces:

using IBM.Data.DB2;

The process of building a connection string is quite similar to other databases like SQL Server.  You’ll need to set the Server, User, Password and Database (which is the UniVerse account name).   In addition, you need to set two less familiar properties — ServerType and Pooling:

private string providerName = "IBM.Data.DB2";
private DbConnectionStringBuilder builder;
private DbConnection conUniVerse;

builder = new DbConnectionStringBuilder();
builder["Server"] = txtServer.Text.Trim();
builder["User ID"] = txtUserID.Text.Trim();
builder["Password"] = txtPassword.Text.Trim();
builder["ServerType"] = "universe";
builder["Pooling"] = "false";
builder["Database"] = txtAccount.Text.Trim();
string strConnectionString = builder.ConnectionString;

DbProviderFactory m_provider = DbProviderFactories.GetFactory(providerName);
conUniVerse= m_provider.CreateConnection();

conUniVerse.ConnectionString = strConnectionString;
conUniVerse.Open();

You’ll then need to create an ADO Data Adapter and feed it the SQL statement.

DbProviderFactory factory = DbProviderFactories.GetFactory(providerName);
DbDataAdapter adapter = factory.CreateDataAdapter();
adapter.FillError += new FillErrorEventHandler(adapter_FillError);
adapter.SelectCommand = DB_GetCommand(strSQL);

Lastly, you create an ADO Dataset to hold the results of the query, then bind that Dataset to a grid on your form to display the data:

DataSet dsData = new DataSet();
adapter.Fill(dsData , "MyData");

bindingSource1.DataSource = dsData ; // bindingSource1 is a BindingSource object that I dragged from the Toolbox onto my form
bindingSource1.DataMember = "MyData";
gridData.DataSource = bindingSource1; // gridData is a DataGridView object on my form
conUniVerse.Close();

You may have noticed that I added a FillError event handler to my adapter.  This is a workaround for the zero-length multivalue problem that I mentioned earlier in the article.  In the error handler, you can examine the Exception’s properties and, if it appears to be the zero-length multivalue error, set the e parm’s Continue property to true.

void adapter_FillError(object sender, FillErrorEventArgs e)
{
    if (e.Errors.Message.Contains("Input string was not in a correct format"))
        e.Continue = true;
}

This will result in the dataset being filled, but without the multivalued " virtual record" that contained the zero-length value. If the zero-length value is just bad data then that’s probably what you want — otherwise, you’ll need Fix Pack 3.

Whew!   That might seem like a lot of work to read some data into a grid, but it’s stable, it’s pretty fast, and it’s still a lot less coding than the only other stable and fast UniVerse API out there, UniVerse Objects.  Once you’ve learned the ropes, UniVerse is now a viable database option for .Net development. 

public static TimeSpan TimeSpan_FromUniverseString(string strUniverseTime)
{
    int intUniverseTime = 0;
    // if the string contains milliseconds, discard them
    int intIndex = strUniverseTime.IndexOf('.');
    if (intIndex >= 0)
    {
        strUniverseTime = strUniverseTime.Substring(0, intIndex);
    }

    if (!Int32.TryParse(strUniverseTime, out intUniverseTime))
    {
        // if not a numeric, then return 00:00:00
        return new TimeSpan(0);
    }
    int intDays = intUniverseTime / 86400;
    int intRemainder = intUniverseTime % 86400;
    int intHours = intRemainder / 3600;
    intRemainder = intRemainder % 3600;
    int intMinutes = intRemainder / 60;
    int intSeconds = intRemainder % 60;
    return new TimeSpan(intDays, intHours, intMinutes, intSeconds);
}

However, an easier workaround (which I stumbled across after using the above approach for months) is to take advantage of the "correlative" code. 

This code only exists in A- and S-type dictionary entries, in attribute 8.  Ordinarily, time fields are defined in UniVerse dictionaries by specifying a "conversion code", such as "MTS" in attribute 7.

    ACCESS.TIME
001 A
002 3
005 S
007 MTS
009 R
010 5

This tells ADO.Net (or UniVerse Basic) that the value in this field can be regarded as a time.  However, by specifying the same conversion code in attribute 8, the conversion to a time value is handled internally by the UniVerse database before ADO.Net sees it.

    ACCESS.TIME.SQL
001 A
002 3
005 S
008 MTS
009 R
010 5

Therefore, a value such as "3600.5" will be seen by the ADO.Net driver as 1:30 am, without that nasty millisecond value that it so dislikes.

3) Multivalues

Ah yes, multivalues.  This is a method for storing multiple values into the same field, separated by a special delimiter.  For example, attribute 1 might contain a list of order numbers, and attribute 2 the amount (in cents) of each of order:

001 C2000ýM3000ýS3000
002 600782ý700422ý101456

Multivalued strings are the primary feature that distingishes UniVerse and its brethern databases from all the others, but .Net doesn’t really understand the concept (nor do many .Net programmers, actually).  UniVerse provides some tools which allow you to dress up the multivalued data to resemble columns in a normalized SQL database — it is up to you whether you want to play dress-up, or just process the multivalued fields in their underlying form: delimited strings.

There are 3 different approaches to definining multivalued strings in the dictionary so that they are safe for consumption by .Net:

i) The documented approach is to use a D-type dictionary entry which contains an "M" in attribute 6 and an "association" name in attribute 7.  For example, the definition of the Order ID and Order Amount fields might look like:

    ORDER.IDS
001 D Order IDs
002 1
004 Order IDs
005 10L
006 M
007 DETAILS

    ORDER.AMOUNTS
001 D Order Amounts
002 2
003 MD0,$
004 Order Amount
005 7R
006 M
007 DETAILS

And the dictionary entry which defines the virtual "DETAILS" table would be:

    DETAILS
001 PH ASSOCIATION OF ORDER DETAIL FIELDS
002 ORDER.IDS ORDER.AMOUNTS

If the main table which contained the multivalued fields was named SALES, then there would now be a virtual table named SALES_DETAILS containing three fields: ORDER.IDS, ORDER.AMOUNTS, and a foreign key field that points back to the primary key of the SALES table. 

For details, see chapter 5 of the UniVerse System Description manual, and for a thorough example see the sample code which is included with Part 2 of IBM’s tutorial on the ADO.Net driver.  In the third part of this series, I will show an example of how you would use a JOIN clause in a SELECT statement to return the fields in the virtual table in the same record as the fields in the main table.

The advantage of this approach is that the resulting SQL statements will closely resemble those that would be used for a relational database, and this should make it easier to port your application from UniVerse to a relational database should you wish to.  The downside is that it can become quite cumbersome to code when your records contain multivalued attributes that aren’t related on one another. For example, if a customer record contains a list of order numbers in attribute 1, and a list of customer contacts in attribute 2, you’ll have to specify a different association name for these 2 entries, and retrieve them using either a very complicated 3-way JOIN, or using 2 different SELECT statements. Also, since UniVerse does not offer any constraints or other mechanisms for enforcing the integrity of your data, you are likely to find that bad data fouls up your application, resulting in missing rows or runtime Exceptions.  In our application, we use this approach only for small and simple tables.

ii) You can return the multivalued strings in their "native" form — strings with delimiters separating the values.  You do this by using an I-type dictionary entry that converts the multivalue delimiter (ASCII 253) to whatever character you want.  For example, to return attribute 1 with the multivalued fields delimited by commas:

   FIELD.LIST
001 I
002 CONVERT(@VM,',',@RECORD<1>)
004 FIELD.LIST
005 50L
006 S
008 VARCHAR,32767

You might be wondering why you can’t just return the strings using the original delimiter, ASCII 253.  For reasons known only to IBM, this results in a runtime exception.  Incidentally, there is is a second type of delimiter which is also common in UniVerse, the subvalue delimiter (ASCII 252).  UniVerse subvalues  occur when the parts of a multivalued fields are, in turn, delimited into smaller fields.  For reasons also known only to IBM, these are automatically converted by the ADO.NET driver to a carriage return and line feed (i.e. "rn").   IBM support has indicated that this delimiter conversion is actually a bug and will be changed back to ASCII 252 to a future Fix Pack.

Once you have the delimited string, you’ll need to use .Net’s string parsing capabilities to separate it into individual fields and convert the fields to their correct data type.  You’ll find that .Net isn’t ideally suited for this type of string processing.  UniVerse, on the other hand, is very much suited to it, and IBM provides a library named UniVerse Objects.Net that contains a variety of methods for extracting data from delimited strings.  Frustratingly, your ability to use UniVerse Objects to parse your data is hampered by the fact that ADO.NET has forced you to replace all the multivalue and subvalue delimiters with other delimiters that UniVerse Objects doesn’t support.

If you decide to read UniVerse multivalued data in its native form, you might want to consider bypassing ADO.NET altogether and reading the data using UniVerse Objects.  This is what our application does when dealing with tables that contain data that is mostly multivalued.  It is much faster, and will likely require less code.

iii)  While "official" UniVerse multivalued fields are delimited using ASCII 253 and 252, many UniVerse Basic programmers grow so fond of multivalues that they use them all over the place, with whatever delimiter character tickles their fancy.  I’ve seen multivalued strings used as the primary key, and I’ve seen multivalued strings that contain embedded dates and time fields.  While that kind of data structure would make SQL programmer cringe and a SQL database crawl, UniVerse is tuned to handle these strings efficiently and provides functions that you can insert in your dictionary to extract these fields relatively easily.

The trick is to place a correlative in attribute 8.  Appendix C of the UniVerse Basic manual documents these correlatives, but be forewarned: they are cryptic enough to make a Perl programmer weep with frustration.  Here are a couple of examples to get you started.

Say you had a key that was delimited with "@" signs.  Just to make things interesting, say that one of the fields was a UniVerse Julian date, and another was a UniVerse Julian time.  A typical key would be: FIRSTPART@14927@49055.75.

If you wanted to return the 1st part of the key ("FIRSTPART") as a separate field in the SELECT statement, you could define it as follows in the UniVerse dictionary:

    THE.STRING
001 S
002 0
005 S
008 F;0;(G0@1)
009 L
010 20

The correlative in attribute 8 means "take attribute 0 (the key), then split it into sections with ‘@’ as the delimiter and return part 0".  If you want all the gory details of what the "F code" can do, see Appendix C of the UniVerse Basic manual. 

To return the date string as a date, you would use a similar "F code" string in attribute 8, along with a conversion code in attribute 7:

    THE.DATE
001 S
002 0
005 S
007 D2
008 F;0;(G1@1)
009 R
010 9

The time field is trickier – as mentioned above, the ADO.NET driver doesn’t like time fields with milliseconds.  The best way to handle these fields is by putting the conversion code in attribute 8, but that’s where we need to put the correlative, and (as far as I know) you can’t put both a conversion code and a correlative in the same dictionary attribute.  You can, however, combine 2 extract commands in a correlative:

	THE.TIME
001 S
002 0
005 S
007 MTS
008 F;0;(G2@1);(G0.1)
009 R
010 12

This correlative tells the database to extract the time field from the key and then, having done that, extract the part before the decimal point.  The conversion code in attribute 7 then converts that value to a .Net TimeSpan.  (The Universe Basic manual refers to this as "reverse Polish format (Lukasiewicz)" — how geeky is that?)

So, there you have it: an overview of the tricks that I’ve used to whip our UniVerse data into a form suitable for consumption by .Net. 

If you are new to UniVerse, you are likely thinking at this point: "um, so how do I edit a dictionary".  As a crash course on UniVerse for Windows programmers, I would strongly recommend the "Learner Pack" put together by the U2 User Group.  The UniVerse manuals are a well written, exhaustive reference when you are ready to dive deeper, but as a crash couse you can’t do better than the U2UG Learner Pack.

In third and final part of this series, I’ll show you the (relatively simple) coding required to load UniVerse data into a DataGridView using ADO.Net.

1. Stability.  UniOLEDB’s intermittent "memory is corrupt" errors were driving me nuts.  Upgrading the database server from UniVerse 10.1 to 10.2 improved UniOLEDB’s stability significantly, but switching to the ADO.NET driver has made things better still.  Stability is no longer a concern — we’ve had no problems with dropped connections or failed queries.




2. Deployment.  UniOLEDB’s archaic install program was driving our clients’ IT guys nuts.  The fact that UniOLEDB had to be installed by manually on each desktop was a dealbreaker for the larger sites.  The DB2 Client’s installation program has its own quirks, as I wrote about here, but once we figured out the trick our clients have had no problems rolling it out with automated tools like Microsoft Systems Management Server.




3. Productivity.  The ability to have Visual Studio treat UniVerse like a full-fledged database would make it a lot easier to developer new forms.  With UniOLEDB we had to laboriously handcraft the DataAdapters for each table: write the SELECT, INSERT, UPDATE and DELETE statements, set their parameters, use them to construct the DataAdapter, then write the code to populate and manage the DataSet tables.  That was a lot of code, and we have a lot of tables!  Wouldn’t it be nice to replace all of that with a simple drag-and-drop?

Well, 2 out of 3 ain’t bad.

We haven’t quite reached that third goal yet.  We’ve got a relatively messy UniVerse database: our master dictionary and most of our data dictionaries are littered with references to things that don’t exist or aren’t correctly defined.  UniVerse itself ignores these obsolete entries, but it seems that the DB2 Client dutifully tries to make sense of them, each and every time that we connect.

As a result, any attempt to access a table through the Visual Studio IDE, be it from the Server Explorer or the Data Sources window, is painfully slooooooow.   I have an idea for doing an end-run around this problem, and when I have time to pursue it I’ll write a blog post describing what I did and how it turned out.

But, for now, drag and drop isn’t an option.

Our internal developers are still accessing UniVerse data through a labour-intensive code-based approach, without using the Server Explorer or Data Sources IDE at all.  That’s fine for us, since we’ve built a framework to support it, but when I was recently asked to help a client’s .Net developers get to our data, I knew I had to give them something easier.

My boss assured them that once they had the UniVerse 10.2 and its new IBM ADO.NET driver, UniVerse would be as easy to use as SQL Server.  Heck, we’d even create Views for the data they want.

Well, not so fast.  Even with an ADO.NET driver, UniVerse doesn’t support Views. There’s a node called "Views" sitting under the UniVerse data connection in Visual Studio’s Server Explorer, but it will always remain forlornly barren.  There is simply no equivalent entity in the world of UniVerse.

When I asked IBM about this, they recommended using UniVerse subroutines instead.  These are regarded by the ADO.NET driver as stored procedures, meaning that they can be viewed and invoked from the Server Explorer under the Procedures node.   You can use them with Visual Studio’s Data Source Configuration Wizard to create DataSets and DataAdapters without having to write any code.

Pretty nifty, and that’s something you couldn’t do with UniOLEDB.  (Predictably, it would throw up a "memory is corrupted" error if you tried).  But the subroutines are written in dowdy old UniVerse Basic, and our clients’ developers didn’t want to go there.  They opted for SQL.

Running SQL SELECT statements against a UniVerse database is a lot easier with the new ADO.NET driver, but it’s still tricky, with quite a few pitfalls that aren’t mentioned in IBM’s tutorials or (as far as I know) anywhere else.

So, having just played matchmaker between UniVerse and some .Net developers, I thought it might be useful to describe the quickest path to getting your UniVerse data into a .Net grid.

My next post will describe what needs to be done to prepare the UniVerse dictionaries for .Net, and the following post will show how to create UniVerse-friendly SELECT statements and feed the results into a DataGridView.

Having made the leap over to IBM’s new "DB2 Client" ADO.Net driver for the U2 UniVerse database, we recently spent some time figuring out how to get it onto users’ PCs.  This turned out to be harder than I expected, and harder than it should be.

The new Data Client’s installer was actually a big selling point for us.  The previous database driver, UniOLEDB, used a proprietary install program that didn’t support automated, silent installs.  This means that someone from the user’s IT department had to visit each user’s desktop and 1) launch the install program, 2) click Next a bunch of times, 3) phone us and complain about how annoying this is.  Every install program that we’ve created or incorporated into our product for the last 10 years has supported silent installs, except this one.  

I was so giddy at the prospect of moving on to a proper installer that I assured our clients that we would now fully support silent installs without (ahem) actually trying to do one.  My optimism was based on the fact that 1) the new database client uses the same installer as IBM’s flagship database, DB2, 2) this installer was created using one of the industry standard tools, InstallShield, which has supported silent installs for ages, and 3) the DB2 Information Center‘s extensive documentation includes a section entitled "Silent Installation".  It sure looks easy when you read about it.

Unfortunately, it doesn’t quite work the way it’s described, and finding the specifics required quite a lot of Googling as well as some trial and error. 

Here’s what doesn’t work:

1) Installing without a response file.  This will launch the installer, but it will quickly abort, and the error message written to the log is enigmatic:

Action ended 20:06:46: DB2SetDB2INSTANCECA.2BC48F01_561E_4906_8321_946A9F5A90AA. Return value 3.
Action ended 20:06:46: INSTALL. Return value 3.
MSI (s) (E4:68) [20:06:46:607]: Product: IBM Data Server Runtime Client – DB2COPY1 — Configuration failed.

Presumably, the response file is mandatory.

2) The standard InstallShield command line switch for generating a response file in the Windows folder (-r).  I never thought that the Windows folder was a great place to put it — apparently IBM agrees.

3) The Response File Generator utility (db2rspgn).  Although this utility does, indeed, generate a response file, we had absolutely no luck when trying to use this response file to do an install.  Googling the problem suggested that many others had the same result, but I still have no idea what the solution is, if any.

4) Manually editing one of the sample response files, as suggested by the Information Center, might have worked if we actually had a sample response file.  The sample response files are apparently located on the "Information Integration Product DVD", but I’ve never seen such a beast.

Here’s what does work:

1) Run the installer normally (with no command line switches), then look in your %TEMP% folder.  Sure, enough, there’s a .rsp file left behind in there. 

2) This response file works just fine with the standard InstallShield parameters for a silent install.  For the current version of the DB2 runtime client, this is :

v9.5fp2_nt32_rtcl_en.exe /v"/qn RSP_FILE_PATH=<path of the .rsp file>.

The contents of the elusive response file are somewhat of a letdown.  Unless you need to install 2 different versions of the database drivers, you’re unlikely to ever want to change a thing.   Should you not have an Information Integration Product DVD at hand, feel free to use our response file as a sample:

PROD=RUNTIME_CLIENT
INSTALL_TYPE=TYPICAL
DB2_COPY_NAME=DB2COPY1
DEFAULT_COPY=YES
DEFAULT_CLIENT_INTERFACE_COPY=YES
DEFAULT_INSTANCE=DB2
INSTANCE=DB2
DB2.NAME=DB2
LIC_AGREEMENT=ACCEPT

string strConn = @"User ID=JoeUser;Password=HisPassword;Database=c:accountpath;Server=ourserver;ServerType=universe;pooling=false";

Incidentally, you can now do away with the cryptic "uci.config" initialization file used by UniOLEDB.  The ADO.Net driver does not require any configuration beyond what is in the code.  Welcome to the 21st century!

5. Certain OleDB database objects need to be replaced with their ADO equivalents.  Note that the syntax for DataReaders, Commands and Parameters is quite similar, with the main difference being that the ADO objects don’t have their own constructors.  The DataAdapter, on the other hand, requires considerably more code to create using the ADO.Net driver:

• OleDbCommand is now DbCommand

// UniOLEDB
public static OleDbCommand DB_GetCommand(string strSQL)
{
    OleDbCommand cmd = new OleDbCommand(strSQL, conn);
    return cmd;
}

// ADO.Net
public static DbCommand DB_GetCommand(string strSQL)
{
    DbCommand cmd = conn.CreateCommand();
    cmd.CommandText = strSQL;
    cmd.CommandType = CommandType.Text;
    return cmd;
}

• OleDbDataReader is now DbDataReader

// UniOLEDB:
public static DbDataReader DB_GetReader(string strSQL)
{
    OleDbCommand cmd = new OleDbCommand(strSQL, conn);
    DbDataReader rdr = cmd.ExecuteReader();
    return rdr;
}

// ADO.Net:
public static DbDataReader DB_GetReader(string strSQL)
{
    DbCommand cmdSQL = conn.CreateCommand();
    cmdSQL.CommandText = strSQL;
    cmdSQL.CommandType = CommandType.Text;
    DbDataReader rdr = cmdSQL.ExecuteReader();
    return rdr;
}

• OleDbDataAdapter is now DbDataAdapter

// UniOLEDB:
public static OleDbDataAdapter DB_GetAdapter()
{
    return new OleDbDataAdapter();
}

// ADO.Net:
public static DbDataAdapter DB_GetAdapter()
{
    DbProviderFactory factory = DbProviderFactories.GetFactory("IBM.Data.DB2");
    DbDataAdapter adapter = factory.CreateDataAdapter();
    return adapter;
}

• OleDbType is now Db2Type (since we needed to support both UniOLEDB and ADO.Net, I changed our code to use the generic DbType enum and added conversion methods for the types that we use):

// UniOLEDB:
public static OleDbType DB_GetOleDbType(DbType type)
{
    switch (type)
    {
        case DbType.Date:
            return OleDbType.DBDate;
        case DbType.Decimal:
            return OleDbType.Decimal;
        case DbType.String:
            return OleDbType.VarChar;
        case DbType.Int32:
            return OleDbType.Integer;
    }
}       

// ADO.Net:
private static DB2Type DB_GetDB2Type(DbType type)
{
    switch (type)
    {
        case DbType.Date:
            return DB2Type.Date;
        case DbType.Decimal:
            return DB2Type.Decimal;
        case DbType.String:
            return DB2Type.VarChar;
        case DbType.Int32:
            return DB2Type.Integer;
    }
}

• OleDbParameter is now DbParameter

// UniOLEDB:
public static void DB_AddParameter(OleDbCommand cmd, string strName, DbType type, int intSize, string strSourceColumn)
{
    OleDbType oleDbType = DB_GetOleDbType(type);
    cmd.Parameters.Add(strName, oleDbType, intSize, strSourceColumn);
}       

// ADO.Net:
public static void DB_AddParameter(DbCommand cmd, string strName, DbType type, int intSize, string strSourceColumn)
{
    DB2Type db2Type = DB_GetDB2Type(type);
    DbParameter param = new DB2Parameter(strName, db2Type, intSize, strSourceColumn);
    cmd.Parameters.Add(param);
}

6. Once all of the tedious but straight-forward "a Quarter Pounder with Cheese is now a Royale with Cheese" stuff is out of the way, you should find that the two drivers speak pretty much the same language and not many other code changes are required. When we fired up the app and tried out our SQL statements, we only found two cases where the same statement didn’t return the same data:

a. Subvalue markers (ASCII 252) are returned by the ADO.Net driver as carriage-return/line-feed ("/r/n").  Subvalues are a method used by UniVerse to stuff 3 levels of data into a single field: if a multivalued field can be thought of as representing a parent-child table relationship, then subvalues can be thought of as parent-child-grandchild.  We don’t use them much, and it seems that most other developers don’t either, since IBM support wasn’t aware of this when I reported it to them.  However, IBM intends to resolve this problem in Fix Pack 3.

b. The ADO.Net driver doesn’t support use of the virtual "@ASSOC_ROW" field in SQL statements.  The UniOLEDB driver stored a row number in this field when converting a multivalued field into multiple virtual rows.  SQL statements that rely on row numbers are almost certainly not a good idea in any database — you should be able to rewrite the SQL to avoid using them.


And there you have it — with a relatively small and straightforward set of code changes, you’re now accessing your UniVerse data with a true-blue ADO.Net driver. 

As for the set of problems listed at the top of this article, I can report that the new driver solves problems #2 and #4.  I haven’t tried doing database updates with ADO.Net yet, so I can’t say if problem #1 is solved.  I noticed that IBM’s series of tutorials for the new driver recommends that database subroutines (written in UniVerse Basic) be used to handle updates, so I plan to stick to using UniObjects for the time being. 

We haven’t completely cracked problem #3 yet, partly because of the unavoidable "multivalued database with lipstick" issue.  The Visual Studio add-in works great with the sample HS.SALES account, and every crusty old UniVerse Basic programmer that I’ve showed it too is quite surprised to see a graphical database design tool that works with UniVerse.  Unfortunately, when we tried accessing our real world database we ran into a serious performance problem.  HS.SALES has 749 entries in its master dictionary — our database has over 5 thousand.  As a result, it took Visual Studio a few minutes to enumerate all of our schema objects.  A few minutes might not seem so bad, but this enumeration is done not only when the database’s object tree is first displayed, but each and every time a new query is created from the Data Connection.  It’s really cramping our style!

However, there is hope that Visual Studio and UniVerse can now get along.  IBM provides a few different ways to filter the number of schema objects that are available to Visual Studio, and we’ll be trying out those methods in the future.  I’ll write about that in a future article.

As mentioned at the end of my last post, my effort to optimize the SELECT statement by changing the .Net code had topped out around the 5 minute mark, well short of the 45-second response time that a query written in Universe Basic could achieve. I couldn’t see any way that .Net and UniOLEDB could replicate the technique used by the Universe Basic code, which involved building a native List structure from the keys, using a native Universe Basic command to extract the matching records from the large table, then running the SELECT on just the extracted records.

Fortunately my .Net code didn’t have to replicate this technical trickery; it could just stand back and let the Universe Basic code do the heavy lifting. Thanks to the fact that Universe Basic subroutines can return resultsets to .Net, I simply passed my SELECT statement to our Universe Basic code, let it do its thing, and processed the results.

OleDbCommand cmdTest = new OleDbCommand(”CALL GET.RECORDSET(‘SELECT yada yada yada’)″, conn);
cmdTest.CommandType = CommandType.Text;
dataAdapter = new OleDbDataAdapter(cmdTest);
dataAdapter .Fill(dataSet, “Table1″);

From the point of view of the .Net application this change is totally transparent — the contents of the resultset are exactly the same as when my code submitted its own SELECT statement. The Universe Basic subroutine had to be modified to extract the table name and key list from the SELECT statement, and it has since worked flawlessly in our testing, without stumbling over any rows or column types.

This turned out to be quite a good overall solution: virtually no .Net code changes, about half-a-day of coding for the Universe Basic programmer, and since the .Net code still uses SQL we maintained close compatibility with other databases. If we were to replace Universe with a relational database, I would just have to change a couple of lines of code in the .Net application.

The performance improvement when reading large amounts of data from the very large table was dramatic: from a previous best of 310 seconds down to just 45. Performance is slightly poorer than before on smaller SELECTs, but not enough to be noticeable to the user.

I’m a little disappointed that we couldn’t find a solution that only used .Net code. We already had to replace our INSERTs, UPDATEs and DELETEs with Universe-specific code as described in this post, and now the last facade of seamless SQL support has fallen. However, it was probably naive to think that we could transparently run SQL statements on a non-relational database like Universe. When in Rome, do as the Romans do. An appropriate axiom, considering that the Universe database is almost ancient history!

First, let me define “large amounts of data”. There is a grid in the application which lists media buys in a campaign — one row per “airing”. A campaign often has hundreds of thousands of airings, and millions of airings isn’t unheard of. In order to populate the grid the application may need to read tens of thousands of Universe records. Due to the multi-valued nature of a Universe record, each of these contains sets of up to 100 subvalues, roughly equivalent to a 1-to-many relationship to another table. The table from which this data is being retrieved contains billions of records – we ran our benchmarks with a table that was 11G in size and contained 11.6 billion records. Because this table only has one index, the primary key, the fastest way to retrieve records is by listing each and every key in the WHERE clause. The resulting SELECT statement is massive. The one which I used when benchmarking contained over 29 thousand keys in the WHERE clause and was 740K in size. Not the data, but the SELECT statement — 740K of text!

Given all this, I wasn’t terribly surprised that, prior to optimization, UniOLEDB performance was pretty poor. The first time I tried it, the 29,000-key SELECT statement required about 12 minutes to run.

From the .Net application’s point of view, this 12 minutes was spent waiting for 1 instruction to complete:

dataAdapter.Fill(dataSet);

Aside from the time spent transferring the data from the server to the desktop, which appears to be about 20 seconds based on the server and desktop CPU activity, all of this time was spent by Universe server process. This, then, appeared to be a Universe performance issue rather than an issue with the .Net code or with the various API layers that sit between .Net and Universe (ADO.NET, the ADO.NET Data Provider for OLEDB, and UniOLEDB).

If 12 minutes is clearly unacceptable, what response time should we reasonably expect for this type of query? Our Universe programmer tried running the same query (or so we thought) in Universe Basic code and it took 45 seconds to complete. So, we had plenty of room for improvement and a reasonable expectation that we could cut the response time by 90%.

Here’s a list of what I’ve tried, and the result:

1. Increase the Universe MAXSELBUF parameter. This is a server-side configuration parameter which defines the size of the buffer used to hold the results of SELECT statements. The default is only 4K, and Universe tech support recommended increasing it to 8K. This seemed a little like using a tablespoon instead of teaspoon to empty out the ocean, but like most Universe configuration parameters it was easy to try — just update the uvconfig text file on the server and restart the Universe service. Result: response time dropped from 12 minutes to 8. Whoa! If an 8k buffer made things that much better, how about a 16K buffer? Oddly, this had no effect on the response time. Maybe if we get out the big guns and and bump it up to a “whopping” 128K? No effect. I began to get suspicious and changed the setting back to its original 4K — response time remained at 8 minutes. I ran the full application rather than just my benchmarking code and retrieved the same set of data — response time was still 8 minutes. Result: no effect, despite initial appearances.

2. Restarting Universe. As you probably figured out, this gets credit for dropping response time from 12 minutes to 8. Of course, this isn’t a practical optimization technique in a production environment — I can’t have my code restart the database before each large query. Result: 33% improvement, but not a practical technique.

3. Change the UVTSORT parameter. This is another server-side configuration parameter — changing it from 0 (the default) to 1 turns on multithreading support for sorting data. This seemed promising, since my query included an “ORDER BY” clause and I had noticed that only 1 of our server’s 4 CPUs was being used for most of the time the query was being processed. But turning on this support had no apparent effect on either response time or CPU usage for this query. Result: no effect.

4. Remove the ORDER BY clause. The UVTSORT parameter did get me thinking about the ORDER BY clause in the SELECT statement. It was originally put in there to improve performance — when processing the results of the query, the code could reduce the number of lookups it had to do if the data was returned in a certain order. But in the course of processing a query only a dozen or so such lookups were required, even for a 29,000+ row resultset. Their cumulative impact on response time would be a couple of seconds at most, and how much processing time was that ORDER BY clause costing me? The answer, it turns out, is “a lot”. Result: 33% improvement.

5. Run the SELECT statement as a Universe paragraph. This idea came from Universe support, and was intended to see if the bottleneck was in one of the API layers between the .Net code and the database. By packaging the SELECT statement as a Universe package, the 740K of SQL text was replaced by the following: “CALL TEST1″, where TEST1 was the name of the paragraph. From the .Net application’s point of view, TEST1 is an ADO Command object that returns a resultset, kind of like a stored procedure:

OleDbCommand cmdTest = new OleDbCommand(“CALL TEST1″, conn);
cmdTest.CommandType = CommandType.Text;
dataAdapter = new OleDbDataAdapter(cmdTest);
dataAdapter .Fill(dataSet, “Test1″);

The 740K of SQL text is stored in the Universe master dictionary, or VOC. Getting it into the dictionary is actually easier said than done, since the usual method of editing a file is a Telnet session and Telnet wasn’t intended to handle 740K strings. Fortunately, to Universe the VOC is just another table, so I wrote a bit of UniObjects code to insert a new record in the VOC with a 740K column. Surprisingly, Universe has absolutely no problem with a 740K field — to Universe, every column is just a string, and 740K is well within its comfort zone for string lengths.

Unfortunately, while the operation was a success the patient died a bloody, inglorious death. Not only did the SELECT statement take over 10 minutes to complete, but it returned the data as strings formatted for a Telnet display, one field per row, 1.3 million rows, totaling 38M of string data. Result: don’t go there.

6. Use the Universe Basic SQLExecDirect API. This idea is taken from an IBM Technote. The .Net code is similar to that in the previous example, but now the CALL statement is invoking a Universe Basic subroutine which passes the 740K SQL statement to the SQLExecDirect API. The .Fill method returns the same rows and columns as you get if you submit the SQL statement to UniOLEDB, so it was a seamless change for the .Net application. Unfortunately, for us it was only marginally faster than UniOLEDB, reducing response time from 5:25 to 5:10. Result: 5% improvement.

So, the search for the holy grail continues. IBM’s new ADO.Net driver for Universe beckons on the horizon.

However, my faith in this crusade is wavering. I’m starting to wonder whether this 45-second goal is a reasonable expectation. It is based on a “query” done using native Universe Basic code that bears faint resemblance to a SQL query.

The apples-to-oranges comparison of Telnet to .Net is something that is never easy to reconcile: users love the extra freedom of the interface, but begrudge the extra sluggishness. It’s DOS vs Windows all over again. Maybe there’s a primal love for the command-line deep inside every user? Or maybe I need to more closely embrace the strange ways of Universe Basic, tightening the link between .Net and Universe.

To be continued.

The new design is having the desired effect, but it resulted in one new problem which looks like a bug in the UniOLEDB / .Net Provider for OLEDB stack. A tester found that “booking” (one of our application’s functions) a certain type of document without first saving it consistently resulted in a “Not all parameter markers have been resolved” error. This exception was thrown by the Fill method which ran a SELECT statement on this document’s record. The same action definitely worked fine with the previous design, where the .Net Data Provider would automatically open the database connection before running the Fill method, then close it again afterwards.

This bug was somewhat puzzling, since there had been no code changes specific to this SELECT statement, and the SELECT statement was pretty mundane: just one parameter, which was the key for the record. Running the statement in the debugger confirmed that the parameter was, indeed, being set.

When UniOLEDB errors occur for reasons that aren’t obvious from the coder’s viewpoint, it can be helpful to look at the problem from UniOLEDB’s viewpoint. This can be done by turning on the UniOLEDB trace facility, by setting the UniOLEDBTRACELEVEL and UniOLEDBTRACECATEGORY environment variables: I use settings of 4 and 4095, respectively. A DRITrace.txt file will then be written to the folder that the .Net application is running in, chock full of mostly cryptic information about UniOLEDB’s activities:

00022990 CArICommandTextImpl(73761420)::SetCommandText
00022991 CDRImCommandUCI::SetCommandText(SELECT …)
00022992 CArCommand(73761420)::Execute()
00022993 CDRImAdmin::ClearError
00022994 CArCommand(73761420)::InternalSetParameters()
00022995 CArCommand(73761420)::ParamData2DRIBuffer()
00022996 CDRImCommandUCI::Execute(SELECT …)

When I looked at the .Execute that failed,, I noticed that the trace entries which indicate a parameter (such as InternalSetParameters in the above example) were not present. Or, as some would, say the “parameter marker” hadn’t been “resolved”.

I went back and the looked at a previous parameterized SELECT statement that succeeded, to confirm that setting the parameter did result in a trace file entry. It did, but I noticed something else that I hadn’t expected: the previous SELECT statement was exactly the same as the one that had later failed. Same command text, same parameter. A check of my own log confirmed that it was the same parameter value too.

This turned out to be the root cause of the problem. The only thing that we had (intentionally) changed between versions was the point that the database connection was opened — it used to be closed after each SELECT statement, and re-opened for the next one. Now, it was left open between SELECTs. So, some piece of code in the stack (maybe the .Net Provider for OLEDB, maybe UniOleDB) wasn’t impressed that we were asking for the same thing twice, and ignored part of the 2nd request. Closing the database connection between these two requests solved the problem, and that’s the solution we went with for now.

In case you’re wondering, running the same SELECT statement twice wasn’t a bug on our part. Booking an unsaved document required that the .Net application 1) save the document 2) invoke a “Universe Basic API” (basically, a stored procedure) to fill in some missing fields in the document record 3) read the document record (the first SELECT) 4) invoke another Universe Basic API to “book” the document 5) read the newly booked document record (the second SELECT). Not the optimal way of handling this action, perhaps, but not such as unlikely series of events either.

I have a feeling that there are other parts of our .Net application which can run the same SELECT statement twice — if there aren’t, then there almost certainly will be some in the future. So, a more generalized solution is going to be required. I’m inclined to just do away with parameterized SELECT statements, embedding the parameter values into the SELECT statement instead. While this will, in theory, adversely affect both the performance and security of the application, conventional coding wisdom doesn’t necessarily apply when sailing the mostly uncharted waters of UniOLEDB development.

And, boy, what a snag. The first time we tried to login to our application on-site, U2′s UniOLEDB driver threw a “Native Error 930065″ Exception. The very first time, an error that I couldn’t remember ever seeing before. Talk about a worst case scenario!

Like many things in the U2 world this particular error code isn’t documented, nor Googleable. (Go on, try Googling “UniOleDB” and “930065″. Feel Lucky? I bet you end up back on this page). However, Google Desktop came through when my real memory failed by dredging up an old e-mail from IBM support stating (inaccurately) that a 930065 error that we hit when evaluating their U2 10.2 version meant that we were out of licenses.

This time, the “out of licenses” scenario seemed more plausible, since this client only had a few spare licenses. But the very first user? While logging in? Our application is a strictly two-tiered architecture, and given that our clients have to pay for database licenses, we are reluctant to use U2′s add-on server-based connection pooling feature. So our database connections are old school — the user desktop connects, does its stuff, and immediately disconnects to free up the license.

UniOLEDB actually co-operates fairly well with the .Net Data Provider for OLE DB. You create your OleDBConnection object, set the connection string, login to the database, then immediately close the connection. Thereafter, the connection is automatically reopened when needed, such as when you call a .Fill method to run a SELECT statement, and (so I thought) automatically closed afterwards . This arrangement looked great during our many months of internal testing. Connection management was never a concern — it just worked.

It turns out that the arrangement wasn’t working as well as I thought. We have plenty of database licenses on our test server and we never come close to reaching the limit. As a result, I never noticed a bug/feature concerning the way U2 handles the database connections. In the course of handling a particular action (e.g. logging in), we might read from a dozen tables, resulting in the database connection being opened and closed a dozen times. There is only one OleDBConnection object on the client end (a public static variable), which means there is only 1 database connection per client application. But closing that connection does not immediately cause the database license to be freed up. Depending upon the speed of the desktop PC and the load on the server, those dozen connections might result in 8-10 database licenses being briefly devoted to a single client.

Actually, much of this is conjecture on my part. Since I couldn’t find any confirmation of this behaviour in IBM’s “Using UniOLEDB” manual, I e-mailed IBM support asking “when I close a database connection is it possible that the Universe license is not freed up instantaneously?” I received a rather philosophical response that “In reality, there is no such thing as instantaneous. Everything takes a period of time, no matter how tiny a period.”

The IBM tech support person went on to say that he tried closing his test application, then entering the U2 license command into a Telnet session, and he was never able to catch the license still being used. He suggested that, when the code encounters a 930065 error, it should pause to let Universe “reclaim” the license, then try to connect again.

I wasn’t keen on the concept of adding speed bumps to the application, but his use of the word “reclaim” was enough of a hint for me that closing the connection doesn’t free up the license synchronously. (Like that word better, Oh Zen Master of IBM?) There was a period of time that would elapse before the license was freed up again, somewhere between 0 and the amount of time required for an IBM tech support guy to type “uvlictool report_lic” into a Telnet session.

While a dozen connections might chew up 12 licenses during that time, 1 connection couldn’t possibly chew up more than 1 license. So, rather than relying on the (admittedly undocumented) ability of UniOLEDB to automatically connect and disconnect as needed, the safest solution seemed to be to take control of the connections in our code. This meant a long evening of furiously adding calls to our connect and disconnect method at the start of and end of each user “action”, and a long day of testing that code, but it fixed the problem. No more 930065 errors!

What I am still seeing, though, is that the license occasionally isn’t freed up at all. The UniOLEDB object gets closed, disposed and set to null and the license is still being used until the next database connection is made from that client. This isn’t a garbage collection timing issue, but a failure to let go. With so many layers of code sitting between the user and the server, it’s hard to know which layer is failing to pass along the request to drop the license.