The new Bin2DataStreamer Component

The new Bin2DataStreamer is provided as an alternative to the existing BinDataStreamer, which, of course, is still supported for backward compatibility.

Data Streamer components are responsible for encoding and decoding data packets that are transmitted between server and client applications over the network. As such it is a very core part of the Data Abstract architecture.

Why Produce an Alternative to BinDataStreamer?

When BinDataStreamer was written for the first version of Data Abstract, data transfer requirements were far different from those needed these days. Although it still provides fairly efficient data transfer, particularly for small loads, it had become increasing obvious that we needed a replacement because:

• Over three major versions of the product, the requirement for full backward compatibility made it increasingly difficult to extend and improve.
• BinDataStreamer was based on streaming functionality provided by Delphi's TReader/TWriter classes, which were not really designed for the type of data transfer we need.
• For other platforms, quirks in Delphi's TReader/TWriter classes had to be reproduced on .NET and FPC for backwards compatibility.
• Several of the planned features for 'Vinci' required a much tighter integration with the data streamer, which was just not possible with the existing component.

Bin2DataStreamer Benefits

The main benefits that you, as the user of Data Abstract, will see from the new streaming formats are: speed, speed and more speed! Except for the different component name, you will see no logical or conceptual changes in how the streamers are used or configured.

Under the hood there are three core features that the Bin2 streamer benefits from:

• New streaming format: better defined and much more efficient, especially for large data transfer.
• Reduction of network traffic: SendReducedDelta mode, pre-validation of the stream structure and non-transmission of null values.
• Integration with TDAMemTable (Delphi edition): for even better performance, TDABin2DataStreamer and TDAMemTable (More) have been written in tandem to be optimized and directly benefit from each other.

New Streaming Format

Bin2DataStreamer provides a completely new well-defined data format (see the extracts below). Each individual data field is stored/read independently. This allows us to parse data in a more efficient and flexible manner. Using custom streaming methods to replace TReader/TWriter provides much better compatibility between the Delphi, .NET and future editions of Data Abstract.

Note for Delphi users: Bin2DataStreamer thus avoids the deadlocks which can occur in the FormDestroy event, caused by TReader/TWriter using global critical sections while reading/writing.

The new streaming format provides efficient validation of the stream structure before it is processed, in order to avoid data corruption. All data in the stream can be validated before it is written to the actual table in the database. Data may be checked by field names, types and size.

Reduction of network traffic

Apart from simplifying data validation already covered in the previous section and general improvements to data encoding that save network traffic, there are two items to discuss here: SendReducedDelta mode and non-transmission of null values.

If SendReducedDelta mode is turned on, this can significantly reduce packet size when applying updates, because only key fields and changed fields are added to the delta and sent to the server. If turned off, Bin2DataStreamer streams all fields, similar to the old BinDataStreamer.

Note: In certain circumstances, there is a small downside to turning the feature on: with the fields being transmitted changing row by row, unique SQL needs to be generated for each record, so the process of applying many record changes in one go will not benefit from having a single SQL statement "prepared" and reused. However, this effect is certainly minor for all but large bulk updates.

non-transmission of nulls values is achieved by analyzing the data row's fields to build a single bit mask, which holds a reference to all null fields. This is transmitted instead of individual nulls and can have a significant effect on the streaming size for tables containing many null values.

Integration with TDAMemTable (Delphi edition)

Bin2DataStreamer is tightly integrated with the new TDAMemDatatable and TDAMemDataset components. Actually, Bin2DataStreamer works with a TDAMemoryDataTable in 'native mode', where the best performance can be achieved. For example, a TDAMemDataTable can fill itself from a received Bin2 stream without having to parse and copy the entire data stream - making opening data tables on the client very efficient.

How Objects are Streamed

In conclusion, the following extracts show the format of the stream produced by Bin2DataStreamer. Note that while the extracts shown below use an XML-structure view for best readability, data streaming in Bin2 format uses binary encoding.

DataTable:

DataTable Schema:

Delta:

A Delta packet contains local changes to be updated on the server.