Transcoding operations with BizTalk Server

Many integration processes include transcoding data activities. BizTalk Server provides several ways to do this and often this can be confusing.

First of all, we begin to understand some basic things. Where to put the data? Where to implement the logic?

The following figure shows a typical integration scenario, which involves the transcoding operations.

clip_image002

The data come from an external system and include proprietary codes that need to be resolved with as many codes present on internal systems.

Then there is the BizTalk infrastructure with its services and operational databases. Optionally, you can use a custom database to support the integration flows (I do often). Finally there are the internal systems with their interfaces and different logical.

That said, where we put the data? Inside the maps? Inside custom assemblies? Inside the custom database? Do we retrieve them from internal systems?

Well, we begin to exclude what should be avoided, that is inside the maps or inside the assembly. So we evaluate whether using the custom database or calling internal systems.

From my point of view, when it is possible, it is best to delegate to the internal systems. This because internal system has staff trained to manage data and probably also the interfaces that they know. Obviously if it isn’t possible, I prefer to put them in a custom database.

Well, but now, where can I put the transcoding logic? Should I make calls inside the orchestration? Can I use the maps? You can imagine many solutions.

I usually use two approaches depending on whether the transaction involves a single instance of the message or if I have to deal with a massive batch of data and multiple transcoding operations.

When I elaborate a single entity, I like to do this transcoding within the maps using a lookup component.

Otherwise, when I’m working on a big batch of data and there are many transcoding operations, I use the custom database.

Transcoding operations inside the maps

In the event that you are working on a single instance of the message, resolve transcoding within the maps is very elegant because it is possible to handle errors, keep the process simple and clear and, at the same time, reduces the number of transformations that otherwise we should implement inside the orchestration.

The point is that whoever solves the transcoding is almost always a service (which can be a call to a database or to a web service). If the service consist of a call to a Database, you can use the Database Lookup functoid as shown in the following figure:

clip_image004

If instead it is a web service, or the logic to access the database is more complex, you can develop a custom functoid that implements the data access logic of the database or of the web service, obtaining a result as shown below.

clip_image006

For additional information about how to writing custom functoid refer the following address http://msdn.microsoft.com/en-us/library/aa560879.aspx.

Using a custom database

If the amount of data is large and the rules of processing and transcoding are numerous and complex I like to use a custom database.

The integration flow is as follows:

1. Data from external sources are loaded in a table.

2. Then I call a procedure that returns the data and that incorporates transcoding activities and complex transformation rules.

Both the operations can be made in a single transaction and using a single connection taking advantage of a composite operation. For additional information about composite operation, refer to the following address http://msdn.microsoft.com/en-us/library/dd788136.aspx.

Advertisements

4 thoughts on “Transcoding operations with BizTalk Server

  1. do you really query the database for every single transcoding??? I can’t believe it, please tell me that you use a cache mechanism

    • We distinguish the two scenarios.

      In a first case, I receive a batch that contains a high number of messages (for example 50000). The first thing that I do is to put the whole batch in a database (typically in a single composite operation).

      Then, still in a batch logic (thus performing a call for each rule), I call the systems owner of transcoding and persist the result in the database.

      At this point in the database I have data that I received and the data that are used to do the transcoding. So I do a last call (typically by calling a stored procedure) that returns transcoded data in BizTalk.

      In the second scenario, the number of messages is limited so I can perform transcoding within the map using the lookup functoid.

      In both scenarios, if the functional rules allow it, it is possible to implement cache systems. In the first case by implementing asynchronous synchronization between the database and the owner of the transcoding systems. In second case using a .Net cache component.

      I used caching techniques, but I did it only when it was necessary, I do not always. In general I do not like to solve performance problems where there are not. I prefer the simplicity of implementation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s