BizTalk CRM Lookup functoid is now ready for CRM Online

In recent years I often integrate Microsoft Dynamics CRM by using BizTalk Server, and as many know, BizTalk Server does not have a native connector for MS CRM.

For this reason, we must work hard to generate canonical schemas, to resolve lookup type fields, Guid type fields and generally to develop maps.

To increase the productivity of my team I have always tried to develop tools or helpers to automate the development of long and repetitive tasks, reducing the possibility of generating errors and standardize approaches to problems.

In recent months, with the cooperation of Nino Crudele and Sandro Pereira we published several functoids that simplify integration with Dynamics CRM and in this article I want to describe the work that has been done to also integrate the CRM Online.

When you integrate MS CRM using BizTalk Server, typically you have at least two problems:

  • Transcode the Lookup and Guid type fields
  • Mapping the input information with the generic schema exposed by CRM

The BizTalk Mapper Extension Utility pack simplifies both tasks by providing specialized functoids.

Today we released a new version where you can find the CRM Lookup functoid with two new features:

  • The ability to define more than one configuration in the SSO. This allows of integrating more than one CRM instance with a single BizTalk Farm
  • Now you can integrate CRM On-Line. CRM Lookup, for performance reasons, performs lookups operation by accessing the CRM database in read only. However, the CRM Online does not expose his database, so we added the ability to do lookups using the native CRM web services that are present in both the online and on-premises versions.

The functoid now takes five mandatory input parameters:

  1. SSO Application Name: the connection parameters are stored inside SSO. Here you have to specify the application name where you decide to store the settings. This allow you to have multiple CRM environments that you can integrate;
  2. CRM Guid Field Name: key field name of which you want to get the value;
  3. CRM Entity Name: CRM entity name on which the resolution will be performed
  4. CRM Input Field Name: imagining having a “code” as input and you want to know the CRM physical key, in this attribute should specify the name of the “code” field.
  5. CRM Input Value: value from source system to resolve


The output of the functoid is a string, Example: 6erg5r-sdrf56-4dfgt5-6ty5r4-456trt 24

Installation

Download and install BizTalk Mapper Extensions UtilityPack from https://btsmapextutilitypack.codeplex.com/.

Download CRM Sdk and register in GAC the following assemblies:

  • Microsoft.Xrm.Sdk.dll (version 5.0.9910.15)

Create a SSO application by using SSO Configuration Application MMC Snap-In available at https://www.microsoft.com/en-us/download/details.aspx?id=14524.

SQL Server Lookup

Create the following SSO configuration parameters

  • CrmConnectionType: “sql”
  • CrmDatabaseConnectionString: specify the CRM database connection string

Open SQL Server management console, connect to CRM database and execute the script under CRMLookup.sql file in project folder.

Local CRM web services lookup

Create the following SSO configuration parameters

CRM Online web services lookup

Create the following SSO configuration parameters

Advertisements

Manage your BizTalk Passwords

During the deployment of BizTalk applications, it is sometimes necessary to set passwords manually. There are other options such as those to prepare a binding file that contains the passwords, but it is not always a quick and easy job.

Then I created a console application that easily sets the password for the send ports and receive locations so that it can create automatic deployment script.

You can get this tool at https://github.com/pellitteris/BizTalkPassword site.

It is just an example that currently manages the adapter FILE, FTP, SFTP, POP3 and WCF *. Of course you can access the source code in order to manage other kind of adapters.

This console is very simple to use, simply specify the port type (if a send port or a receive location), a user and a password.

Following an example of use:

Microsys.EAI.Framework.PasswordManager.exe -set -receive -name:MyReceiveLocation -user:John -password:@Passw0rd1

Microsys.EAI.Framework.PasswordManager.exe -set -send -name:MySendPort -user:John -password:@Passw0rd1

In addition to setting the password, it allows you to view the property of the ports that normally contain credentials (TransportTypeData).

Following are other examples of use:

Microsys.EAI.Framework.PasswordManager.exe -list -application:[application name]

Microsys.EAI.Framework.PasswordManager.exe -get -receive -name:[receive location name]

Microsys.EAI.Framework.PasswordManager.exe -get -send -name:[send port name]

Microsys.EAI.Framework.PasswordManager.exe -set -receive -name:[receive location name] -user:[username] -password:[password]

Microsys.EAI.Framework.PasswordManager.exe -set -send -name:[send port name] -user:[username] -password:[password]

Composite Operations vs User-Defined Table Type from BizTalk Server

Normally, when I integrate SQL Server with BizTalk server, I prefer to use stored procedures. This because I believe that a database within an integration scenario should not expose data models but should be a service that exposes methods.

When you need to insert data sets into a SQL Server through a stored procedure, you can use a composite operation or develop a stored procedure that accept a user-defined table type as a parameter.

For more information about the use of a composite operation, refer to the article “https://pellitterisbiztalkblog.wordpress.com/2013/07/01/insert-master-detail-data-structure-on-sql-server-in-a-single-transaction-using-biztalk-server”.

Develop a stored procedure that accepts a user-defined table type as a parameter is very simple. Simply create a user-defined table type as shown below.

CREATE TYPE dbo.Lines AS TABLE
(
    lineId INT NOT NULL,
    lineField1 VARCHAR(10) NULL,
    lineField2 VARCHAR(10) NULL,
    PRIMARY KEY (lineId)
);

Then create a stored procedure that uses the type previously created.

CREATE PROCEDURE dbo.BizTalkSample
    @headerId INT,
    @headerField1 VARCHAR(10),
@headerField2 VARCHAR(10),
@lines Lines READONLY
AS
BEGIN
    INSERT INTO sampleHeader
   (
        headerId,
        headerField1,
        headerField2
   )
   VALUES
   (
       @headerId,
       @headerField1,
       @headerField2
   );
    INSERT INTO sampleLines
   (
       lineId,
       headerId,
       lineField1,
       lineField2
   )
   SELECT
       lineId,
       @headerId,
       lineField1,
       lineField2
   FROM
       @lines;
END

At this point, it is sufficient to generate a schema from a BizTalk project.

There are different contexts in which it is preferable to use a method or the other. The main difference is that a composite operation invokes the stored procedure several times while a stored procedure that accepts a user-defined table type as parameter is called only once.

If the need is to send a dataset to SQL Server, the second method can introduce significant advantages in terms of performance. The BizTalk message is a few percent smaller, but the main point is that the number of communications between BizTalk Server and SQL Server is drastically reduced and SQL Server performs the transaction much faster.

I conducted several tests using a 100MB file which contained about 8 million lines. The improvement was amazing. The time has decreased significantly from about 20 minutes using a composite operation in just over three minutes by running a single stored procedure that accept a user-defined table type as a parameter.

BizTalk Business Activity Monitoring “remove-view” issue

When you remove an existing BizTalk Business Activity Monitoring (BAM) view using BM.exe command, you could get the following error message: “XML parsing failed at line 1, column 0: A document must contain exactly one root element”.

This occurs only if the view contains Analysis Services cube objects such as measures and dimensions.

When you deploy a BAM view that contains OLAP objects, BM.exe generates an Integration Services package that is able to process data contained in the database BAMPrimaryImport to update the Analysis Services cube.

The package, however, is not scheduled therefore you must configure a SQL Server job that periodically process BAM activities data.

If this is not done and the process has never been performed, you could have the problem.

To resolve it, you have to manually process the cube and re-execute the remove-view command.

To manually process the cube, open Microsoft SQL Server Management Studio. When “Connect to Server” window appears, select “Analysis Services” as Server type and specify the Analysis Services server name.

Then expand the “Databases” node and choose the BAM OLAP database (in my case BAMAnalysis). Then right-click on the database and select “Process”

Leave all default settings and click on Ok.

Once Process is complete, re-execute the BM.exe command.

How to generate a BizTalk schema for a SAP Web Service

 

I have several clients who own a SAP system, so I often develop integration flows that invoke BAPI exposed as Web Service. However, every time, I lose a lot of time in verifying the SAP configuration that allowed me to download the correct WSDL and generate a XSD schema. So, I decided to document every step, from the access to SAP to check the correct configuration up to the necessary actions on BizTalk side.

 

Configure the SAP logon settings

 

Start the SAP GUI program

 

 

Add new connection entry

 

 

Select “User Specified System”

 

 

There are two ways to access the SAP system, in this document I choose to connect directly to the application server.

Then, specify a connection Description, Application server name, Instance number and System ID.

 

 

Accept default settings and click Next

 

 

Select your language

 

 

At the end you have the configuration ready to use.

 

 

Access to SAP

To access SAP, use an existing connection or one configured previously specifying user and password

 

 

You should get an interface similar to that shown below

 

 

So specify “soamanager” in the command field

 

 

The SOA Management web interface appears. Click on “Web Service Configuration”

 

 

Search the BAPI you want to generate a WSDL

 

 

Select the BAPI and click on “Apply Selection”

 

 

You must be sure that the “WSDL format” section is set to “Standard”. Unlike the wizard of visual studio will not be able to generate correctly the XSD schema and the bindings file.

 

 

Click on “Open WSDL document for selected binding or service”

 

 

The Internet browser appears with the WSDL. Save the file to a folder.

 

 

 

Open the BizTalk Visual Studio project and Add a generated items.

 

 

Select “Consume WCF Service”

 

 

Select “Metadata Files (WSDL and XSD)”

 

 

In the Metadata Files form, add the WSDL file that you had previously saved and then click Next

 

 

Specify a Namespace and click on Import

 

 

That’s all. You have your SAP web services schema ready to use.

 

BizTalk Configuration Reports

Documenting the integration flows is an activity very expensive in terms of time. If we think to the configuration of the receive ports, send ports and how they are linked to the orchestrations we become crazy.

In a context where they develop many BizTalk objects you end up having documents constantly obsolete.

For this reason, I decided to develop a small package of reports that allow having this part of the documentation always up to date. Simply run reports and export them in the required format such as Microsoft Word, PDF or Microsoft Excel.

You can download the installation package at https://github.com/pellitteris/BizTalkConfigReports.

The solution includes a package of reports developed with Microsoft SQL Server Reporting Services and a database to be installed on the same SQL Server instance where the database BizTalkMgmtDb resides. This database contains the views and stored procedures performed by the reports.

Once the installation is completed you have three reports:

 

  • Orchestration Summary
  • Orchestrations
  • Receive Port with Port Subscription

 

 

Orchestration Summary

The report “Summary Orchestration” displays at a high level the relationship between the orchestrations, the receive ports and the send ports.

Each report allows you to select the application, and optionally a search string to filter the objects you want to examine or document.

 

 

Following an example of this report.

 

 

Orchestrations

The report “Orchestrations” adds more details to the previous report.

 

 

Finally the report “Receive Port having Send Send Port or Port Group Subscriptions” that displays all subscriptions to a specific port configured on the send port or on the send port group.

 

 

Now you have the source code of this solution and you can add features or simply customize reports to fit your needs.

Integration Monday – High available BizTalk infrastructure on Azure IaaS

I’m really pleased to announce that on 4 May Massimo Crippa and me will present my work about BizTalk HA on Microsoft Azure to the integration Monday (http://www.eventbrite.com/e/integration-monday-2015-05-04-massimo-salvatore-tickets-16540234286?aff=estw).

I am very honored that Michael Stephenson gave me this opportunity.

I hope to meet his expectations.

It is a pleasure to work with Massimo who was a colleague of mine at Microsys for 10 years, and now is doing a great experience of working and living at Codit.