Upload Flat File on Azure SQL Database using Azure Logic App

In this article, I’m going to describe how to upload on-premises flat files on Azure SQL Database using Azure Logic Apps.

Logic Apps provide integration and workflows capabilities in the cloud. It provides a visual designer to model and automated processes as a series of steps.

In this sample, I also use Azure Service Bus as an entry point to the cloud because the file size is less than 1024 KB. Consider to use Azure Storage if you plan to upload file greater than that size.

Initial Consideration

Before you start, consider installing Microsoft Azure Logic Apps Enterprise Integration Tools for Visual Studio 2015 (https://www.microsoft.com/en-us/download/details.aspx?id=53016) in order to facilitate the creation of schemas and maps.

In addition, you can install Azure Logic Apps Tools for Visual Studio (https://visualstudiogallery.msdn.microsoft.com/e25ad307-46cf-412e-8ba5-5b555d53d2d9) to develop Logic App integration using visual studio.

At this link (https://azure.microsoft.com/en-us/tools/), you can find several tools for the development of solutions based on Microsoft Azure.

The following picture shows the data flow that I am going to describe.

clip_image002

Following the check list

  • Create Resource Group
  • Create Service Bus Namespace
  • Create Service Bus Queue
  • Create SQL Server
  • Create SQL Database
  • Create SQL Objects (Table, Schema and Procedures)
  • Configure API Connection to Service Bus
  • Configure API Connection to SQL Database
  • Create an Integration Account
  • Upload Schemas and Maps on Integration Account
  • Configure Transform Function (this step should be unnecessary in the next future)
  • Create and Test Logic App integration flow

To complete each step, you must first access the Microsoft Azure portal (https://portal.azure.com).

I think it is unnecessary to describe each step in detail, so I will just provide some screen shots.

Create Resource Group

clip_image004

Create Service Bus Namespace

clip_image006

Create Service Bus Queue

clip_image008

Create SQL Server (Logical Server)

clip_image010

Connect to SQL Server and create database objects

Before you can connect to SQL Database virtual server, you have to configure Azure firewall to allow the communication.

clip_image012

Use SQL Server Management Studio to create database and tables

clip_image014

Create database
   1: /*

   2: For additional information refer to address https://msdn.microsoft.com/en-us/library/dn268335.aspx

   3: */

   4: CREATE DATABASE [int-fftosql-sqldb]

   5: (

   6:     EDITION = 'basic',

   7:     SERVICE_OBJECTIVE='basic'

   8: );

   9:

  10: GO

Create table and procedures

 

   1: CREATE XML SCHEMA COLLECTION [dbo].[OrderXml] AS

   2: N'<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

   3:   <xsd:element name="Root">

   4:     <xsd:complexType mixed="true">

   5:       <xsd:complexContent mixed="true">

   6:         <xsd:restriction base="xsd:anyType">

   7:           <xsd:sequence>

   8:             <xsd:element name="Order" maxOccurs="unbounded">

   9:               <xsd:complexType mixed="true">

  10:                 <xsd:complexContent mixed="true">

  11:                   <xsd:restriction base="xsd:anyType">

  12:                     <xsd:sequence>

  13:                       <xsd:element name="Lines" type="OrderDetailType" maxOccurs="unbounded" />

  14:                     </xsd:sequence>

  15:                     <xsd:attribute name="orderDate" type="xsd:string" use="required" />

  16:                     <xsd:attribute name="orderNumber" type="xsd:string" use="required" />

  17:                     <xsd:attribute name="channel" type="xsd:string" use="required" />

  18:                   </xsd:restriction>

  19:                 </xsd:complexContent>

  20:               </xsd:complexType>

  21:             </xsd:element>

  22:           </xsd:sequence>

  23:         </xsd:restriction>

  24:       </xsd:complexContent>

  25:     </xsd:complexType>

  26:   </xsd:element>

  27:   <xsd:complexType name="OrderDetailType" mixed="true">

  28:     <xsd:complexContent mixed="true">

  29:       <xsd:restriction base="xsd:anyType">

  30:         <xsd:choice maxOccurs="unbounded">

  31:           <xsd:element name="productName" type="xsd:string" />

  32:           <xsd:element name="quantity" type="xsd:string" />

  33:           <xsd:element name="value" type="xsd:string" />

  34:         </xsd:choice>

  35:       </xsd:restriction>

  36:     </xsd:complexContent>

  37:   </xsd:complexType>

  38: </xsd:schema>'

  39: GO

  40:

  41: CREATE TABLE [dbo].[OrderHeader]

  42: (

  43:     [orderId] [uniqueidentifier] NOT NULL,

  44:     [orderNumber] [varchar](20) NOT NULL,

  45:     [orderDate] [datetime] NOT NULL,

  46:     [channel] [varchar](50) NULL,

  47:     CONSTRAINT [PK_OrderHeader] PRIMARY KEY CLUSTERED

  48:     (

  49:         [orderId] ASC

  50:     )

  51: )

  52: GO

  53:

  54: CREATE TABLE [dbo].[OrderLine]

  55: (

  56:     [orderLineId] [int] IDENTITY(1,1) NOT NULL,

  57:     [orderId] [uniqueidentifier] NOT NULL,

  58:     [productName] [varchar](100) NOT NULL,

  59:     [quantity] [int] NOT NULL,

  60:     [value] [numeric](18, 3) NOT NULL,

  61:     CONSTRAINT [PK_OrderLine] PRIMARY KEY CLUSTERED

  62:     (

  63:         [orderLineId] ASC

  64:     )

  65: )

  66: GO

  67:

  68: /*

  69: TEST STATEMENT:

  70: -------------------------------

  71: EXECUTE OrderInsert N'<Root>

  72:   <Order orderDate="20160101" orderNumber="000001" channel="Internet">

  73:     <Lines>

  74:       <productName>Portable Radio</productName>

  75:       <quantity>10</quantity>

  76:       <value>23.5</value>

  77:     </Lines>

  78:     <Lines>

  79:       <productName>6 red wine bottles</productName>

  80:       <quantity>20</quantity>

  81:       <value>30.9</value>

  82:     </Lines>  

  83:   </Order>

  84: </Root>'

  85: -------------------------------

  86: */

  87: CREATE PROCEDURE [dbo].[OrderInsert]

  88:     @orderStream VARCHAR(MAX)

  89: AS

  90: BEGIN

  91:

  92:     -- Fix Incoming XML: Logic App add a "?" char at the beginning of the string

  93:     SET @orderStream = REPLACE(@orderStream, '?<', '<');

  94:

  95:     DECLARE @orderXml XML(OrderXml) = CAST(@orderStream AS XML(OrderXml));

  96:     DECLARE @orderId UNIQUEIDENTIFIER = NEWID();

  97:

  98:     INSERT INTO OrderHeader

  99:     (

 100:         orderId,

 101:         orderNumber,

 102:         orderDate,

 103:         channel

 104:     )

 105:     SELECT

 106:         @orderId,

 107:         orderNumber = OrderObject.Header.value('./@orderNumber', 'varchar(20)'),

 108:         orderDate = OrderObject.Header.value('./@orderDate', 'datetime'),

 109:         channel = OrderObject.Header.value('./@channel', 'varchar(50)')

 110:     FROM

 111:         @orderXml.nodes('/Root/Order') AS OrderObject(Header);

 112:

 113:     INSERT INTO OrderLine

 114:     (

 115:         orderId,

 116:         productName,

 117:         quantity,

 118:         value

 119:     )

 120:     SELECT

 121:         @orderId,

 122:         productName = OrderObject.Lines.value('./productName[1]', 'varchar(100)'),

 123:         quantity = OrderObject.Lines.value('./quantity[1]', 'int'),

 124:         value = OrderObject.Lines.value('./value[1]', 'numeric(18,3)')

 125:     FROM

 126:         @orderXml.nodes('/Root/Order/Lines') AS OrderObject(Lines);

 127:

 128: END

 129: GO

You should see something like that.

clip_image002[4]

Create API Connection to Service Bus queue

Before to configure the connection, you need to know the service bus connection string.

clip_image004[4]

clip_image006[4]

Create API Connection to SQL Server

clip_image008[4]

Create Integration Account

Integration accounts allow you to manage artifacts such as schemas, maps, certificates, partners and agreements in order to build Integration Application cloud based apps.

For additional information, refer to https://azure.microsoft.com/en-us/documentation/articles/app-service-logic-enterprise-integration-create-integration-account/.

clip_image010[4]

Configure Integration Account

clip_image012[4]

Add source schema (PurchaseOrderFF.xsd)

clip_image014[4]

   1: <?xml version="1.0" encoding="utf-16"?>

   2: <xs:schema xmlns="http://int1demoa1project.Order" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://int1demoa1project.Order" xmlns:xs="http://www.w3.org/2001/XMLSchema">

   3:   <xs:annotation>

   4:     <xs:appinfo>

   5:       <schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.Azure.Integration.DesignTools.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />

   6:       <b:schemaInfo standard="Flat File" codepage="1250" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" root_reference="Order" />

   7:     </xs:appinfo>

   8:   </xs:annotation>

   9:   <xs:element name="Order">

  10:     <xs:annotation>

  11:       <xs:appinfo>

  12:         <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="postfix" />

  13:       </xs:appinfo>

  14:     </xs:annotation>

  15:     <xs:complexType>

  16:       <xs:sequence>

  17:         <xs:element name="Header">

  18:           <xs:annotation>

  19:             <xs:appinfo>

  20:               <b:recordInfo tag_name="H0" structure="delimited" child_delimiter_type="char" child_delimiter=";" child_order="prefix" />

  21:             </xs:appinfo>

  22:           </xs:annotation>

  23:           <xs:complexType>

  24:             <xs:sequence>

  25:               <xs:element name="orderDate" type="xs:string">

  26:                 <xs:annotation>

  27:                   <xs:appinfo>

  28:                     <b:fieldInfo justification="left" />

  29:                   </xs:appinfo>

  30:                 </xs:annotation>

  31:               </xs:element>

  32:               <xs:element name="orderNumber" type="xs:string">

  33:                 <xs:annotation>

  34:                   <xs:appinfo>

  35:                     <b:fieldInfo justification="left" />

  36:                   </xs:appinfo>

  37:                 </xs:annotation>

  38:               </xs:element>

  39:               <xs:element name="channel" type="xs:string">

  40:                 <xs:annotation>

  41:                   <xs:appinfo>

  42:                     <b:fieldInfo justification="left" />

  43:                   </xs:appinfo>

  44:                 </xs:annotation>

  45:               </xs:element>

  46:             </xs:sequence>

  47:           </xs:complexType>

  48:         </xs:element>

  49:         <xs:element maxOccurs="unbounded" name="Detail">

  50:           <xs:annotation>

  51:             <xs:appinfo>

  52:               <b:recordInfo tag_name="D0" structure="delimited" child_delimiter_type="char" child_delimiter=";" child_order="prefix" />

  53:             </xs:appinfo>

  54:           </xs:annotation>

  55:           <xs:complexType>

  56:             <xs:sequence>

  57:               <xs:element name="productName" type="xs:string">

  58:                 <xs:annotation>

  59:                   <xs:appinfo>

  60:                     <b:fieldInfo justification="left" />

  61:                   </xs:appinfo>

  62:                 </xs:annotation>

  63:               </xs:element>

  64:               <xs:element name="quantity" type="xs:string">

  65:                 <xs:annotation>

  66:                   <xs:appinfo>

  67:                     <b:fieldInfo justification="left" />

  68:                   </xs:appinfo>

  69:                 </xs:annotation>

  70:               </xs:element>

  71:               <xs:element name="value" type="xs:string">

  72:                 <xs:annotation>

  73:                   <xs:appinfo>

  74:                     <b:fieldInfo justification="left" />

  75:                   </xs:appinfo>

  76:                 </xs:annotation>

  77:               </xs:element>

  78:             </xs:sequence>

  79:           </xs:complexType>

  80:         </xs:element>

  81:       </xs:sequence>

  82:     </xs:complexType>

  83:   </xs:element>

  84: </xs:schema>

Add SQL Server stored procedure schema (PurchaseOrderDB.xsd)

clip_image002[6]

   1: <?xml version="1.0" encoding="utf-16"?>

   2: <xsd:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003" attributeFormDefault="unqualified" elementFormDefault="unqualified" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

   3:   <xsd:complexType name="OrderDetailType" mixed="true">

   4:     <xsd:choice minOccurs="1" maxOccurs="unbounded">

   5:       <xsd:element minOccurs="1" maxOccurs="1" name="productName" type="xsd:string" />

   6:       <xsd:element minOccurs="1" maxOccurs="1" name="quantity" type="xsd:string" />

   7:       <xsd:element minOccurs="1" maxOccurs="1" name="value" type="xsd:string" />

   8:     </xsd:choice>

   9:   </xsd:complexType>

  10:   <xsd:element name="Root">

  11:     <xsd:complexType mixed="true">

  12:       <xsd:sequence>

  13:         <xsd:element minOccurs="1" maxOccurs="unbounded" name="Order">

  14:           <xsd:complexType mixed="true">

  15:             <xsd:sequence>

  16:               <xsd:element minOccurs="1" maxOccurs="unbounded" name="Lines" type="OrderDetailType" />

  17:             </xsd:sequence>

  18:             <xsd:attribute name="orderDate" type="xsd:string" use="required" />

  19:             <xsd:attribute name="orderNumber" type="xsd:string" use="required" />

  20:             <xsd:attribute name="channel" type="xsd:string" use="required" />

  21:           </xsd:complexType>

  22:         </xsd:element>

  23:       </xsd:sequence>

  24:     </xsd:complexType>

  25:   </xsd:element>

  26: </xsd:schema>

Add XML Transformation Map

clip_image002[8]

clip_image004[6]

   1: <?xml version="1.0" encoding="utf-16"?>

   2: <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var s0" version="1.0" xmlns:s0="http://int1demoa1project.Order">

   3:   <xsl:output omit-xml-declaration="yes" encoding="windows-1250" version="1.0" indent="no" method="xml" />

   4:   <xsl:template match="/">

   5:     <xsl:apply-templates select="/s0:Order" />

   6:   </xsl:template>

   7:   <xsl:template match="/s0:Order">

   8:     <Root>

   9:       <Order>

  10:         <xsl:attribute name="orderDate">

  11:           <xsl:value-of select="Header/orderDate/text()" />

  12:         </xsl:attribute>

  13:         <xsl:attribute name="orderNumber">

  14:           <xsl:value-of select="Header/orderNumber/text()" />

  15:         </xsl:attribute>

  16:         <xsl:attribute name="channel">

  17:           <xsl:value-of select="Header/channel/text()" />

  18:         </xsl:attribute>

  19:         <xsl:for-each select="Detail">

  20:           <Lines>

  21:             <productName>

  22:               <xsl:value-of select="productName/text()" />

  23:             </productName>

  24:             <quantity>

  25:               <xsl:value-of select="quantity/text()" />

  26:             </quantity>

  27:             <value>

  28:               <xsl:value-of select="value/text()" />

  29:             </value>

  30:             <xsl:value-of select="./text()" />

  31:           </Lines>

  32:         </xsl:for-each>

  33:       </Order>

  34:     </Root>

  35:   </xsl:template>

  36: </xsl:stylesheet>

 

Temporary operations

Create Storage Account

clip_image002[10]

Create App Service Plan

clip_image004[8]

Create Function App

clip_image006[6]

Add the transformation function

To add the transformation Function for Logic App run the following link

https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F201-logic-app-transform-function%2Fazuredeploy.json

clip_image008[6]

Create Logic App

clip_image010[6]

Associate Integration Account to the Logic App

clip_image012[6]

Create a blank solution

clip_image014[6]

Configure trigger

clip_image016

clip_image018

Modify the polling frequency if you need to process the file in near real time.

clip_image020

Add Flat-File decoding to parse flat file and fill into XSD schema message.

clip_image022

clip_image024

Add the map to transform flat file message to SQL Database XML schema type.

clip_image026

Configure the call to the stored procedure.

clip_image028

clip_image030

Write and run client application

Now you able to run the client application. To create a sample application, follow the steps below.

1 – Install WindowsAzure.ServiceBus NuGet Package

clip_image032

2 – Get service bus Connection String

clip_image034

3 – Write your code

   1:

   2: using Microsoft.ServiceBus.Messaging;

   3:

   4: ...

   5: using (FileStream file = File.Open(<FilePath>, FileMode.Open, FileAccess.Read, FileShare.Delete))

   6: {

   7:

   8:   // Send

   9:   file.Seek(0, SeekOrigin.Begin);

  10:

  11:   var queueClient = QueueClient.CreateFromConnectionString(<ConnectionString>, <QueueName>);

  12:   BrokeredMessage serviceBusMessage;

  13:

  14:   serviceBusMessage = new BrokeredMessage(file);

  15:

  16:   queueClient.Send(serviceBusMessage);

  17:

  18: }

Advertisements

Integrate Azure SQL Database and OneDrive with Logic Apps

This sample shows how to read information from an Azure SQL Database and send them to OneDrive folder using a simple Logic Apps.

I will use a polling statement in order to intercept new data available on SQL Database, transform to a new information schema and then write a XML formatted file in an OneDrive folder.

Following I will demonstrate how to setup the sample environment, how to configure the connection to Azure SQL Database and then how to create the integration flow using Azure Logic Apps.

Setup Environment

First of all, you need to create a resource group that will contains the SQL Database instance and all the other items needed for Logic Apps.

To create a resource group, navigate the new azure portal, click on Resource groups and click on Add.

Specify a Resource group name and then click on Create.

Now you are able to create a SQL Database instance. To do that select SQL Databases from the portal and select Add.

You can choose to add the database to an existing server or create a new server. In this demonstration, I have created a new server.

Once you have specified the server name, the user name and password you will use to connect to the database, the resource group previously created and the database name click on the create button to complete the operation.

Now you need to open the Azure firewall in order to connect to the database using Microsoft SQL Server Management Studio. So click on server name link.

Then click on Show firewall settings link.

The interface will show you the current IP address of your machine. So click on Ass client IP.

Save the setting.

After a few seconds, the firewall will be updated.

Now you are ready to connect to the SQL database instance. Open SQL Server Management Studio console and connect to the database using the SQL Server Authentication. Provide the credential you have chosen in the previous step.

Now you are connected to the Azure database.

Now, we have to create the table that contains the information that the Logic Apps will intercepts and will sent to OneDrive.

Click on new query and run the following statement.

— Source Table

CREATE
TABLE [dbo].[SourceTable]

(

    [sourceTableId] [int] NOT
NULL,

    [transactionStatus] [varchar](10)
NOT
NULL,

    [transactionId] [uniqueidentifier] NULL,

    [sampleContent] [varchar](100)
NOT
NULL,

    CONSTRAINT [PK_SourceTable] PRIMARY
KEY
CLUSTERED

    (

    [sourceTableId] ASC

    )

)

GO

— Available Statement

CREATE
PROCEDURE [dbo].[AvailableStmt]

AS

BEGIN

    SELECT

        COUNT(*)

    FROM

        SourceTable

    WHERE

        transactionStatus =
‘To Process’;

END

GO

— Polling Statement

CREATE
PROCEDURE [dbo].[PollingStmt]

AS

BEGIN

    DECLARE @transactionId UNIQUEIDENTIFIER
=
NEWID();

    UPDATE SourceTable

    SET

        transactionStatus =
‘Processed’,

        transactionId = @transactionId

    WHERE

        transactionStatus =
‘To Process’;


    SELECT

        sourceTableId,

        sampleContent

    FROM

        SourceTable WITH (NOLOCK)

    WHERE

        transactionId = @transactionId;

END

GO

Create the Microsoft SQL Connector

Once you have created all you need on SQL Server, you can define the database connection. From the portal home page click on Marketplace.

Find for Microsoft SQL Connector.

Select the connector and click on Create.

Specify a Name, a Logic App Plan (or create a new one) a resource group and then click on Create.

Now you have to specify the server name, the user name and password, the database name.

Scrolling down you have to specify the most important things: The Available Statement and the Polling Statement. The first one is the statement that checks for new information on the database. The second one, gets the information and make them available for the process.

Create the Artifact

In this demonstration, we will transform the information coming from the SQL database to a different information scheme. To do this we will need a map.

You can define a map using Visual Studio and Microsoft Azure BizTalk Services SDK available at http://www.microsoft.com/en-us/download/details.aspx?id=39087.

First, we need to download the schema that will act as a source schema.

To do that, navigate the portal in the all resources list and select the connection to SQL Server previously created. Then select the Download Schema link.

After installed Microsoft Azure BizTalk SDK, open Visual Studio and create a new project of type BizTalk Service Artifact.

Delete al unneeded objects created by the project template.

Then import the schema downloaded in the previous step.

Now, create a new schema that will acts as the target schema. Following just a simple schema.

Finally you can create a new map to transform the source schema.

Define the mapping rules.

Close and save the project.

Now we will define a transformation service on Azure that will contains the map.

From the Azure Marketplace find for BizTalk Transform Service.

Select and specify a name, an App Service Plan (this must be the same used by the SQL Connector) and click on Create.

Navigate the Transform Service and select Maps.

In the right panel will compare a new form with a list of maps. Clock on Add.

Select the “trfm” file created by Visual Studio Solution.

Create the Logic App

Now you are ready to create the Logic App.

Navigate the Azure Portal, select Logic Apps and click on Add.

Specify Logic App name, the resources group and click on create.

Once the Logic App is created click on Edit.

There are many Logic App templates. In this demonstration I choose a blank Logic App selecting “Create from Scratch”.

On the right panel select the “Microsoft SQL Connector” created previously.

Specify “Poll Data (XML)”.

Specify the polling interval. Depending on the Pricing Tier you can specify different measures as polling interval.

Confirm your selection and select “BizTalk Transform Service”.

Click on Arrow to start the configuration.

Then click on Tranform.

Choose the output of Microsoft SQL Connector as map input transform.

Then specify (manually) the map name.

Finally select the OneDrive connector.

You have to autorize the access to your OneDrive subscription.

Then I chosen “Upload a file”.

Specify a File Path and the Transform output as file content.

In this case no content transfer encoding is needed so specify “none”.

Once the configuration is completed, you should see the following integration flow.

Test the sample

To test the sample is enough to insert a row into SQL database table. You can use the following statement to do that.

insert
into SourceTable

(

    sourceTableId,

    transactionStatus,

    sampleContent

)

values

(

    1,

    ‘To Process’,

    ‘Content to transfer’

);

After a few seconds, you should see the file into your OneDrive subscription.

In the following picture, you can see the content of the result file.