Upload Dataset on Azure SQL Database using Azure Event Hub and Azure Stream Analytics

There are several ways to upload data to SQL Database, including SSIS, BCP, API and several other ways.

For those who work with SQL Server it is natural to use tools that historically belong to the SQL platform, so I wanted to write this article to expose an alternative and to talk about different techniques that apply to different scenarios.

When designing an architecture, you may find yourselves at a crossroads, and often you do not know what to expect once made a decision. How requirements will change over time.

Azure Event hub allows us to decouple who collects the data (the client) from who receives the data, allow you to change kind of receivers or add additional targets.

For this reason, I consider it the entry point to Microsoft Azure.

The following picture shows the data flow that I am going to describe.

Following the check list

  • Create Resource Group
  • Create Event Hubs Namespace
  • Create Event Hub Queue
  • Create SQL Server
  • Create SQL Database
  • Create SQL Table
  • Create Stream Analytics Job
  • Configure Input
  • Configure Output
  • Configure Query
  • Start Stream Analytics Job

To complete each step, you must first access the Microsoft Azure portal (https://portal.azure.com).

I think it is unnecessary to describe each step in detail, so I will just provide some screen shots.

Create Resource Group

Create Event Hubs Namespace

Create Event Hub

Create SQL Server (logical server)

Connect to SQL Server and create database objects

Before you can connect to SQL Database virtual server, you have to configure Azure firewall to allow the communication.

Use SQL Server Management Studio to create database and tables

Create database

/*

For additional information refer to address https://msdn.microsoft.com/en-us/library/dn268335.aspx

*/

CREATE
DATABASE [int-dstosql-sqldb]

(

    EDITION ‘basic’,

    SERVICE_OBJECTIVE=‘basic’

);

GO

Create table

CREATE TABLE OrdersByCity

(

    entryId INT NOT NULL IDENTITY(1,1) PRIMARY KEY,

    city VARCHAR(50NOT NULL,

    orderValue NUMERIC(10,2NOT NULL,

    orderDate DATETIME NOT NULL

);

Create Stream Analytics Job

Create Stream Analytics Job Input

Create Stream Analytics Job output

Define Stream Analytics Job Query

Refer to the link https://msdn.microsoft.com/en-us/library/azure/dn834998.aspx for additional information about Azure Stream Analytics query language.

Save and start stream analytics job.

Write and run client application

Now you able to run the client application. To create a sample application, follow the steps below.

1 – Install WindowsAzure.ServiceBus NuGet Package

2 – Get Event Hubs Connection String

3 – Write your code

using Microsoft.ServiceBus.Messaging;

using System.Threading;


DataTable dataTable = <Put here your code to retrieve data>;

var data = JsonConvert.SerializeObject(dataTable);

var eventHubClient = EventHubClient.CreateFromConnectionString(<Event Hub Conn String>, <Queue Name>);

eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes(data)));

eventHubClient.Close();

Refer to the link https://azure.microsoft.com/en-us/documentation/articles/event-hubs-csharp-ephcs-getstarted/ to additional information.

Advertisements

Integrate SQL Server with Power BI

One of the most common scenarios for those who use Microsoft technologies and intend to publish its data into Power BI typically use SQL Server as a data source.

Power BI provides several ways to access SQL Server information, among them there are some that need simple configurations and others that require developments. Below we summarize the different possibilities that you have available:

Power BI has a wide range of connector to access to several types of data sources, these include a connector for Azure SQL Database and one for Azure SQL Data Warehouse.

To access to a SQL Server, whether it be an instance of Azure IaaS or On-Premises instance you can use Power BI Gateways. There is a Power BI Enterprise Gateway for enterprise deployments and a Power BI Personal Gateway for personal use. You can download it directly from Power BI portal and install in a server near your SQL Server.

For additional information about Power BI Gateways refer to https://powerbi.microsoft.com/en-us/gateway/.

Both these features support direct query option, that is SQL Server is queried when the user accesses the report or dashboard. Typically you prefer to schedule the frequency of the update. You can schedule updates every few minutes.

If you want to get a dashboard that receives continuous updates getting the effect of a live dashboard you need to perform simple developments.

To update a Power BI dataset you can use different Microsoft Azure functionality:

Windows Azure Event Hubs + Windows Azure Stream Analytics

Power BI REST API

Both techniques need configuration on Microsoft Azure side and a Windows service that resides in the same network of your SQL Server that is responsible for reading the information and send them to the Cloud.

This approach allows you to make updates with a frequency close to the second.

Use Windows Azure Event Hubs and Microsoft Azure Stream Analytics is simpler and the Power BI dataset is created automatically, but with this technique, you can only insert new information.

The Power BI APIs are a bit more complex but allow several types of operations. For example it is possible to empty the dataset before inserting new information.

However, you must manually create the data set.

You can download a Windows service sample at the address http://db2powerbi.codeplex.com/. This simple solution it able to use both techniques. It is enough to apply a configuration to the QueryConfig.json file and write a query file. The project includes a console (DbToP Svc.Power Bi.UI.exe) that allows you to create the dataset and retrieve its id (required in the configuration file).

Below is an example of Query Config.json file.

As I said, before starting the service you need to apply some Microsoft Azure side configurations.

You can follow the article https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-power-bi-dashboard/ to configure Windows Azure Event Hubs and a Stream Analytics Job.

You have to follow the article https://powerbi.microsoft.com/en-us/documentation/powerbi-developer-register-a-client-app/ before use Power BI REST API.