How to Process Streaming Data in Power BI

How to Process Streaming Data in Power BI

Organizations produce, consume, and store large amounts of data. Streaming is data that is continuously generated by various sources. It is processed incrementally and analyzed in near real-time. This type of data gives businesses a live view of all aspects of the organization. It enables business leaders to quickly make decisions and respond to issues faster than other forms of data pre-processing.

What is Data Engineering?

Data Engineering is the process of taking raw data sets and creating the infrastructure required to transform them into data that is meaningful and store it so that it is accessible in a usable format. This involves sourcing (databases, data warehouses, application programming interfaces, etc.), moving (source to target), storing, securing, and governing data (quality).

What is Public Cloud?

The public cloud is defined as computing services offered by third-party providers over the public internet, making them available to anyone who wants to use or purchase them. They may be free or sold on-demand, allowing customers to pay only per usage for the CPU cycles, storage, or bandwidth they consume. Azure is a Microsoft Public Cloud Platform with three main capabilities: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).

What is an Analytical Data Flow?

Dataflows that load data into analytical entities are categorized as an analytical dataflow.

A picture showing a data flow
https://docs.microsoft.com/en-us/azure/architecture/example-scenario/data/small-medium-data-warehouse

Assumptions

While it is essentially free to upload data into the cloud and relatively inexpensive to store, it can be costly to extract data from. A common theme in Data Management is to hit an application or data source once, grab everything, and store it for future use. This approach lends itself well to cloud computing because it has a large storage capacity and reduces hardware maintenance costs that businesses incur with on-prem data centers. Further, organizations need to protect their data against hackers and data breaches. Beyond the loss of data, these can be costly to resolve and negatively impact the organization’s brand and customer experience. Therefore, security should be part of every data decision you make.

A donut chart showing the month's top products by charges

Let’s get started!

Log in to your azure portal at https://portal.azure.com/

Set up a Resource Group. This is a logical grouping of resources. For an Enterprise these will be broken out by department, line of business, or subject matter.

Then you want to create a Storage Account for the data to be stored as a Binary Large Object in Azure Blob. Azure Blob (and Azure Data Lake Storage) is like a Hadoop Distributed File System (HDFS) managing large data sets running on commodity hardware within Azure.

Note: Tie the Storage Account to the Resource Group and Region. It is important to be consistent on the Region because you will be charged a penalty for downloading data across locations. As always, choose your performance based on the data needs of your organization.

Azure Blob Storage

A screenshot of Azure creating a storage account

Next, create an Event Hub Namespace. This is a logical grouping of Event Hubs. Create your Event Hub within the namespace. This is your message broker that ingests streaming data and feeds it to Azure.

Azure Event Hub

Azure event hub screen recording

Now that your Event Hub is set up, you want to authenticate the connection strings and SAS access policies. This governs the access to manage, send, and listen to a specific Event Hub.

Connection

A screen recording to authenticate the connection strings

Azure Stream Analytics allows you to run queries on a live stream. A Stream Analytics job consists of three main parts: an input, output, and query which defines how Stream Analytics will process the data.

Azure Stream Analytics

a screen recording of a new stream

Now that we’ve established that you can send messages through Event Hub and store them in containers, the Stream Analytics job will take messages from Event Hub and put them in your desired output, which in this case is a Power BI data set that will be consumed in Power BI Pro to show updates in real time.

Power BI

Now that we’ve established that you can send messages through Event Hub and store them in containers, the Stream Analytics job will take messages from Event Hub and put them in your desired output, which in this case is a Power BI data set that will be consumed in Power BI Pro to show updates in real time.

What About the Data?

I used an API from CryptoCompare.com to see current pricing for Bitcoin (BTC), Ethereum (ETH), and Litecoin (LTC). I executed a python application to pull the data from CryptoCompare and push it to Azure Event Hub using the connection string from the shared access policy.

A screenshot of Spyder

How can I use Streaming Data in my Organization?

Previously restricted to Media and Finance, data streaming is now available across all industries. Streaming connects decision makers with data, providing them with the insights they need to take action. For instance, streaming of coal reclaimer idle time or aircraft sensor data could enable leaders to forecast maintenance more proactively and efficiently, thereby minimizing downtime and revenue impact.

What are you waiting for? Get started with Azure today!

Still have some curiosities? Connect with us!

About Tessellation

We are a modern analytics consultancy. We enable and manage organizations’ analytics and self-service teams by educating people, optimizing technology, developing world-class products, and providing sustainable results. Curious to know how we can level up your organization’s analytics? Click here!

Interested in joining the Tessellation Team as a data analyst, Tableau expert, dashboard designer, or data scientist? We’re hiring! Check out our latest job listings on our website and on our Linkedin page!

Subscribe to Our 'Data Cap' Newsletter!

Want a monthly insights from the Tessellation Team with tips, tricks, and secrets to improve your analytics?

Sign up below and we'll deliver articles, events, and how to's straight to your inbox.
Comments are closed.