Exciting new and upcoming Microsoft Fabric updates

Written by Ryan McNaught | May 29, 2024 3:37:00 PM

As the Head of iTalent Digital’s BI, Data & Analytics consulting practice, I had the privilege of attending the first-ever Microsoft Fabric Community conference in Las Vegas in March. It was an energizing experience that brought together the Microsoft Fabric product team, customers, and technology partners like us to dive into the latest developments in data analytics in the age of AI.

We got an exclusive look at some of the exciting new features and updates coming to Microsoft Fabric which I have learned even more about since. These enhancements are transforming how Microsoft Fabric early adopters manage and analyze data, making the process more intuitive and powerful than ever.

Here’s my take on the pivotal developments and what they mean for the future of data analytics.

Quick overview of Microsoft Fabric

As someone who engages with the complex world of data analytics every day, I've found Microsoft Fabric to be a formidable platform that our clients are eager to learn more about. It’s an end-to-end analytics and data platform that integrates a wide range of functionalities into a unified solution that simplifies complex data operations. That includes data movement, processing, ingestion, transformation, real-time event routing, and Power BI insights with metric monitoring (e.g. Data Activator).

What sets Microsoft Fabric apart is its ability to enhance data analytics and integration. It enables users like us to manage and analyze large datasets with efficiency and precision. With Fabric, you can significantly simplify multiple services from different vendors, since it provides a comprehensive suite of services such as data engineering, real-time analytics, and data warehousing all in one place!

In the Age of AI, Fabric offers robust data management and analytics tools that are embedded with AI capabilities. This allows for automatic integration and makes it easier to leverage AI for more intelligent and efficient data processing.

Ryan's key takeaways from the conference

Here are some observations from the Microsoft Fabric Community Conference.

1. Upcoming features sneak-peek

One of the most exciting previews we got at the conference was the introduction of task flows in Microsoft Fabric. Task flows are set to revolutionize how we visualize and manage data projects from start to finish. This new feature promises to enhance workflow clarity and improve project management by providing a clear visual map of the entire data process.

Task flows will allow users to:

Visualize data projects: See your entire data project in a single, comprehensive view, from data ingestion to analysis and reporting.
Improve workflow management: It becomes easier to identify bottlenecks, track progress, and manage tasks efficiently across different stages of the data pipeline.
Enhance collaboration: With a clear view of the project workflow, team members can better understand their roles and collaborate more effectively on complex data tasks.

2. Recently announced Power BI Semantic Link benefits

The Power BI Semantic Link has been a game-changer, allowing business analysts and data scientists like us to analyze data using various languages such as MDX, DAX, SQL, or Python.

This capability allows us to:

Combine multiple semantic models: This enhances our ability to analyze complex datasets and derive more nuanced insights.
Perform comprehensive data analysis: By using different languages, we can tailor our analysis to fit the specific needs of each project, making our work more precise and insightful.

The ability to create scripts for various purposes is another significant enhancement in Fabric. It helps on several fronts.

Data validation and orchestrations: We can now build data validation rules and smart orchestrations to run refreshes more efficiently.
Enhanced scripting with TSQL, PySpark, and Scala: There's an increased focus on these languages within Fabric that enable us to automate development and operational tasks. That includes the creation and maintenance of workspaces and lake houses and optimizing the performance of semantic models.

3. OneLake updates

The updates to OneLake are particularly exciting, starting with the release of shortcuts within OneLake, to the Azure Gen2 ADLS, Google Cloud Platform, Amazon S3, and to on-premises data. This new feature simplifies how we integrate and manage data across different cloud and on-premises environments and enhances our ability to unify diverse datasets virtually with ease across storage locations.

Also worth mentioning is that mirroring is now in preview for Azure Cosmos DB, Azure SQL DB, and Snowflake customers. This capability allows us to mirror our data in OneLake and unlocks all functionalities of the Fabric Data Warehouse, Direct Lake mode, notebooks, and more without incurring additional storage costs.

Microsoft is also introducing a new external data-sharing experience for Microsoft Fabric data and artifacts. This development is set to revolutionize how we share and collaborate on data projects. By enabling seamless and secure data sharing, we can work more closely with partners and clients while ensuring everyone has access to up-to-date information.

4. Data management enhancements

Microsoft has also enhanced the support for data pipelines and data warehouses within Fabric Git integration and deployment pipelines. That means managing these resources is now more integrated and streamlined for robust data operations.

Spark job definition

One of the key updates is the inclusion of Spark job definition in the Git integration. A Spark job definition is essentially a blueprint that outlines how to execute a data processing task using Apache Spark - an open-source, distributed computing system that provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. This includes the Spark code and any dependencies or parameters needed to run the job.

By defining Spark jobs within Fabric, users can manage and run large-scale data processing tasks more effectively, making it easier to process vast amounts of data quickly and reliably.

Spark environment

Along with the Spark job definition, the Spark environment will also become available in Git integration. The Spark environment refers to the configured setting in which your Spark jobs run. This includes the version of Spark, the configuration of cluster resources (like memory and CPU), and any required libraries or dependencies.

Having a well-defined Spark environment ensures Spark jobs are executed in a consistent and controlled manner, which is crucial for maintaining the reliability and efficiency of data processing tasks. This integration into Fabric means users can manage and adjust their Spark environments easily and ensure their data processing workflows are both scalable and reproducible.

APIs and Fast Copy in Dataflows Gen2

The introduction of Fabric Git integration APIs and deployment pipelines APIs is another significant enhancement. These APIs enable us to integrate Fabric more deeply into familiar tools and enhance our ability to automate and manage data workflows.

Fast Copy in Dataflows Gen2 also allows us to ingest a large volume of data using the same data movement backend as the ‘copy’ activity in data pipelines. For data pipelines, we can now access on-premises data using the on-premises Data Gateway—the same gateway used with dataflows. This integration is crucial to ensure that data ingestion processes are as efficient and scalable as possible.

5. Workspace and security updates

Enhancements to workspaces and security are critical to ensuring our data management and analytics efforts are both effective and safe.

Here are the latest updates that Microsoft Fabric is rolling out in these areas:

Improved organization with folders: Microsoft Fabric is introducing the ability to create folders within workspaces. This feature allows users like us to organize our content more effectively, much like we do in Windows. That means we can now group related data projects, models, and reports into dedicated folders, making them easier to navigate and manage.
Enhanced compliance and discoverability with tags: Soon, we will be able to add tags to Fabric items, which will enhance compliance, discoverability, and reuse. Tags act as customizable markers that can be applied to various Fabric resources, making it easier to sort, identify, and retrieve these resources based on specific criteria. This feature is especially beneficial for maintaining a well-organized and compliant data environment, as tags can help categorize items based on security levels, project phases, or any other organizational needs.

Microsoft Fabric is stepping up its game with significant enhancements to its security framework. These updates fortify the platform against a range of threats and ensure that our data handling is both secure and compliant with the latest standards.

Here’s a closer look at the new security features and what they mean for us:

Azure Private Link support: Microsoft Fabric is enhancing its connectivity options with Azure Private Link support. This feature ensures data interactions within Fabric are securely handled through private links, minimizing exposure to public networks and potential threats. By using Azure Private Link, we can secure and privatize our communication with Microsoft Azure services for enhanced data protection and privacy.
Trusted workspace access and managed private endpoints: To further secure workspace access, Microsoft Fabric now supports Trusted Workspace Access and Managed Private Endpoints. Trusted Workspace Access lets us define and enforce who can access specific workspaces based on predefined security criteria for enhanced control over data access. Managed Private Endpoints provide a secure way to connect to Fabric services from within a virtual network and ensure data flows are isolated and protected from external access.
VNET data gateway: The VNET data gateway is now available. It offers robust connectivity options for integrating Fabric with various data sources. This gateway facilitates secure data transfers between Microsoft Azure and on-premise or cloud environments and ensures data ingress and egress are safeguarded.
Deeper integration with Microsoft Purview: Fabric's security capabilities are further extended through deeper integration with Microsoft Purview’s industry-leading data security and compliance offerings. Soon, security admins will be able to define Purview Information Protection policies directly in Microsoft Fabric. This integration will automatically enforce access permissions to sensitive information for enhanced data security across our projects.
Extension of Purview Data Loss Prevention (DLP) policies: The extension of Purview Data Loss Prevention policies to Fabric means we can better prevent potential data breaches and leaks. These policies help monitor and protect sensitive data across Fabric while ensuring any data mishandling is detected and mitigated promptly.
Integration with Purview Insider Risk Management: Another upcoming feature to look forward to is the integration with Purview Insider Risk Management. This addition will help us detect, investigate, and act on malicious and inadvertent data oversharing activities within our organization. By leveraging this integration, we can enhance our ability to safeguard against internal risks and ensure our data use remains within compliance and security guidelines.

Empower your data journey with Microsoft Fabric

As we've explored, the new features and enhancements to Fabric signify Microsoft's commitment to data analytics and security. From innovative task flows to advanced security measures integrated with Microsoft Purview, these updates are set to transform how we manage, analyze, and protect data.

If you’re looking to harness the full potential of these developments in Microsoft Fabric or need expert guidance to optimize your data analytics infrastructure, iTalent Digital is here to help. Contact me at itbi@italentdigital.com or visit our website to learn more about our BI, Data & Analytics services.