Editorial Summary :

Organizations across various industries are using artificial intelligence (AI) and machine learning (ML) to solve business challenges specific to their industry . Large enterprises sometimes set up a center of excellence (CoE) to tackle the needs of different lines of business (LoBs) with innovative analytics and ML projects . To generate high-quality and performant ML models at scale, they need to do the following: This can reduce the long cycle time for converting ML use cases from experiment to production . A data mesh architecture strives to solve these technical and organizational challenges by introducing a decentralized socio-technical approach to share, access, and manage data in complex and large-scale environments . Each LoB defines their own data products, which are curated by people who understand the data and are best suited to specify who is authorized to use it, and how it can be used . In contrast, other LoBs and application domains such as the analytics and ML CoE are interested in discovering and consuming qualified data products . Organizations need to balance the agilities illustrated earlier with proper mitigation of the risks associated with data leaks . In regulated industries like financial services, there is a need to maintain central data governance to provide overall data access and audit control . We address data consumption by the analytics and ML CoE with Amazon Athena and Amazon SageMaker in part 2 of this series . In this post, we focus on the data onboarding process into the data mesh and describe how an individual LoB such as the consumer banking domain data team can use AWS tools such as AWS Glue to prepare, curate, and enhance the quality of their data products . The core AWS service for enabling data mesh governance is Lake Formation . Lake Formation offers the ability to enforce data governance within each data domain and across domains . To set up a data mesh architecture, you need at least three AWS accounts: a producer account, a central account, and a consumer account . After you run the templates, you can go through the step-by-step guide to add a product in the data catalog and have the consumer subscribed to it . The guide starts by setting up a database where the producer can place its products and then explains how the consumer can subscribe to that database and access the data . All of this is performed while using the tag-based access control for Lake Formation . Hasan Poonawala is a Senior AI/ML Specialist Solutions Architect at AWS . Hasan helps customers design and deploy machine learning applications in production on AWS . Benoit de Patoul is a Data Scientist, Machine Learning practitioner, and software developer . In his spare time, Hasan loves to explore nature and spend time with friends and family . He likes to watch TV documentaries and play video games with his son, and play piano in his free time, he likes to play piano and play with friends .

Key Highlights :

  • Organizations across various industries are using artificial intelligence (AI) and machine learning (ML) to solve business challenges specific to their industry .
  • Each LoB defines their own data products, curated by people who understand the data and are best suited to specify who is authorized to use it, and how it can be used .
  • We address data consumption by the analytics and ML CoE with Amazon Athena and Amazon SageMaker in part 2 of this series .
  • In this post, we focus on the data onboarding process into the data mesh .
  • To set up a data mesh architecture, you need at least three AWS accounts: a producer account, a central account, and a consumer account .
  • To deploy a data . mesh environment, you can use the following GitHub repository .
  • Hasan Poonawala is a Senior AI/ML Specialist Solutions Architect at AWS .
  • Hasan helps customers design and deploy machine learning applications in production on AWS .
  • Benoit de Patoul is an AI/ml Specialist Solutions .

The editorial is based on the content sourced from aws.amazon.com

Read the full article.

Similar Posts