Data Collection and Deployment With Azure

Introduction

The ability to train and deploy custom models is at the forefront of artificial intelligence, but the journey through the complexities of data handling, format conversions, and automated scheduling can often feel like wrangling a herd of digital mustangs. This blog presents a robust solution for taming the Artificial Intelligence Technology wilds – automated Azure Open AI custom model training and deployment with Logic App

A Symphony of Azure Solutions

Through meticulous research and testing, a harmonious orchestra of Azure tools emerged.

Microsoft Form

The maestro, collecting data through a custom-designed form.

Azure Storage Account

The vault, storing both raw data (Storage Table) and processed Jsonl files (Blob container).

Logic App

The conductor, orchestrating the entire workflow with three separate Logic Apps.

Data Collector

Captures each Form submission and deposits it in the Storage Table.

Training Data Generator

A diligent daily worker, it retrieves all Table data, transforms it into the required Jsonl format, and deposits it in the Blob container.

Custom Model Deploy

An AI development agency, fetching the latest training data, uploading it to Open AI, creating and deploying the model, and keeping it up-to-date.

The Challenge

In a recent project, a formidable challenge arose: train a custom Open AI model based on real-world Logic App scenarios. The obstacles were imposing

Data Avalanche

Jsonl Enigma

Automation Rodeo

Data Avalanche

Jsonl Enigma

Automation Rodeo

Implementation

To join this AI symphony, you’ll need a few instruments:

Azure Open AI Resource: Available only in North Central US or Sweden Central regions (for fine-tuning models).
Three Logic App Instances: Consumption tier with Managed Identity, each assigned specific roles for storage and Open AI access.
Your Own Microsoft Form: Tailored to your data collection needs.
Data Collector: Sample code awaits in the GitHub repository ([Logic App GitHub Repository](https://github.com/Drac-Zhang/Logic-App-Azure-Open-AI-custom-model-automation)). Remember to adjust the Form ID.
Training Data Generator: Set the "[Table Base Url]" parameter to your Storage Table.
Custom Model Deploy: Modify the "[OpenAI name, Management Url, Deployment Name]" parameters to match your settings.

The Benefits of Automation

By embracing this automated approach, you’ll enjoy:

Conclusion

This Logic App-powered solution tames the complexities of Azure Open AI custom model training and deployment, opening doors to a world of Artificial Intelligence Technology possibilities. So, saddle up your AI spirit, embrace automation, and let your models ride into the sunset of success!