Secure Large-Scale Data Analysis and AI with Polyglot Data Sources
summary
Concrete setup showing SQL Server Big Data Clusters running on Red Hat OpenShift to provide a secure HTAP data hub that serves OLTP, OLAP and Spark/HDFS workloads together. Demo pipeline: data engineer writes transactions with PySpark to HDFS, DBA creates external tables to query that data from SQL Server Big Data Clusters, data scientist trains models in Jupyter with sklearn and a small neural network where the neural net shows higher loss despite slightly better accuracy so a classical model wins. Everything deploys via containers, GitOps and tools like Red Hat OpenShift and polyglot data sources.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:38Data at Scale
- 2:08Applications of Data
- 4:48Requirements
- 6:57Platform
- 10:35Demo
- 13:40Red Hat Openshift
- 16:07Azure Data Studio
related talks
43mSQL Server 2019 Big Data Clusters Architecture
02h 05mBig Data and the Data Lake from the Analyst Perspective
28mFlexible Cloud, Whatever The Workload
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
06mDriving enterprise open source adoption, from data lake to AI
42mWeb Development To Big Data. A Journey
17mDemocratising data analytics
30mProcess large-scale data and computational function in serverless manner