~/talk

Synapse Q&A with PG

on YouTube 50m
Talkat SQLBits 2023 · 2023-03-14..03-18· [data-engineering, databases]

speakers

name:Stijn Wynants
links:none on file
name:Luke Moloney
links:none on file
name:Filip Popović
links:none on file
name:Pawel Potasinski
website:https://infinite-services.com
name:Mark Pryce-Maher
links:none on file

resources

video:https://youtu.be/yU3M6PHxUXs
slides:none on file

summary

Serverless now reads Delta tables visible to serverless when Spark writes them, but pipelines cannot write Delta (only Spark engines can), and mapping data flows can read Delta but not write it. Views versus pre-transformed gold layers depends on volume, reuse and skills — large, repeated transforms push to Spark; long nested views hit resource grants and slow queries. Synapse ML adds AutoML, MLflow and Azure OpenAI hooks; self-hosted IR private routing is unsupported; dedicated pools cost-effectiveness depends on sizing — 500 DWU ≈ one node.

AI-written summary · about

chapters (29)

  1. 0:00Introduction and team intro
  2. 1:49Synapse Espresso videos overview
  3. 3:40Lake house data transformation patterns
  4. 9:12Database designer and templates direction
  5. 11:36Databricks and Synapse integration
  6. 13:01Consistent data science experience
  7. 15:37Fast operational reporting methods
  8. 19:08Spark notebook session reuse
  9. 21:46Notebook type support plans
  10. 22:51Open table formats beyond Delta
  11. 23:42CI/CD in Azure DevOps plans
  12. 26:31Indexing in dedicated SQL pools
  13. 27:55Identity columns in Spark
  14. 28:37Reading Delta files in pipelines
  15. 29:39Consuming gold layer data
  16. 33:51Quick-fire audience questions
  17. 34:32Self-hosted IR private routing
  18. 35:37Synapse Link data lake support
  19. 36:20Dedicated SQL pool cost concerns
  20. 38:28Online scaling in Gen2
  21. 39:09KQL to SQL transition tips
  22. 40:10Spark pool Python library install
  23. 40:53On-prem GitLab server support
  24. 41:36Best data transformation tools
  25. 43:45CI/CD experience improvements
  26. 45:53UTF-8 support in dedicated pools
  27. 46:13Production workload under 500 DWUs
  28. 47:39Learning resources for beginners
  29. 49:04Closing and community thanks

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.