~/talk

Software Engineering Skills for Data Engineers

on YouTube 28m
Talkat DataEngBytes Sydney 2024 · on Tuesday, 2024-09-24· [data-engineering]

speaker

name:Yousry Mohamed
links:none on file

resources

video:https://youtu.be/3R8glhadjMk
slides:none on file

summary

Covers concrete fixes: pick numeric types to avoid floating point surprises and cut storage (example dataset stored as 22 GB vs ~11 GB when switching doubles to floats), store hashes as binary (32 bytes) instead of hex strings to save space, use pydantic for typed configs, use Databricks notebook code debugger, handle Spark driver vs worker execution and Python GIL pitfalls, resolve a real soda vs dbt core dependency conflict with Poetry, and apply partial application and memoization for reusable, testable pipeline code.

AI-written summary · about

chapters (7)

  1. 0:00Software engineering intro
  2. 1:49Data types matter
  3. 8:31Avoiding spaghetti code
  4. 17:04Distributed computing
  5. 22:21Debugging pipelines
  6. 25:13Managing dependencies
  7. 27:39Engineering curiosity

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.