BrainTrust Investment Lab › Band 4 — Managing the ETL Environment
Band 4 — Managing the ETL Environment · book 486-487 · grain: one station per walk sheet
Every load is built to die safely: rerun it and nothing double-loads, kill it and nothing is left half-written. Failure is an event we planned for, not an adventure.
No walk sheets yet — this station begins when the ladder reaches it. Nothing is walked ahead of the lowest incomplete rung.
The contract — what each step must produce
| # | step | page | what the book requires |
|---|---|---|---|
| 2 | the book's six, plus ours | p.486 | 'Network failure - Database failure - Disk failure - Memory failure - Data quality failure - Unannounced system upgrade' |
| 3 | small sets, and keep track | p.487 | 'a load process should commit relatively small sets of records at a time and keep track of what has been committed.' |
| 4 | which, and why | p.487 | 'In many cases, it may be best to back out any rows that have been loaded as part of the process and restart from the beginning.' |
| 5 | the key that makes back-out possible | p.487 | 'With the fact table surrogate key, you can easily resume a load that is halted or back out all the rows in the load by constraining on a range of… |