BrainTrust Investment Lab › Band 4 — Managing the ETL Environment
Band 4 — Managing the ETL Environment · book 483-485 · grain: one station per walk sheet
The schedule knows its dependencies; nothing runs on hope. A job runs when everything it needs has finished and proven itself — and when something upstream fails, downstream waits rather than loading garbage on time.
completed
0 of 1
running now
fact_price
completedrunningawaiting its turn · owedfailedexcluded
The contract — what each step must produce
| # | step | page | what the book requires |
|---|---|---|---|
| 2 | the REFUSAL, dated and argued | p.484 | 'After the ETL process has been developed and deployed, it should execute in a hands-off manner. It should run without human intervention, without… |
| 3 | p.483-484 | if you elect not to use the scheduler included with the ETL tool, or do not use an ETL tool, you need to utilize existing production scheduling or… | |
| 4 | the scheduler's real job | p.483-484 | 'The scheduler needs to be aware of and control the relationships and dependencies between ETL jobs... if the customer table is not properly updated,… |
| 5 | recognizing that a source is ready | p.483-484 | 'It needs to recognize when a file or table is ready to be processed... monitor database flags, check for the existence of files, and compare… |
| 6 | no black box | p.484 | 'We prefer a system that logs to a database because the structure makes it easier to create graphs and reports.' |
| 7 | p.484 | Logging to text files is the minimum acceptable level. We prefer a system that logs to a database. | |
| 8 | somebody needs to know | p.485 | NOTE: 'Somebody needs to know if anything unforeseen happened during the load, especially if a response is critical to continuing the process.' |