BrainTrust Investment Lab › Band 2 — Transform (Cleaning & Conforming)
Band 2 — Transform (Cleaning & Conforming) · book 458-460 · grain: one source per walk sheet
Every quality failure ever caught lands in its own fact table — what failed, where, when, how bad. Data quality here is not a slogan; it is a measurable, queryable history that improves because it can be seen.
completed
0 of 48
running now
sec_cik
completedrunningawaiting its turn · owedfailedexcluded
sec_cik↻rank 1 · walking · 8 step(s) certified
security_master↻rank 2 · walking · 8 step(s) certified
prices↻rank 3 · walking · 8 step(s) certified
edgar_filings↻rank 4 · walking · 8 step(s) certified
companyfacts↻rank 5 · walking · 8 step(s) certified
submissions↻rank 6 · walking · 8 step(s) certified
overview↻rank 7 · walking · 8 step(s) certified
form345rank 8 · queued
fsdsrank 9 · queued
ftdrank 10 · queued
finra_shortrank 11 · queued
sec_name_cikrank 12 · queued
notesrank 13 · queued
framesrank 14 · queued
fredrank 15 · queued
waybackrank 16 · queued
pbsrank 17 · queued
wikipediarank 18 · queued
edgarrank 19 · queued
av_transcriptsrank 20 · queued
shred_8krank 21 · queued
shred_13dgrank 22 · queued
aapbrank 23 · queued
bdcrank 24 · queued
crowdfundingrank 25 · queued
form13frank 26 · queued
formdrank 27 · queued
market_structurerank 28 · queued
mfp_rrrank 29 · queued
ncenrank 30 · queued
nmfprank 31 · queued
nportrank 32 · queued
regarank 33 · queued
sec_enforcementrank 34 · queued
sec_filings_statsrank 35 · queued
sec_registriesrank 36 · queued
transfer_agentrank 37 · queued
viprank 38 · queued
fomc_textrank 39 · queued
lda_lobbyingrank 40 · excluded
av_options↻rank 41 · walking · 8 step(s) certified
wikidatarank 42 · queued
dividends↻rank 43 · walking · 8 step(s) certified
splits↻rank 44 · walking · 8 step(s) certified
gdeltrank 45 · queued
usaspendingrank 46 · queued
finra_regshorank 47 · queued
edgar_logsrank 48 · queued
edgar_docsrank 49 · queued
The contract — what each step must produce
| # | step | page | what the book requires |
|---|---|---|---|
| 2 | the star EXISTS, measured against Figure 19-1 -- and its DDL of record is named | p.458 | 'The error event schema is a centralized dimensional schema whose purpose is to record every error event thrown by a quality screen anywhere in the… |
| 3 | EXACTLY ONE ROW PER ERROR THROWN -- and every counted defect has its row | p.458 | 'Its grain is every error thrown (produced) by a quality screen anywhere in the ETL system. Remember the grain of a fact table is the physical… |
| 4 | the calendar date RESOLVES -- to a dimension that exists | p.458-459 | 'The calendar date is not a minute and second time stamp of the error, but rather provides a way to constrain and summarize error events by the usual… |
| 5 | the run is ONE batch row, and the batch ENDS | p.458 | 'The dimensions of the error event fact table include the calendar date of the error, the batch job in which the error occurred, and the screen that… |
| 6 | every declared screen has ONE dimension row -- closed BOTH directions against the signed spec | p.459 | 'The screen dimension identifies precisely what the screen criterion is and where the code for the screen resides. It also defines what to do when… |
| 7 | the LOWER grain -- one field, in one record, per detail row | p.459-460 | 'The error event schema includes a second error event detail fact table at a lower grain. Each row in this table identifies an individual field in a… |
| 8 | THIS LANE'S SCREENS WRITE -- in THIS run, by a writer that exists | p.460 | 'You should now appreciate that each quality screen has the responsibility for populating these tables at the time of an error.' |