BrainTrust Investment Lab › Band 2 — Transform (Cleaning & Conforming)
Band 2 — Transform (Cleaning & Conforming) · book 460-461 · grain: one source per walk sheet
Every fact row in the star will carry its audit grade — where it came from, when it landed, what condition it arrived in. Ask any number for its papers, and it has them.
completed
0 of 48
running now
sec_cik
completedrunningawaiting its turn · owedfailedexcluded
sec_cik↻rank 1 · walking · 8 step(s) certified
security_master↻rank 2 · walking · 8 step(s) certified
prices↻rank 3 · walking · 8 step(s) certified
edgar_filings↻rank 4 · walking · 8 step(s) certified
companyfacts↻rank 5 · walking · 8 step(s) certified
submissions↻rank 6 · walking · 8 step(s) certified
overview↻rank 7 · walking · 8 step(s) certified
form345rank 8 · queued
fsdsrank 9 · queued
ftdrank 10 · queued
finra_shortrank 11 · queued
sec_name_cikrank 12 · queued
notesrank 13 · queued
framesrank 14 · queued
fredrank 15 · queued
waybackrank 16 · queued
pbsrank 17 · queued
wikipediarank 18 · queued
edgarrank 19 · queued
av_transcriptsrank 20 · queued
shred_8krank 21 · queued
shred_13dgrank 22 · queued
aapbrank 23 · queued
bdcrank 24 · queued
crowdfundingrank 25 · queued
form13frank 26 · queued
formdrank 27 · queued
market_structurerank 28 · queued
mfp_rrrank 29 · queued
ncenrank 30 · queued
nmfprank 31 · queued
nportrank 32 · queued
regarank 33 · queued
sec_enforcementrank 34 · queued
sec_filings_statsrank 35 · queued
sec_registriesrank 36 · queued
transfer_agentrank 37 · queued
viprank 38 · queued
fomc_textrank 39 · queued
lda_lobbyingrank 40 · excluded
av_options↻rank 41 · walking · 8 step(s) certified
wikidatarank 42 · queued
dividends↻rank 43 · walking · 8 step(s) certified
splits↻rank 44 · walking · 8 step(s) certified
gdeltrank 45 · queued
usaspendingrank 46 · queued
finra_regshorank 47 · queued
edgar_logsrank 48 · queued
edgar_docsrank 49 · queued
The contract — what each step must produce
| # | step | page | what the book requires |
|---|---|---|---|
| 2 | the audit context, DECLARED | p.460 | 'The audit dimension in Figure 19-2 contains the metadata context at the moment when a specific fact table row is created. You might say we have… |
| 3 | one audit row per run condition | p.460 | 'you would generate only one audit dimension row, and it would be attached to every fact row loaded today.' |
| 4 | the quality flags, named | p.460 | Figure 19-2: 'Overall Quality Rating - Complete Flag - Validation Flag - Out Of Bounds Flag - Screen Failed Flag - Record Modified Flag - ETL Master… |
| 5 | metadata queryable beside the data | p.460 | 'we have elevated metadata to real data!' |
| 8 | the audit dimension stands where Figure 19-2 draws it -- in the star, column for column | p.460 | dim_audit is in braintrust_v8 beside the error event schema, carrying every Figure 19-2 column, the context UNIQUE that IS p.460's grain, and p.511's… |
| 9 | the quality indicator's domain is the BOOK's two values | p.511 | every audit row this run wrote carries 'Quality Checks Passed' or 'Quality Problems Encountered', agreeing with its own screen_failed_flag |
| 10 | the audit key on the fact table -- counted here, attached at SS13/SS14 | p.511-512 | fact_tables_lacking_audit_key = 0 of however many business fact tables exist; today that set is empty and the requirement is UNEXERCISED |
| 12 | one audit row per DISTINCT metadata context in the run | p.460 | 'Suppose today you have a perfect run with no errors flagged. In this case, you would generate only one audit dimension row, and it would be attached… |