BrainTrust Investment Lab › Band 1 — Extraction (Remote Sources → Local System)
Band 1 — Extraction (Remote Sources → Local System) · book 453-455 · grain: one source per walk sheet
Getting data out of a provider is easy; proving you got all of it is the job. Extraction here reconciles against the provider's own totals — every file, every row accounted for, with the gaps on a list rather than in the dark.
completed
0 of 48
running now
sec_cik
completedrunningawaiting its turn · owedfailedexcluded
sec_cik↻rank 1 · walking · 7 step(s) certified
security_master↻rank 2 · walking · 7 step(s) certified
prices↻rank 3 · walking · 7 step(s) certified
edgar_filings↻rank 4 · walking · 7 step(s) certified
companyfacts↻rank 5 · walking · 7 step(s) certified
submissions↻rank 6 · walking · 6 certified · 1 owed
overview↻rank 7 · walking · 7 step(s) certified
form345↻rank 8 · walking · 1 step(s) certified
fsdsrank 9 · queued
ftdrank 10 · queued
finra_shortrank 11 · queued
sec_name_cikrank 12 · queued
notes↻rank 13 · walking · 1 certified · 1 owed
framesrank 14 · queued
fredrank 15 · queued
waybackrank 16 · queued
pbsrank 17 · queued
wikipediarank 18 · queued
edgarrank 19 · queued
av_transcripts↻rank 20 · walking · 7 step(s) certified
shred_8krank 21 · queued
shred_13dgrank 22 · queued
aapbrank 23 · queued
bdcrank 24 · queued
crowdfundingrank 25 · queued
form13frank 26 · queued
formdrank 27 · queued
market_structurerank 28 · queued
mfp_rrrank 29 · queued
ncenrank 30 · queued
nmfprank 31 · queued
nportrank 32 · queued
regarank 33 · queued
sec_enforcementrank 34 · queued
sec_filings_statsrank 35 · queued
sec_registriesrank 36 · queued
transfer_agentrank 37 · queued
viprank 38 · queued
fomc_textrank 39 · queued
lda_lobbyingrank 40 · excluded
av_options↻rank 41 · walking · 7 step(s) certified
wikidatarank 42 · queued
dividends↻rank 43 · walking · 7 step(s) certified
splits↻rank 44 · walking · 7 step(s) certified
gdeltrank 45 · queued
usaspendingrank 46 · queued
finra_regshorank 47 · queued
edgar_logsrank 48 · queued
edgar_docsrank 49 · queued
The contract — what each step must produce
| # | step | page | what the book requires |
|---|---|---|---|
| 2 | file-or-stream declared, bulk-first, the door-shape CITED from the registry | p.454 | 'There are two primary methods for getting data from a source system: as a file or a stream.' |
| 3 | the extract file is SAVED with its digest -- rerun without touching the source | p.454 | 'They are easy to restart at various points. As long as you save the extract file, you can rerun the load without impacting the source system.' |
| 4 | row counts compared before and after | p.454 | 'it is easy to verify that all data has moved correctly by comparing file row counts before and after the transfer.' |
| 5 | the load restarts at declared points -- proven where the house already has the receipt | p.454 | 'extracts to file have some advantages. They are easy to restart at various points.' |
| 6 | shred-or-persist decided per payload | p.454 | 'the DW/BI team must decide whether to “shred” the XML into a large number of destination tables or persist the XML structure within the data… |
| 7 | compression and encryption posture declared | p.454-455 | 'Remember to compress before encrypting because encrypted files do not compress very well.' |
| 8 | the source-system courtesy contract | p.453-454 | 'try to do the comparison on the source machine, so you don’t have to transfer the entire table... Of course, the source support folks may have an… |
| 9 | THE LANDING (blocked until step 9's SIGN) | p.421 (Develop Physical Database Model, under the Physical Design heading on p.420) | "There are several additional sets of tables that need to be designed and deployed as part of the DW/BI system, including staging tables to support… |
| 11 | IMPROVING DATA QUALITY CULTURE AND PROCESSES -- the section's nine, disposed | p.455 | p.455 'Improving Data Quality Culture and Processes': Declare a high-level commitment to a data quality culture. | Drive process reengineering at the… |