Assumptions about Data Platform Architecture Scaling-Out is better than scaling up. In-Memory is better than on disk (spinning or otherwise). Having data hosted within your control plane or tenant, either on-premises or in the cloud (AWS, Azure, GCP) gives you more control and visibility into the security and access for this data than data hosted […]
Data Platform Architecture Read More »
Data Provenance is loosely referred to as where the data came from. This includes everything from the source application, database type, database instance, database name, schema name, table name, and attribute name. Data Provenance specifically refers to the first instance of that data or its source. Definition from DataVersity – Quoting W3C “Provenance is defined
Assumptions about Change when working with data Always be flexible because technology and vendors are constantly changing and evolving. Be a life-long learner because if you are not keeping up with the advancements in your industry, technology, tools, etc. then you are being left behind.
Object Stores are better because: Cloud storage in object storage (AWS S3, Azure Storage, Google Cloud Storage) is widely used and it has well defined standards and protocols. Object Storage has the best cost structure of any data storage mode/method. Object Storage allows separating Data Storage from Compute which eliminates inefficiencies when both share a
Object Stores vs. RDBMS (Relational Database Management Systems) Read More »
Assumptions about Code when working with data Owning your code is better than Renting, Licensing, or Subscribing to someone else’s code. Code built to perform an operation like ETL based on patterns and driven by property files and parameters to repeat the same operation on thousands of tables/files/objects is more consistent and efficient, easier to
Assumptions about Platform/Vendor Agnostic standards, structures, tools, and languages: Open standard formats, structures, tools, and languages are better than proprietary vendor specific formats, structures, tools, and languages. The ability to move data, code, workflows, processes, and solutions between vendors and platforms without re-architecting or lengthy migration efforts provides the flexibility needed to consider and change
Data Platform Vendor Agnostic Read More »
Guiding Principle: Maximize Automation – Human Intervention is Inconsistent and Error Prone Processes are not always started at the scheduled time as vacations, illness, job changes, and person availability impact the ability to kick off the process Humans often miss key values or make data entry mistakes when choosing date ranges, account ranges, well identifiers,
Maximize Automation Read More »
Minimize Data Movement Data Inconsistencies – Every time data is moved it has the potential to be corrupted, dropped, or miss-aligned Reconciliation must be performed to ensure all records made it from source to target and all data returns the same results regardless of which copy is used. Increased Latency – Data takes time to
Minimize Data Movement Read More »
Minimize Data Duplication Each copy can become a data silo Each copy consumes resources – Storage, Compute, and Memory Each copy must be kept in sync with the data source / system of record Schema Changes and Key Changes must be accounted for in all copies Each copy must be secured, monitored, and governed Multiple
Minimize Data Duplication Read More »
Data Architect Guiding Principles What I rely on when evaluating Data Architectures and Solutions Minimize Data Duplication Minimize Data Movement Maximize Automation I arrived at these three guiding principles over many years of developing and supporting data architectures in a variety of industries and platforms. Experience taught me that for most situations simpler is better
Data Architect Guiding Principles Read More »
