Cloud Lake Storage - Files vs Tables

The cost and scale of cloud storage makes it possible to store large amounts of data in almost any format.  We can build abstractions on top of this simple cheap highly available storage that lets us access it as if it were something more sophisticated.

Let's divide the data into two coarse types, unstructured and structured.

 Unstructured data, in this case, is any data where the content is un-typed, is not easily machine parse-able or does not store data with a tight schema.  Sound, media, pictures and PDF documents are examples of unstructured data for this discussion.

Structured data has a deterministic content format and if often designed for machine consumption.  Let us break structured into columnar-record and non-columnar record formats.  Most columnar record formats are designed for many records per data object.  Non-columnar formats may be single record per file or may not be easily flatten able into record format. We ill ignore them for the tabular discussion

TypeStructureMulti-object ParseableUpdate-able
UnstructuredIrregularn/aNoNo
StructuredDocument or GraphMaybeYesNo
StructuredColumnar Record YesYesYes

Cloud storage is organized as a giant blob-space of objects where every object has its' own unique identifier. It is a global file system, except different.

Cloud Storage Paradigms


Cloud Data analytics operates on huge amounts of data most often arraigned in row and column format.  Big Data compute treats those square entities as data tables that are joined and filtered en-masse as if the data store was a relational database.  Many organizations formalize this Data Warehouse style treatment of lake data. 

 In general, cloud storage does not support in-place partial updates.  Object portions cannot be replaced.  Updates are done by replacing the entire cloud copy of the object.  Big Data works around this by mapping virtual tables onto many files organized in a specific way.  Each file contains some subset of the total records. Big data tooling like Spark or Presto understand this and can transparently merge large numbers of files into high performance virtual tables.  They actually leverage this storage in order to execute highly parallel operations where each process executes against some subset of the objects that make up pseudo-table.





Big Data Cloud tables are broken up into partitions.  Each partition can contain many objects (files) that in union make up all the data for that partition. This makes it possible to easily add data to slow changing tables by adding new files into the data's partition.

Same Data - Multiple Formats

Cloud storage is relatively cheap.  Individual companies can store 10s or even 100s of Petabytes without worrying about running out of space. This is more data than the sum of all global storage 25 years ago.  

Cheap storage means we can retain multiple copies of data in different formats optimized by consumption or regulatory needs. CSV files can be stored as objects in CSV format and as virtual partitioned tables in something table friendly formats like ORC or Parquet.

Wrap Up

Cloud based, storage, maps to objects, tables or other types based on the data mapping tier.   This makes it possible to turn immutable objects into mutable (insert) tables.  You can treat your data either one way or the other or both.    

Video


Modification Log

Created 4/2020

Comments

Post a Comment

Popular posts from this blog

Installing the RNDIS driver on Windows 11 to use USB Raspberry Pi as network attached

Image
I do a lot of my development and configuration via ssh into my Raspberry Pi Zero over the RNDIS connection. Some models of the Raspberry PIs can be configured with  gadget drivers  that let the Raspberry pi emulate different devices when plugged into computers via USB. My favorite gadget  is the network profile that makes a Raspberry Pi look like an RNDIS-attached network device.  All types of network services travel over an RNDIS device without knowing it is a USB hardware connection. A Raspberry Pi shows up as a Remote NDIS (RNDIS) device when you plug the Pi into a PC or Mac via a USB cable. The gadget  in the Windows Device Manager picture shows this RNDIS Gadget connectivity between a Windows machine and a Raspberry Pi. The Problem Windows 11 and Windows 10 no longer auto-installs the RNDIS driver that makes magic happen. Windows recognizes that the Raspberry Pi is some type of generic USB COM device. Manually running W indows Update  or  Update Driver  does  not  install the RNDI
Read more

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Image
The Windows Subsystem for Linux operates as a virtual machine that can dynamically grow the amount of RAM to a maximum set at startup time.  Microsoft sets a default maximum RAM available to 50% of the physical memory and a swap-space that is 1/4 of the maximum WSL RAM.  You can scale those numbers up or down to allocate more or less RAM to the Linux instance. The first drawing shows the default WSL memory and swap space sizing. The images below show a developer machine that is running a dev environment in WSL2 and Docker Desktop.  Docker Desktop has two of its own WSL modules that need to be accounted for.  You can see that the memory would actually be oversubscribed, 3 x 50% if every VM used its maximum memory.  The actual amount of memory used is significantly smaller allowing every piece to fit. Click to Enlarge The second drawing shows the memory allocation on my 64GB laptop. WSL Linux defaults to a maximum RAM size of 5
Read more

Almost PaaS Document Parsing with Tika and AWS Elastic Beanstalk

Image
The Apache Tika project provides a  library capable of parsing and extracting data and meta data from over 1000 file types.  Tika is available as a single jar file that can be included inside applications or as a deployable jar file that runs Tika as a standalone service. This blog describes deploying the Tika jar as an auto-scale service in Amazon AWS Elastic Beanstalk.  I selected Elastic Beanstalk because it supports jar based deployments without any real Infrastructure configuration. Elastic Beanstalk auto-scale should take care of scaling up and down for for the number of requests you get. Tika parses documents and extracts their text completely in memory. Tika was deployed for this blog using EC2 t2.micro instances available in the AWS free tier. t2.micro VMs are 1GB which means that you are restricted in document complexity and size. You would size your instances appropriately for your largest documents.   Preconditions An AWS account. AWS access id and secret key.
Read more