Streaming is Micro-Batching when feeding a Data Lake
Streaming data in Data Lakes is one of those ideas that doesn't work exactly as you expect it to. Your pretty big data streams are converted sent into your data lake by old school batch process.
The primary reason for this is the way that data is stored in the cloud in non-append-able file/blob storage. Data files cannot have data appended to them. New files/objects must continually be generated for any kind of producer performance.
There are other issues that may drive a project to total old school batch style ingestion. This often happens due to the batch oriented nature of data producers, the need to store data in exactly the same format , the need to store data in exactly the same order as it was presented to the lake.Streaming Data
Streaming sends individual documents or records to interested parties. Those streaming listeners then write the data to their appropriate storage. Messages are often limited in size due to streaming and streaming storage restrictions. Uniqueness …
The primary reason for this is the way that data is stored in the cloud in non-append-able file/blob storage. Data files cannot have data appended to them. New files/objects must continually be generated for any kind of producer performance.
There are other issues that may drive a project to total old school batch style ingestion. This often happens due to the batch oriented nature of data producers, the need to store data in exactly the same format , the need to store data in exactly the same order as it was presented to the lake.Streaming Data
Streaming sends individual documents or records to interested parties. Those streaming listeners then write the data to their appropriate storage. Messages are often limited in size due to streaming and streaming storage restrictions. Uniqueness …
