Blog de Joe Freeman

Remote works often connect to remote desktops over Cisco AnyConnect VPN links.  Those remote connections secure the remote corporate network by forcing all network traffic over the VPN connection.   You free up your computer to access the internet and other services by making your remote desktop connection from a Virtual Machine.  The cheapest way to do this on a Windows 10 machine is to run a Windows 10 guest  inside Microsoft Hyper-V.  You connect to the guest machine via Remote Desktop.  This leaves you with two desktops, the host  desktop which is still attached to the Internet and a guest  desktop connected soly to the corporate hosts via the Cisco AnyConnect VPN connection. Hyper-V Manager Remote Desktop This information originated  https://devhammer.net/2014/05/22/cisco-anyconnect-and-hyper-v-connect-to-a-vpn-from-inside-a-vm-session/ Hyper-V Manager supports two Remote Desktops.   Enhanced: This is the new and improved ...

The only way to protect sensitive information is to remove the sensitive values everywhere they are not absolutely needed. Data designers can remove the fields completely or change the field values so that they are useless in the case of data theft.  Data tokenization and Data encryption are two possible solutions to this issue.  Both approaches must be implemented in a way that they return the same non-PII value for a given PII value every time they are invoked. We're going to talk about tokenization here. Tokenized field values must be changed in a repeatable way so that the attributes still be useful for joining data in queries or reports. This means every data set with the same value for the same PII field will have the same replaced value.  This lets us retain the ability to join across datasets or tables using sensitive data fields.  Every PII field has a typecode or a key.  That type is used whenever...

The only way to protect PII is to remove it from your Lake or other Analytical Stores.  New regulations and laws create stiff penalties for data leaks and give consumers or customers right to know all the place their data is used.  We want to remove PII to meet new regulations while still retaining enough information to join across datasets. Recorded Talk Speakers Notes Speaker Notes not yet available. Speaker Notes not yet available. Speaker Notes not yet available. Speaker Notes not...

Streaming Data into a Lake is a powerful tool, modern, approach that replaces traditional ETL.  There are use cases where streaming  can make things difficult for business systems or direct data users. Your data ingestion tier may have to support both data streaming  and bulk processing. Speaker's Notes Notes to be added Notes to be added Notes to be added Notes to be added Notes to be added Notes to be added Notes to be added

There are other issues that may drive a project to total old school batch style ingestion. This often happens due to the batch oriented nature of data producers,  the need to store data in  exactly the  same format  , the need to store data in  exactly the same order  as it was presented to the lake. Streaming Data  Streaming  sends individual documents or records to interested parties. Those streaming listeners then write the data to their appropriate storage. Messages are often limited in size due to streaming and streaming storage restrictions.  The goal is to use our common data streams to update various consumers and populate our Data Lake and Data Warehouse. This diagram shows three consumers, one of which is used to write data to the Data Lake. Uniqueness Not Guaranteed Messages/data may be presented to the consumers or to the lake multiple times.  This can happen because of failover events in the streaming system, consumers ...

Streaming data in Data Lakes is one of those ideas that doesn't work exactly as you expect it to.   Your pretty data streams are packed up and sent into your data lake by old school batch process.  The primary reason for this is the way that data is stored in the cloud in non-append-able file/blob storage. Data files cannot have data appended to them . New files/objects must continually be generated for any kind of producer performance.  Streams are Micro Batched Groups of records are written as a single operation to cloud storage.  Data lake writers must batch up  streaming data in order to write it to cloud storage. Writing individual records is inefficient and may not even be possible depending on the run rate of the message stream. Files in the Data Lake in the same dataset may be different sizes based on the batch writing trigger definitions. ...

Data immutability was one of the core tenants of Data Lakes when they first became big.  Mutable data went to Relational and Document databases while immutable data and and documents were store in the lake .   Emerging privacy regulations and data sharing regulations are adding data retention, data visibility and data management rules and behaviors that may drive companies to re-think which data should be stored and how data should be stored in data lakes .  Video blog Phase 1: Data Set Storage Retention Retention times are are set on the file(s) that make up a dataset. Datasets are managed as files. Entire datasets are removed at the end of the retention period. Phase 2a: Partition Storage Retention Retention times are stored somewhere and bound to partition keys.  Data is organized as tables in a a table/partition/file format.  P...

Azure PaaS services are an example of how the cloud has raised the raised the bar in the commodity platform space. More functionality is baked into the platform and less has to be built by software developers and enterprises. Some of the PaaS tenants are that it should be simple to scale-up and scale-out.  Networking should be simple and port exposure should be simple. Microsoft has at least 3 different Container services with different levels of PaaS-ness. They appear to be targeted more at Enterprise customers than the original SMB oriented PaaS services. It may be that the original ones just didn't support enterprise security, networking and other needs. Tech Comments The demonstration deploys a Tika Parser, Java Docker container running on each Azure Linux Docker PaaS services. Microsoft's move towards explicitly containerized PaaS services has both improved and degraded this model. Sample Scripts Use these scripts to experiment with Java Linux deployments on Azure ...

Agile can really burn people out with a never ending stream of sprints  or iterations .  One way to keep people fresh without breaking the team is to rotate different people though the lead roles on a Program Increment / Planning basis.  This gives people the opportunity to both reach  and recharge . Program Level We had had great success on a project over a 2 year period where the tech lead / for the Program Incremented rotated between three people.  This meant they could each do technical work 2/3 of the time while driving a PI the other 1/3.  It also made it possible for them to do some pre-planning while not trying to manage the current PI. Team Level This can also be done on a per sprint basis for intra-team responsibilities.  Story Grooming or product/technical preparation roles can be rotated every sprint. Video ...

What does it mean to move from a Relational Database style Schema on Read to  Schema on Write ? Schema on Write  is used to stage data in a consumer friendly form.  It can also be used in poor-join-performance environments to restructure and stage data in consumer read  format. It is pretty much mandatory for Document Databases.  Ingestion stores data in its original format for compliance, audit or other purposes.  This copy may be called True Source. Format Standardization converts the raw information into and agreed on standard format.  Examples include  Data Tables  in a lake or documents in a document store.  This is purely a mechanical conversion. Consumption Model are built from raw data, reference data and applies view and business rules creating a consumer ready dataset...