Blog de Joe Freeman

We use performance tests to verify the raw throughput of some subsystems and to verify the overall impact some subsystem has on an entire ecosystem.  Load tests act as documentation for performance indicators and re-enforce performance expectations. They are vital in identifying performance regression. Load and performance tests are an often overlooked part of the software release lifecycle.  Load tests, at their most basic level, are about stress testing a system by dropping a lot of work onto it. Sometimes it is a percentage of expected load, other times it is the expected load, and other times it is future expected levels of load.  A failure to test  expected  near-term load can lead to spectacular public failures. Video Measurements  Your  business requirements  determine requirements for throughput, latency. Your  financial requirements impact the cho...

This article originally discussed "Sliding Windows" but actually refers to a variant called "Tumbling Windows" The first step to using ML for intrusion analysis detection is the creation of Features that can be used in training and detection.  I write in  another blog  about creating features from tumbling windows bound aggregates of packet streams. Inbound packets are analyzed and then grouped with other packets that happen near each other.  We can walk through the steps of   GitHub repository   contains Python code that creates features from Wireshark/tshark packet streams. The program accepts live tshark output or tshark streams generated from captured .pcap files.  Network Traffic into Tumbling Windows The example program requires Python and Wireshark/tshark.  The Python code uses 4 multiprocess tasks making this essentially a 5 core process.  It is a 100% CPU bound on a 4 core machine so I suspect it ...

This article originally used the term "Sliding Time Window".  This article actually discusses a variant called the "Tumbling Time Window" Feature creation is one of the first steps toward creating Machine Models that apply to network monitoring or other stream-oriented data processes.  We massage independent variables into a form that can be used by ML models or other statistical tools. This often involves transforming source data through numerical conversion, bucketing, aggregation, and other techniques. For this project, we'd like to try and train a machine model to detect intrusion events by having it look at network traffic. People sometimes try and  directly consume events  as inputs. An individual network packet does not contain enough context to be useful on its own. A Tumbling time window makes it possible to create features with more context than you would get with a single message. This GitHub repository contains Pyth...