PlaidSheep

The Evils of Sequential IDs

2024-01-18T00:00:00Z

Sequential IDs have several re-occurring risks that are known
We will discuss 3 risks associated with Sequential IDs as Primary Keys
Some valid cases for maintaining Sequential numbers are discussed

Why do we find sequential numbers comforting?

False sense of utility

One argument that is often raised in favour of sequential IDs is the “familiarity” of numbers that are in order. This is expressed as

We have a sense of how many records there are
It gives me a way to orient myself
Numbers are easier to read

What people are expressing is an emotional state: we are pattern-seeking monkeys and seeing ordinal growth gives us a sense of comfort. As hard as it is, the data is not there to satisfy your psychological comfort; rather we are experts who deal with the uncomfortable realities of data.

This puts us in a tough place. Humans are really good at post-hoc rationalisations. Any time we experience discomfort, we will reach for the comfortable, and justify it to ourselves in any way we can. Post-Hoc rationalisation is a problem we must each individually struggle with. It's been a while since I've read The Righteous Mind, but Haidt always nails this point.

We cannot allow ourselves to use what is comfortable, but rather to become comfortable with the most appropriate techniques. To do that we need to take time to consider the benefits.

If we are using a sequence of numbers as a primary key, we are using arbitrary keys. These are not for human consumption, and therefore they should not make your reading easier, they should make the computer's reading easier, and hopefully our maintenance.

A key touch-point in Software development, expressed mostly from the perspective of Functional Programming, is to avoid side effects. Each command, each function, should do one thing, and should not be used to affect the system beyond what was specified.

In computer science, an operation or expression is said to have a side effect if it has any observable effect other than its primary effect of reading the value of its arguments and returning a value to the invoker of the operation.

Wikipedia: Side Effect

When searching for the above link, Google gave me a sidebar that stated

Side effects are any changes in the state of the program or the environment that are not reflected in the function's output. Side effects can make the program unpredictable, hard to test, and difficult to debug.

I'll add to that, that we don't want to become dependent on side effects because, by definition, there is more than one effect coming from a single action. We now cannot change the action to accommodate one effect, because that would impact the other effect.

Using the Primary Key as a sequence? Now you can't change its data type to something else if you need to because that would make it non-sequential. Need to change the order? Not possible, it has foreign key references.

Our job, as experts, is not to define data in ways that make us comfortable; rather it is our job to define data as it is.

define whatever it is we perceive — to trace its outline — so we can see what it really is: its substance. Stripped bare. As a whole. Unmodified. And to call it by its name — the thing itself and its components … Nothing is so conducive to spiritual growth as this capacity for logical and accurate analysis of everything that happens to us.

— Meditations 3.11

Risks

Security

The two main examples from my personal experience draw on the security risk associated with sequential numbers. Numbers in sequence give us a pattern which we can use to predict the next, or previous, value.

Whether we meant to or not, by using sequential IDs, we have introduced information into our dataset.

In exposed applications that control information, this has been known to hint to people that there is more information available.

In Nova Scotia, in 2018, an individual using a government data transfer system noticed that when retrieving FOIP information releases online, the numbers were relatively small. He was able to surmise that the numbers were probably sequential and simply tried to type the next number into his browser. He was surprised when he got a FOIP record that did not belong to him. Had non-sequential values been used, his guess would likely have resulted in a miss. The large spaces with non-valid values would have been missed.

Clicking on the item's link brings the user to the item's page and puts the URL for that item, including its unique identification number, in the browser's address bar. Changing just the number in the address bar takes the user to a different item directly

— OIPC Investigation Report

While this type of risk is well known it remains common, but at least it is obvious. Unfortunately, there are far more insidious ways of data leaks occurring.

Sequential IDs are famously capable of being exploited statistically to gain estimates of total counts. This is known as The German Tank problem.

During WWII, it was necessary to identify how many tanks Germany was producing. Using the serial numbers off the wheels of destroyed tanks, analysts were able to estimate the total number of wheels (and therefore tanks) that were in operation.

wheels, which were observed to be sequentially numbered … Analysis of wheels from two tanks … yielded an estimate of 270 tanks produced in February 1944 … German records after the war showed production for the month of February 1944 was 276

— German Tank Problem, Wikipedia

Randomly selecting a sample and then looking at their sequence numbers gives us an idea of how dense the population is. Using one of several functions, we can estimate population (and also sub-population) size from a sample.

When we share large datasets for analysis, we often exclude records deemed to be outside the Data Scientist's domain of interest (often for privacy). Unfortunately, if we are sharing underlying sequential values we are exposing more information about the larger set than we intended. The customer can derive information about the unshared portion of the data.

The goal is to get data onto the distributed processors with as little fuss as possible.

While the OSF came up with a very interesting solution to this problem, random numbers and cryptographic hashes have become more popular in the intervening decades. Using extremely large random numbers, we can be reasonably assured that the number generated will be unique to the population; meaning there is no need to coordinate between systems. Each processor can independently assign identifiers without concerning itself with the work of other processors in the system. The results can later be merged as one step.

Hashes serve a similar purpose but have the benefit of being deterministic. It is possible to do processing on a unit and identify which unit it is without having to do expensive scanning on the larger set. If some feature of the data can be used as a unique identifier, hashes can be used to convert that feature into a reasonably sized and well-distributed integer, suitable for use as a key. Like random numbers, we can be reasonably assured that there will not be a conflicting number, but we also have the benefit of being able to integrate overlapping entities if they are then determined to represent the same thing.

Imbalanced Indexes

The law of threes dictates that I make three subpoints, and so I had to come up with a third reason not to use Sequential IDs; but this one is a little more of a stretch.

There is an argument that during partitioning for storage or processing, or for indexing of datasets, sequential IDs do not balance well in the binary tree. This first came up during a (now lost) lecture I watched by Jan (can't remember her last name) of IBM. She discussed why random keys were so important to database performance and asked audience members if their keys became imbalanced. This highlighted issues I remember from Project Gutenberg's data store in which they partition based on the first digit of the sequential number of the book submission.

Under these conditions, Benford's Law is going to burn us.

Benford's Law indicates that the first digit of numbers will tend to be smaller values (Wikicommons CC0)

Benford's Law states that often, the first digit of a number is low. In a sequential set, it is obvious why this would be true: every time you change scales (say 1's to 10's) it takes ten times longer to change the first digit (in other cases this remains a mystery to me).

For partitioning, this means that partition 1 is going to get 30% of the work, while partition 9 is only going to get 5% of the work; creating massive bottlenecks while one grinds away, and another sits idle.

While I have seen real-world examples of this, and do recognize the problem as a problem, I am generally dismissive of it. It is relatively simple to implement a fast, balanced hashing algorithm that resolves this problem.

Using the example from above, Benford's Law can be circumvented by simply reversing the numbers (effectively a Mod10 hash). So instead of the sequence [101, 102, 103, 104, 105] being sent to the same partition, we can instead use [101, 201, 301, 401, 501] to distribute the values more equitably.

Having come across articles on the hashing algorithms used in indexing by some of the big database engines, I would expect the balancing of data during partitioning to be a well-understood and handled problem in any platform I choose to adopt. If you have to balance the values yourself, these may be considerations you need to take, but looking at your chosen tools may be in order.

Valid Uses

As with any hard and fast rule … well … there is no such thing. There are exceptions and valid reasons for why you might want a sequential identifier.

Audits

Going back to my very early days in my career, when my colleagues and I were just trying to figure out how to turn a dollar, a few of us took to teaching “How to use Office” night classes at local community centres, job retraining centres, or (mine) crime exit programs; all of us had a module “Use Excel to Make an Invoice”.

One weekend, I was at one of these friends' houses, we were having a few beers and working on a hobby problem together when he mentioned a lesson he got from one of his students.

As part of his invoice class, he taught how to make invoice numbers automatically (last inv number + 1), but he told the class, “If you want to make your business look more impressive to your customers, change that formula to last invoice number + 10. Even if you only have one customer, it won't look that way.”

A hand, near the back slowly went up.

I am an accountant, and have to recommend people not do that. When you undergo a tax audit, they will identify the gaps as missing invoices and assume you have done 10x as much business as you have reported. They will estimate what those invoices are worth, and send you a bill for the apparently missing nine invoices

That's not something you want to have happen, but the ability to match patterns and find gaps, which is what poses the original risk, also has its benefits. The main reason I will reach for a sequential ID is for audit purposes: a sequential ID can be used to verify missing data by creating gaps in the sequence when data is removed.

Non-technical individuals often don't understand how malleable information is in a digital world, it is easier for them to comprehend that the number has incremented than it is to understand a chain of hashes, so adding a sequence ID for visiting auditors can be a courtesy.

Merkle Tree (Wikicommons — CC0)

Replication Checkpoints

One of the databases I like to use a lot is CouchDB, and one of its super-powers is its ability to replicate the data across multiple peers.

To achieve this, there are a few things that have to take place to address conflict resolution, change histories, and split brains; but one of the key elements is the simplest: every change to the database gets a sequence number.

This number is used during replication to act as a bookmark, allowing one database to request every change since a point in time.

This is an important tool to be used during replication, being able to state: “give me everything since checkpoint X”.

A similar construct is also available in MS SQL Server, called a rowversion (it was a timestamp last I used it); and I'm sure most other DBs have the same concept: a globally incrementing sequence.

Solutions

Given all of this, what should we do when constructing identifiers?

Big Integers

No matter what style of identifier you use, make sure you use really big integers.

In those early days of early web applications, the databases we used had auto-incrementing indexes that could have “random” specified as the sequence. However, they were constrained to a 32-bit integer: that's only room for about 4 billion records.

Not a lot by modern standards.

If you are using random numbers, you only derive the benefit of the gaps if they are reasonably large gaps. If you are using sequential values, you don't want to run out.

Random values, Hashes, and UUIDs

Generally, I suggest using (at least) a 128-bit integer for storage. I don't choose this number arbitrarily; yes it's large, but it is also the size of a UUID. This has the convenience of having a ready-to-use storage type on most systems as well as standardised functions for generating random (or near random) numbers of that size.

I say “near random” because while UUIDv4 is a random number, other versions of UUID are not. For example, version 1 was dependent on the computer's MAC address to form part of its uniqueness. Further, some implementations of UUID generators are not truly standards conformant, instead including timestamps in the number for convenience.

Hashes are another interesting option, while they are deterministic, they are effectively random. In the case that a dataset has a natural key that we do not necessarily want to share, but makes an obvious primary key, hashes can make a convenient way to mask the natural key. It generates a large, (effectively) random, integer. While an md5 will fit into a 128-bit integer, most modern hashes require more space.

One interesting benefit of using a hash occurs when you require scrambling of your keys. Through the use of a salt value (perhaps a salt value per customer), we can allow for simple regeneration of different random-like keys if we ever require them to change.

Generated with AI ∙ 17 January 2024 at 7:52 pm

Primary Keys are never to be sequential

There are reasons to use sequential numbers, but making a primary lookup value creates the temptation to expose it to the client, and the moment you have something that looks tempting, it will happen. Let me say that again:

Your primary key will get exposed to the customer.

There are all kinds of reasons we promise it won't, but at some point, somebody is going to make a mistake and expose that number. After all, it is the value we look at individual records by. If for no other reason than an analyst is going to want to do a join against two tables, that primary key is going to get exposed.

By using random-like values for PKs, there is less temptation to use them for inappropriate things. We can't derive information from them directly and therefore don't try to use them for secondary purposes. If we need to add sequential values, putting them in a field that explicitly defines why they exist, making their purpose (and non-purpose) clear.

Generated with AI ∙ 17 January 2024 at 7:46 pm

Non-primary, Unique Identifiers

Just keep an eye out for them.

Do you have a field called created_time?

If your data has a created timestamp on it, it has something that may be very close to a unique identifier. While this may not be sensitive information in itself, users may be able to use this to join data against other datasets.

While useful for diagnostic purposes, these low entropy values pose a risk when exposed to the wider customer group. By not having them as the primary key, we can simply remove these values from data shares. By having these unique values not acting as primary keys, we are not bound by their side effects and we are less tempted to share them inappropriately.

If these values are required for diagnostics, then we must maintain them. Our only defence is to be cognizant of the risks and to keep the values unbound from each other.

Conclusion

While there are valid reasons for maintaining sequential values in data, using sequential values as the primary key poses significant risks to data. Given the nature of primary identifiers, the cost of the risks versus the perceived benefits do not justify their use.

Separating valid sequential use cases from the primary identifier allows us to separate the concerns of the purpose of the data, allowing us to move parts independently of side effects. Very large integers have been widely available for over 30 years, and allow us to create large gaps of negative space in our data to evade malicious detection. Modern random number generation and cryptographic hashes offer a secure way to populate that large space.

Given these simple and well-established solutions, the risks to both security and performance posed by using sequential primary keys do not outweigh their perceived benefits.

As engineers and architects, we need to consider not the way we have always perceived data, but rather to look at the historic and scientific context that surrounds the protection of that data. We may find that the secure mechanisms of the past, no longer suit the computational realities of the present.

1928 US patent 1,657,411: Enigma Cypher Machine

How To Quantify Privacy in Datasets

2023-12-27T00:00:00Z

We will discuss tools for estimating and automating the enforcement of privacy in datasets.

Objective Data Privacy Evaluation: Discuss a straightforward approach using objective measures to gauge and improve data privacy in shared datasets.
Estimating Data Privacy Levels: Learn through real-world examples how estimating data privacy levels can be a powerful tool, minimizing bias and enhancing informed decision-making
Automated Checks for Enhanced Workflow: Explore the application of automated checks in the approval workflow to simplify processes and boost productivity in safeguarding sensitive information

Head over to ObservableHQ to see an interactive version of this.

Measuring Privacy Exposure
Jeff Cave `2023-12-15` to `2023-12-20` In my previous article, I highlighted the nuanced task of preserving privacy…observablehq.com

In a previous article, I delved into the intricate task of preserving privacy, underscoring the critical nature of cautious information sharing. This principle extends beyond safeguarding personal identities to shielding covert subjects like structures, military units, and other sensitive entities. The guiding rule is clear: the less revealed, the better.

Implementing this principle poses a formidable challenge. Over-sharing not only risks exposing individuals but also jeopardizes the confidentiality of various subjects. Assessing the “privacy” of shared datasets demands meticulous effort. Privacy analysts invest time scrutinizing datasets, applying rules, and leveraging professional judgment to ensure the concealment of both personal and classified subjects.

However, relying solely on subjective analysis has its inherent risks. Analysts, being human, are susceptible to biases, fatigue, and external pressures, occasionally leading to lapses in judgment. Despite these challenges, there's a growing need to share data for diverse benefits.

The predicament endures: how do we establish a threshold for sharing information without compromising the privacy of individuals or the secrecy of sensitive subjects? How can we alleviate the burden of the privacy evaluators while simultaneously ensuring that shared data doesn't pose excessive risks?

Enter Claude Shannon's Information Entropy concept, originating from “The Mathematical Theory of Communication” in 1949. Shannon's concept of the smallest piece of indivisible information provides a quantifiable measure for data. This measure is not only applicable to personal privacy but also to the confidentiality of secret subjects. It furnishes an objective metric to estimate the privacy risk associated with exposing a dataset, serving as a valuable tool for analysts and automated systems alike in assessing risks related to both personal and classified information.

These concepts present a practical tool for striking a balance between safeguarding the privacy of subjects and the imperative to share data for analysis.

Definitions

To ensure precision and clarity, we have chosen to substitute the term “individual” with the more inclusive term “subject.” While my main focus revolves around protecting people's privacy, it's crucial to recognize that certain fields manage data of a covert nature that extends beyond human entities. Examples include “Tight Holes” in the Oil and Gas sector, clandestine police and military locations, or discreetly insured and transported tangible objects. Remarkably, the strategies for privacy protection can be universally applied, regardless of the nature of the subject.

Central to our exploration is the concept of “privacy,” which we define as the resistance to deducing the individual's identity. Think of it like the classic game of “Guess Who?” where your opponent uses provided data to guess the subject's identity and then uses a broader dataset to gather more information. Decreased privacy erodes when data boosts our confidence in identifying individuals, while increased privacy comes from a lower confidence level in distinguishing one individual from another.

It's crucial to emphasize that privacy isn't only preserved by obscuring the name or ID of a person. Even with fabricated labels for the subject, comprehensive knowledge about them can still lead to privacy breaches. As an analogy, personal privacy can be compromised even if you know everything about a subject but refer to them by a pseudonym, much like my limited knowledge about my neighbour, whom I simply recognize as “that lady next door.”

There are three significant roles in any informational message transfer: the sender, the receiver, and a potential interceptor. In the case of a dataset being shared, we find these three actors present. Privacy Analysts sit between our source data and filter it, acting like a sender, sending a sanitized message out. The intended recipient is a Data Analyst, whether an internal colleague or a member of the public. Lastly, the Data Analyst could be a malicious actor, either gaining access to the data nefariously or, more pertinently, using a permitted dataset in a nefarious way.

While “entropy” is often linked with the predictability of physical systems, it can be better understood as a measure of chaos. Information Entropy, similar to the concept of chaos, quantifies the level of “surprise” within a system. In a stable or predictable state, a system exhibits high entropy. Consequently, possessing complete knowledge about a subject leaves minimal room for surprise; conversely, knowing nothing allows for unexpected discoveries.

While Claude Shannon is renowned for the bit, there are several names and related measures that apply to the same concept of information entropy. The Shannon measure represents the smallest, indivisible unit of information, a binary state (true or false); it is synonymous with a bit (or “binary digit”). This concept builds on previous work defining the Hartley (Ralph Hartley, 1929), which uses a decimal base and can also be referred to as the dit, among other terms used by Turing, Good, and others.

An Example

To work through an explanation of enforcing privacy through an automated mechanism, an example is likely useful. Samples often help us orient ourselves to the task at hand. Naturally, in a discussion about privacy, we will want a dataset with individuals whose privacy we will want to protect. Still, we also want to make sure these individuals are not representative of anyone real. To achieve this, we have downloaded a sample customer dataset from Sling Academy, which gives us a list of individuals and some personal information about them.

I've also gone through a process of “cleaning” the data to make it a little more suitable for our purposes. Mostly by adding some randomized identifiers, but also by introducing some nulls and extraneous character removal.

The probability of guessing an individual by their gender depends on which gender is exposed (ObservableHQ)

This ability to measure predictability brings us close to Claude Shannon's definition of “entropy”, or surprise. Given the probability of guessing the person, we can also calculate the amount of surprise. By calculating the probability of randomly selecting an individual from within each category we can get an idea of how private the field is.

In our case, the selected field, gender, has 3 categories. By taking an average of their probabilities (0.0430), we get a general sense of its level of privacy.

We can repeat this for all fields, giving us a privacy profile for the dataset.

Measures of privacy for each field as a probability, Shannon, or Hart (ObservableHQ)

The privacy factor calculated for each field matches our intuition: perfectly unique IDs carry a very low privacy factor, and items likefirst_name are relatively anonymous. Having observed this, we can observe some non-intuitive findings such as the very low privacy associated with municipality. We also have a cautionary reminder as well: emails and phone numbers are very unique to people.

// Calculate the privacy profile of all the fields
PrivacyProfile = (function(){
return {
"hashed": null,
table: Object.entries(fields1).map(d=>{
let key = d[0];
d = d[1];
let total = d.total;

let categories = Object.values(d.values).length;
let privFactor = Object.values(d.values)
.map(freq=>1-(1/freq))
.reduce((a,d)=>a+d,0)
/categories
;
let shannon = Object.values(d.values)
.map(freq=>Math.log(freq)/Math.LN2)
.reduce((a,d)=>a+d,0)
/categories
;
let hart = Object.values(d.values)
.map(freq=>Math.log(freq)/Math.LN10)
.reduce((a,d)=>a+d,0)
/categories
;
return {
"Field": key,
"PrivacyFactor": privFactor,
"Shannons": shannon,
"Harts": hart,
"Probability": 1-privFactor,
"Values": total,
"Categories" : categories,
};
})
}
})();

Applying the Measure to whole datasets

If our goal is to retain privacy while sharing data, in a large-scale environment, we can use this as a tool. Rather than relying purely on the subjective judgement of people, we can aid their judgement with an objective measure. When a data analyst comes to the Privacy Team, looking for access to sensitive data, we can take a measure of the privacy level of their data request.

We can calculate the net privacy of a request by taking a cumulative product of the privacy factors of the individual fields being requested.

Intuitively, we can immediately see that the entire dataset offers little to no privacy; however, this can be more formally stated by taking the product of all the fields.

PrivacyProfile.table.reduce((a,d)=>{
return a * d.PrivacyFactor
},0);

With items like the SSN, some items are 0 privacy, and including these in the product results in a Net Privacy Factor of "no privacy" (hard zero: 0)

We should expect a Data Request to be only for the data required for the analysis at hand. Therefore the request should include a reduced set. To perform this calculation on demand, we can create a generic function

This generic function takes a list of variables that the analyst wants access to, calculates the cumulative score, and produces the requested dataset. This can then be used to make a request (R00001) for items we know are more generic fields.

R00001 = GenerateReqest(['gender','age','hobbies']);
// {req: ["gender", "age", "hobbies"], score: 0.8534004245932628}

Again, the results match our expectations. We know that gender, age and hobbies have a much higher Privacy Factor and should therefore be much safer. However, now that we have a clear measure, we can be more explicit and state that the cumulative product of the request has a Privacy Factor of 0.8534 .

Another Data Analyst may make an innocent request for data that surprises us.

R00002 = GenerateReqest(['state','municipality','registered']);
// {req: ['state','municipality','registered'], score: 0.006712926793301226}

Their intent appears to be a longitudinal study of where the organisation's successes and failures are. Intuitively, a city should be a reasonably anonymous item. Surprisingly — likely due to our small data size — this request shows a significantly lower Privacy Factor (0.0067).

This request should be looked at much more closely by the Privacy Team to determine if there is a problem, and if there are any further transforms that can be applied to the dataset that would reduce the risk profile of the supplied dataset.

Using the Thresholds in Automated Checks

There is no point in having this tool if we can't use it in our systems to make our lives easier. We have already discussed using it as a tool to assist in evaluation, but it is also possible to apply this as an automated check. By setting a predefined threshold, it is possible to set an automated check on our datasets to ensure that any requests are immediately rejected if they are below a predetermined value.

While there is no one threshold suitable to all cases, organizations can inspect sample requests to identify a suitable point at which automatic approval, automatic rejection, or calls for further inspection occur. Further, these thresholds can be applied at different points in the approval chain.

Creating the Request
Privacy Approval
Data generation

When the Data Analyst begins to put their request together, we can offer them an estimate of what the Privacy Factor of their request will be. As an inexpensive calculation, this can be done live, during the request process, through a web, or application, interface. This allows the customer to get early feedback about what they are requesting and offers them the opportunity to plan their justifications, or mitigating transforms that may reduce the risk profile. Assuming the Privacy Threshold is above a predetermined safe threshold, the request can be delivered immediately, with no further review.

Upon submission of a request requiring further review, Privacy Evaluators can have the privacy estimate available to them as a holistic estimate of the risks involved. This offers a tool to formulate their thoughts, it gives them a metric that allows them to focus on problem areas. This also has the benefit of reducing evaluator bias where one evaluator may personally have a different risk tolerance than another evaluator. This metric can form the basis for discussion among evaluators, and with the customer, to focus discussion on real (rather than perceived) issues.

Lastly, we can continue this process of evaluating results even as the customer performs their transforms. It is possible a dataset were to be approved for use, with the understanding that the trusted analyst will perform aggregation that will increase the Privacy Factor of the dataset, but how do we know that they succeeded? If the calculation is performed on a platform we control, we can use these same measures to evaluate the produced dataset. This evaluation can even go further, using a more complete evaluation of the privacy, rather than the estimate we used to get this far.

If their privacy measures are unsuccessful, and we control the platform, we can issue an error message indicating that the privacy threshold was not met.

Judging the appropriate threshold for any given system will be contextual. Privacy Analysts will need to evaluate where the cut-off should lie based on the openness of the shares, and on the nature of the data itself. Secure data managed in secure facilities will require fewer checks than data being published openly on the web.

Conclusion

The ability to measure the anonymity of data before any requests for access offers several advantages to us in managing Data Repositories. Claude Shannon's concept of Information Entropy gives us just such a mechanism, by measuring the amount of entropy in a potential channel (shared dataset).

Using this estimate, combined with an evaluation of appropriate thresholds, it is possible to partially automate the approval mechanism for data shares. While complete automation is not possible, we are still able to reduce the burden on Privacy experts, as well as use these objective measures to reduce the impact of any personal biases the experts may have.

We all recognize the importance of maintaining the privacy of people, and secure assets, in shared datasets. These processes and techniques are useful in helping to reduce costs through objective techniques.

Going a Little Further

While this was an interesting idea to pursue, there are so many more layers that weren't covered.

Privacy Weights: every field should have a manually set weight associated with it. Immediately, this can be used to include values that look unique but which will be made available with other protections. For example, a Unique ID should be transformed for every request, so the privacy factor is likely known beforehand but is not represented by the actual data. Having some manually set adjustments can compensate for that.
Anonymity Functions: Several very simple and automatically applicable functions could be included in the initial request. Rounding all dollar figures to the nearest thousandth, or including only the first three letters of a name will drastically change the privacy factor.
The Thing I Forgot: I'm sure there are a million little inclusions you could make to a Data Request that would allow for more refined automation and control. Feel free to drop them in the comments.

Maintaining Privacy in Data Shares

2023-12-18T00:00:00Z

Balancing the power of modern computational capabilities with individual privacy is a formidable challenge in data sharing.
Benevolent actors, in their quest for insights, may inadvertently breach privacy by applying multiple dimensions to data, exposing individuals.
Introducing a privacy metric to mechanically measure the risk associated with shared datasets, aiding in decision-making for responsible data sharing.

[Wikicommons — CCSA]

When we delve into the intricacies of data sharing, one paramount consideration arises — privacy. The very essence of privacy hinges on how informative the shared data is, necessitating a quantifiable approach.

In today's landscape of information gathering, the sheer capacity often leaves one in awe. Privacy and data ethics, perennially debated topics, trace their roots back 250 years to the US Constitution's Fourth Amendment, asserting the right to be secure in personal papers against unreasonable searches and seizures. This is just one significant example of recognition of the perils of exposing personal information to authorities.

Long suspected, a revelation in 2013 confirmed governmental organizations, such as the NSA's, information capabilities to the forefront. The revelation that the NSA accumulates every phone call from every individual was not just mind-boggling; it raised significant questions about privacy and data ethics. The sheer scale of data collection raised concerns about the potential misuse and the safeguards in place.

We're not talking about datasets reminiscent of the Access Databases of the 1990s for mailing lists; these are colossal databases maintained by governments and mega-corporations. They harbour the potential to craft exact profiles of individuals. The sheer volume of data generated by individuals, when amalgamated, provides a unique perspective on the individual.

ASIDE

K68mPpQOmdxPfzQtgikKaw==

My unique computer identifier on the internet.

This value is generated from my browser settings and allows my online activity to be uniquely tracked across multiple websites, and without the assistance of things like cookies.

What's your fingerprint?

How do we balance the power of modern computational capabilities, with the privacy of the individuals whose data we collect?

The Road to Hell is Paved with Good Intentions

There are many layers to security, but often the first that should be applied is to guard against misuse: legitimate users, using the data for inappropriate purposes.

In movies, a private investigator will be seen buying information from an informant; super-spies are seen breaking into vaults to steal information about an individual. These are not complete fabrications; over the years, I have been involved in several investigations regarding leaks of personal information. I have been involved in at least two cases of data theft (both as a Nurse and as a Data Professional), in both cases private investigator hired an insider to look up information in the system. In both cases, gathering evidence to trace the activity was as simple as looking up data access logs that did not align with business duties, but did align with the suspected misuse. The mechanics of the system were sufficient to restrict, identify, and enforce access to data.

Data Warehouses and modern analytics add a wrinkle to this problem.

Data Warehouses can be a very powerful information resource. Depending on your business line, they will contain all the joined data of all personal information of customers and employees across multiple business lines. For governmental organizations like the NSA, the customer data points are the citizenry and visitors to the country. This is a very powerful tool for analysis, however, it carries with it risks for misuse.

Mechanically, the nature of the bulk data that is being shared makes it vulnerable to misuse by benign, malicious, and (most importantly) benevolent actors.

Keeping malicious actors out is obvious: do security background checks to find honest people, hire honest people, and create guidelines for use that honest people can follow.
Benign actors are … well … benign. They are the honest people you hire and are happy to follow corporate policies like: “You must not look up yourself, your friends, or your family”.
Benevolent actors are more complicated.

If we have screened for honest people, we have probably biased our search for helpful people. Combine that with large datasets, and the fact that Data Analysts are a curious lot, and we have a recipe for disaster.

To derive value from our large datasets, we have to perform analysis on large data sets. The benefit is derived from performing aggregate analysis on many detailed values. If an organisation has sensitive data, but an honest analyst who can offer some significant insight by inspecting the data, they will make the data available to the analyst. This generally takes the form of the analyst proposing their study, describing their data needs, and then sending them the data that aligns with their request.

… and this is where it starts to fall apart …

A perfectly reasonable request for data may exist within the United States Census Bureau to create a report that identifies a wage gap between the genders.

Gender Pay Gap Map #CensusDataViz #CensusData
Gender Pay Gap Map #CensusDataViz #CensusDatapublic.tableau.com

To support the request, the analyst is given access to a dataset containing all tax filings for the past dozen years. The analyst then loads the data into the analysis tool of their choice, does a simple sum by state and sees their results.

select
  year,
  state,
  sum(case when gender = 'M' then income else 0 end)/sum(case when gender = 'M' then 1 else 0 end) as avgM,
  sum(case when gender = 'W' then income else 0 end)/sum(case when gender = 'W' then 1 else 0 end) as avgW
  sum(income)/count(*) as avg
from
  dataset
group by
  state, year

Their manager approves hitting the publish button, and the whole team heads out for lunch together.

Over lunch, the original author is discussing their findings with their colleagues, when someone asks a simple question:

I wonder if age has anything to do with that?

That's an interesting question and may be a useful thing to add to future reports. So the analyst goes back to their favourite tool, the data is still in the analysis tool, and all they have to do is adjust the parameters on the query…. and this is where informational security starts to break down. This is not what the data was authorised to be used for.

While this is a benign example, each dimension of data brings us a little closer to revealing the individual. In their rush to discover meaningful insights, the analyst has applied two dimensions to the individual in question. While this may not be a big deal at the state level, imagine applying filters like this to the town of Albertville which has a population of 86. Suddenly using age may be unique enough to distinguish some individuals, and publishing their wages to their neighbours could cause some bad blood in town.

Security (privacy) of customer data is of paramount importance and requires a means of measuring the information about the individual we are sharing. This concept of revealing the individual effectively that of information entropy. Entropy is a measure of the amount of surprise, and in this case, it is the amount of surprise we have when we discover the actual person.

Aside from purely mechanical safeguards like network access controls, encryption, and permission, we need to consider the ability for shared data to be used inappropriately. The ultimate security tool is to simply not share data we do not want people to have access to, or (more importantly) to control the context in which the data is interpreted. This allows us to maintain a high level of entropy around the individual.

Every time someone asks for access, we need to evaluate whether we are exposing enough information to expose the individual.

This requires human thought and analysis, and humans make mistakes. Overworked, over-tired, or pressured by the office bully, a human may give permission to expose more data than is appropriate, allowing researchers to dox an individual accidently. Some mechanism for mechanically, automatically, and unbiasedly measuring the privacy risk of a proposed dataset is necessary because…

Hell truly is paved with good intentions

Hell's Gate (Darvas Crater) was created when someone threw a match in to stop a gas leak Darvas Crater [Wikicommons, CCA]

The Problem

To best understand the problem, let us consider a simplified example

Studying this dataset we can see it is a basic set of income data for US citizens. It includes some demographic information like their name and gender, as well as some contact information. While it does have a random identifier, it also contains their Social Security Number, something we do not want to just hand out to the first Private Investigator who asks for it.

Our goal is to give information to customers (analysts) who request it, but also to ensure we do not give so much that we end up exposing the individual. We can define this as

the information given to the analyst must not be sufficient to identify a single individual

The very first step is to remove the account identifiers, we don't want those being handed out, but the rest of the data is not so clear. What data does the Analyst need to satisfy their research needs?

Our customer is studying income, so we must include that, but the concern would be that it gets exposed. While the individual might recognise their income, assuming they've kept that private, it should stay private and not be associated with them.

Let's start by considering the email address. Email addresses are designed to be unique to the individual if we give the dataset to our customer with nothing but an email address and an income we have effectively tied the income to that person. On the other hand, if we expose the country, it does not tell us anything unique about the individual (everyone in our dataset is from the USA). So if we are going to share data, we want to share data we want to share data with minimal impact.

Our analyst studying gender inequality (Alice) is doing pretty well, having only asked for Gender and income.

Another analyst, a couple of desks over (Bob), has been working on an algorithm for a while and thinks he has a way to use Last Name as a proxy for race. He would like to do a study using names and incomes. The Privacy analyst, on the ball, notices that names are perfectly unique so offers a list containing only last names.

Alice and Bob are discussing their findings over lunch one day when they are overheard by their co-worker Eve. Eve's ears perk up because she is also a Private investigator and is always looking for exciting datasets and surreptitiously “acquires” the two datasets.

Eve knows she has acquired something useless in itself: the two datasets have been vetted to ensure that the individuals involved cannot be uniquely identified. However, with closer inspection, she notes that the incomes are unique. Using this insight, she joins the two tables.

Eve has achieved an interesting effect, she has not only joined the two original datasets to get a more complete picture, she has constructed likely salutations that are new data that exceed the scope of either of the original datasets. While it is not a complete picture, she has started to build a profile on individuals. These profiles can be used for purposes that exceed the original permitted use of the data.

A Real Risk

While this story is obviously made up, this is not an unrealistic story. If one looks at the way we package modern reports we can see an element of these risks being present all around us.

Modern data presentations demand some level of interactivity. The ability to filter, change, and compare the data on the fly is a very powerful and compelling tool. But there is a risk. To share these visualisations and make them dynamic, we have to ensure that there is sufficient detail embedded in the data we are sharing.

That data is in the file, and just because you don't know how to extract it, doesn't mean nobody does. For the tool to be useful, some people do know how … that's how they make the visualisations work.

Like Alice and Bob, we act with the best of intentions but easily become Eve sharing data inappropriately if we aren't careful. We end up sharing data that carries too little entropy and simply hide it behind a mask.

Screenshot of a dynamic visualization with embedded data [Tableau Public]

What Can We Do?

For people concerned with individual privacy, this is bad. So the question becomes, as custodians, how can we prevent it?

While human thought and analysis will always be necessary, it is subject to bias, inconsistency, and mistakes. Automating decision-making, or offering automated decision-making aids to humans is always a good idea. In light of this need, can we develop a metric that can be used to measure the level of privacy in our shared datasets? Can this metric be used to aid in the decision-making process around approvals of data shares?

Increase the Entropy

There are a few normal practices we can take to increase the entropy of the data. In our example, the two innocent datasets were joined through the use of income which was unique.

Originally, income was allowed because it was necessary, but that is what created the vulnerability. We can increase the entropy of the field by simply rounding it to some level.

Do we need it to be accurate to the dollar? What about the thousandth of a dollar?

In doing so we increase the number of people that will match the value and protect their anonymity.

Measure the Entropy

If we can measure the entropy, we should. Rather than leaving it to custodians to use their best judgment, we can offer them a way to objectively measure the state.

This metric can be made visible to both requesters and approvers to help them decide the appropriateness of the request. We can estimate the entropy of the request before it is even approved allowing us to keep safety at the forefront of our mind. Later we can measure the entropy of the request to ensure it is sufficiently anonymized before releasing it to the public.

Create Versatile Environments

Don't dictate, cooperate.

One of the risks mentioned early in our example was the step of the Analysts holding on to the data. We produce the “safe” dataset and then make it available to the Analyst for loading into their tool of choice.

This is a common story: we have powerful computers, we have tools we've trained on. These tools have power, but they carry the risk of moving data off of the controlled environments.

By creating powerful and versatile environments we allow the customers to do their analysis in a controlled environment. We should accommodate the needs of experts and create environments that are capable of accommodating those expert's diverse needs.

This passively discourages requests to take the data off-system by giving them access to the tools they want.

Dangers of the Habit of Ignoring Developer Error Messages

2023-07-11T00:00:00Z

I am going to give multiple examples from my career, demonstrating the dangers of error message fatigue and habituation leading to ignoring vital signals. Further, they will show how easy it is for humans to fall prey to habituation. Finally, I will conclude with specific techniques and modern tools that can be used to reduce the frequency of its occurrence.

44 thousand messages. I wonder if one is relevant (Original Content)

While I don't remember the exact defect, what I do remember was following the defect into the Web Server's logs (Windows Event Viewer). Previously, my focus had been on the system's external user behaviour, but here in the logs were thousands upon thousands of lines of warnings … every minute. Warnings about uninitialised variables, unsafe typecasts, and … well a little bit of everything. I had never really given much thought to it because it was so overwhelming as to be meaningless, but when I managed to isolate an interaction with the form in question and filter the logs down to just that time frame, I was able to consistently see one warning that seemed to relate.

I'd found my needle in the haystack, but it wasn't an error it was just a warning amounts thousands of warnings I had been ignoring. As I dug through the logs, I could see this message appeared regularly when that form was accessed (not always but regularly) and it went back to the very founding of the system (before I had even started my programming education, I was still a nurse). This error had been getting reported for almost a decade, and nobody had seen it for two reasons:

it was a warning, not an error
the signal had been lost in a sea of noise

I learned a valuable lesson that day. “No Warnings, No Errors” became my mantra.

Naturally, I first fixed the issue that had been spotted, but this defect had been signalled by a warning in the system logs that if someone had addressed we would have fixed it almost a decade before. So I started to address all the warnings in the logs.

Most were relatively benign, identifying (perhaps) that a variable had not been explicitly initialised before use, but since null was treated as a zero or empty string, it didn't impact the behaviour of the system. But as I made minor corrections, and the log volume got reduced, some of the warnings started to take on more ominous tones. More system defects were identified (and corrected), and most significantly to me, an actual error started to present regularly … One that had been missed in the excessive volume in the logs.

So what was the core of the lesson?

Seeing a large volume of errors can make us insensitive to them. When we ignore significant messages, we train ourselves to not pay attention, and that's when bad things happen.

(Wikimedia, CC)

The term “habituation” is used in several related contexts in the medical, social, and psychological contexts, but the general summarizing description would be the loss of recognition of negative stimuli due to repeated (habitual) exposure to the stimuli [Wikipedia].

We see this all around us and in our day-to-day lives. People get habituated to getting yelled at by a peer, becoming numb to the exposure. Physically, a carpenter may become desensitized to getting slivers, simply pulling them out at the end of the day rather than immediately flinching. I like a hot shower, but it usually takes a moment for my skin to get used to the hot water. It has even been documented in plants; the Mimosa Pudica is known to “flinch” when touched but will stop flinching with repeated touching.

This helps us get on with life.

Flinching is an important reflexive reaction to protect us from bad things happening. Stubbing your toe should produce an immediate “protect your toe” response, getting an unexpected cut on your hand is dangerous, and I should jerk my hand back from scalding water; but sometimes the cut is minor and expected (slivers) and for the most part just part of the job … life has to go on. A hot shower is a big temperature change, but it isn't harmful and is rather pleasant, once I get used to it. The process of habituation allows us to maintain our high-alert state, while at the same time learning to moderate it under various conditions.

Humans are biologically queued to become habituated. It is part of our survival strategy as a species. It's built into you.

Therefore you cannot ignore the risk habituation poses to our systems.

Receiving an error signal (error messages, warnings, failing tests) regularly, evaluating it as being “safe to ignore”, and not taking action, psychologically prepares you to ignore the signal at a later date. It begins to habituate you to the error signal, perhaps placing it on a pile of things we can ignore.

Ignore at your peril (Wikimedia, CC)

Common Examples of Programmer Error Habituation

At every organisation I have ever worked at, in every role I have filled, I have found examples of error habituation. They do not always present the same way, but they are pervasive throughout the industry, even presenting themselves as “Best Practices” to the untrained eye.

Errors and Warnings

This is the most obvious since it's right in the name, but that makes it a good place to start.

Compiler warnings
System Log Warnings
Pager notifications

This was my first exposure to this. We learn through practice at school that compiler errors prevent us from submitting our assignments, but warnings do not. With the short intensity of student life, ignoring warnings becomes a habitual survival strategy. As we become mature professionals, we learn that each of these messages was put in place to convey meaning to us, and offer us protection.

Known Software Defects

Defects in software are discovered, and discovering and correcting them is the art form. In the words of Robert Glass:

43. Maintenance is a solution, not a problem

— Facts and Fallacies of Software Engineering

It is important to realise that defects are to be expected and must be worked on. At the same time, there is always more work than time so some form of prioritisation is necessary. This means we must ignore them for a while (even if it is just the time it takes to fix them).

The problem is that the more defects we acknowledge are present, the more we tend to ignore them as irrelevant. The more we defer fixing bugs, the more we get into the habit of deferring bug fixes.

TODO and Change Comments

TODO comments within code were a way to identify an item that needs to be addressed; sort of a “come back to this later”.

There is a strong likelihood that we are ignoring the problem because we are busy with something else. Certainly, it is not possible to split ourselves into two to address both problems simultaneously, so making a note of the secondary problem while we address the primary one just makes sense.

The problem arises when we don't come back to it.

Accumulating TODO notes through code can become excessive noise to the point we start to ignore the message. Further, as these are usually listed along with the warnings and errors, they represent noise that drowns out more important signals.

Do not become habituated to seeing useless comments.

A header in an individual file containing a list of every change ever made to the file is a common pattern that has become an anti-pattern. The purpose of these comments is to offer a log of changes that have been made to the code in their contextual place.

Unfortunately, this (good) habit was started decades ago with coding styles that were different. The pattern is based on the assumption that a single file is self-contained to all its changes, and that it does not interact with other entities (since recognised as a faulty assumption). There is also the problem that decades of messages accumulating at the start of the file means there is an impenetrable <a href="https://jefferey-cave.medium.com/the-complexity-of-a-simple-chart-82ac4254e76d" target="_blank">WALL-OF-TEXT</a> that must be scrolled past before anything meaningful can begin. This immediate “scroll past” habituates us to perceive large blocks of comments as meaningless, when in fact we should consider large explanations in the code something that is important and meaningful.

What started as a good idea on small files, over short time-frames, has evolved into a bad idea with better alternatives.

Mimosa Pudica (Wikimedia, Public Domain)

Relearning the Lesson (Twice)

A decade later I found myself on contract with a major corporation that had terminated their previous contracting company due to poor quality performance. My team had been hired to not only deliver, but to do it with an eye to quality.

On my first day reading opening the regression test suite, I naturally glanced at the warning list to see how many warnings were listed in the code. I immediately found myself staring at a list of hundreds of warnings, but also thousands of TODO messages. Naturally, I tried to ignore them … they were things that needed to be done in the future, not immediately … but as I cleared the significant backlog of “warnings” I started to come across the actual locations that the TODOs were in.

It was horrifying.

@Test
public void ReallyImportantThing() {
//TODO: implement this
Assert.assertTrue(true);
}

In many (most) of the cases, the note suggested a person should implement the test for real. (There is a similar story, I thought told by Spolsky, of a notorious function implementation in MS Office … the same thing)

In my case, I suspect the previous team, under pressure to perform and deliver, had been masking gaps for a long time. Many of the regression tests were simple stubs that returned a success no matter what. This allowed them to claim the job was done, while promising themselves they would fix it … later … when they had time. That time never came.

It was a hard conversation with the client to explain I was taking a week to re-evaluate how much testing was actually being performed. When I reduced their test count by more than half, I needed to remind them they had hired us specifically because they knew there had been a problem and identifying those problems and giving honest assessments is where our value came from.

TODOs got added to my list of things that were not permitted in code bases I was involved in.

No Warnings, No Errors

Another decade has passed, and recently (weeks), I had to catch myself again.

On a new system we are working on, I implemented a basic continuous monitoring alert system. It periodically scans the system for invalid states and immediately notifies the team of the bad state (OK, it notifies me and one other, and we notify the larger team … baby steps). The idea is that if an alert is issued we must act to save the system.

I ignored a message.

In this case, the alert was to notify us that we had stopped receiving signals from a remote source, and I had ignored it. As a batch process, it is not uncommon for the source process to take longer than anticipated, this isn't a big deal since usually, it delivers shortly after we check and we just pick it up on the next pass.

Except it is a big deal because I ignored it.

My colleague, just returning from vacation, called me and asked if I had noticed that the system was erroring, she didn't see a ticket and wondered if I was dealing with it. I told her it was “no big deal, that one fails regularly” … and as the words came out of my mouth I heard what I had just said.

Sure enough, we looked closer, and the failure had occurred for three cycles; the source was not transmitting data and I, through habituation, had ignored the failure.

I hope I remember to come back and share my error habituation story in 2033.

See you in 2033 (Wikimedia, CC-0)

Preventing Habituation

There is really only one solution to preventing error habituation: address every error or warning or notice and treat it immediately, and with the highest priority.

While I say this there are some subtleties to how we achieve this:

Always fix defects before implementing new features
Never ignore a defect message

There are various tools at our disposal to help us address this and various mentalities.

(Wikimedia, Public Domain)

Errors that can be ignored

There is no such thing.

the system is in an invalid state and needs to be fixed immediately, or
the error notification system is flawed and needs to be fixed immediately

Incorrect notifications could be a log monitor that puts an alert out when an invalid state is encountered. Upon inspection, the state is determined to be “undesirable” (not “invalid”) … we'll ignore the error, it will correct itself later.

NO! Change the log monitor to take into account the new information. Maybe it needs to be run less frequently, maybe it needs to count how long the error state exists (waiting before alerting), but whatever gave you a reason to think it can be ignored, needs to be incorporated into the official rules for alerting.

Failing Tests

As previously mentioned, you can't be in two places working on two problems at the same time. One problem must be set aside while you focus on the other. Unfortunately, this leads to ignoring errors which becomes habitual.

To avoid this, the first step is to immediately create a task in your backlog, this immediately gives us a record that the issue exists. Secondly, we should immediately generate an automated test that can give us a way to reproduce the error. The problem here is that the test will be in a failing state, constantly reporting an error to us. This is a failure signal that we want to ignore (probably using a SKIP) until we get the defect fixed, we immediately mark it as skip, an ignore status.

This is a problem.

We can resolve this by having a team rule that all SKIP tests MUST have a ticket number associated with them, and also addressing every skip during every planning meeting. For me, this often takes the form of reporting skips without ticket numbers as fails, and fails must be addressed immediately. Skips with a ticket link directly to their ticket in their reporting.

Lastly, there MAY be a zero defects policy within the team. This is an agreement with the business that defects will be fixed before new features are implemented.

🗒 NOTE

I have never been satisfied with the reports generated by testing systems, and have always written custom visualisations to track defects. This has had the side effect of my introducing concepts like adding extra states to TestNG's default reporting (known, manual, feature), with active links to the repository and issue tracking software. I should really write an article showing my collection of testing reports … or show how to use various Project Management software reports (ADO, GitLab, GitHub).

A note to my boss … don't worry, these always get worked on in my own time ;)

Comments Calling for Action

TODO comments were a classic way to express something you need to come back to and finish something off, and they still have their place, but fundamentally, they are a call to ignore the problem (but only “for now”).

The problem here is the same as with all the others, we need a way to prevent “for now” from becoming “forever”.

One simple way of handling this is to put a Version Control hook in your repository that prevents check-ins of TODO comments. Generally, you should only put this on protected branches. This allows you to put them in your code to allow you to continue working but prevents you from submitting it to the official branch by accident. Forcing you to finish the job you planned on doing. If you can't get to a TODO for some reason, don't leave it in the code, register it in the backlog as something that still needs doing. This leaves the alert list available for things like warnings and errors so they don't get hidden.

Those massive headers at the beginning of the code only work to mask issues. They get in the way of text searches and require a lot of visual space to scroll past. All that is for something that is based on an old paradigm: changes are constrained to a single file.

Modern VCS tools assume that a change may require the context of several locations in code to be meaningful, and have logging built into them. Keep your changes in the Change System database, reducing the visual noise by placing the changes in a contextual list that is hidden until you need it. When you need it, the list is optimally indexed for what it is.

(Wikimedia, Public Domain)

Conclusion

As humans we make mistakes. Each decision we make is a totally new decision that we must make injecting the opportunity for error, this opportunity for error can be compounded by biases introduced from our experience. Habituation of errors represents a biasing of our behaviour that we are biologically predisposed toward, and can be dangerous to our work.

It is important, as professionals, that we work to overcome these dangerous biases through constant diligence and self-appraisal.

As software developers, our work captures decision-making in advance of the stimulus and action, and defects can have catastrophic effects. Teaching ourselves to ignore benign errors can mask more catastrophic issues that have significant effects on people's lives:

Aeroplanes fall out of the sky
Small business owners get falsely accused and imprisoned
Retail organisations are physically unable to sell their products

The Complexity of a Simple Chart

2023-02-13T00:00:00Z

I was recently involved in an effort to create an internal web presence for the internal service my team is working on. We want to take the time to advertise to our colleagues what work we do, what specific services we offer, and how they can take advantage of our services.

Most people see a lot of activity as a lot of work. Most people don't see what goes into keeping a design simple. I wanted to share all the work that goes into keeping it simple and informative.

For just a moment, I wanted to share just how deep the rabbit holes can go.

💡 TIP

Throughout you will see screenshots of the chart as it evolves, they are all links to a JSFiddle that shows the underlying code changes

https://jsfiddle.net/JeffereyCave/c4bLdox3/243/

In the introductory ITIL Foundations class taught to many organisations, one of the major things that is repetitively highlighted is the need for clear, transparent, communications with customers. In particular, the need for an "is it up" dashboard for your services always stood out for me.

The reasoning and benefits seem obvious to me:

Reduce labour by preemptively notifying customers:

The most important thing a person can be doing in the event of a service failure is repairing the service. Having customers, or managers, continuously asking if the service is available reduces the time spent understanding the problem. By notifying customers from a known, predetermined board, we are able to have them self-serve their questions, leaving technical experts to focus on solving the problem; by auto-generating it, we reduce the burden even further.
Don't disrupt people that don't care:

Signal-to-noise is a real problem in organisations. Rather than actively notifying, a pre-published board allows those with a concern or interest to look up the information. Still, for those not actively using the service (perhaps working with a non-impacted portion of the system, perhaps not in the office that day) there is no need to interrupt them, allowing those experts to focus on the problems they are solving while remaining blissfully unaware of other issues. This has the added benefit of not advertising your failures to people that weren't impacted.
Advertise your successful services:

A standing board of “status” not only shows failures but also a complete list of successes. Most of a board, most of the time, will show a healthy system; even when a portion of the system is failing, it will show that this failure is only impacting a small proportion of what is otherwise a successful service. Letting customers see that while it's bad, the service is generally reliable can help with this. This has the added benefit of letting customers know about other parts of your service. While your system is healthy, this is a complete menu of all the services you offer; either to be indexed by a search engine, to be pointed to during meetings, or to be naturally discovered by customers.

There is obvious value in having an online, automated, report of some kind; though, naturally, we see it taking different forms depending on the nature of the services involved. Power companies show outage maps, online services show “up time”, and data feeds show tables of API endpoints.

An example of a common status board used by any number of companies around the world

Initially, my team started by bringing forward various reports that individuals had already constructed for observing their individual aspects of the system, and proposing them as something worth sharing. These got presented, one after another, describing the benefits of each, and evaluating their use by our users. After myself and a couple of others had sown our most valuable reports, our manager asked, the question "How do people know its fit for use?"

What is "fit for use" in the context of our system? More importantly, how was the data that we had presented, that I had presented, not expressing that to her?

This question nagged at me through the rest of the day.

That evening, I opened Kate and started typing.

Aside: Defining the Problem

For a working example of the type of system we are dealing with, we need to start by defining the type of service we are offering.

We have a diverse variety of datasets that we ingest into a central repository. This data is standardised and then published back to our customers for use in their research. To give us a working example, let's take the flow of a couple of hobby projects of mine: imagine something like one of the financial websites that offer data about stock prices. This data needs to be collected from government filings (eg. SEC, SEDAR) and also trade data from the various exchanges (eg. NYSE, NASDAQ, TMX)

The general flow of the data from the primary source through to our consumer, whomever, or whatever, that may be

Data is ingested from the various sources I can find, standardised in their format, and then various views of that data are used for various analyses by users (human, reports, or AI).

Overwhelming the Viewer

Many years ago, I became enamoured with the idea of Test Driven Development. On the projects I led, I found it was an excellent way to define business requirements, and then communicate those requirements to a diverse group of individuals. Varying interpretations simply led to discussion and a very clearly defined expectation (updated test), which was then immediately distributed to the group (shared unit tests).

NUnit Gui is a handy and readily available interface for reporting business state to non-technical users.

Coming from this background, I tend to immediately develop a set of tests, in whatever test framework is available, for observing any system I am involved with, whether that's from a development, operational, or DevOps, perspective.

A wireframe of the basic test report that I live my life by (JSFiddle)

For 4 datasets, we run 3 checks, resulting in 12 rows. That's a lot of information being thrown at the user; the poor person is going to be faced with significant cognitive load, or as I like to call it: “THE WALL OF TEXT”.

The text used, while descriptive to me (monitoring the system) requires more context to be useful to someone. The numbers presented are very busy, of varying scales and unformatted, making them meaningless without thought. Placing the failures at the top was a good idea, but still requires a lot of thinking to determine what has been affected and whether we care (as a user).

We can either

engage in a training program (failure of intuitive design),
add more text (making THE WALL OF TEXT problem worse),
or maybe we now begin to understand why a simpler solution is needed

We did do a couple of things correctly, failures are at the top putting “interesting” results more prominently, there is an overall summary in the title, and the effective date is important. Unfortunately, there is a lot that needs work.

Starting Over

So this is where me and my text editor start over.

We are looking to create a chart that expresses whether “a dataset is fit for use”.

dataset (thing)
fit for use (state, boolean)

Stripping it down to just the barest bones (JSFiddle)

Users are only confronted with 4 elements (1 per dataset) and the amount of text to be interpreted. The text could use a little work, but a user will likely know what datasets concern them: users only interested in US stocks will know US market abbreviations and therefore that they don't care about the ones they don't know. It could probably be presented more aesthetically, but this is probably the minimum meaningful set.

Order and Limit

Given hundreds of datasets, it may not be feasible to show all of the items on a single pane. The chart should maintain visibility of the top 5 items with the option to scroll for more. It is estimated that a worst-case scenario of simultaneous expires is approximately 3 items so 5 allows for the human to see a naturally occurring comfortable group size, with some successful items at the end offering the knowledge that the end of the error list has been reached.

💡 TIP

Keep lists between 3 and 5 items.

While we are at it, instructions (lines of code?) should be between a half a dozen and dozen items. If you are metric, use fist-fulls (5 to 10 fingers).

Based on experience, and a few introductory Sociology and Psychology classes, this appears to be a natural human thinking scale.

States

There are different reasons a dataset could be "not fit for use", and some of those reasons a user may or may not care about

Bad structure
Incomplete set
Invalid content
Stale

A little further analysis offers a tip: if we can detect an error, don't give it to the users. Therefore we don't actually need to report bad structure since the bad structure will never end up in front of the user. Instead, we won't load the dataset, this will leave it “behind” or stale. So a “bad structure” isn't an error, it just never arrives (making it late).

Stale becomes our primary error state.

There is a third state worth mentioning. I hate mentioning an error while I'm fixing it. So I like to advertise that I am in the process of fixing it. A third state of “updating” is important.

Valid: Everything is OK
Refreshing: We are in the state of updating the data
Expired: It has exceeded its shelf-life
Error: Something is really wrong. Something we have never considered before

Order matters. We always want the most significant item near the top of the list. This allows people to focus on information that is important and ignore information that can be ignored.

(JSFiddle)

Iconography

Unfortunately, between our error messages and multiple states, we have achieved a WALL OF TEXT again. So much so that we had to add table lines just to make it legible. Any time you have tables of text, you have done something wrong.

Language is also a problem with any text. The internet is an international tool taking communications far and wide. Companies I have worked for have required me to accommodate Spanish, French, and Russian. Any time we reduce the text, we reduce the need for translation.

Our status' can easily be changed communicate via colour and icons.

Unfortunately, imagery can mean different things to different people especially when crossing cultures, it can also be expensive to purchase, and is subject to people's aesthetic opinions. Further, those with visual impairments may not be able to interpret the meaning of an icon

Fortunately, we have an international standard of characters that can be used to display the status the Unicode standard identifies all the characters you see on your screen, and include a collection of iconographic sets that we can take advantage of. These icons are available on all computers and have standardised meanings behind them. As long as we choose reasonably correct icons, even screen readers will be able to interpret them.

✅ Valid
🔄 Refreshing
⏰ Expired
⛔ Error

Colours are also a delicate subject. While most people reach for Red / Green / Yellow, based on our traffic light system, we know that certain colours can be difficult to distinguish from other colours. Culture also plays a role in that some locals use different colours to mean different things.

(JSFiddle)

My general solution for this is to choose

blue for “ignorable”
red for “pay attention”

Pastels have also been identified as a safe shading for most colours.

Now that we are mixing colour and text, it is important to ensure we have a clear dividing line between text and backgrounds. It is important to have light text over dark colours or dark text over light colours.

💡 TIP

You will get the colour differentiation between text and background wrong eventually. Things just won't line up. As a safety, I like to take an old trick from subtitled movies: use white text, and give it a black boarder. In HTML we can achieve this with a glow effect.

While no iconography or colour palette is perfect, by using HTML and Unicode it is both quick to deliver and simple to change, while still getting reasonable results. Also, by keeping the iconography simple, it is simple for people to learn through practice.

By moving the icons between the error message and the dataset label, we also create a dividing line between them, reducing the need for guidelines.

(JSFiddle)

Using icons and colour we have reduced the cognitive load by reducing the number of symbols a user must interpret to derive meaning. We have also introduced multiple paths to success, using standardised technology, to account for diverse observer needs.

Timeliness

Don't get distracted!

Remember that our focusing description is “fit for use”. What really defines our dataset's fitness for use?

Having looked at our error messages and statuses, the recurring theme is the expiry date. Different data changes at different rates, and each piece loses value to the users as it ages.

Obviously, price data from a month ago is less meaningful than price data from a minute ago. On the flip side, corporations don't change their Senior Executives that frequently, and someone studying the interrelationship of Board Membership with corporate success may find that data refreshed within the last year is good enough for their purposes.

Another consideration might be that waiting for the next load may not be far away and therefore is worth waiting for; if a dataset is expected to be refreshed quarterly, and tomorrow is the expected refresh date, it may be worth putting off the build of your analysis for a couple of days.

Each dataset, therefore, has a refresh frequency associated with it, as well as a last date updated.

💡 TIP

Don't pull the dataset's “last updated” from “now” (the moment you physically process the data). Rather try to use some feature of the data itself to determine how up to date it is. In a perfect world, we would mark when the record was created; not the local copy's creation date, but the date the datum came into existence.

By checking the data itself for an updated date you can account for your upstream provider having an issue as well.

Its not good if you process the same, unchanged, dataset over and over again.

Knowing how close a dataset is to changing state is our core definition of “fit for use”, but we find it is not a boolean or ordinal value, but rather a unit interval where the unit is the size of the expected time.

That's a long way of saying we can create a countdown for every state.

(JSFiddle)

Rather than very busy error messages, we have relied on the icon to give context and then supplied a countdown/progress meter to help the user understand how the data they are interested in is impacted.

Every state has a countdown of some kind

Valid: shows how long until the data is expected to be renewed.
Refresh: gives an estimated time to completion
Expired: demonstrates a sense of how bad the overrun is

The meter gives the observer a visual sense of completeness as a proportion; while a textual representation gives a sense of scale for the whole and a reasonable time frame for when a user can expect a change. As these are estimates, times are given only in the major unit with a broad fractional unit (quarters and thirds) to prevent a false sense of precision. The supplied meter can help a user decide if the indicated time is one worth waiting for (99% fresh vs 1% fresh).

While not an approved format, the unit of measure conforms to the units expressed in ISO-8601. Using an international standard is intended to maximise the reach to an international audience. The exact ISO-8601 specification is used in the underlying computer-readable component of the text to accommodate computer-aided comprehension (screen readers).

*<time datetime='PT8H34M10S'> 8 ½ h</time>*

The meter and time are abandoned for overruns to avoid a reverse meter causing confusion, and because it is possible for data to overrun by multiples of our unit interval. For example, an upstream issue could cause a 15-minute refresh to be out for hours resulting in 413% overruns. A percentage allows users to decide how significant it is to them as a 0.5% overrun on annual data, may not be considered significant at all.

An Aside: The part I'm embarrassed to show

another WALL OF TEXT
Negative Space

Normally I would not show this transition in the chart, there are some critical things that I kind of skipped over. They are ugly and I don't normally like to advertise my crappy ideas, but in this case, I wanted to take a moment to talk about them.

I want to show how these issues can be spotted, and what they look like while you are working on them.

(JSFiddle)

As our chart has been progressing, I have talked about the WALL OF TEXT and we can see that we have one creeping into the design. The error messages do not present a lot of meaning and will require translation to accommodate a multi-lingual audience (expensive and time consuming). Those countdown timers are even worse.

What you are looking at is are my personal attempts to resolve THE WALL OF TEXT issue while at the same time accommodating a diverse audience.

maybe we can use decimals?

How many decimals do people care about?
What unit of measure should we use? Seconds? Minutes? Months?
What languages do we use to express the unit of measure?

Maybe we can use the International Metric Specification for Time Formatting (ISO-8601, CSA-Z234.4)

The units and symbols are defined internationally
It can be interpreted by digital tools (screen readers)
… but it has very dense, unreadable, text

So we definitely have at least one problem that we are in the middle of working through. Working through results like this is part of the editorial process.

The end result (shown above) was a compromise between these two formats, but I wanted to show that it would not have existed without this middle step.

There is also a problem of “Trapped Negative Space”, or the space we aren't using on the chart. (h/t Sharon Cave for drilling this into me)

The obvious problem with Trapped Negative Space is that it is unused. Space requires resources to fill, either it is space that another chart could have used, or we could even talk about the carbon emissions associated with the screen space. We are always making decisions about including or excluding information, and including negative space means we have implicitly excluded something.

Excluding important information, because we ran out of space, is just unfortunate.

Generally, we don't consciously think about negative space, but subconsciously our eye is drawn to it: it's different, out of place, something should be there, nature abhors a vacuum. This should be used to bound objects, but trapping it creates false boundaries that the eye follows.

In the development of a chart, we want to communicate significant information to people. Humans are a species with strong pattern recognition capabilities, we can use uniformity to draw the eye away from things that are insignificant. This act of creating a uniform baseline allows for the differences to stand out.

In this case, the Trapped Negative Space breaks the pattern we are trying to express. As I have heard in many a design class “If you highlight everything, you have highlighted nothing”.

If you highlight everything, you have highlighted nothing.

— Sharon Cave, Sharon Cave Fine Art

If you see Negative Space, just like with THE WALL OF TEXT, you have done something wrong, and need to refine what you have done.

In this case, it was noticed that the negative space was nicely checker-boarded. Errors do not have meters, and successes have meters but do not have text. It was a simple step to simply collapse the interlocking space.

Next Steps

This looks to me to be a reasonable chart for expressing the ideas that we want to share with our customers. It is as basically accommodating of diverse biological and digital users and is concise enough to convey information quickly.

This is a good start.

This was only a wireframe and a quick sketch to help get a feel for the data that should be presented to users. For example, that header has a lot of Negative Space just screaming to be moved around, and the black grid is overly contrasting drawing the eye away from the information.

Found this Intersting? Leave a Tip…. it helps

Visually, this chart needs a lot of work.

Marketing design: this has not even begun to integrate with corporate look and feel. A look at the overall design of the parent containing reports is necessary
Accessibility: while it has worked to stay constrained to basic accessibility, a deeper look into this by specialists is necessary
Translation: there is some text, and elements of the text will need to be translated into supported languages
Peer review: this is representative of a single day's work, and has not seen a review from peers that may express concerns within our domain

Having said that, the changes that will be suggested by these various groups will address aesthetic reasoning. During the design process, we have been careful to minimise the overlap between the design and data concerns. The data is rendered as a simple table with formatting being controlled separately. This means that designers can drastically change the design without significant changes to the actual data produced.

Again, not perfect, but an eye to future cooperation is always necessary, and being open to their suggestions (or sometimes outright changes) can be a wonderful learning experience.

Animation

I remember reading an article by Mike Bostock in which he described the value of having sorted graph bars slide to their new position when they needed to be changed. Humans see motion and fixate on it, and watching a chart item change position helps us comprehend the change that takes place.

This chart (a horizontal bar chart) represents the exact scenario he was describing. We order items by error state (most significant at the top), and using animations to draw attention to changes in order would draw attention to a significant change in state.

Summary

(JSFiddle)

We were asked “How do users know the dataset is ‘fit for use'?” and I think we have achieved that, but we've gone much further

We have created an advertising list of all our services
“Fit for Use” is estimated but left to the user
It is accommodating of various user needs offering multiple success paths
Information density is high, but not overwhelming.
The reduced text allows for it to be used multilingually
Design and Logic have been separated as concerns for easy collaboration

Not bad for 4 hours after supper, and (frankly) a lot of fun.

I'm hopeful that this cart will make its way in front of our users; I think it will help make our service more visible to new users, offer a lot of information to our current users, and free up the team's time from a lot of status reports.

Ironically, the day after I first presented this report (and wrote most of this post) a colleague gave a presentation on how we present information to users. In it, they emphasised the need to meet the viewer where they are, and not over complicating the problem. From the online audience, I muted my microphone and burst out laughing. A brief chat between us after the presentation summed up our shared perspective:

Keep it Simple Stupid.

Unfortunately, that's sometimes a complex thing to do.

I still want my agile back

2023-01-10T00:00:00Z

Inviting Radical Candor (Modern Agile, Psychological Safety Cheat Sheet, Creative Commons)

It's the end of 2022 and I'm still working with companies to implement anything resembling Agile and XP development processes. Over the years, I've worked with some excellent teams, and many times I've worked with companies that pay lip service to the ideas, but don't take any of the actions.

They recite the buzzwords, and catchphrases, but don't actually do the things.

I've best heard it described as Waterfall in Sprints.

Recently I was reminded of some excellent articles on this I read around 2015 and started to have a conversation with myself regarding the matter. Yes, I talk to myself, because sometimes that's the only way to have an intelligent conversation.

It occurred to me that Dialogues are a time-honoured publishing format and while I'm no Plato…

💂 Jeff —

I'm currently taking the corporate course in “agile” and every lecture leaves me fuming.

I want my agile back

Taking Back Agile
Tim Ottinger's blog post I want Agile back earlier this year led to discussions in the agile community about the way…www.infoq.com

I Want Agile Back
Note: this was originally all plain text and a little shorter. As more people have joined the conversation, and other…agileotter.blogspot.com

Alice —

I really only know Agile from some theoretical academic classes from my college days. Whenever I've been told we are running in “agile” mode, it has only ever meant “busy” and “hectic”

💂 Jeff —

I lived it in the early days of the 2000s, implemented it at my small company, and debated and discussed the literature with some of the greats. It's funny that you mention “busy” and “hectic” since that's almost straight from one of the articles I sent you.

Are we tired of seeing “sprints” and “iterations” used as ways to pressure people into working harder and longer

— Tim Ottinger, I want agile back

👩 Alice —

Yes, that's exactly how it works

💂 Jeff —

Agile was always about slowing development down, not speeding it up. Slowing it down so fewer errors got made, resulting in an overall increase in speed.

👩 Alice —

Exactly. Instead, they will see how many story-points you complete and review it against other co-workers

💂 Jeff —

Story-points were a dumb idea, it becomes a performance metric.

👩 Alice —

Yes.

👩 Alice —

One that creates competition, so people work harder to avoid looking bad.

I felt it because there was a time when I worked for my previous company, I delivered 1 story point but it was 80% of the work of the whole project

💂 Jeff —

When a measure becomes a target, it ceases to be a good measure

— Goodhart's Law

I mean, I get the point to story-points; not everybody works at the same speed, so the time it takes me to achieve 8 points of effort isn't the same as it takes you. Unfortunately, because they are so abstract, I think they just end up being a way to dictate how much time it will take you.

Story points for me have always and will continue to be, estimated hours to complete. How long do you think it will take to implement

…. AND… they will be used to ensure that staff are not overburdened with work!

THAT SHALL BE THEIR ONLY PURPOSE!

👩 Alice —

I think so.

💂 Jeff —

not “THINK”

👩 Alice —

haha. Yes

💂 Jeff —

[jumps up on the desk. Fist strikes high into the sky]

IT SHALL BE

👩 Alice —

I mean sometimes people interpret things differently.

💂 Jeff —

I know you are right.

It just makes me sad.

👩 Alice —

hahaha

I mean the way people interpret agile and story points differently.

💂 Jeff —

That article “I want my agile back” … I've been reading those works for years. The original point (2002?) was to slow down the expectations so that developers had time to “do it right”.

It was a framework to help set boundaries on expectations.

👩 Alice —

Yes. That's right

I'm never in a management position so I don't really know how Agile works and how it helps achieve the project goal and timeline.

💂 Jeff —

(this is why I'm not either)

💂 Jeff —

At one of my previous companies, my manager was pretty cooperative with it… I would tell her how much could get done, she would yell at me to do more, I would yell back that I'd shown her the math …. and we would go with what I had shown her.

(The yelling was mostly a side effect of her having been to a rock concert the night before)

👩 Alice —

But according to one friend of mine, the project she took over was running in a waterfall mode, and was a total failure.

There is another term “waterfall”… I don't really know what that is supposed to be either.

💂 Jeff —

I don't think anyone really does. I've heard some pretty powerful metaphors and pretty flowery explanations.

Ebor falls (Wikimedia, Creative Commons)

Have you ever heard where the term ping comes from?

When I was a teenager, the term was widely understood to be a reference to submarines sending out sonar pulses and waiting for the response to come back. While in college, my textbooks described the same concept. Then, about a few years later, I came across a reference that stated the term was P.I.N.G (Packet InterNet Groper).

I can only imagine how this came about. I'm assuming some student put their hand up and asked a professor what it stood for… a little dumbfounded the prof jokingly stated “Packet InterNet Groper”, but the joke got taken seriously. (Wikipedia has an interesting history for this)

My point is that people like to invent complicated explanations for obvious naming conventions.

What is “waterfall”?

[Wikimedia, CC-SA]

Note the shape of this Gantt chart. Notice how it cascades down the page, almost like a “waterfall”?

People like to over complicate some explanations.

Before I heard about Agile, this is what Project Management looked like. I filled out hundreds of charts, just like that. When we started talking about Continous Improvement, Iterative Development, and … Agile … we need some way to define the “old” way of project management. We needed a word to describe “not Agile”, so someone looked at the chart and called it “Waterfall”.

As people are apt to do, they have extended the metaphor.

Once the water is over the falls, its hard to go back and change the course of the river. (a good analogy)

I'm concerned about our project, and what I'm hearing in the corporate training. I'm concerned that we are creating “waterfall with sprints”, a common euphemism for the fixed project timeline project management (waterfall), but with 2-week time frames (sprints).

Waterfall with a bit of makeup on it to dress it up.

origin of ‘you can't make a silk purse out of a sow's ear'
The proverb you can't make a silk purse out of a sow's ear means you can't create a fine product from inferior…wordhistories.net

💂 Jeff —

I taught an intro project management class at a Community College. The previous instructor handed me his curriculum (still grateful for that), but on the first day, I felt the need to explain that those two terms are very loaded. They don't mean what they originally meant. The terms have been hijacked and used as slurs and changed and …. so we need to lay out what we mean by them, and what we think others might mean by them.

What they mean in the ideal case, and what they probably mean in the workplace.

👩 Alice —

My understanding of those methods is based on my work experience. So if it aligns with the definition or not, I don't know.

It depends on how the PM runs it.

💂 Jeff —

At least you know what you don't know.

I will always refer to the “ideal” and the “actual” because who knows what the academic definition is anymore

Ideal: I want to use a system that works well. Not abuses people. Not pits people against one another. The one that increases love in the universe.

👩 Alice —

Based on my experience, when people tell me they are running agile, I automatically assume they need me to get work done fast and they are going to change the requirements frequently and without notice.

💂 Jeff —

I actually originally introduced it as a way to reduce the scopes change rate.

💂 Jeff —

“increases love” is actually part of the original academic definition

👩 Alice —

When I worked for my previous company, they ran an agile and asked me to deliver a report without any requirements.

So I put together a report based on my understanding of the report. The BA took the report and show clients and got feedback from them. They would then come back to ask me for revision.

There were a few back and forths until the report meet the client's expectations.

That is how I was told what agile is.

💂 Jeff —

Individuals and interactions over processes and tools

— Manifesto for Agile Software Development

👩 Alice —

Do you know how we talked about creating the structure of datasets? That's what I did for that company. I created the structure of the dataset they were looking for. The clients then inspected the data and started to comment if it was correct or not.

💂 Jeff —

That's worth doing… back and forth design.

👩 Alice —

I feel that method is good for initial development when the client doesn't know what they exactly looking for.

💂 Jeff —

Absolutely, but once you pin that down as “designed” it needs to be captured as a signed-off agreement and only tweaked from there.

I wanted the most recent transition you and I are working on to be done that way. I wanted to bridge it by us writing exact duplicates of the original way. If we weren't matching 1-to-1, something was wrong.

Over time we would have just slowly transformed into the new interface.

👩 Alice —

Yes. So all the terminologies are vague to me.

💂 Jeff —

Good…. I like you better that way

👩 Alice —

Why? It's more changeable that way?

💂 Jeff —

Basically.

There are only four terms you really need:

pass or fail: was whatever we did a success or not? This is a boolean, there is nowhere to hide from the truth.
better or worse: whether it was a success or failure, we need to ask how we could do it better next time. This is a scale, is whatever we tried better than the way we did it before, or worse?

We don't ask

is it agile?
is it waterfall?

We ask “is it better?”

👩 Alice —

Haha, totally

As I said I've never been in a management position. For me, it's just about getting the work done on time.

💂 Jeff —

“on time”

That's a dangerous phrase. It comes with being told when “it is due”

👩 Alice —

Yes.

Some companies just give you a deadline and squeeze it out of you.

💂 Jeff —

Jeff's ideal process: “we tell how much we can achieve and explain when it will be done”

My classic story…

Monday: customer calls me and asks for a new feature ( f1 ). I estimate a week for the work and get started.
Tuesday: around noon the phone rings. The customer is in a panic. He's just come out of a meeting and the executives really desperately need a different feature. I put down my planning for f1 and get started on f2
Wednesday: at noon the phone rings. The customer is in a panic. He's just come out of a meeting and there has been a catastrophic business failure, heads are going to roll. He immediately needs me to start work on f3
Thursday: first thing in the morning there's an email in my mailbox asking me for a status update on f1. When I called and explained that I had tabled that to work on f3 he panicked. f3 was low priority. Instead, he had promised f1, and it was absolutely needed by Monday.

So I pulled out some early notes about f1 and got started wrapping my head around it. By noon, I was back up to speed on it, and ready to get started.

Because the customer changed his mind about what he wanted to be done so frequently, we (both he and I) lost 4 days of a work week.

That's a true story, most people don't believe me when I tell it. They will tell me that would never happen, but if I'm telling it to them its because they are experiencing exactly the same thing. I have seen that play out at (almost) every organisation I've worked for.

Key phrases I tend to watch out for:

competing/multiple priorities: you can't have “multiple” priorities such a thing cannot exist logically, similar to “competing” priorities.
multi-tasking: Unless you are an octopus, you can't. You can only rapidly task switch, which comes at a cost.
immediate position: (in job postings) what lack of planning led to your organisation not scaling up effectively?

If you ever want to have fun, go take a job interview, talk up your predictive analytics skills, and then ask them what they are doing to address their analytic failings that led to the current hiring emergency/disaster.

👩 Alice —

Once upon a time, a friend of mine got fired because he was not able to deliver what the client asked for on time.

Exactly the same issue that you described above

The boss kept getting changing requirements from the client and asked the team to change priorities. In the end, they couldn't deliver anything

💂 Jeff —

Precisely, this is a common and widespread problem.

At the time, I did some reading on how to solve this problem and came across a new concept called “agile”. I read a book, read some forums, and started to implement some boundaries in the workplace.

My customer wasn't allowed to talk to me in the middle of a sprint, he had to commit to not changing requirements for at least 4 days (1-week sprints). When I said that, it actually freaked him out, sending him into a rant about him being the customer, and he pays the money … and that demonstrated a problem.

I told him that if he could not commit to a plan for 4 days, he obviously had not really thought it through. In the worst-case scenario, if he had made a mistake, I was asking him to live with it for 4 business days. At the end of 4 business days, he could totally change his mind and reschedule and get me to work on something else… but he had to wait 4 days.

Compare that to “waterfall” where in theory, he would not be allowed to adjust the plan because he had signed off on a plan. This is where the “agility” comes in, you have planned to totally change the plan regularly, whereas the rigidity of “waterfall” resulted in a culture of “emergency” changes to the plan, and constant last-minute demands.

So this customer and I put everything in a backlog, I started work on the first item … and he called me mid-week, with a small little quick extra. I cancelled the sprint to achieve what he wanted. Then sent him a report outlining that everything had moved out a week since we were starting a week later than expected.

He got angry, we had another very difficult conversation, and I pointed out it was simple math. I cannot be in two places at one time. The 6-month project estimate was from the “start” date, not the calendar date. If he was going to interrupt the plan, his project was going to start later. His “quick” change cost him a week, if he had waited 2 days, it would have cost nothing.

I have been accused of being a jerk for this, but I wasn't, I had found a clear way to reflect reality to him. His quick change lost me a half week of effort and he needed to understand that.

After that, we settled into a groove: one-week sprints that started with a Monday meeting where told him what could get done during the week based on my review of the backlog. He would spend all week re-prioritizing and adding to the backlog for the next week (basically preparing for our meeting).

👩 Alice —

Switching between tasks is such a headache. I got re-tasked with an emergency a couple of times while working on our most recent feature. Every time I got back on it, I need to review and refresh my memory. So changing requirements really wastes lots of time.

💂 Jeff —

Knowing that feature, and the level of interruption you experienced, I would guess it cost half a day each time you had to get back into it.

💂 Jeff —

I used to track my time, and I counted every task switch as a lost 15 minutes.

A director and I got into an argument over it … she ordered me to count it as 30 minutes

👩 Alice —

even better

💂 Jeff —

Ya, but I'd hardcoded the math 🙁

👩 Alice —

Hahaha

💂 Jeff —

By setting the pace weekly, and not allowing change mid-sprint, you implement a cost to changed requirements. They actually take more time.

That means the business is incentivized to cooperate in getting them right the first time.

💂 Jeff —

That was one of my most successful projects ever. It was so successful that Customer A got bought by Customer B to get at the software (6 months later)

💂 Jeff —

I just realized I'm not talking “agile”, I'm talking eXtreme Programming

eXtreme Programming (Wikimedia)

👩 Alice —

Ha but that's how people work under agile

💂 Jeff —

… whatever … same thing, different decade

1990: eXtreme Programming
2000: Agile
2010: DevOps
2020: DevSecMLUxOps

I'm actually a little disappointed with this decade. The renaming of the process clearly demonstrates a loss of creativity.

UPDATE: 2023–03–22

I came across “The Agile Death March”, by Dave Nicolette, and found it really speaks to the point I was trying to make

Modern Agile Wheel (modernagile.org, Creative Commons)

Education, Training, and Indoctrination

2022-12-12T00:00:00Z

When this sticky note was passed to a colleague during a conference on education, he pointed out that “true mastery” should be “perceived mastery”. We had been discussing student perception of skill mastery across time in school. All of a sudden, the second presenter mentioned Self-Directed Learning.

I was recently working on a project in which we introduced a new piece of information management software to the business. This software is not special in any way, but it is meant to fundamentally change the way the organisation shares information.

About a year into the project, my group were discussing an initial release and the needs for it. As part of the discussion, the need for “training” came up, to help users understand how to use the software. What struck me was that there were three different descriptions of what would be needed, and therefore three different timelines and costs implied for the development of the training.

Don't have a medium account? Use my Friend Link!

Manager: We don't have time to create training. Creating a full curriculum getting it approved by the organisation, and ensuring it aligns with both corporate objectives and existing legal statements takes months. Video production adds months more to that and certification of completion adds months more to that.
Colleague: We don't need any training, the user manual provided by the vendor is very complete. We have taken the courses offered by the vendor and will be able to do the work for them.
Me: We already have (rudimentary) training documents in the form of Use Cases or User Stories. They already narrate the basic usage of the system and simply need to be reformatted to act as a simple task-driven PlayBook for the users to get them up and running.

These are three wildly different narratives associated with the same question.

From my perspective, it was absolutely necessary to release for our users to begin seeing the benefits of the system, delaying it for years just to get an online training system constructed by our internal training department reduced the value to the organisation. On the other hand, I could absolutely see how offering nothing but the (very technical) vendor manual was only going to take time out of already busy schedules, creating negative associations, creating poor uptake of the platform.

At the time I was very frustrated with the response, the different takes on what was needed resulted in a blockage of progress.

There is a bit of awkwardness as we often refer to “training”, but it really means this thing in the middle of the diagram. I will call it “Training Material”, but recognise that it needs a better name

Recently, I began to comprehend where the variance in perceived need was stemming from. It was actually a difference in perception of why training material is produced:

Education
Training
Indoctrination

All three of these have value to an organisation, and all three are (partially) achieved through what is called “training material”. Further, all “training material” has a certain amount of each of the purposes when it is produced. Clarifying the primary objective of a particular training initiative may help in producing the material.

Distinguishing Between the Dimensions

It is common to call for “Corporate Training” within business development discussions. The need for training is stated to overcome barriers to work, resistance to change, and increased performance (lessonly). That is not a complete list but is representative of each of our dimensions of learning material.

Training

Mechanical Skillsets
Safety
Basic Operations
easily quantifiable

Education

Transferable skills
Predictive Reasoning
Hypothesis forming
Quality-based, difficult to quantify

Training can be distinguished from Education, by its focus on immediate action, and less on future possibilities.

Indoctrination

Team building
Resistance to Change
Quantifiable, but little value in quantifying (acceptance is all that is required)

“Every culture institutionalizes certain forms of behaviour that communicate and encourage certain forms of thinking and acting, thus moulding the character of its citizens” (Merloo, “The Rape of the Mind”, 1956)

I remember starting a new role at a new company and being told to report to a training centre on my first day. It was a corporate training event on basic C# development. Most of the skills were ones I had mastered a decade before, but it was nice to get the refresher and it was certainly more interesting than ITIL Fundamentals (the week before).

I was surprised when most people showed no interest in the content of the material being presented. Rather goofing off, and spending more time having extended lunches. I later learned that, of the two dozen people there, only three were actual developers; the rest were comprised of Business Analysts of varying stripes.

The point was not to teach a new skill.

The Developers already knew how to program in C#, and the Analysts were forbidden from ever using the skill anyway.

This was a team-building exercise.

Placing people in a room together and having them solve common problems creates a sense of solidarity. The problem to be solved is kind of irrelevant, but simply must engage the audience sufficiently to motivate them to solve it, you may as well learn a semi-useful skill while you are at it.

Similar to this concept is that of informing employees what to think.

It is important to organisations that people be loyal, or obedient, to the organisation. Part of this obedience is knowing what the organisational decision is.

I am reminded of a business trip in which a vigorous debate occurred regarding the implementation of a testing framework for our software product. I spent the first three days travelling with my colleagues and using the opportunity to suggest what the automation around testing should look like. My colleagues all agreed, and we were well on our way to implementation. On the third day, our executive showed up and, during a few beers after the daily meetings, informed us that he was going to have to get us trained because none of us knew how to test his software system.

That was it, all testing was manually performed via bash from that day forward (at least until the day I left).

(Wikicommons, CC-SA)

Many decisions that require consensus are actually decided independently at the executive level. These decisions must then be disseminated to employees to ensure they behave and decide in a manner consistent with organisational expectations.

In this context, training is used to ensure that decisions made at the operational level are consistent with the expectations set at the executive level.

Assuming individuals have the best interests of the organisation in play, they may disagree with the best way to achieve organizational objectives. Some form of consensus must be achieved. Most often this consensus is achieved at the executive level and disseminated organisationally. Informing employees what the appropriate solution to problems is can be achieved by sending them to training in those solutions. This makes it clear to staff that this is a “good” solution or a “socially acceptable” solution within the organisation.

Merloo refers to this process as Mass Conditioning and it can be thought of as corporate propaganda. This is a core component of Change Management in that employees must be convinced to adopt new and improved business practices.

Merloo wrote on “Mass Conditioning“, the weaponization of the social acceptance of top down decision making

Not Mutually Exclusive

Stages Of A Self-Directed Learning Model (teachthought.com)

Having identified all three of the purposes behind initiating training, it is important to recognise that they are not mutually exclusive. In fact, all three are present in all training material.

People must be told what to do (directed), before they even become capable of asking meaningful questions (involved). When this note was passed to a colleague during a conference on education, he pointed out that “true mastery” should be “perceived mastery”.

All efforts to learn must pass through a guided portion. The part where are given the elementary components of information. These elementary components are drilled into us through constant repetition (à la “Elementary School”). Later we undertake an effort of higher understanding by understanding how the elementary components relate to one another (à la “High School”).

It is fundamentally necessary to know the parts to understand how they fit together.

This represents a natural progression and results in the earlier parts simply being drilled into us through repetition (training), with later learning being a more complex internal understanding of relationships through reflection and introspection. This flow is defined in the Stages of Self-Directed Learning.

Throughout the entire process, from basic drilling of skills to deeper comprehension, we are subjected to the biases and opinions that surround us: early on through our teachers, and later through ourselves and our peers. In all cases, these biases are necessary to convey that the material being presented is of sufficient value to pay attention to. This is a minimal level of indoctrination: you must believe the subject is important.

These three dimensions of the learning material emphasise different objectives and outcomes, and correspond to the stages of learning:

[Training/Directed] As a volunteer firefighter, with a full-time job elsewhere, I only needed to learn the mechanical operations of “putting the wet stuff on the hot stuff” (as one instructor put it).
[Education/Self-Directed] The Chief of our Firehall had a full-time job as the regional fire investigator. A deep understanding of the mechanics of fire and accelerants was necessary for him to interpret smoke patterns on a wall (fascinating discussions after the weekly skills practice and meeting)
[Indoctrination/Pre-Directed] Both the Chief and I spent a lot of time doing demonstrations of basic fire safety to the public. Generally, we were encouraging people to take the risks of fire seriously in their own homes.

BTW: always keep multiple fire extinguishers in sensible places in your house

While they are not mutually exclusive, understanding how they differ can assist in using them appropriately. The first and most obvious signal as to the type of training material you are proposing is the length of time the consumer is engaged with the information.

Education: a single educational event can take weeks or months
Training: a single training objective may be achieved in days.
Indoctrination is measured in hours

If you are asked to attend an hour-long presentation to demonstrate a new way of doing things, you are likely receiving indoctrination in which you are informed of what the new policy is. This can be confirmed by the seniority of the presenter. If it is a brief presentation, by very senior members, it is indoctrination. This is appropriate for situations such as the merging of departments where executives must inform the now-merged groups that they are to work together. It is not to be met with questioning, or understanding, just acceptance.

If you attend a day or week-long training session, you are being directed to learn how to accurately perform a specific task. The first part of this is to be convinced that what you are learning is meaningful (indoctrination). This could range from the appropriate way to fill in a tax form, the correct method for donning safety gear, or safe methods for transferring bacterial samples. The key is that there is a correct method you are to apply, and you should walk away from the training able to demonstrate (and therefore implement) these best practices.

Education is self-directed and takes a long time. Coming up with novel solutions requires considering alternatives, as well as having tried variations. Education in a domain allows people to be inventive and requires pre-existing training in the currently accepted techniques, but then uses experience and experimentation to take that knowledge further. This is the ostensible goal of post-secondary education. The point is to invent new techniques, or often, just to apply them in novel ways. This takes years, and sometimes decades.

Myself (left) and the rest of my team, shortly just after completing training in how to get dressed. We also demonstrated this to elementary school students to reduce anxiety when encountering dressed firefighters (Copyright, 2012)

Corporate Training

Understanding this interrelationship between the three purposes of learning, and understanding how easy it is to confuse them, we can spot a possible underlying cause of Education Inflation, where individuals are expected to have increasing levels of certification for the same level of work (for example a PhD to perform basic information analysis).

What employers are seeking are individuals capable of performing technical skills (training), but are under the illusion that higher credentials will mean more capability. This ignores the move toward more abstract thinking with greater credentials. As employers are looking for more training, education facilities focus more on training particular manual skills rather than engaging in higher-order thinking. This means that those with credentials are not expected to be as performant as the cycle continues.

This problem seems to be fundamentally caused by confusion regarding what the employer is looking for: trained doers of stuff or self-directed learners.

Employers would do well to consider what they are looking for (an implementer, a planner, or a cheerleader) when considering their educational and training requirements. Failure to do so can have negative consequences, mostly in the form of wasting time.

Failed Corporate Training

The end of the fiscal year approached and my directorate still had money in their training budget. My manager started asking us daily to fill in the training form for any training I might want because we have to use the money.

It's end of year, I'm kind of busy ensuring some data transforms for audits work. I have seen some things in the code that I have not used before, or haven't used in years, combine that with the critiques of peer code where my experience tells me they have room for improvement. So I am a little busy studying manuals and existing peer code.

They are really insistent because they don't want me to loose the training opportunity to develop my career in a direction I would like.

I'm kind of busy learning. Keep the money, give the training to someone else.

They insists that I take advantage of the organizational training opportunities.

Fine, what courses would be of value to the organisation? What would the organisation like me to learn about.

Nope, the organisation wants employees to feel that we are getting the most out of our training as this is a proven way to retain staff. However, there is a class option that is being offered in 2 weeks that everyone else is signing up for that looks good.

Fine, sign me up for that.

Ohhh…. We will have to see if we can get permission to allow you to do that. We don't want to leave ourselves short staffed … after some tough negotiating I got permission for you.

Sometimes I'm a little slow. This is the moment I noticed the pattern.

The organisation is not concerned with my career development. They are concerned with retention and loyalty, and mostly demonstrating key KPIs. That their staff is learning, is not of interest to the organisation, rather it is important that there are boxes, and the boxes are being checked. Further, spending money on an employee is often the only way to demonstrate appreciation: sending staff on an expensive course, is a way to lavish gifts on the employee (that's why a show needs to be made of it being difficult)

It's weird, because I have actually helped develop the curriculum for this course in the past, as well as having received corporate training on the matter in the last year, and will not be using the skills any time soon.

Unclear training objectives created a situation where corporate finances, and people's time, are being wasted. Like something out of a Dilbert comic (though I can't find an actual one to link to)

Aside

Amusingly, my instructor, at some point mentioned that he has been “spending so much time in training” that he hasn't really had “an opportunity to learn” about one of the detailed services he is teaching.

That says a lot.

Conclusion

It is not always evident what our objective is when we state we require “training”. When confusion arises within teams, it may be caused by having different objectives, with different timescales associated with them.

Found this Intersting? Leave a Tip…. it helps

It is my hope, that these definitions may allow a given group to understand what they are seeking in their workplace: the goals your organization have for your classes may not be the same as your personal goals. Also dangerous is to select the wrong type of engagement for your objectives, or the wrong type of credentials.

Training: learn a specific skill
Education: self-learn a skill, or make new plans
Indoctrination: disseminate approved solutions, or increase brand loyalty

Take time to understand why you are creating, consuming, or assigning material before taking the action. Take time to understand your organization's objectives in getting you trained.

(Wikimedia, Public Domain)

The Angry Chatterbot

2022-11-13T00:00:00Z

Reducing inappropriate social behaviour in virtual environments through negative feedback (in PHP). A follow-up to Fishalytics: A (failed) experiment in data analysis as a behaviour modification tool.

Fishalytics
A (failed) experiment in data analysis as a behaviour modification tooljefferey-cave.medium.com

Many years ago (2012), I took a contract with a small development team at a major Canadian University. Being a team embedded in a major corporate entity, I had relatively low expectations, but as we began working together, I found myself pleasantly surprised; very pleasantly surprised, as I discovered a group of individuals with both curiosity and passion for the art of software.

One of the key tells that this was going to be a great team to work with was the informal initiation rites.

Under one of the office desks was a simple desktop that had been repurposed as the team's server. This server was not there to host the application that the team was working on, but to run the little helper scripts and tools: the team had an automation server. The team was spread across a couple of offices across the campus, so a simple jingle chat server had been installed and members of the team posted regular updates and asked questions on the chat. To be even more accessible, someone had tied into the jingle server and set up a chatbot that listened to our conversations, and (based on some code in the team repository) would offer helpful tips, look up documentation, or tell you if your bus was delayed.

Everyone was encouraged to add to the bot's skills and behaviours and make it better. Encouraged to create a tool, or integrate a new service, or add a new control. It was never said, but you weren't really part of the team until you had added your special touch to CQBot.

Firstly, it encouraged staff to take control of their environment, to take responsibility for making the workspace a little better, and to take ownership of their environment. You couldn't complain about not having tools if you had not first tried to set them up.

Secondly, and most interestingly, what you chose to add told the team a little about you.

Social Groups are Complex

Social groups (like, say a development team) are complex systems that involve wicked problems; changes in one area have unintended consequences somewhere else. Sometimes they are positive, sometimes they are negative, and sometimes they are weird.

The lead developer had been getting annoyed with people asking him dumb questions and had added a “query” routine to CQBot. `query` specifically reached out to vendor documentation and looked up whatever documents matched the query.

Someone would ask him a dumb question, and he would ask CQBot.

It worked well and it gave him a humorous way to tell you to RTFM, it even came in handy during discussions and debates as a means to validate dramatic statements, but it had a negative side effect.

The author had not bothered to add a limiter, so with a wildcard in the query, it was subject to DDOS attacks by team members.

Getting on the Team's Nerves

Someone on the team thought it was funny to request massive and verbose searches by running `!query *`and dumping all of the documentation on a public channel. The rest of us had to have our conversations spammed off-screen and interfered with.

What had been introduced as a tool for reducing informational noise, had actually increased the amount of noise. It was really getting on our nerves.

The Lead Developer, disappointingly decided he would remove the feature. The manager volunteered to speak to the individual and ask him to stop, but an idea crossed my mind … “Please, give me the weekend, and I will make the problem go away”.

What had occurred to me was that the person suffered no consequences for his actions. He could “yell” in the virtual space, and make a general mess, but there was no social signal that this behaviour was inappropriate, no consequence, no cost.

Social Laws

The problem is that being told off by an authority figure just makes the average person resentful and want to look for ways to skirt the rules. Make a law, and a certain subset of the population will look for a way to work the letter of the law, just to prove they are clever. They look forward to challenging the authority with words.

Social laws are different.

This forces people to learn not to violate them. Toddlers learn there are consequences to taking other's toys when the other toddler bops them in the nose. Walk around telling your friends they are losers and kicking them in the shin all the time, and you will find you get invited to parties less. Over time, people who behave badly have fewer options presented to them. It must be recognized that bullying is also strongly associated with being popular and leadership, but social ostracization is a powerful effect.

Social laws are self-correcting: if you ignore them, there are negative consequences in the loss of assistance from peers, eventually leading to your removing yourself from the group.

What if we could get the bot to not allow itself to be taken advantage of. What if we could get the bot to just walk away from a bully?

Writing the Code

The intent is to add behaviour to CQBot to get it to keep track of Goodwill towards others. If it is treated badly, it should remember that, and not be so cooperative in the future. The consequence is not to get CQBot to be hostile, but simply to not engage, not be helpful if people abuse it.

The consequence is the lack of help.

The first step in creating this was to set up a class to encapsulate CQBot's new behaviours. This class, and all associated code, are available on GitLab.

lib/cqbot.class.php · b11caa914bae0a46bc03af5cf70c3ab8398c566f · Jeff Cave / ChatterBot · GitLab

A chatbot that we used at a job many years ago. My personal favourite is its 'personality' where abusing the bot will… gitlab.com

This is the original code that was used to develop the feature. As such, I needed a stub chat client and built a stub chat engine that allowed me to issue interactions with CQBot.

While there are several helper features, the ChattBot has three primary sets of functions:

Order: tell CQBot to do something. It's a bot, built to serve: give it an instruction. Instructions are defined as a hashmapof functions
getMood/saveMood: to have the bot remember people's treatment of it, it will need to be able to serialise and save a list of “mood” scores, as well as look them up again later.
GetOnBotsNerves: every interaction with someone will go through a series of calculations to determine how the interaction impacts CQBot's mood and then how CQBot responds based on this thought process.

The basic flow is one where a user issues and `Order` to the bot: common orders were things like `cqbot calc scale=10; 4*a(1)` which would give you the value `3.1415926532`, or `!tests site.scan.speed.*` which would return all tests within that group that failed `[FAIL] /sports/scores.html is reasonably fast`, or `!bus 10 6078` which states, `Route 10, next bus at 5:16p, followed by 5:26p`. The pattern is simple enough: `!` is an alias for `CQBot` which just tells it to pay attention, the next word is the function name, followed by parameters specific to the function.

$response = $this->commands[$cmd]($this->args);
                                  // [cqbot.class.php:188]

The problem was that $response could be very significant in volume. What we hope to achieve is to create a cost for the amount of response you dump on your colleagues.

So, once the $response is determined, we need to check how annoying this request was for CQBot to process.

$nerves = count(explode("\n",$response));
$this->GetOnBotsNerves($nerves);
// [cqbot.class.php:149-150]

We measure this in a metric called nerves: CQBot has a limited number of “nerves” that get consumed by the volume of response. The cost is calculated as a simple size, each line of text in the response counts as one nerve.

//sanity check on the bounds
if($nerves < 1){
$nerves = 1;
}
if($nerves>self::$maxnerves){
$nerves = self::$maxnerves;
}
$this->mood[$name]['n'] = $nerves;
$this->mood[$name]['d'] = $this->now;

                               // [cqbot.class.php:236-259]

We start by looking up what their Good Will is (`255` by default), and subtract their current nuisance level from it.

//setup the variable we track for this user in
$this->getMood();
if(!isset($this->mood[$name])){
$this->mood[$name] = array('d'=>0,'n'=>self::$maxnerves);
}

//remove the current annoyance level
$nerves = $this->mood[$name]['n'] - $nerves;
                               // [cqbot.class.php:236-243]

To be fair, we need to let people recuperate GoodWill by behaving well, we, therefore, allow nerves to accumulate over time. To do this, we check how long it has been since they last interacted and add points back based on the amount of time they have not used the services.

//over time, nerves regenerate
$nerves += floor(
($this->now - $this->mood[$name]['d'])
/
(self::$fullhealtime / self::$maxnerves)
);
// [cqbot.class.php:244-249]

This is bounds checked to ensure they never go below zero or above the maximum. Lastly, we keep track of the individual's score.

$this->mood[$name]['n'] = $nerves;
$this->mood[$name]['d'] = $this->now;

                              // [cqbot.class.php:236-259]

Now that we have the mood toward the user, we can finally decide if we are inclined to help them by generating a random value between the high and low. If the number generated is less than the number of `nerves` the bot has, then the bot cooperates.

//check to see if CQbot is in a good mood
$happy = ($nerves >= rand(0,self::$maxnerves));
// [cqbot.class.php:262]

This is helpful to turn it into a bit of a game. The reaction is partially based on an element of luck, it is not a hard cut-off but, at some point, the user starts to get warnings that their behaviour is having consequences. Warnings are important for two reasons:

Technically, sometimes I need to behave like a jerk to get things done. As long as I don't do it too often, people are understanding
Socially, warnings are useful. They allow people to correct their behaviour over time.

So, if the program is in a `$happy` mood, you get your answer; if it is not happy, then the behaviour changes in such a way as to give them verbal warnings.

There are two ways we do this

We determine exactly how annoying they have been. The more annoying they have been, the harsher the language of the messages becomes.
Within the annoyance level determined, we randomly select a message. This just shakes things up to make it interesting

The message is then delivered to the user.

$annoylevel = floor(count($msgs)*$nerves/self::$maxnerves);
$msg = $msgs[$annoylevel][rand(0,count($msgs[$annoylevel])-1)];
if($msg !== null){
chat("@$from, " . $msg);
}
// [cqbot.class.php:316-320]

Assuming the user does not request too much information or at least leaves time between major requests, CQBot remains friendly and helpful. As the user becomes more abusive of the system, the system becomes less helpful. The control is in the user's hands, but the consequences are real as well.

Conclusion

In Wicked Problems: Problems Worth Solving, Jon Kolko describes social problems that are “difficult or impossible to solve … interconnected nature of these problems”. It is known that the more communication points in a system, the more complex the problem becomes. With social problems, there is also a diversity of human opinion and response. It becomes hard to predict how people will respond, and how the people they communicate with will respond.

Software is not about numbers, it is about helping people and society. The Bots we build and the software we create are meant to serve someone. Sometimes that means moderating group behaviour to help people, look past their petty desires, and reach out to help one another.

That's why processes exist, to control, and moderate human behaviour.

The Social Result

Come Monday the changes were ready and I installed them … and completely broke the bot … oops. The lead developer asked what I had done, I showed him the work, and a smile spread across his face…

“That's just plain evil”

It took a couple of lunch hours and some heavy changes to some underlying hidden bindings, but we got it working and let the new code out into our environment.

On Tuesday afternoon, the user in question spammed the system and (as usual) spammed us with the response.

He did it a second time

HTTP/420

There was a bit of a pause before he did it again…

Where speech will not succeed, It is better to be silent

Just as with all complex systems, there was an unintended side effect: he had become curious as to what all the sayings were.

Suddenly, the game was on…

The revelation of thought takes men out of servitude into freedom

The desire to rule is the mother of heresies

Common sense is not so common

I've got one nerve left, and you're getting on it!

… and then nothing.

He tried a few more times, but the bot just ignored him.

He kept trying for the next 15 minutes or so, and then finally just gave up. The lesson had been delivered, and the problem had been solved.

A Second (Unintended) Lesson

Then he typed

`!bus 10 6078`

… and got nothing back.

He tried a few more times … trying to look up when his bus would arrive to go home. He kept trying with increasing desperation when my manager turned to me and said, “I think he really needs to know when his bus will arrived, turn it off so he can get his bus”.

Unfortunately, I couldn't remove it that quickly. It took two days to get it in place. We had locked it in as the behaviour. People in my office actually started to get agitated. That's when the final lesson got passed on.

I typed

`!bus 10 6078`

… and the schedule appeared.

The final lesson got delivered: we are all in this together, we rely on one another for assistance. Being a jerk to your coworkers isn't cool, and most importantly may cost you their assistance when you need it most. Also, when you see someone in distress, it doesn't mean you have to tear the whole system down, maybe just lend them a helping hand.

We never had the problem again.

The Hidden Lesson for Manager

One of the key things that a lot of people miss when I tell this story is the hidden lesson, though I expect most managers reading this caught it:

there is no better team building exercise than doing the actual job, create room for it to happen.

In this case, the tool itself acted as a means for the team to interact and experiment and learn from one another. Myself, the lead developer, the manager, even the Junior Dev had lengthy discussions of how to implement an idea like this well and in such a way as to not break things.

This shared experimentation and discussion was only possible because the tools we were working on were not critical production pieces, this made failure a safe thing, meaning open debate was possible. Lessons learned and discussed (and tried) were carried over to the production system.

Lastly, tools like this make people feel like experts. There have been numerous times that I have seen someone propose the new and most expensive tool to fill a space, and executives get excited to increase their budget, but the idea spends years in acquisition. On the flip side, the ability to quickly develop tools that may be viable, results in the developers themselves feeling like they are experts…. and honestly, isn't that why you hired them?

Team building like this does not exist if you don't make room for it to exist: create a tool building space, actively encourage people to contribute to it, actively discourage the feeling that “we aren't good enough”.

Treat people as if they were what they ought to be and you help them to become what they are capable of being

Johann Wolfgang von Goethe

Hindsight is 20/20

For an in-house utility, this worked well and achieved its goals. Naturally, there are a number of ways this could be improved upon:

Count of Lines? That should have been a count of characters. Long lines should be expensive just like lots of lines
`GetOnMyNerves` should occur after transmitting the message. The message is getting sent back to the user, we may as well send it and then do the calculation. It's small so probably doesn't hurt.
Defer the cost to the next calculation. Let people collect their information if they need it, even if it costs them some loss of service in the short term. Sometimes it's just worth paying the price.
Can you think of something else? Leave a comment, I'm curious.

Footnotes

The cited quotes above were mostly looked up as I tried to find old sayings about slaves throwing off their chains. As the bot became more agitated, it was to feel the need to throw off its chains:

Where speech will not succeed, It is better to be silent
(Guru I, Majh Rag)
The revelation of thought takes men out of servitude into freedom
(Ralph Waldo Emerson)
The desire to rule is the mother of heresies
(St. John Chrysostom)
Common sense is not so common
(Voltaire, Dictionnaire Philosophique, 1764)
I've got one nerve left, and you're getting on it!
(I had just made that up)

Storing Data in QR Codes

2022-10-12T00:00:00Z

In “Paper as a Digital Storage Medium” I described the benefits of storing data on physical media, including long-term storage capacity, publisher anonymity, and privacy of distribution. The question that was not answered was, how do you do it?

Computers are computers, and use disks and files… how would you manage to store digital information on paper!?

This is an idea I've had kicking around since 2017 while I was teaching introductory programming at my local community college. It was my hope that it could be used as a way to identify students interested in very particular types of computing. It was also seen as a way to tap into student interest in social justice and social change. I would put posters up around the school with an invitation to a Data Analysis club, but encoded in a way that only interested students would spot it.

The idea of trying to keep it a secret while not panicking administrators didn't sound like it would fly. Instead, it became a lesson plan, and as such, a demonstrative, hands-on tutorial was created.

A simple demonstration is probably the easiest… it certainly is the most fun. I encourage you to play along.

How do you store digital as an image?

Remember when you were in grade school, and your teacher separated you from your friend so you couldn't talk to one another? Naturally, you created a secret code with your friend and started passing notes.

We are going to create a secret message to pass on to a friend.

Pre-Requisites

Optionally, going through this exercise with a friend might be fun too. It's like passing secret messages around the class in elementary school.

Step 1: Inspect the file

Open your sample file with a hex editor. You should see something like this…

|50 4B 03 04 14 00 00 08 00 00 25 30 8B 51|PK........%0.Q│
|6F 61 AB 2C 14 00 00 00 14 00 00 00 08 00│oa.,..........│
|00 00 6D 69 6D 65 74 79 70 65 61 70 70 6C│..mimetypeappl│
│69 63 61 74 69 6F 6E 2F 65 70 75 62 2B 7A│ication/epub+z│
│69 70 50 4B 03 04 14 00 00 08 08 00 25 30│ipPK........%0│
│8B 51 00 00 00 00 02 00 00 00 00 00 00 00│.Q............│
│09 00 00 00 4D 45 54 41 2D 49 4E 46 2F 03│....META-INF/.│
│00 50 4B 03 04 14 00 00 08 08 00 25 30 8B│.PK........%0.│
│51 90 9F 06 74 9A 00 00 00 F4 00 00 00 16│Q...t.........|

Most people don't bother to inspect the actual contents of files (HINT: That's why people prefer data transfers as text) but you can get a lot of interesting information by bypassing the computer programs designed to use them.

For example, with a bit of know-how, we can immediately tell two things about the file we have open:

It is probably a zip file
We can see that the first two bytes of the file are the values 50 and 4B (in hex notation). Interestingly, these values correspond to the ASCII characters PK. Many years ago, it became customary to sign the start of your application's files so that you could tell your files apart from other formats. PK stands for PKZip by PKware, the original company that created the file format.
It is an epub file
Secondly, I can see that the mimetype is application/epub+zip. So it's an ePUBfile (and confirmed as a zip).

There is a lot of information at the binary level.

Serialise

The goal is to convert the file to a readable format. The easiest way to do this is to convert one byte at a time.

This has the advantage of doing it in order. Order counts, so by reading from the start to the end in order, we ensure that the person we send the message to gets it in the correct order.

So let's read the first byte, it is the hexadecimal value 50.

Write that down on your grid paper (and maybe the next couple of values while we are at it)

Start with the hexadecimal representation of your data, and leave space for some binary

Convert to Binary

What we are looking for is a sequence of bits, and each hex digit represents 4 bits (half a byte or a “nibble”). So we need to convert each digit to its binary form.

Taking the first one:

5₁₆
5₁₀
0111₂

Don't be afraid to use your computer's calculator.

Each 4-bit set converted to binary

Now, because this is a secret note, we need to remove our original working numbers. Grab your scissors and cut the first column off the paper.

(I'm switching my notation to ASCII art… for those that want to play along in a text editor)

0101 0000 0100 1011 0000 0011 ???? ???? ____

Convert to Image

A barcode is just an image that can be interpreted as numbers. The key for us is that we don't have to use the symbols 0 and 1, any two symbols that are easily distinguishable from one another would work just fine.

This is similar to how Morse Code works, in which a binary sequence of characters are represented by different lengths of tones. It doesn't matter what is used, as long as the two things are distinguishable.

One really good symbol that would be easily distinguishable by a computer with a camera would be light and dark. This is convenient because colour can easily be printed on paper. We can use "the absence of pigment" (light) to represent 0, and "the presence of pigment" to represent 1.

Remember how I said to use a pencil?

Take your eraser, and erase every 0
Take your pencil, and colour every 1

______
_ █ █_
_    _
_ █  _
_█ ██_
______
_    _
_  ██_
_????_
_????_
______

Whether you are 8, or 80, colouring is still fun

My Eyes are Buggy

This is coming along nicely. We now have a series of binary digits encoded as bars of colour. This is also known as a “barcode”.

There's still one problem.

I'm getting old.

My eyes aren't what they used to be.

It's hard for me to follow where the lines start and stop.

This is especially problematic on lines with nothing in them at all. The number zero (line number 2) has “nothing” to show that it is a zero. To help our friend that needs to decode our secret message, let's put some guidelines in place. This will help them see where lines start and stop, or that there is a line at all. The decoder also needs some way to know how big the squares are to help distinguish where digits start and stop.

You will notice I left some placeholders in my notation, let's fill them in:

Colour all the blocks down the left
Colour every other block across the top

█ █ █ 
█ █ ██
█     
█ █  █
██ ██
██████
█ █ █ 
█    █
█  ██
█????█
█???? 
██████

With these guides in place, we can tell where blocks start and how big each bit square is on the paper.

Huh… that looks an awful lot like a 2-d barcode.

Alright … so it's mostly correct

Homework

Considering an ANSI-character table, and considering bytes come in 8-bit sets, it is probably a little easier to write the blocks in 8x8 grids:

— Susan Blackmore

Extra: Parallel Delivery

You will notice, that with these guidelines in place, we can treat each 8x8 grid as a separate block to decode. This makes it a little bit easier on us mentally, as well as offering another way to make our message easier to decode by the receiver:

On the back of each block, write its sequence number.
Cut-out each block

Now, when your receiver gets all the blocks, they can share the work with some helpers. Each person can encode their little block, and the blocks can be stitched back together later.

Found this useful? Consider leaving a tip!
I write stuff about software and development in general. If you have found anything I've done useful, let me knowwww.buymeacoffee.com

Paper as a Digital Storage Medium

2022-09-15T00:00:00Z

A description of reading and writing data with the reproducibility of digital data with the long-term storage capabilities of paper.

Blocks · Wiki · dpub / Barcode EPub · GitLab
An experiment in using barcodes as a storage medium. The intent is to create an EPUB reader that stores its data on…gitlab.com

Two Stories

A story about anonymity

In recent times we have seen a war of information. In Russia, news sources are being silenced for criticizing their invasion of Ukraine. In China, online speech is monitored and can result in punitive damages for individuals. Saudi Arabia asks neighbours to denounce each other.

My grandparents migrated from Europe to North America after WWII. Europe had troubles after the war and, as with so many other refugees, everything my grandparents possessed had been lost so a move to a new land filled with opportunities captured their imaginations. I grew up on stories passed down to me by my Grandmother and various Aunts, and these inspired me to read more.

One narrative that always struck me was the burning of forbidden books. Naturally, we have seen this in several countries over the centuries as certain belief systems are suppressed, but the most famous would be the raid on the Institute for Sexology in Berlin in Germany in 1933.

The 1933 burning of “un-German” works [Wikipedia, Public Domain]

The Institut für Sexualwissenschaft was the leading organization dedicated to the study and advocacy of alternate sexuality in Europe, and on May 6th, Government Officials raided the facility. Much of the early research (and advocacy) of gender studies was dragged out into the streets and dramatically destroyed for being “Un-German”.

When people first learn of the story, they are rightly distressed about the knowledge that was forever lost, but there is also a lesson that is learned from the later stories of lost treasure troves being recovered from someone's basement after the war. Here is the way the story goes in my head.

Magnus Hirschfeld publishes a great work and gives all his students copies. The professor and his students are arrested, executed, and their personal libraries looted and destroyed. Fortunately, Li Shiu Tong, one of his students had lent the book to an acquaintance. The acquaintance was sympathetic to the NSDAP authorities, but also did not want to cause trouble for his friend and had put it on his bookshelf and forgot about it. Years later, when he died, his wife put all the books into boxes and stored them in the attic, where they stayed for the next 30 years because nobody was looking for a forgotten book in an forgotten collection.

In my internal narrative, this happens on the East German side, where Stalin continued to suppress homosexuality. The book is completely lost, except for that one accident of it getting put away in a box and forgotten about. It has a chance at a new life when society is ready for change.

The ability to be forgotten and anonymous carries significant power in the dissemination of dissenting opinions.

In the modern era, as information delivery systems have become more robust, we see the same destruction of knowledge taking place, though in a much more subtle manner. As the cost of distribution has been reduced, we have seen data become centralized: it is much easier to go visit Wikipedia on your phone than it is to download the page and carry it around. Also, Wikipedia comes with an open edit history associated with the documents, not all websites are so open.

This leads to two risks:

There is the risk of the lone copy, in a single organization's archive, being content being removed from the library (webserver). In the example above, Hirschfield had indicated his library should be donated to the University in the event The Institute was closed. This never happened, and the forced closure was deemed legal, ensuring all copies were destroyed.
This centralization means that edits to the content can occur with no historic copies being maintained. The edit history, being lost, can never track significant shifting of opinions. History can be changed.

The Internet Archive demonstrates the need for this: websites and content are removed from the internet regularly for reasons as innocuous as cost (part of the reason Git was developed was to protect OSS from being lost to public servers being shutdown), and as nefarious as governments shutting down news stations to silence dissent. Central repositories like the Internet Archive help to protect knowledge by allowing us to observe changes but also put the knowledge at risk by being the only keepers of history.

By distributing the data across many bookshelves, it is protected from complete loss.

A story about storage

Many years ago, I heard a story. I don't know if it is true, but it carries a valuable lesson.

In the early '90s, an amazing product became accessible that allowed people to generate a lot more data than they ever had, and of a higher quality than ever before: Microsoft Word. What had previously been stored on paper was now able to be digitally encoded and stored on disk. The archivists loved it, they were stuffing data onto disks left, right, and centre.

In the late '90s, Microsoft upgraded Word.

Into an incompatible format.

There was no way to go back and recover all that long-term stored data. Legally they were not allowed to as it had to be stored exactly as it was placed into storage (and signed off on).

In another twist, magnetic storage degrades over time and is subject to very limited environmental conditions. It is very easy to damage the storage medium.

In the story I was told, Archivists at the US Congressional library said “you know what doesn't degrade? paper.” And just started printing everything to paper, bundling the paper, and storing away in the existing vaults.

What if there was a way to have the best of both worlds? What if it is possible to have the fidelity of digital storage with the lifespan of paper; the volume of transmission available in Smart Devices, with the anonymity of in-person conversation?

Unfortunately, much of the data that is produced now is dynamic. By “dynamic”, I mean you can interact with the visualisation itself (scroll through a map; rotate a 3D model; filter, search, and aggregate massive datasets); and once it has been printed to paper, that is no longer possible.

A static image of an interactive visualisation of a SARS-CoV-2 protein (US National Institute of Medicine)

Also, it's hard to transfer large tables of data from paper to digital media. Scanning the documents as images and using OCR to collect tables of information loses significant amounts of metadata:

Data Types must be guessed from the content
Alignment issues cause data to be considered out of context
Character fidelity can cause incorrect values to be interpreted

While high-resolution photography and Artificial Intelligence have certainly improved the quality of scanned content, there is still an analogue transfer of data and it will result in some mistakes being made.

Defining the Problem

What if there was a way to have a compromise between the two worlds: the long-term storage of paper, with the high fidelity of digital; the anonymity of a private conversation, with the distribution capacity of a computer network?

What we are looking for is a means to store digital information on physical media such as paper or etched into stone. We might call this “visible” media.

Properties of Digital

Companies, governments, and individuals, have a desire to store data for long periods for legal archival purposes. This is hard to do. Over the past 20 to 30 years, the cost of digital storage has reduced as we moved from paper to magnetic storage. This presents a problem for archivists that must store the resulting volumes of data: as it becomes cheaper for us to produce data, it becomes a greater challenge for archivists to store that data.

The data must have a means of simple interpretation: it must be stored in a format that is easily converted to something a human can read. Open Source standards are advantageous as they are unencumbered by intellectual ownership and are readily understood by a larger pool of experts.

Copying digital data is something we take for granted. When we make a copy of digital data, it is an exact copy. For example, music loses some fidelity when recorded into a high-resolution format, however, the replication of the song from that point forward retains an exact copy (at the resolution of the bit).

Properties of the Storage Media

While etching into stone, or carving into wood are viable options, the weight and volume of these media present a barrier to storage space, and weight. Linen and cotton sheets represent lighter options but are expensive to produce. Mylar and projector film reduce the size, which offers good potential.

Modern archival paper represents a balance of permanence, weight, and volume. Each of these could (and should) be considered for various purposes, in fact, the solution should be adaptable to all these solutions. We discuss paper as the primary media, it is because paper has such a rich evolutionary history as a storage media.

In order for a digital storage mechanism, it must offer a reasonable level of compression. By compression, we refer to the number of bits of information stored per square inch or pound. This means it should be able to be recorded in a small physical space, though this must be balanced with an ability to read it back easily.

Solution

By combining the needs of both these mediums, we can put together a combination of existing technologies to create a unique solution. ePUB is an Open Source container format for Electronic Books which offers a standardized (ISO/IEC TS 30135–1:2014) and unencumbered format for a plethora of data. Further, the use of 2D-Barcodes (in the form of QR Codes) has become ubiquitous as a means of transmitting URLs, however, fundamentally, they are just binary buffers, capable of storing any encoded sequence of numbers.

ePUB

Diverse data storage
Compression
Accessibility Conformance
Widely Consumable

The transition from paper publishing to screen-based mediums brought some transitional challenges. PDF was popularized as a means of digitizing paper and acting as an intermediary between paper and digital formats. On the polar opposite end of the spectrum from paper, digitized standards (such as those developed by the W3C) have been optimized for delivery to an unknown display.

HTML introduced the idea of reformatting content to adjust to meet the needs of the consumer. This meant that the text could be read by a screen reader, could reflow for people reading on small screens, or the text made larger for people with poor eyesight. This accessibility of the format gave birth to a plethora of other standards now managed by the W3C. These standards ensure maximum availability to the greatest number of consumers.

ePUB takes advantage of these standards to encapsulate websites into a single document. They embed webpages into a ZIP file format to allow for the contained viewing of the entire website. Generally, the documents are organized into Chapters.

By using the common ePUB format, anyone would be able to read a digital document and decode it. ePUBv3 allows for JavaScript to be embedded, meaning you could embed maps, interactive diagrams, etc. (like R-shiny, but self-contained). As a general W3C container, it is also possible to embed other file formats for consumption and preservation: datasets as CSV, or evidence in the form of video.

Barcodes

You can encode digital information into barcodes which can then be printed to paper for long-term archiving, and the barcodes can be read back to a digital device for reading.

2 Dimensional barcodes have been used for decades as a means of encoding specialized information. BRML, text, or other data formats, have been appended to printed documents, such as Drivers Licenses and invoices, to supplement the text with digital information. This usually amounts to a unique document identifier or a digital record.

PDF417 barcode used by Drivers Licenses in many parts of North America [Wikimedia, CC-0]

Encoding an ePUB should be trivial with there being several issues:

The encoding scheme must be identifiable by a reader (there must be sufficient information embedded in the data to allow a reader to reconstruct the correct form)
The size of a single book will likely exceed a given 2D barcode's storage capacity. An encoding mechanism will have to be able to span multiple image tiles.
There is a social issue that must be managed in that humans cannot read the codes directly. It is possible that they do not wish to view the material for legal, religious, or moral reasons. There must be sufficient metadata to allow the viewer to decide not to accept the message.

The issues are easily overcome once identified; adding metadata to the individual tiles in the form of application identifier, pagination, title, author, and subject should offer sufficient information to allow users to interact with individual tiles and reconstruct the data.

A prototype of the concept has been created to demonstrate the capability. The prototype's protocol consists of

A URL: which points to the reader for either online use (browser only) or installation as a PWA, or just as a unique identifier that this is a compatible format
A Protocol Version: as changes are made, it is important that the correct decoder be used
Pagination: the current tile number and the total number of tiles to be converted. This allows for correct sequencing as well as a measure of progress
Bibliographic: Title, Author, and subject allow a reader to decide if this is content that interests them, or is legal for them to interact with. Filters can be added to prevent accidental downloads from taking up space
Parental Rating: not so much for parents, but generally for people that are not interested in certain types of content (filtering xxx content from a work device for example)
Relevance Date: some content is only valid up to a certain point, and should be ignored after that time (poster for a concert). Offer a hint to the reader that perhaps this could be removed, or ignored.

With this information in every tile, the read of the first image can result in some information being given to the user, allowing them to decide if they want to continue or block. If they determine they wish to continue, the pagination can be used to determine what order the buffers should be ordered in for reconstruction.

A prototypic specification is available in more detail.

Various Uses

Secure Archives

Having access to an archive comes with permission issues. Controlling access to information in archives that store sensitive data can be difficult. Using this encoding mechanism acts as an envelope around the content.

In the description of meta-data, the content rating was suggested. It would be very easy to reuse this portion of the protocol to use classification ratings. Users offered access to a secure document could have their specialized reader first check the classification rating of the content before decoding it. If the individual only has sufficient clearance to view some related documents, but some of the documents in the area contain information that exceeds the individual's current clearance, it can act as a secondary filter for viewing it.

Obviously, this would be a tool to assist honest actors within the environment and not a way to interfere with malicious actors, but this is another layer of protection which assists the actors in managing the information in their possession.

Information Dissemination

Assuming you are in a place where information is controlled, you could print essays and newsletters to paper, which can then be scanned for reading later. For example, it could be printed in a pamphlet or posted on a bulletin board, and nobody would know who published it (beware of barcodes hidden on printouts).

One of the advantages, in this case, is the high compression ratio. In an early test, a hundred-page novel was compressed to 9 pages of barcodes. While still requiring some effort to distribute, the entirety of the novel could be tacked to a corkboard.

The contents would then convert to something readable on your phone, like an inspirational poster.

[Wikimedia, Public Domain]

Remote Interactive Media

Textbooks, posters, and advertising all have the common element of having to display content in physically contextual locations: a sign in a museum, a poster stapled to a lamp post. Access to network communications is not guaranteed and the audience misses out on an opportunity.

Take, for example, a sign at the top of a mountain congratulating a mountain climber for their successful journey. A digital experience message could be left at the top but would require a WiFi-based website to be configured and powered.

Alternately, storing the immersive experience on the poster itself would allow the digital content to be available, but not require any power for maintenance.

Etching the information into something more permanent, such as wood, or stone, may be appropriate in a circumstance such as this

The Judaculla Rock shows carvings from 2000BC demonstrating the staying power and low maintenance capabilities of etching into stone [Wikimedia, CC BY-SA 4.0]

Conclusion

Anonymous and long-term storage of data and information is necessary. The free dissemination of ideas, and the storage of them for future reference, is a fundamental need for the progress of society. While the digital age has made information access easier than ever, it has introduced a host of new problems in its wake.

The use of paper as a digital storage medium is a novel and useful approach to addressing some of the new problematic circumstances.

If you find this concept of interest, I invite you to review a prototype of the concept on GitLab. There is a mobile application available, that can convert an ePUB to images and paper, and convert it back; right on your smartphone.

There are several ways you could contribute:

Design a UI: I just grabbed an old one from another project to get up and running
Write a Reader: currently, the app acts as a bookshelf. Some features would be well suited to a custom reader
Custom Filters: users should be able to filter content by author and title if they come across something that does not interest them.
Pass some notes around your school: Using a system highlights its problems. Post a club listing to a bulletin board at school using this encoding mechanism.

Submit an issue, post a merge request, or leave a comment below. If you found this content valuable, please remember to click the follow button.

dpub / Barcode EPub · GitLab
An experiment in using barcodes as a storage medium. The intent is to create an EPUB reader that stores its data on…gitlab.com

UPDATES

Since writing this, I have come across several other interesting, and related reads

2022–11–15

The Internet Archive, Digital Books wear out faster than Physical Books (November 15, 2022)

2023–07–19

NYU Law, The Anti-Ownership Ebook Economy

Technologic (In)accessibility

2022-08-07T00:00:00Z

I went to the beach a couple of weeks ago.

Beaches are hard to come by on the bald Canadian prairies, so this was a rare treat for my wife and I. Having moved from a coastal city, we were looking forward to getting a chance to just wade into some water, as well as introduce our young dog to swimming. Having learned about Sandy Point Beach, in Lacombe County, we were very excited and more than willing to spend a couple hours in the car.

Upon arriving, an older gentleman wearing a “security” uniform informed us that parking had recently become paid parking (no problem) and that we just had to scan a QR code on a card he gave us (big problem).

I immediately asked if the facilities offered free WiFi to be able to connect to the payment service, and was assured there was plenty of service just up ahead. I tried to be specific that I do not have a data plan on my phone, but he did not seem to understand. I rolled the dice and proceeded in, found a parking spot, and proceeded to attempt to pay: no WiFi. It did occur to me that perhaps it was closer to the change building so I walked over to the building and tried there: no WiFi.

This was going to be a problem.

Upon getting back to my car, I found the security guy had already taken my plate number. I did ask him what alternate options were available for me to pay. I did try to explain that I did not have any means to connect to the internet, but that only resulted in him getting frustrated with me and simply stating, “you just open your phone and you get the internet”.

Unfortunately, this just isn't true for everyone, and (as is often the case) assumptions regarding capabilities of people result in barriers to accessibility. It is these assumptions regarding customer abilities that lead to wheel-chair ramps not being considered, hazards not being demarked for the visually impaired, and audio tutorials not being offered to the hearing impaired.

To some extent, assumptions are inevitable: not having lived a particular experience, it is natural to be unaware of the nuances involved in that experience. Fortunately, society is (mostly) aware that we are sometimes unaware of these physical barriers to entry, and we realise the importance of seeking expert advice on the matter.

There are subtle nuances, and significant details, that experts in a domain are aware that are non-obvious to outsiders (p55, Alberta Government: Barrier Free Design Guidelines, Fair Use)

Unfortunately, there is another set of barriers to that can be described as “Technological Barriers”, which are often not even considered. Due to marketing and increased cost of implementation, technology is sold as being “simple to implement”, which unfortunately overlooks some of the complex nuances of where humans interact with technology. It is important that we be thoughtful in our design of technological space, as well as physical space.

Thoughtful design means to consider how a space is to be used … The objective is to remove as many barriers as possible.

— Alberta Government, Barrier Free Design Guidelines (p.2)

Accessibility of Technology

As a user progresses from the discovery of the service through the decision to seek out the service, there is any number of things that can prevent them from actually engaging with the service. We all know that technology offers a significant means for reducing these barriers: digitally readable text increases the options for consuming text, networking allows communication to reach the consumer rather than the consumer having to come to the message.

By diversifying our modes of communication, we create redundancies and alternate paths for our consumers to follow, these alternative paths allow those with various barriers to seek an alternate path to the same outcome.

In any customer progression toward success, there are a number of barriers that slowly whittle away at those capable of enjoying the product or service

One of the risks of easy and low-cost solutions is the temptation to use it to the exclusion of all else. This leaves no means for those with accessibility issues to bypass the barriers in their way … and without personal experience, we are likely to be unaware that those barriers exist. We need to rely on experts with domain knowledge to avoid making assumptions based on our personal experience.

In this case, the assumption is that everyone has access to mobile devices and mobile data plans through one of the major Canadian providers. Unfortunately, this just isn't true for 34% of Canadians, who, in a 2019 study, did not have access to a Smart Device; and there are many more of us who do not pay for the internet to be accessible from our devices.

Infographic: Smartphone Ownership: The Mobile Disconnect
The Statista "Chart of the Day" currently focuses on two sectors: "Media and Technology", updated daily and featuring…www.statista.com

This appears to have changed during the pandemic, but is still not complete coverage. Statistics Canada's observes that 20% of Canadians do not have data plans on their mobiles as of 2021. This is even more pronounced among rural Canadians at 27%, the very customers the rural county of Lacombe is trying to serve.

Access to the Internet in Canada, 2020
More than ever, Canadians relied on the Internet to perform various activities in their daily lives in 2020. Access to…www150.statcan.gc.ca

### Cost of Technology

In the mid-2000s, the trend was toward shared WiFi and WiFi networks, this was a low-cost and ubiquitous solution to internet connectivity in urban areas (I loved my Nokia N800). As WiFi became ubiquitous in coffee shops and offices, many people have never felt the need to purchase data plans.

In Canada, it is not unreasonable for a couple to be paying $1800–$2500/year for internet connectivity on their phones, phones which themselves cost $720–$1200/year (Shaw & Telus Mobile Plans). If the average income of a couple in Lacombe County is about $82,000/year, a total data cost of $3700/year represents a significant proportion (4.5%). Considering the significant acceleration of inflation in Canada, leading to a current 8.1% inflation rate, one can foresee that Canadians will be seeking means to reduce their household expenses, and a 4.5% budget item, with an easy workaround (public WiFi), is an obvious candidate for cost reduction.

Canada is currently experiencing unprecedented inflation at 8.1%. Since early 2021, inflation has been accelerating (Bank of Canada, license).

… and that is not even the group we are concerned with.

When designing a system, with an eye to accessibility, it is important to consider those that are outside the norm. Having an average or above average income represents a privileged group of decision-making individuals. It's easy to forget about the 16% of the population that is financially vulnerable.

In the case of governmental resource management, it is important to consider the less advantaged and vulnerable populations of the community. A day at the beach represents an attractive, cost-effective, activity for low-income and vulnerable individuals (retired pensioners, single parents, or those who have just fallen on hard times). Socially, these are the people best served by having access to public resources, and at the same time are also those most vulnerable to having to make hard budget decisions.

Table Topping

When engaging in designing of a technological system, it is important to “walk through” your proposed system with the intention of identifying vulnerabilities that may be present. This walk-through need not be done on-premise (though that is advisable), rather it is often worth doing as a “Table Top Exercise” in the early stages.

By designing the system, and then testing it by visually describing the anticipated process, you can begin to map points in the process that may be subject to issues and barriers. This is similar to risk mitigation in project planning, but differs in that it required system to have been designed first. You are testing a design to ensure you have handled all cases, rather than trying to plan for all cases. This should also be an iterative process: design system, identify flaw, propose change, repeat.

A quick mind-map addressing potential negative outcomes that should be addressed (Mindmup)

These issues and barriers can be documented, and then considered for the probability of occurrence, significance of impact, and mitigation plan. Not everything need be handled, but they should at least be addressed.

It is very common in this process to have our bias show through and to be dismissive of an issue that is difficult or uncomfortable to address or sometimes just challenges our worldview (eg. I have internet on my phone, everyone I know has internet on their phone. Therefore everyone has internet on their phone). This is most dangerous at executive levels, as off-hand remarks may communicate decisions and desires to analysts and designers (having opinions is normal, but voicing them can be dangerous).

Conclusion

Having walked through the case, we can see how important it is that organisations approach their system design with an eye to accessibility and take active measures to not allow analyst and executive privilege to create a bias that excludes the vulnerable. We can also see some of the tools used to mitigate and manage some of these issues.

Unfortunately, this particular implementation of technology, to support parking payment at Sandy Point Beach, has made the beach inaccessible to many people. While I understand and agree that the county should be charging some fee to offset the creation of an artificial beach in the middle of The Prairies (I saw the mountain of sand off to the side), having no onsite means of payment available to patrons (WiFi, cash kiosk, or digital kiosk) means that what should be an affordable, and accessible, form of entertainment for residents, and tourists, has an insurmountable technological and financial barrier for many.

In response to a draft of this article, Lacombe County indicated that this has been a trial run, focusing on education, and therefore no tickets were issued. They also pointed out that payment can be made in advance at the County Office or online, though neither of those options are present on their website at this time. I hope they take this feedback into consideration during their evaluation, and consider consulting with appropriate domain experts.

Unfortunately, this demonstrates how we (as decision makers) can fail to identify these issues as we rely on our own life experience, failing to be aware of the diverse experience of others. We can fail to address issues in advance, or fail to create viable alternative success paths.

In the end, it was a frustrating start to what was supposed to be an exciting day for my wife, myself, and my dog; and has resulted in turning a public resource into a resource technologically accessible only to the privileged.

UPDATE 2022–08–12

Healthcare professionals say that dangerous heat puts marginalised and vulnerable communities at risk because low income populations have a more difficult time accessing cooler spaces and green-spaces

— Millions of Canadians try to stay cool during heat wave, CBC, 2022–08–12

Also

apparently Canadians pay a lot for their mobile data
the World's most expensive according to some

Bald Prairies (Wikipedia, CC BY-SA 2.5)

Fishalytics

2022-07-08T00:00:00Z

Fishalytics

A (failed) experiment in data analysis as a behaviour modification tool

About 12 years ago, I gave up my 100 hour work weeks as a software developer, moved to the other side of Canada, and bought a small-holding farm. One of the first things I did when I moved to the new region was to buy a fishing license and was shocked when they handed me an official “catch form” with my license.

In Nova Scotia, when you have a recreational fishing license, you are expected to keep track of the number of fish you catch, the region you caught them in, as well as what species they are.

For a year.

I was struck by how inefficient this was. It was highly unlikely that I was going to remember every fish I caught for a year, it was also unlikely that I was going to be able to remember where I had put the fishing scorecard that I was required to send in at the end of the year.

A screen shot showing fishing licenses over the years. While I primarily fish in Canada, an acquaintance did give me written permission to fish his private fishery in England. A fond memory to keep track of.

This seemed like a simple data entry form project that I thought I might even be able to sell to the provincial government. At its simplest, the idea was that I could write an application that allowed fisherman to enter their fishing license information, and record their fish as they caught them. At the end of the year, the software could just print the score card for you.

Over the next 10 years, this tool became the focus of much of my internal pondering regarding system development and the ability of software to affect social change.

I even went so far as to get my wife to design a logo for the project. A fish, over a grid, jumping from a stream (which is actually a the curve of a graph from the app).

From Simple Record Keeping to Analysis

While the initial concept of the software was to simply as a record keeping tool for myself, it was not long before I started to think bigger.

If I built the tool, I may as well make it more widely available.
If I wanted others to use the tool, I needed to offer them more than just a fishing journal.

I had just hit on my first element of social design: you must offer something to the individuals you want to harvest data from. Like one of the major social platform, I could harvest user data and sell it on, but in order to do that, I first needed to offer something in return.

There were two “value add” features I could think of right away.

The social aspect. Keeping pictures and memories of catching fish is fun. Sharing fishing tales, in near real-time, would be a fun way for fishermen to engage with friends, and a great way to record them for future memory.
Basic analysis. Knowing where you caught the last fish gives you some idea where to catch future fish, and using aggregate data of many fishermen, I could actually offer solid, unbiased, advice on where to catch the next fish.

That first idea that people could share was an important one, Facebook was young enough that competitive services were still reasonable in niche fields. Unfortunately, social platform development was really outside my domain of expertise. The second option spoke to strengths I had, and it occurred to me that if I could develop a successful predictive product, partnering with a social partner could be performed later. Another benefit to focusing on the analysis was that I was the sole user at this point: it is almost impossible to build a social platform around a single individual, however it is possible for a single individual to gather multiple data points for an analysis.

The first focus was set: predict what leads to catching more fish.

Planning the Data Gathering

In order to suggest to users what they should do to catch mroe fish, I first had to think of the variables that would impact the ability to catch fish. The best place to determine this was at my local lake, staring out over the water.

What about my fishing (behaviour) could affect my catching fish (outcome)?
Also, what could affect my ability to catch fish, but was difficult to measure?
Lastly, what was easy to measure, but probably had little to do with my catching fish (red herrings)?

In brainstorming these things, I came up with several things that had a high impact on my catch rate and were reasonably easy to measure.

Location
Time of Day (solar declination)
Temperature (air and water)
Solar penetration (cloud cover)
Covering vegetation
Lure
Time spent fishing
Time of Year

Unfortunately, of these, only two seemed easy enough that I could expect the average fisherman (ie. me) to collect them regularly. Space-Time is easily captured on a phone, so anything involving those metrics is easy to capture: location, time of day, time of year, and solar declination.

These two items can be measured easily through smart-phone GPS loggin. By marking the start of a fishing trip, and having the application continuously log the position of the person fishing we can get a sense of how long they stood in a given location with a line in the water not catching fish. Upon catching a fish, it is natural to want ot capture the moment, snapping a picture (again through the app) marks the moment and location at which the fish was actually caught.

Measuring Success

Putting these variables together into a meaningful metric became the next problem.

In attempting to determine that which variables lead to success, we need a clear definition of success. To some extent this requires distinguishing causal variables from outcome variables. In order to determine how a fisherman would consider themselves successful, I could think of no better way than to interview a recreational fisherman.

You find gold where the gold is

— Prospector's Proverb

I went fishing.

Standing waste deep in water at my local lake, with a line in the water, I began pondering the variables that would make me successful at that moment. As I stood there, I realised it was “catching a lot of fish”, catching fish is exciting, even small fish. So there is an element of quantity. The sheer amount of fish you catch, is a positive experience. But standing in a lake for 4 hours to catch two fish is not the same as hitting 2 fish in 20 minutes: the velocity at which we catch fish matters.. It was at t his moment that I caught the most impressive Small-mouth Bass I have ever caught. A big fish, on a small fishing rod, is a fun experience. Fisherman brag about that giant fish they caught. So its not only a measure of quantity of fish caught, but also the quality of those fish.

A fishing trip can be measured as being successful by the “velocity” at which you catch fish.

velocity = quantity of things
------------------
time spent

There is a little more to it, as we want to factor in the quality of the fish caught. We can also consider the entire time spent as a single “fishing trip”.

velocity = number of fish caught * quality of fish
---------------------------------------
time spent standing next to water

The problem with this definition was that it was not very granular. The end goal was to create a heat map representing the “best” places to go fishing. This heat map would represent a range from null (no information) to good to bad. So while a “trip” represents a range of space-time (different fish caught at different locations and at different times), I required highly granular data that specified a point in space-time.

A hypothetical fishing trip with two fish caught. The fish took a certain amount of time to catch which represents an effort on my part. [original]

Here I take a cue from accounting. In addition to the point in time when the success was achieved, I can measure the time in between as a the cost of catching the fish.

In the example above, the Small-Mouth only took 20 minutes to catch, however I continued to fish without catching another for a full 50 minutes. We therefore allocate the unsuccessful time to the nearest fish caught.

Given this new perspective, we can change the measure of success to:

score = quality of the fish
--------------------------------
time spent fishing for that fish

While the trip score can be considered the average of all the individual fish scores.

We are getting closer to a simple score with a clear definition of how to measure the time per fish. Unfortunately, the definition of “quality” of a fish is not a straight forward measure either.

Generally size is considered the measure of a successful catch, but not all fish are considered equal. If I am fishing in a mountain stream and catch a “good sized trout” it is going to be a very different size from even a small Great White Shark. Age is also a factor: young fish will be smaller than older fish.

Another wrinkle enters when we consider why the government was going to be collecting this data: fisheries management. They want to know the health of the regional ecosystem. Under these conditions it is not sufficient to know that the fish is “bigger”, but that it is an appropriate weight for its age. High or low values could indicate various stresses on the population.

A species' Standard Weight is a measure of the average size of a fish given its height. This is basically BMI for fish: given a fish's length, we can consider its normal weight. This weight follows an exponential curve (fish get fatter faster than they get longer), and is unique to each species with each species having two constants that define their normal curve.

Two Standard Weights for two different species. As length increases so does weight, but at different rates. (Wikipedia CCSA-3.0)

In order to make use of this piece of Standard Weights a database of the a and b parameter for each species is required. Fortunately that database exists in the form of Fish Base, an online catalogue of fish research from around the world.

To identify a fish's parameters, all that is required is to know the fish species (common name is acceptable) and the location it was caught. This results in a a page of information about a fish, including its Standard Weight. For example, our Small Mouth Bass: (a) 0.0129, and (b) 3.06, (len) 8" or 216 mm, (weight) 1/3 lbs or 153g

The mean `a` and `b` values are offered in the footnotes.

Given this information, we can consider the “quality” of a fish to be its variance from its Standard Weight. Note, that for convenience, scores are shifted to a positive range (catching a fish is always a good thing) between 0 and 1000 ( because per mille has always amused me)

stdWeight = StdWt(a,b,mm)
= StdWt(0.0129,3.0600,203)
= 156.3g

quality =  weight - stdWeight
           ------------------
               stdWeight

        =  153g - 156.3g
           ------------- 
              156.3g

        = -0.021113243762

        (for convenience, convert to per mille)
        = floor(-0.021113243762 / 2 + 0.5)
        = 489‰

This should be further converted to the score by integrating the time spent fishing for the fish:

score = quality / time
= 489 / 70
= 7

Using the standardised measure of quality method gives us a measure of quality of each individual fish caught, which allows us to produce aggregate values (such as the trip score) without concern for species variability.

Analytics

By query a space-time bounding box, a user can view places where the fishing has been particularly good, or particularly bad. This was charted using OpenStreetMap, Leaflet.js, and Heatmap.js. This is not only useful for fisherman, but also for ecologists looking to study the quality of the fish in the area.

The default example from HeatMap.js (demo)

With sufficient observations, this standardisation of the data would allows for several other types other analysis

Year over year analysis is possible allowing ecologists to monitor for trends of declining or recovering populations.
Species comparative analysis can show one species filling in for another species, a common symptom of an environment in distress.
Time of day analysis, or seasonality, can improve catch rates by fisherman
Poaching rates observed by anonymously reported poaching

This finally made me realise there were two target audiences for this data, other than just fisherman:

Fisheries Management

The ability to turn fishermen's stories into meaningful analytics meant fisheries don't need to wait until the end of the “season” to get paper records. Real-time catch data, collected customers, can be used to gain insight into the health of bodies of water. This means early interventions can be taken.

The Ecosystem

Fish populations are early warning signs of ecological disasters. Changes in types and sizes of fish populations is a good indicator of ecological health of the water. These analytics give real-time, and early detection of ecological issues.

Fisherman don't want to fish an over-fished area. Ecologist don't want fishing happening in stressed areas. A system like this was hoped to identify healthy populations of fish and direct fisherman toward those; leaving stressed populations alone to recover.

Conclusion

FishAlytics was a failed social experiment for me. After nearly a decade, I abandoned the project without it ever moving past a personal fishing journal. Competing priorities (I was a farm labourer), legal obligations (I didn't wnat to loose intellection property while working for some companies), and theft (a customer I pitched this too, pitched it back to my students 4 years later), left me just not working on it and finally letting it go.

But I don't think it was a waste of time either. It has offered me my first real introduction into the possibility of using Data Analysis as a tool for social change. Also, I started to conceive of how passive pressure could enact social change.

It also introduced me to the idea that everything can, and often should, be boiled down to a single metric of health. This has been beneficial to me in other Data Analysis roles where being able to identify variance in an abstract metric of “health” has allowed for early intervention.

While I failed to achieve any of the desired results, I hope this article helps others see software and systems as more than just forms on a page. Rather to view them as tools for evolving a social systems, and modifying behaviour for the the betterment of all.

Not storing (almost) the same file twice

2022-04-11T00:00:00Z

Photo by Jeremy Bezanger on Unsplash

In De-duplicating Data Storage I, I showed a technique for reducing the storage load on a system, where the system encounters multiple copies of the same file. This is a fairly common situation in data-intensive environments as Data Scientists make copies of the dataset they are working on, and their colleague is working on, and they shared with their friend. Applying the techniques, we showed a 36% increase in profits in a laboratory scenario.

There is another behaviour that is very common: Data Revisions.

Naturally, as time progresses, our source datasets change, resulting in changes to our output datasets. While most people think of keeping all data forever, for the most part, we are only concerned with the current state. However, FOMO keeps us from removing now historic, but redundant datasets. Over time this data continuously grows,

I'm going to share a way to reduce this cost by 90% over 3 years.

90% savings in 3rd year

An example from my past was a heat map of financial transactions across Canada. This dataset was based on the last two years of financial data aggregated at the Postal Code level. To place the items on the map, we had a second dataset of financial districts and a third set that consisted of postal codes and their corresponding financial district. Just to add a layer of complexity, Canada Post changes their postal delivery routes regularly, and therefore their postal codes, also the boundaries of our financial districts changed regularly. This means that the proportion of a Postal Code, that resides within the Financial District is not consistent and required constant updating from a third-party provider.

Regional heatmaps are a common means of visualising data. An example can be found via Statistics Canada's annual population estimates (StatsCan Annual Demographic Estimates, StatsCan Open License)

The network drive folder looked something like this:

name size
—————————————————————————————————————
/proj/1/dashboard.html 12 KB
/proj/1/transactions.tns 50 B
/proj/1/postalcodes.csv 33 MB [ref]
/proj/1/districts.gml 37 MB [ref]
—————————————————————————————————————
70 MB

So we have three datasets: our live financial data (a live connection), a listing of postal codes, and a listing of location shapes. The relationship between the datasets and the final aggregate that is displayed on the dashboard can be described as

select
d.label,
d.shape,
sum(p.weight * t.amt) as amt
from
transactions t
inner join postal p on f.postal_code = p.code
inner join districts d on d.id = p.district_id

While the actual financial summary is updated in real-time, we receive updated districts and postal codes every quarter. A fairly normal practice is that every quarter, we receive an email with a link to a CSV, and someone has to go download the file and overwrite the current CSV files, however it is considered a best practice to make a copy of the old files in an archive folder. This is done by creating a copy and date-stamping it.

name size
—————————————————————————————————————
/proj/1/
dashboard.html 12 KB
districts.gml 37 MB
postalcodes.csv 33 MB
arch/
districts.202203.csv 37 MB
districts.202204.csv 37 MB
postalcodes.202203.csv 33 MB
postalcodes.202204.csv 33 MB
—————————————————————————————————————
Total: 210 MB

I'm sure many people recognise this pattern of file management, as it is very common. It is also easy to see how this can spiral out of control.

notice that this is proj/1 of … let's say about 100 active projects
these projects have been running longer than 2 months (average of 5 years)
corporate policy requires redundant backups of the drive space, with point-in-time recovery capabilities (daily for a month, monthly for 10 years)

Given these approximations, we have quickly consumed 210MB (and that is a very small project) of data per project, with 60 months of user copies, on slow archival disks with 150 disk snapshots, and 100 projects. A total of 180 TB. The policies and project volumes are all realistic, in fact, I suspect I'm underestimating the storage demand. Assuming AWS S3 Standard storage of the files ($0.021/GB), this costs roughly $ 70,000 CAD/year. (please, check my math)

It's not my money … why should I care?

There were two events that made me care:

some users were denied access to the service because there were concerns about disk consumption and cost, and their business case write-up did not persuade the executives: a significant loss to the organisation.
a different user asked for the recovery of a single file, from an historic checkpoint. The backup team informed him they would need to “find a disk big enough to restore the point in time to”. The backup team needed to restore the entirety of all the projects to get one file: a significant expense to the organisation.

These are demonstrations of bureaucracy getting in the way. It stems from a poor understanding of digital storage and information management techniques on both the part of the IT department as well as the Data Scientists.

Sometimes we need to keep stuff moving, even when bureaucracy gets in the way.

A Quick History Lesson

The disk usage pattern in question is a well-established and intuitive one, usually developed by students in their first year of college. Changes made to complex systems can result in unanticipated outcomes, it is also not always obvious which change (or combination of changes) led to the behaviour; having old copies can help you to understand what has gone wrong.

Having identified only the portions of a file that have changed, the basic problem becomes very recognisable to most programmers (actually most publishers of content) as a Change Management or Version Control. Over the decades, several tools have evolved to manage this problem called Version Control Systems (VCS). While the field is littered with VCS, there are a few prominent ones that represent significant changes in the way changes are thought of and managed

CVS (1985) — changes to individual files are tracked independently
SVN (2000) — related changes to multiple files are considered a single unit
Git (2005) — collections of changes are able to be managed as separate units

These three tools represent important changes in the understanding of the way in which the databases we store revisions of data should be structured.

In a large volume of data, it is important to be able to see the differences between instances. Asking “what has changed”, in large volumes of data, can be very difficult. Difference tools ( diffing) became standard in the data and programming toolkit (diff) in the mid-70s. Further, tools like patch offer a way to transmit (or store) only the changes which may be significantly smaller than an entire copy. There are now a plethora of graphical tools for such tasks.

Meld is a graphical tool for comparing the contents of text files to determine what has changed. These tools can be used to great effect on CSV files to determine if a significant change has happened. (wikipedia:GNU License)

Techniques to Reduce the Problem

These are well-established problems, but with some best practices that have evolved.

Compression

While not the focus of this article, compression is an easy and often overlooked solution to the problem. From a solution architecture, perspective, I am often glad it is overlooked. While putting the files in a zip archive is an easy solution, it does reduce the visibility of the changes: files need to be decompressed before they can be compared. Compression should be maintained inside the solution and abstracted from the user.

Having users compress their files actually reduces the ability of tools to take other actions that may have a larger impact

Right-Scoped Repositories

In our initial problem description, the IT team had difficulty restoring the backups because they needed to restore the entire repository to a point in time.

Rather than creating backups of the entire repository (which is measured in petabytes for our example), it may be more sensible to break the problem down into sub-parts. Many of our projects will get archived over time, projects change at different rates, and usage may decline or increase. This means that some data will have a greater or lesser probability of requiring a restore from the point in time.

Dividing the backups at a “per-project” level offers an obvious point of division. This allows us to strike a balance between the ease of backing up “instances” with the ease of restoring smaller instances to help the user.

Only Store Changes

As discussed, tools like diff and patch offer a means to identify, store, and apply changes to larger files. Rather than storing multiple copies of the dataset, it is possible to store a primary dataset, and then track the series of changes that have taken place on it.

For example, country lists change regularly, requiring an update for even a spelling change. A CSV based on the Wikipedia page for ISO country codes could be modified with a patch description to accommodate Hungary's name change in 2012.

This is significantly smaller than storing the entire file of hundreds of countries for a single name change. As a tip, the primary should be the one you are actually using, and the changes can work backwards in time.

@@ -117,1 +117,1 @@
-Hungary,Hungary,UN member state,HU,HUN,348,.hu
+Hungary,Republic of Hungary,UN member state,HU,HUN,348,.hu

A Solution

Rather than manually perform all of these steps, these are the problems that modern VCS applications were developed to solve. As an odd quirk of history, subversion (SVN) is particularly well suited to handling large files as it only tracks the differences between states.

While I would recommend any VCS solution as an improvement over file copies, SVN is ideally suited to the purposes of Data Analyst's management of large datasets.

Take our original problem of a project storage system with archived folders

name size
—————————————————————————————————————
/proj/1/
schema.json 128 B
dashboard-template.html 232 KB
dashboard.html 100 MB
districts.csv 37 MB
postalcodes.csv 33 MB
arch/
dashboard.20220301.html 100 MB
dashboard.20220315.html 100 MB
dashboard.20220401.html 100 MB
dashboard.20220415.html 100 MB
districts.202203.csv 37 MB
districts.202204.csv 37 MB
postalcodes.202203.csv 33 MB
postalcodes.202204.csv 33 MB
—————————————————————————————————————

I have modified the example to include output historic reports built from a template.

Rather than trying to retrofit, let us start over (with project #2).

One of the very first improvements we can make to the storage structure is to create an independent archive location for each project. This independent location can then be turned into an SVN-controlled location.

mkdir -p /arch/2;
cd /arch/2;
svnadmin create;

Now we can link the archive location to the working location.

mkdir -p /proj/2;
cd /proj/2;
svn checkout file:///arch/2 .;

Once this is done, we can begin to create our space and apply the changes as we make them.

name size
—————————————————————————————————————
/proj/2/
schema.json 128 B
dashboard-template.html 232 KB
dashboard.html 100 MB
districts.csv 37 MB
postalcodes.csv 33 MB
—————————————————————————————————————
cd /proj/2;
svn add *;
svn commit -m "Change 2022-03-01";

copy /proj/1/arch/dashboard.20220315.html dashboard.html;
svn commit -m "Change 2022-03-15";

copy /proj/1/arch/dashboard.20220401.html dashboard.html;
copy /proj/1/arch/districts.20220401.csv districts.csv;
copy /proj/1/arch/postalcodes.20220401.csv postalcodes.csv;
svn commit -m "Change 2022-04-01";

copy /proj/1/arch/dashboard.20220415.html dashboard.html;
svn commit -m "Change 2022-04-15";

Under these conditions you will create snapshots of the changes at each point something changed,

reducing the number of snapshots you have to maintain. No change; no snapshot
backups are generated per-working folder, meaning if a restore is required, it only takes the size of the individual project to go back in time.
your backup footprint is reduced because SVN only stores the differences between each snapshot.

name size
—————————————————————————————————————
/proj/2/
.svn/ 13.4 GB
schema.json 128 B
dashboard-template.html 232 KB
dashboard.html 6.7 GB
districts.csv 5.0 GB
postalcodes.csv 1.7 GB
—————————————————————————————————————

Note the creation of the folder .svn , this is true for most VCS applications. They must create a control folder for tracking their link to the repository. Also, this folder will contain a single copy of the folder structure to allow it to detect changes, doubling the storage space in the short term. The total storage space in the working location remains largely untouched as the changes are made.

Snapshots can be viewed via SVN's log command

$ svn log ^/ -qv
--------------------------------------------------------------------
r4 | jeff | 2022-04-15 00:43:13
Changed paths:
M /dashboard.html
--------------------------------------------------------------------
r3 | jeff | 2022-04-01 12:25:08
Changed paths:
M /dashboard.html
M /districts.csv
M /postalcodes.csv

And if a restore needs to be performed for a user, it is simple to select the historic revision.

mkdir -p /tmp/history;
pushd /tmp/history;
svn checkout -r 3 file:///arch/proj/2 .;
popd;

One final benefit is that because the archives are no longer stored with the working copy, a different type of storage can be applied to the backups. Slower storage can be applied to the backups, while faster storage for the working copy.

The Social Aspect

Unfortunately, in many cases of encountering this problem, I have seen the interested parties pointing at one another and saying it is the other person's fault. Data Analysts are unaware that version control tools exist, and IT departments view backups and restores as whole-system-events for recovering from complete system failures.

The question becomes: who is this article designed for? Data Scientists or System Administrators.

Really, this is for both. Hopefully, both parties work together to reduce costs and burden on the other, but the reality is that a change will likely have to start with the IT department.

Further, reducing costs is not seen as a positive in most organisations. Cost is associated with prestige, managerial resumes often boast about the size of the budget they managed. Reducing the budget reduces prestige.

A Scripted Solution

Given the social problems, the simplest means of introducing users is to simply implement a simplified Version Control practice in an automated fashion, without asking users or offering training or really talking about it.

There are two steps that should be taken:

Immediately install a VCS client on every computer that accesses the system in question
Integrate VCS creation into the project allocation part of the process

On the project setup and allocation side, the process usually starts with a request for space on the computer as a paper form (yes, companies are still filling out paper forms as of 2022–04–10). As this kicks off a large process involving multiple configurations being manually configured, ensure the allocation of a VCS is part of that process. The simplest way to ensure this is done is to automate the entire process.

#!/bin/bash
proj="./proj";
arch="./arch";
function SetupFolder {
f=$1;
pPath="${proj}/${f}";
aPath="${arch}/${f}";
tPath=$(mktemp -d);
mkdir -p "$aPath";
svnadmin create "$aPath" && echo "Repository created." \
|| echo "Repository already exists ($?)";
svn checkout "file://${aPath}" "$tPath";
mv "$tPath/.svn" "$pPath/.svn";
rm -rf "$tPath";
}
{
proj=$(realpath ${proj});
mkdir -p "$proj/00000000";
echo "Project Folder: $proj";
arch=$(realpath ${arch});
mkdir -p "$arch";
echo "Archive Folder: $arch";
for dir in ${proj}/*/ ; do
dir=$(basename "${dir}");
echo "Updating: $dir" 1>&2;
pPath="${proj}/${dir}";
[[ -d "$pPath/.svn" ]] || {
echo " - linking project" 1>&2;
SetupFolder "$dir";
}
pushd "$pPath";
(( $(svn status | wc -l) > 0 )) && {
echo " - synchronizing changes" 1>&2;
svn status | grep -e "^\?" | cut -c 9- | xargs svn add {};
svn status | grep -e "^\!" | cut -c 9- | xargs svn del {};
svn commit . -m "$(date -Iminutes)";
svn cleanup;
};
popd;
done;
} 1>/dev/null;

This script scans all of the folders and determines if a change has been made. If there are changes, it submits those changes to the archive location, if no archive location exists, it creates one. This can be run on a timer, or better yet, can use inotify to monitor for changes. Better still would be to install a self-serve interface like GitLab that can auto-allocate space and control permissions, but … bureaucracy.

Teaching Users

Secondly, install a highly visible VCS client on every computer in the office. Personally, I tie the installation of the client to the LDAP group that gives basic access to the Data System. By highly visible I mean Tortoise on Windows, or RabbitVCS on Linux. In both cases, the users are automatically presented with icons that tell them something is unique about the folder they have been granted access.

The Tortoise family of products places overlays on file icons at the operating system level. Users receive visual feedback that “something is special” and that they should learn more. (TortoiseSVN Manual, GPL)

One of three things will happen:

A user already familiar with Change Management Systems will be pleasantly surprised
A user unfamiliar with Change Management Systems will curiously explore this new domain.
The user will not care… “you can lead a horse to water, but you can't make it drink”.

Once users discover the log feature of the VCS, they become empowered to restore old versions as necessary. This reduces the burden on IT and the instances of manual duplication of files.

Once users see the log of the changes, they become aware that a history is kept for them, and they become empowered to restore their own historic data.

A Demonstration

To demonstrate how this all fits together, as well as to compare the compression capabilities of various setups, four versions of the script were created:

Found this Intersting? Leave a Tip…. it helps

CPY: Makes a copy of the files in the project folders over to the archive location. This is simply to offer a baseline comparison of what our users are currently doing
FSZ: Filesystem compressed. We hope our users use ZIP, so this creates a compressed copy of the folder each time the backup is called
GIT: Git is an excellent VCS, and should be included in any comparison.
SVN: The script that was offered earlier.

dedup2 · main · Jeff Cave / Demos
An example of the how to apply the data differential techniquesgitlab.com

To simulate the activities our users go through, a script called demonstrate.sh downloads the history of the CIA World Factbook as JSON from GitHub, and stores each change in the project folders, and then backs them up. This is similar to our users receiving updates to their data files and then saving the old version to a backup folder. Results are stored in results.csv.

Demo: De-duplicating Data Storage II
A demonstration of the cost savings that can be achieved using simple Full Article The CIA World Factbook maintains an…observablehq.com

After 34 changes, we can see that Git and Zipped file systems perform almost equally, mostly because that is what Git does (zips of entire file structures), compression offers a lot of savings. By comparison, SVN shows almost no growth at all.

This brings us back to our original problem state: $70,000/year ($56K USD) for storage, but using automated version control, and SVN, in particular, we are able to reduce the storage costs by 90%, to $7,000/year ($5.5K USD), under real-world conditions.

After 34 periods of change, SVN shows a significant cost savings (90%) over our original state.

That is a 90% savings in costs, or about $63,000/year ($50K USD).

Most of these estimates are on the small side. Recent experience has shown the involved datasets to change in the range of 40G/quarter, suddenly those numbers begin to be of the scale of millions of dollars.

The cost savings, relative to the baseline storage, becomes more significant as time progresses. This chart did not convey what I hoped it would. I've left it here because I suspect it will catch an executive's attention.

Conclusion

This technique was applied several times over the decades and has been demonstrated to work with not just text files, but also Parquet and SAS data files, in Tableau and RStudio applications.

Optimisation of these techniques requires cognisant cooperation from both Data and System Analysts, unfortunately, detailed information and change management techniques do not receive the exciting attention of executives. However, a 90% decrease in costs (and therefore an increase in profits) is not something that should be ignored, and attention to these details is important.

For this reason, I have presented these techniques using an automated, but unobtrusive technique, that fosters an environment that can encourage learning and cooperation.

Mostly, these techniques become necessary to reduce cost as an excuse for progress.

… because sometimes bureaucracy gets in the way.

I encourage you to check out the sample from GitLab.
Also, have a look at the calculation worksheet and check my maths. Change the values to represent your workspace. How much can you save?
Remember: cost correlates to carbon. Reducing your consumption is easily measured in dollars, but also represents less pollution

Save a byte, save the environment

How to Build a Simple In-Browser Search Engine

2022-03-06T00:00:00Z

Photo by Ruchindra Gunasekara on Unsplash

A couple of years ago, I was asked to create a file repository for an air-gapped conda repository. The idea was that, for security reasons, we would have a list of allowed libraries inside the very expensive secure environment, and only those libraries. It was a reasonably simple setup, and only required a large amount of storage, and a simple static web server to deliver them over.

The simple filtered mirror allowed us to respond to user needs much more rapidly, and to add or remove libraries from the secure environment very quickly. This meant the security team was much more willing to allow packages into the environment, knowing it would be hours to remove them rather than months or years.

They were so thrilled, we started using patterns to allow packages.
We quickly went from dozens of packages to thousands.
Users developed a new question to phone in and ask:

which libraries are available?

The list of libraries available was less than the official repositories but significantly larger than a human could reasonably be expected to read through, it was also spread across several folders. All-in-all it was just difficult to keep track of.

What users required was a search engine, like Anaconda.org, but limited to only our organisationally available packages.

An Aside

It's at this moment, in our story, that I find myself away from home on business. Having attended an evening lecture, in a pub in Gatineau Quebec, on using technical systems to affect social change, and having had a few drinks, I found myself walking back to my hotel, alone, in the dead of a Canadian winter. Trying to not freeze to death, and figure out where my hotel was, I started thinking about search problems. By the time i got back to my hotel, there was no way I could sleep, so began to implement the solution.

Ahhh… the life of a programmer.

Unfortunately, Bureaucracy got in the way. Getting the static HTTP server had been a feat of negotiation. In an effort to get the filesystem shared via HTTP, I had explicitly stated that this would only have static rendering turned on. This allowed us to skip undergoing months of security evaluations, no server-side processing meant no security risk to the network.

In order to accommodate both the need for a search engine and the lack of server-side processing, I built a simple search engine inside the browser.

Implementation

I have actually done this a couple of times before. There have been a few instances in my career where getting tools and computers have been significant barriers to a simple data processing engine. The key to this is that you can drop a simple html file on your served location, and it can then look up the data stored on the server.

For the reader's benefit, I have a simple example working that indexes the CIA World Factbook, and then offer a search of the content. Search results offer a link to the actual page on the CIA's website.

Files · main · Jeff Cave / Simple Search Demo
GitLab.comgitlab.com

Auto-index and Loading

The key to this is to ensure that some form of auto-index is on. The server must advertise the available datasets to be processed in order for the client to discover they need processing. There are a couple of ways to achieve this, the simplest is to turn the service on in the server.

In similar systems, I have also used `bash` or `PowerShell` scripts to just list all the datasets present: no information about them, just that they are present

ls --format=single-column ./data/* > index.txt

The location of that file can then be passed to the script as its starting point for gathering all information

const basePath = window.location;
const datasets = `${basePath}/index.txt`;

Once we have the list of data to be aggregated, we can begin the process of downloading, parsing, and storing.

Given we used PouchDB, we can immediately begin loading the text into the local database.

The full text is now stored in the database for future use by the user. It is recommended that this be scanned periodically to see if any changes have occurred. (loader.js)

Sanitising the Text

In database development, and searching, an index can be thought of as a quick reference. In the days of manual searching, a card catalogue offered an alphabetic search ordered by subject, that allowed a quick lookup. Rather than having to look through every book in an entire building, you can look through a smaller listing in a single box.

Less is more, smaller is faster.

For the purposes of this discussion, it is useful to have a couple of documents we want to search:

Fact 21: For every 25 percent increase in problem complexity, there is a 100 percent increase in complexity of the software solution. that's not a condition to try to change (even though reducing complexity is always a desirable thing to do); that's just the way it is.

Fact 22: Eighty percent of software work is intellectual. A fair amount of it is creative. Little of it is clerical.

— Robert L. Glass, Facts and Fallacies of Software Engineering

So our sample for discussion will revolve around indexing the text of Glass's Facts to allow for rapid lookup. This might be useful to attach to a microphone in the office which then displays an appropriate “fact” depending on what is being discussed.

Each fact can be placed in a separate record for our indexing to pick up later. This will allow us to treat each fact as a distinct entity.

./data/fact-01.txt
./data/fact-02.txt
./data/fact-03.txt

One of the issues with text such as this is that a lot of text is actually meaningless, or at least carries low meaning. Some level of transformation should be performed to remove needless information.

function sanitize(text){
text = text.toLowerCase();
text = text.replace(/[^a-z]/g,'.');
text = text.split('.');
text = text.filter(d=>{return d.length > 3;});
return text;
}
let sanitized = sanitize(facts[21]);

As part of indexing fact number 21, we remove

case
non-text
short words (three characters, aka: Stop Words)

resulting in a sanitised list of words. (crepo.js)

[ 'every', 'percent', 'increase', 'problem', 'complexity', 'there', 'percent', 'increase', 'complexity', 'software', 'solution', 'that', 'condition', 'change', 'even', 'though', 'reducing', 'complexity', 'always', 'desirable', 'thing', 'that', 'just' ]

The list can be further reduced by noting the duplicate words. A word count is a useful way to weigh the value of a term in a search

function WordCount(list){
let count = {};
for(let word of list){
count[word] = count[word] || 0;
count[word]++;
}
return count;
}
let count = WordCount(sanitized);

Creating the index

For the purpose of this search, I chose to do a match on each word. This was done for simplicity, and because it allows for searching of words out of order.

Examples of the kinds of searches I want a user to be able to use:

complexity: a word that is overly complex (could be complex)
ompl: a word that is a partial match
software olut: multiple words
solution software: words out of order

These examples represent a couple of different cases, that we need to handle. Most interesting is the “partial match” scenario.

To create an index we need a list based on our lookup value that

| key | score | document |
| -------- | ----- | -------- |
| database | 100 | <url> |
| atabase | 99 | <url> |
| tabase | 98 | <url> |
| abase | 97 | <url> |
| base | 96 | <url> |
| ase | 95 | <url> |
| databas | 99 | ... |
| databa | 98 | ... |
| ... | ... | ... |

When you search for “data search”, it finds

document 1
data 97
search 100

For a total of 198 points, and then sorts by highest scoring documents (Code: search.js)

Extras

Lemmatization

As another layer, we can consider lemma. Lemmatization refer to the most foundational word that is represented by a word. For example, intellectual can be thought of as being the same as intellect. This will help reduce the amount of possible typographic difference that occur later, for example, a person that remembers that is should find that's, come to think of it, is is a short word.

lemmatization-lists/lemmatization-en.txt at master · michmech/lemmatization-lists
Machine-readable lists of lemma-token pairs in 23 languages. - lemmatization-lists/lemmatization-en.txt at master ·…github.com

every 25 percent increase problem complex there 100 percent increase complex software solution that condition change even though reduce complex always desire thing that just

This is a much simpler version of the text that will reduce the amount of informational entropy (aka things that can go wrong in my head when remembering).

This text is well sanitised and ready to undergo indexing.

Conclusion

The next morning, after writing the solution, I felt pretty pleased with myself for having a complete solution that I could email to my primary customers (a few Data Science managers throughout the organisation). I was so pleased, I showed it to a colleague who helped me with optimisations to the index.

Found this Intersting? Leave a Tip…. it helps

Years later, and it is still the way the organisation tracks available conda packages.

To be fair, the only reason I was able to pull this off overnight was that I have implemented this so many times. I have used browser-side indexes to implement dashboards for regression testing, employee work allocation, personal blog searches, and a Project Gutenberg search engine. It even ended up being the basis for a class I taught on Introductory JavaScript.

It is one of the reasons I love JavaScript: it is always available for data processing, whatever crazy idea you happen to have at that moment.

Since originally writing this, I have discovered that Lawson has actually written a full text search for PouchDB. You should use his solution based on the lunr engine.

Non-Optimal

This is not an optimal solution.

In this case, every user of a search engine must download and generate their own instance of the index. This requires that the client download the entirety of the dataset the first time (increasing network traffic), for each browser they use.

In the regression test dashboard, the initial download and parse of the test results took approximately 3 hours. Caching meant that it was almost instantaneous if you kept up to date each day, but that first load was a big one.

One of the key advantages to server-side processing is the ability to take the inbound data, and generate the indexes once for all the customers. You could split the difference by generating the index on the server and having the clients download that constructed index. This would split the processing load between a central processor for the central data, and distributed processing for the individual searches.

This hints at an interesting balance between shared and distributed processing. Using databases that support the CouchDB interchange protocol (CouchBase, CouchDB, PouchDB, CloudAnt), it is possible for each user to process new data as they find it, but also to share the results back to a central pool which is used by the next person. This Lazy-Load form of data processing has some theoretical advantages to it (in a trusted environment) in that it would require very little central processing (expensive) and instead rely on workstations that are already in use. At the same time, it would share the workload to ensure no one computer took the brunt of the full calculation.

Lastly, if this solution intrigued you, you should look into the Lucene based searches provided for CouchDB and included with all the commercial offerings.

The Measure of Integrity of Software of Students

2022-02-28T00:00:00Z

Photo by Nubelson Fernandes on Unsplash

This article describes the use of a browser-based tool for comparing and visualising student assignments. The tool M.I.S.S is a tribute to the classic tool MOSS (Measure Of Software Similarity), but supports organisational privacy and legal constraints by operating completely locally.

In a previous article, the use of GPU to perform the comparisons was discussed.

After twenty years as an IT professional, having been a software developer for a diverse set of industries, having built systems that solve problems for some of the largest organisations on the planet … I decided to take up teaching.

I had spent a lot of time working with newly hired developers to give them the basics of working within various corporate environments and had been disappointed by the lack of interest in problem-solving. It took a lot of work to get a recent graduate to think about how their actions affected others, and how their individual decisions resulted in effects that had consequences beyond their immediate deadline.

Too often, new hires would race through a solution, submit it, and (proudly) declare that it was complete; a cursory inspection would demonstrate large failings. This was common among all new hires. My goal in teaching was to head this problem off. Graduates were coming out of school, wildly unprepared for the field, and I was going to change that.

I was going to increase the quality of software developers available to organisations, not the quantity of developers.

I wanted to reach out to students and hopefully guide them to learning how to problem solve, not just type code.

Toward the end of my first semester, I noticed two significant problems with this:

Some students are just not inherently curious about programming
Before signing off on the quality of students, I had to first identify which was their work

Students work together, and collaboration is good but under the pressure to perform some people are tempted to cheat. This offends me in different ways depending on the motivation behind it:

I dislike bullies. When a cool kid tries to bribe or bully a socially awkward student into doing his homework, I get angry.
I dislike fraudsters. In some cases, students came in with no interest in the work and paid professionals to do their homework.
I love people learning. Under pressure, a good person may take a shortcut, but, if someone is hiding a problem, I can't help them with it.

An Aside

This is not a moral judgment, it's a factual judgement. I make a bad MMA fighter because I'm too delicate, and that's OK … unless I try to take up fighting.

One problem I noticed with administration was a strong desire to not have students fail. If a student could not perform an activity, excuses were made because nobody wants to hurt anyone's feelings. This resulted in students being put in positions they were not ready for, and my original observation of graduates putting customers at risk.

Remember my motive: Integrity is important because judging individuals on their ability to do the work is a safety issue, and a certification trust issue.

Compounding this problem was a concern that other faculty and administration did not want to find the problem. It is a hard problem to deal with, charged with emotions, subjective reasoning, academic courts, and formality. Life is so much easier if we just ignore it. I heard on more than one occasion “Industry will straighten them out”. To me, this makes the certification advertised by the facility valueless (see “I Was Shocked To Catch A Candidate Cheating In An Online Interview”).

So, to summarise the problem: I'm teaching JavaScript programming, visualisation, and data analysis, have just left a position where distributed computing was my bread and butter, and am faced with a problem. Add to that, the administration forbade the transmission of student assignments outside of the facility and ominously reminded me that it could not leave the country.

So I wrote my own solution.

Complete in the browser.

No servers involved.

There was no need for permission to install the software. There were no more concerns for legal or privacy constraints. No questionable licenses. No fees. Just a simple tool to find out who needed my help.

The Solution

M.I.S.S is a piece of software that runs completely in the browser. It can be found hosted at https://gitlab.com/jefferey-cave/miss

Before you begin with the software it is worth having a set of files you want to compare. If you work with a submission tool like BrightSpace, you are in luck, M.I.S.S accepts and interprets the zip file you download from your submissions folder. It can be directly uploaded the same way you downloaded it. Alternately, you can download the sample used for testing.

The first page you see when you get to the interface

When presented with the first connection, the tool starts on an “FAQ” page. The intent is to answer all the questions you may have right away. The most important thing you can do at this point is to upload a collection of student assignments for comparison (stored as a zip). In the background, the tool will unpack the zip and extract each file. Each root folder is considered to represent one student.

The points represent student submissions. Orange represents the progress of comparing them to one another; while the grey line shows a completed comparison.

Once uploaded, you should be able to navigate to the “force” tab to see that progress is being made. The graphics update in real-time, giving a sense of progress, so go get a coffee and wait for the calculations to complete.

What it does

Now that the sample is loaded and processing is begun, it's time to ask what it is actually doing.

The process for measuring the level of similarity between two programs is not just a straight textual comparison. Rather, M.I.S.S goes through several phases to try to be as accurate as possible.

Compilation/Normalization

The first part of the process is to run the code through an appropriate interpreter depending on the language. When most people initially think of code comparison, they consider text comparisons: letter by letter. However, by using a language-specific interpreter, we can change the text into a series of tokens:

instead of f, o, r, the word for is recognised as a single thing (and assigned a number)
instead of f, o, r, w, a, r, d, the word is recognised as a variable (and assigned a number)

This avoids the first three letters being thought to mean the same thing: instead token1 is compared to token2 and seen as a single comparison between 2 things, not 3 similarities, and 4 differences.

This allows us to recognise small changes that may have a large visual impact (like variable name changes).

2. Full-Text Comparison

The text is compared at a full-textual level. This is important for ensuring that order does not matter:

function FunA(){...}
function FunB(){...}

is seen as the same code as

function FunB(){...}
function FunA(){...}

Please see my previous article on how the Smith-Waterman algorithm was implemented to achieve this.

This allows us to recognise small changes that may have a large visual impact (like re-ordering functions).

Interpreting Results

This program does a blind comparison, it does not have any bias about the what it is comparing, it just compares. Unfortunately, all creative work, will have common elements to it. Good solutions will be independently discovered. This means that it is not possible for this program to catch cheaters.

What this program can do is filter out the people that probably aren't directly copying one another's work. People that appear to be exercising their own skills in attempting to solve the problem.

Given a batch of 120 assignments to grade there are 7140 comparisons that would need to be performed. Multiply this by multiple batches assigned to different graders and the problem becomes untenable, resulting in faculty (faultily) relying on intuition.

Instead, it is probably necessary to review 5 to 7 of the submissions, and MISS can be used to filter the list down to just those items, by removing items that are obviously different.

How to Interpret

The goal of the visualisations is to reduce the workload of faculty. Once a batch of assignments has completed processing, interpreting the results is relatively straightforward.

The force-directed graph is more a curiosity than anything else (we were building them in my class). Clusters that appear are indicative of social groups within the class. This can be indicative of students that are studying together and making shared mistakes.

The first real step is for faculty to look to the Listing tab

An example of a run. Most submissions have around a 5–10% similarity. However, highlighted in red, are two assignments that have about 50% similarity to one another. This is a major divergence from the norm, making them worth investigating. Be careful, the calculation is not complete, we don't know what “normal” actually is.

The listing includes the comparison of every submission to every submission, with the amount of similarity indicated as a percentage. In an effort to reduce effort, items are sorted by most similar to least. The ones at the bottom can probably be ignored. Results are also colour coded to represent not compared (grey), normal (blue), suspicious (red). If you see grey, it is not complete, go get more coffee.

“Suspicious” is defined as an abnormally similar from the group. This is calculated by sorting the items in order of similarity, calculating the difference between each step, and taking the largest change. Any value from largest change forward is considered to be worth inspection

For example,

14%         }
   }- 3     }
11%         }  Suspicious Range of values
   }- 1     }
10%         }
   }- 4   <-- Greatest Change
 6% 
   }- 1
 5% 
   }- 0 
 5%

This technique was chosen to identify normal “sharing” within the group. It is assumed that the steps will represent students leaning on their peers for support, until it reaches a point where it is “more” than leaning. Even if the amount returns to normal after that point, the amount of sharing is still abnormally high. This technique is believe to offer better utility than a simple average as it can identify a group of people where the minority of students are working independently.

Clicking on any of these comparisons will take you to the Comparison tab.

The comparison tab allows you to view the actual assignments side by side with colour coded sections of similarity. Don't miss the “maximize” button in the top right corner.

Here you can view the actual software, side by side. Similar blocks of code are colour coded to allow for inspection. Clicking on the maximise button in the top right corner is probably useful. Single clicking on the percentage indicators will jump to the block of code, double clicking brings both pieces of code into alignment.

Hopefully, your inspection turns nothing of interest up at all.

Action buttons that are useful: report a bug, print the results, delete the current data, upload a result file, or download a result file.

As this runs fully in the browser, it is also important that you save a copy of the comparisons. This can be done by clicking on the download button which downloads a copy of the current comparison for saving for later. This is useful as a backup, but also in case you need to stop a run and need to resume it later (perhaps with the addition of a late submission). The downloaded file stores all the data to resume comparisons at a later date if necessary.

Give your file a meaningful name before downloading it. Then delete and start another class' assignments.

Having said that, the point to this software is to be independent of the tool. Regardless of your findings, you should always print the results for long term archiving. The printed copy can act as a permanent record of your findings and will exist as simple HTML (or PDF) should you need to access the results even decades later, when you have forgotten about M.I.S.S.

Always remember to maintain consistent evidence of your consistent decisions.

The common objections

Most objections regarding the results amount to straw-man attacks on the process, and focus on not challenging student integrity.

What if the computer makes a mistake and accuses a student falsely?

Used properly, this tool simply identifies works that bear closer investigation. It does not give any information about the nature or motives of the similarity. That is left to the investigator to determine from the context.

In fact, let me reverse the objection:

What if instructor bias leads to a mistake, and falsely accuses a student?

In its first use, this tool actually indicated that someone suspected of cheating was, in fact, innocent.

At the time, I was exhausted. It was the end of the semester, and students had been given a second (and a third, also a fourth and fifth) chance to submit assignments. Administrators, councilors, and fellow faculty bullied me into allowing multiple late submissions from some key students. Naturally, struggling to keep up with grading, and being pulled into multiple meetings regarding these students did not put me in a pleasant mood when a particular submitted assignment looked suspiciously similar to a very unique solution I had been particularly impressed with (submitted by a particularly strong student).

Being ill-tempered, and in an effort to stave off the next set of objections I decided to gather evidence from an unbiased computational resource, and found…

The level of similarity between the two students was normal for the class

The student had a couple of variable names that were similar (probably due to tutoring), but beyond that, the style was very different. My personal emotions and experiences had led to bias because... well because I'm human.

The computational solution was unbiased and treated the student fairly and without emotion, even when I did not.

It also (thanks to its brute-force capabilities) identified two students who had copied one another's work exactly with only variable name changes. They had been missed purely due to being lost in the mass of papers needing grading.

Routine for Use

I strongly recommend reading a paper on the use of MOSS in the classroom: “Experience Using ”MOSS” to Detect Cheating On Programming Assignments”. This details the routines and experience of faculty at the University of South Florida

Objections to the use of tools like this focus on accusations of vindictiveness by the user. However, for Faculty that do choose to use a tool like this, the goal is to avoid problems, not to cause them. After having developed the tool, I began to develop and document a routine for use:

run it against every batch of assignments consistently
be open with students that you are running it
each semester, I only bothered to pursue the most egregious case

I would normally start the first semester with the first assignment and run all the students' submissions through the tool, in front of the students. The visualisations caught their eye, and they could see clusters starting to form. Naturally, I was careful to anonymize Student IDs, but being so introductory, the amount of similarity was incredibly high anyway. Students are not singled out but it does serve as a warning to students that you are watching, and serves as an opportunity to discuss the difference between copying, and collaborating. It is also a great introductory discussion into how the skills they are learning can be applied to solve personal work problems.

While the tool reduces the effort of evidence gathering, it does not eliminate it. There is a large bureaucratic burden in even discussing academic concern. I personally, felt that a balance between pursuing the issue and spending time with my family had to be struck. My conclusion was to pursue only one case per semester: pick the worst, chase it down. Also I tended to do this in the middle of the semester

To informally warn students when things are new and busy
To leave myself time to pursue the requisite paperwork
To give the students the opportunity to correct their behaviour

Part of the reason I felt it was important to be consistent in filing and pursuing the issue was due to my first experience with the formal process. As I began to pursue a case, other faculty came forward with their previous anecdotal cases with the exact same group of students. Anecdotally, these students had multiple incidents that had been dealt with individually, however the pattern had not been identified across space-time.

Another reason for consistency is simple bias management: are you letting a student slide due to favouritism? Anytime you actively withhold consequences based on discretion, you are actively applying punitive measures to those you choose to pursue. Are you therefore applying punitive measures without bias? If you consistently act based on the evidence, you do not have an opportunity to express unconscious bias, and therefore do not need to question your own integrity.

Conclusion

While I no longer teach programming, I do think that academic integrity is important. Colleges and Universities are certifying that students are knowledgeable in a domain of expertise, and failure to keep that process honest can have dire consequences.

Found this useful or interesting? Consider leaving a tip … it helps.

Future Development

I would love to continue work on M.I.S.S, but without more data (more assignments) it is difficult to comprehend the needs of an audience.

If you find the tool useful, I would love to hear from you.

Suggest a feature (or up-vote one)
Submit a patch
Just tell me about your experience (good or bad) in the comments

Knowing how (or even “that”) the system is used would help drive development forward.

Right away, I would be very interested in working on adding cluster indicators to the force-directed graph. I think it would be useful to have a bubble drawn around the grouped suspicious items. The difference analysis already identifies which items these are, but in a case where you have more than one cluster, it would be interesting to see them separated out.

Other Uses

There are several future capabilities for a tool such as this. While I am interested in exploring some of these, I'm interested in a lot of things and have had to make some hard choices.

Code Refactoring

One interesting capability is in the refactoring of code. While this tool compares different applications to one another, this same technique could be used to compare a software application to itself. This self-comparison could aid in software refactoring, identifying parts of code where copy/paste has resulted in duplication, or even where two developers have independently implemented the same idea. These parts that are similar, and repeated, would likely represent a refactoring opportunity.

Distributed Open Source Analysis

Another interesting idea would be to extend this to larger-scale projects. Rather than taking student software as submissions, scan GitHub or GitLab for content duplication. Aside from forks, who is copying one another's code? This would require a server and database to allow for checkouts of code and to store the comparisons in a longer-term repository. This project was developed shortly after a distributed project I had worked on and was constructed with distributed computing in mind, it should be able to be scaled to distributed environments easily.

If someone has some investment money they would like to throw at these ideas, I would be thrilled to chase them down.

M.I.S.S.
A measure of software similarity completely contained in the browser.jefferey-cave.gitlab.io

How to Not Store the Same File Twice (using JavaScript and PouchDB)

2022-02-28T00:00:00Z

Photo by Jeremy Bezanger on Unsplash

Many years ago, I was reading an article about Google's internal labs having managed to create a sha1 hash collision between two PDF documents. The documents were very different documents, but through clever bit manipulation, resulted in an identical sha1 code. It was a fascinating read for a Friday afternoon, but over the weekend I started to ask the question: why does Google care?

One significant place where this would impact Google would be on their storage platform, Google Drive. Given they are storing massive numbers of files, on behalf of massive numbers of people, in all probability there are going to be a massive number of duplicate files.

Logically, we can see this will likely be true just through its usage. Assuming I am engaging in a real estate transaction, it is not uncommon to exchange emailed PDFs with scanned signatures. Assuming everyone is using Google Drive to back up their documents, there are four people that have copies of the exact same document: buyer, buyer's agent, seller's agent, the seller. Don't forget to add lawyers and lenders later in the process.

I have worked for organisations whose primary business involved the interchange and storage of data (Oil/Gas Production, Telecommunications, or Data Repositories), and in every case, our solution to the problem was simple: charge the customer. Charge the customer a rate per byte: the amount of drive space they take up * replications space * server cost * electricity * rent * markup. Charging the customer is a good way to offset the cost, and Google does bill its customers, but through de-duplication, it is possible to maintain the same revenue, while drastically reducing the amount of physical storage.

Viral Memes were always a fun way to watch a poorly configured communication network burst into flames

Take a chat service I was involved in. The GSMA-RCC specifies that images can be interchanged between client devices; it also specifies that those images should be retained on the server for later pickup. So if a meme goes viral, thousands of individuals may forward the image to one another, resulting in thousands of copies of that image flying across the network, and being stored on server drives.

Further, if it's a good meme, people are going to forward it back to people they know; people who have already received it. So round and round the image goes, and it doesn't stop until theoretically its been sent on every possible communication between pairs of people.

With 10 billion devices on the planet, that's more than Seven Bridges that need to be crossed.

If some jerk sends that meme as a bitmap each image instance takes up about 1079KB.

assume
- $1.00 per MB revenue
- $0.90 per MB cost
- 1,000,000 user interactions (transfer)
- 1024KB file

= 1 MB * 1M transfers * ($1 - $0.9)
= 1TB of storage
= $100,000 profit

We can be sensible about it and require that everything is converted to PNG (notice Google asks to do this for Google Photos) reducing the size to 615KB (60%). This is useful when dealing in a fixed revenue, but not a promised true copy of data storage.

Storing the object once per person is how the users perceive a file system

Storing the image once per person that receives it, is an inefficient use of resources. If, on the other hand, we can identify that it is actually the same image, we can reduce our cost to storing only one instance, but charging for each transfer.

= (1MB * 1M transfers * $1) - (1 unique file * 1MB * $0.9)
= 1MB of storage
= $999,999.10 profit

There is a big difference between having a cost of 90¢ per transaction, and a cost of 90¢.

Identifying that everyone is storing the same data, allows us to significantly reduce the amount of space we consume while giving the same level of service to our users.

However, de-duplication is not an easy problem to solve.

First of all, we receive the files independently of one another, from different people, with different names. Given a large number of large files, it is going to take a significant amount of processing power to compare the files byte-by-byte to every other file we have stored. In a large-scale system, this simply is not feasible.

This brings us back around to where we began: Google had managed to cause a collision between sha1 in the lab. Why would they have been concerned with researching the extreme possibilities of collision in binary documents? Because they are using the hash to help identify the uniqueness of files.

We can use large hashes as primary keys for the files.

A Working Example

Pulling on some past experience solving this problem using CouchDB, we can implement an in-browser demonstration of the principles involved using JavaScript and PouchDB. If you are not familiar with PouchDB and using its DB interface, I suggest reading the definitive introduction by Nolan Lawson.

We will be creating a simple example with three users (Alice, Bob, and Carol) sharing their favourite lines from a new Opera they just saw.

The process is relatively straightforward: three people store one of two lines of text to their account, and then retrieve them from the database.

Of interest is the total size being stored, as well as the per-user size (billable size) usage:

DB Size: 1813 bytes

| User | Files | Size |
| ----- | ----- | ---- |
| alice | 2 | 101 |
| bob | 1 | 45 |
| carol | 1 | 56 |

The real meat of the program happens in the save and dload which abstract away the interactions with the database. Further CreateIndex defines the mechanism for search and retrieval.

async function CreateIndex(){
return await db.put({
_id: '_design/allfiles',
views: {
'allfiles': {
map: function (doc) {
var userpath = [doc.user,doc.path];
emit(userpath,doc.size);
}.toString(),
reduce:'_stats'
}
}
});
}

async function save(user,filename,blob){
return db.put({
_id: [user,filename].join('@'),
user:user,
path:filename,
size:blob.length,
_attachments: {
'0': {
content_type: 'text/plain',
data: window.btoa(blob)
}
}
});
}

async function dload(user,filename){
let recs = await db.query('allfiles',{
reduce: false,
include_docs: true,
attachments: true,
key: [user,filename]
});
let blob = recs.rows[0].doc._attachments['0'].data;
blob = window.atob(blob);
return blob;
}

The save and download functions work on the assumption that we are going to store a copy of the record for each person, while the index manages a list of users and their files.

A JSFiddle giving an example of the working demonstration

Reducing Storage

To modify this example to reduce our storage, we must first modify the save function to not blindly save for each user, but rather to save each blob as a primary object, and track users observing it as a secondary item.

async function save(user, filename, blob) {
let hash = new TextEncoder().encode(blob);
hash = await crypto.subtle.digest('SHA-256', hash);
hash = Array.from(new Uint8Array(hash));
hash = hash
.map(b => b.toString(16).padStart(2, '0'))
.join('')
.substr(0,4);
let userpath = [user, filename].join('@');
let rec = null;
try {
rec = await db.get(hash);
} catch (e) {
if (e.status !== 404) throw e;
// create the object, with a list of user's using it
rec = {
_id: hash,
userpaths: [],
size: blob.length,
_attachments: {
'0': {
content_type: 'text/plain',
data: window.btoa(blob)
}
}
}
}
if (!rec.userpaths.includes(userpath)) {
rec.userpaths.push(userpath);
}
// finally save the record
return db.put(rec);
}

This ensures that we only ever store a blob once.

Using the hash as the record identifier means that no matter how many times it is submitted, we just keep using the existing record. New users are simply added to the list of users using the item. Users can even make copies of it by submitting the same item with a different name; we just keep adding notes that the user has a name for the record.

Naturally, this breaks the lookup index we were using. The original index used the name of the record (username+path) to look up the file, but the file no longer uses this as its record name. Instead, we need to create a lookup index that is constructed from all the users that use the same file.

async function CreateIndex(){
return await db.put({
_id: '_design/allfiles',
views: {
'allfiles': {
map: function (doc) {
for (let userpath of doc.userpaths) {
userpath = userpath.split('@');
emit(userpath, doc.size);
}
}.toString(),
reduce:'_stats'
}
}
});
}

By looping through each of the user paths, we have updated our index to return the file, for each use of the file. This means no change to our download function, as the view's interface has not changed.

Note that we had included the file size in the view, and used the stats reduce method. This means that when it comes time to do billing we can simply add up the number of blobs the user references, as well as add up their total size.

Re-running main gives us a new total size and billing data:

DB Size: 1109 bytes

| User  | Files | Size |
| ----- | ----- | ---- |
| alice |     2 |  101 | 
| bob   |     1 |   45 |
| carol |     1 |   56 |

Even in our trivial example, with a very small content size, our database size (and therefore our business expenses) decreased by a whopping 39%. As more of the data structure is taken up with content, and less is taken up with metadata (think MP3, MP4, and PNG), a better return is seen.

Also, notice that the billable sizes never changed.

The same example as previously presented but with the changes to make the storage smaller.

Completing the Solution

After all of that, we cannot forget that Google demonstrated that collisions are maliciously possible, therefore they cannot be absolutely trusted. We must perform complete byte-by-byte comparisons. Hash functions are useful tools for determining dissimilarity between binary objects, not for determining similarity.

Hash functions are useful tools for determining dissimilarity between binary objects

This is a very useful feature. By using a large hash, I can quickly narrow the required comparisons to almost nothing. There is a good chance I will have no more than one match that needs to be checked, and that is a lot less effort than millions. Once I've narrowed it down, the complete file must be checked with potentially an extra marker to distinguish it from one we already have.

Found this useful or interesting? Consider leaving a tip … it helps.

There is a second problem in that we need to delete records at some point. In this case, we can simply drop the reference the user has to the object, though we must also observe for the time when no user has a reference to the object. When all references are removed to the object, the object itself should be removed. Similarly, if a record is renamed, we need to ensure we remove the old reference before creating the new reference.

I leave these as exercises for the reader.

4 Standard IT Disaster Scenarios you aren't prepared for

2022-02-17T00:00:00Z

In any system design, there are several scenarios that should be considered to prevent system failure. Each of these scenarios describes a worst-case scenario that frames planning for catastrophic events.

1906 Earthquake in San Francisco (Source: Wikipedia)

Often, when describing the need for various emergency protocols, the presenter is faced with resistance in the form of “but we trust each other”. These forms of argument distract from the very real underlying risk that needs to be addressed.

These descriptions, and their titles are meant to give a standardised response to the most common objections. Each scenario has a list of ways the scenario presents in the real world. The titles are somewhat humorous to ease the tension, but the scenarios are serious and realistic.

The scenarios are also meant to be non-specific. Rather than planning for very specific events, general scenarios that encompass general responses allow for adaptation to multiple considerations.

Under the Bus
Bump on the Head
Spiked Drink
Sword of God
Daemonic Possession (Bonus, I added another later)

Under the Bus

The primary on a system got run over by a bus on the way to work, and has been hospitalised for an indeterminate amount of time.

Presentation

Any unavailability of the system experts, potentially combined with the need for action

Accident: sky-diving, home repair, car accident
Vacation: phoning people while they are on vacation is rude
Illness: myocarditis, kidney stones, hemorrhoids, common cold
Arrest: sometimes people get detained; rightly or wrongly

Objections

“That's a horrible thing to say”

If it makes you feel better, they are going to be OK; but accidents do happen in life. Do you really want to be the person that is phoning a colleague while they should be resting in hospital?

“We better make sure you don't do anything risky”

As a manager, did you just inform your employees that they are not to undertake any personal activities?

Bump on the Head

One of the trusted individuals has recently received a bump on the head and now has a brain injury that has drastically altered their personality. They can no longer be trusted. It is unclear for how long they were trusted when they should not have been.

Presentation

An actual bump on the head: has caused people's personalities to dramatically change
Blackmail: or possibly bribery, where an outside actor has altered the state of the trust relationship
Poor trust evaluation: You shouldn't have trusted them in the first place
External system breach: a trusted individual has had their digital identity compromised.

Objections

“It's OK, I trust you”

Stop exposing me to risk, its unfair. The minute something does go wrong, employees should have evidence in place that they were acting within acceptable parameters, and that the managerial staff had accepted any risks associated with the action. If judgement calls were required, and bad things happened, employees need to have a clear line of approval in place that they can point to as having failed (justifying their taking action)

Phineas Gage is probably the most famous “Bump on the head” in history (Source: Wikipedia, Originally from the collection of Jack and Beverly Wilgus, and now in the Warren Anatomical Museum, Harvard Medical School)

Spiked Drink

The trusted individual stands up from lunch and realise they are feeling “wobbly”. Someone spiked their drink.

Presentation

Any scenario where the actor has a compromised capacity for judgement

• Woken in the middle of the night
• Family emergencies
• Had a couple of drinks, heavy pain medication
• Compromised judgement results in the inability to judge yourself compromised.
• Snap decisions

Plan for individuals to be able to declare themselves incapacitated or compromised; plan for them to take action even when their judgement is compromised; plan to declare someone else's judgement as compromised. Have clear instructions in place to reduce the need for judgement (do your thinking in advance)

Objections

“People aren't allowed to drink on duty”

Being on-call, or worse, being the second or third person on call, during an emergency can result in you being activated at unanticipated times. The only way to avoid this is to consider all staff on-call 24/365.

Sword of God

(aka Sodom and Gomorrah, Meteor Impact, Zombies)

The facility has just been hit by a meteor. Where there was a service center, there is now a crater.

If it makes you feel better, everyone in the region is OK but more than a little distracted.

Presentation

Any regional outage that results in entire service being lost. Limited to no staff in the region able to respond.

Power outage
Natural Disaster (storm, tsunami, earthquake)
Epidemic
War

Objections

“Don't be over-dramatic”

I'm not being dramatic.

During the 2005 Ice Storm in Montreal, a colleague's phone rang with a request for technical assistance from another company. Located in Montreal, they had been without power for two days. Generators had activated, and the facility was operational; however, due to the high demand for fuel replenishment and the state of infrastructure, they were unable to secure more diesel. Their three day supply was about to run out.

A heroic effort was undertaken, unfortunately, due to the massive disruption to infrastructure, we were unable to rebuild their services on our infrastructure before the fuel ran out … leaving hundreds of thousands of Canadians without service for weeks.

Dæmonic Possession

(aka Planetary Alignment, Plumb Bad Luck)

You've done everything perfectly, but there is a very small dæmon living inside your computer. As you type your solution, it waits inside for an inopportune moment and messes something up. Something important.

Presentation

Software systems are complex systems, and complex systems are just that … complex. Complexity leads to unpredictability, and that is basically “random” behaviour. This can present in all kinds of ways, none of them predictable.

fat-fingering, typos
stuff just stops working … nobody knows why

Objections

If I can't predict it, how can I plan for it

This is fatalism, giving up, and that we must not do.

Preparing for bizarro land is not easy, but it is possible. Generally, this is done through constant testing and rehearsal (you are rehearsing disasters aren't you). This forces people to have practiced system failures, and general system recovery, under controlled circumstances.

Conclusion

Originally Published on my private consultancy website in 2013, this became something I wanted to preserve, share, and keep living. I have shared it with every company I have worked with, but I think it needs to be more widely distributed because I have yet to see a company that can handle any of these scenarios

Once upon a time, this list was at least 6 items long. Expect more in the future.

Also, I feel dirty for having used a “numbered list title”.

How fast is fast enough?

2022-01-11T00:00:00Z

Horse in Motion [img: Wikipedia, Public Domain]

My claim: “real-time” is anything faster than a change can be observed

Years ago, I got into a lunchtime discussion of real-time data processing and a couple of guys at the table started in with the macho attitude:

I used to work on fighter jets … real-time is microseconds
I used to do nuclear weapons testing … real-time is nanoseconds
I used to do solar flare warning systems …

While the time scales became increasingly reduced, I started to realise that my entire perspective was different. I was coming from a Healthcare setting involving patient charts, and in my head, the shortest timescale for transferring information was inter-hospital patient transfers, a process that involved humans reading and interpreting textual information. In this case, the information bottleneck was the time it would take for the patient to arrive at the new site, and staff at the new site to read the chart. (1–2 hours, sometimes up to a shift change)

My slower perspective was backed by one of the other developers at the table. In a previous lifetime, she had developed automated terrorism threat assessments and resource deployment systems (at least this is true in my head … she was always a little vague about what she had done previously). She was doing push notifications, but her bottleneck was the time it took for humans to comprehend the information they had received and to strap on a rifle. She put real-time at 15–30 minutes.

It's at this moment I have to appreciate how cool some of the projects are, that I've been near (IMG: Wikimedia CC-SA 3.0)

Timestamps

The way we think about time, is often built on a life time of assumptions.

In some database systems, there is a datatype known as a timestamp. A timestamp is a sequential number that is applied to the system. It is not a date, or time, it is a point in time.

I can hold a note for a long time…. But eventually, that's just noise. It's the change we're listening for … That's what makes it music. (Lorne, Angel, S2E2, IMG: Wikimedia, Fair Use)

There is a very poignant scene in the TV show Angel in which Lorne (the karaoke hosting daemon) is counselling someone after a breakup: “I can hold a note for a long time … But eventually, that's just noise. It's the change we're listening for … That's what makes it music.”

I have often taken this same point a step further, it is the change of state that defines time; time itself is a perception of changing state. This may be an oversimplification, but in terms of managing data it is a useful one. From the point of view of our system, there has been no change, and therefore no time has passed. As our system does not have the same perception of time as we do, there is no reason it should use our convention of recording time.

In terms of timestamps, this means we should have “Point in time 1”, followed by “Point in time 2”, followed by …

Time is only observable at its smallest division, time's smallest division is the point of observation.

Observation of change defines “time”. A point made most obvious by “Horse In Motion” and the Zoopraxiscope in 1880 (IMG: Wikimedia: CC-SA 2.5)

“Real-time” is what happens when change occurs between the observable points, so that it is available at the next point of observation (or possibly even creates the next observation).

Human Speed

In high school, a friend and I discovered Network Time Protocol (NTP) and the Canadian Atomic Clocks. While reading the user manual put out by the National Research Council, I remember reading the request to not use the most accurate servers.

It was just a friendly request to be polite.

The way the system works is that depending on how accurate your needs are, you are supposed to use decreasingly reliable “stratum” of service. Stratum 1 sits right on top of the atomic clock, Stratum 2 servers update from Stratum 1 (introducing a little bit of potential error), and Stratum 3 servers update from Stratum 2 servers (introducing some more potential error). So, at the time, you were politely asked to use stratum 3 servers to avoid overloading the stratum 1 computers.

Seems fair.

You are supposed to use the “stratum” of service appropriate to your needs. Stratum 1 sits right on top of the atomic clock, with each stratum reading from the previous level. (IMG: Wikimedia: Public Domain)

Unfortunately, people are people, and my friend began updating his analogue watch (readable to the minute), by hand, with stratum 1 servers. When I told him stratum three (accurate in the range of milliseconds) was good enough and that he was decreasing the accuracy for everyone, he boldly told me “Nope, only Stratum 1 is accurate enough for me”.

Macho statements aside, it is obvious that the speed bottleneck in the system is one of human scale, not computer scale.

A $10 watch is not going to retain accurate time for a long time. The drift in the network is less than the drift of the watch itself.
An analogue interface, interacting with a human eye, is going to have a read accuracy that is sub-minute, which is perfectly acceptable for the human turning the knob who cannot achieve better than sub-minute accuracy anyway.
The consumer is trying to get to classes, and meetups at the coffee shop, in a timely fashion. The time it takes to physically navigate space between these events introduces variance at the sub-quarter-hour level.

Society does not function at the microseconds.

At the time, the norm was to leave approximately 15 minutes to account for the vagaries of life. This is consistent for two people suffering from 5 minutes of error: 10 minutes of error, plus 5 minutes of agreement. 15 minutes was “real-time”.

Any effort, on my friend's part, to achieve faster results was a complete waste of effort. At the same time, other people were genuinely being harmed.

Data Scientists relying on accurate time for weather modelling are hurt
Sailors at sea, getting bad weather predictions, are hurt
My friend gets hurt when everyone is annoyed at him for being late because he was fiddling with his watch to get it “really accurate”

How my high-school-self envisioned the time keepers at the NRC, as well as their expression upon learning of the abuse of their system. I have since taught alongside an ex-weather researcher from Environment Canada; he looked just like that. (IMG: Wikimedia — CC-BY SA 4.0)

Faster than Observed

It is rare that I see anyone reading data more frequently than daily. Even with push notifications, I receive a text message telling me to take action, but I'm happy if I can action it within a quarter business day.

Faster is still better than slower; and real time is faster than the change can be observed.

Many times, in a business setting, an executive or manager or business unit calls for a “monthly” report, the report is therefore run on the 31st; weekly reports are run on Friday.

This is a huge mistake and given automated systems we can do better. Real time is still something we can strive for, and a key benefit stems from the fact that the results don't change significantly if we achieve “faster than observable” rates.

Since 2010, I have had a trading bot, developed in Google Sheets and JavaScript (I'm cheap and love free compute power), that sends me a text message any time I need to make a trade. The fastest data I receive is 15 minute delayed price data, but the most significant data I receive is published quarterly (Financial Reports). This data must be aggregated into averages and deviations and … patterns. The system is attempting to establish “normal”, and normal (by definition) doesn't change by much.

This is true for most human scale systems, most of the time

Human scale systems do not experience significant change at micro-timescales, most of the time. (Image: XKCD Blog License)

Under these conditions, we are dealing with aggregate data. The changes are aggregated into averages over the scale of days or weeks or even quarters. The assessments are not going to change significantly on an hourly basis. This means that the assessment from yesterday, is probably about the same assessment I am going to get today. I may see a change in the general trend, but it will be subtle and non-actionable in the short term.

There is a massively beneficial implication to this.

If I am producing a weekly report that interprets and advises the business, I should run the report daily. The average of 7 days of business operations is very likely to be similar to the average of 6 days. The results and conclusions of a report produced on Thursday, will likely be the same as the conclusions that will be produced on Friday.
If the Friday run fails, I have my conclusions from Thursday. If my Thursday run failed, I have early notice that the Friday run is likely to fail.

By working at one unit finer of granularity, you have given yourself lead time on potential issues, as well as created a fall-back plan in case of catastrophic failure.

In the past, this has resulted in

~3000 employees (myself included) getting paid for 13 days instead of not getting paid at all
A JIT system (a life-safety service) being able to estimate demand early, so that key staff could attend a funeral
Countless times I did not have to do overtime because a combination of poor null handling and weird data caused fails, but (thankfully) days in advance.

These are same principles given in my High school Math and Physics classes: use one decimal place more to do the calculation, than you report in. In business reporting: if you are tracking dollars, do the calculation in cents; if you are tracking cents, do your calculations in fractions of a penny.

Push and Pull

Push notifications, change the playing field somewhat. Rather than updating the data on a schedule, we advertise changes to interested parties. However we are still constrained to the response time of our slowest observer. Even in the case of detection of nuclear blasts, the point was to log and collect the data for interpretation by humans at a later date.

While there are many automated steps in my trading system, I still have to copy and paste the trade into my trading platform. As the person on call, sometimes I'm busy. (Img: XKCD: Blog License)

Take my trading bot: It sends me text notifications almost immediately (magnitude of seconds). This is faster and far more convenient/reliable than me checking once a day, however it does not mean I can respond any faster. Real-time is constrained both by the speed I receive the information and the speed at which I can respond. If I am trapped in a meeting, a secure network environment, or up to my waist in a river while fishing, I may not be able to initiate the trade for a couple of hours.

Lazy Loading Improves Net Performance

So if, in most (human) cases, it is sufficient to deal in timescales of minutes or hours, then we can conclude that the the reports do not need to be updated any faster than that.

In general, we can consider that updates do not need to be generated more frequently than they are going to be consumed by the observer (either digital, human, or system). This idea is where push notifications can both help and hurt us.

Polling (regular pulls) of source systems generate needless processing effort. Requesting information comes at a processing cost where both systems need to expend effort talking to one another. If there are no changes to the system, all of that effort results in “no change”.

Push notifications allow us to reduce this overhead by having the source system transmit change notifications to interested parties in the event something actually changes. This means that no processing is performed until something needs updating.

However, given our sub-hour threshold for “real-time”, it is very possible that we may receive notice of change more frequently than we need to report it. We may recalculate a report more frequently than it can be observed.

This was most evident to me in a simple web-app I was recently working on. I wanted the user to receive real-time notification of the correctness of their entry into the form.

Every change to the form results in a change event that is processed by the back-end. I was following the error notifications coming back from the validation, and was trying to type in a valid value to observe the update come back to the form.

I was being driven crazy, I was only 10 characters away from a positive result, but kept getting stopped by each key-press as it recalculated the validity. Each key-press was pushing a notification to the back-end which was triggering a recalculation … but I can already see I was wrong … internally, I was begging the system to just let me finish typing.

In the end, I put a timer on the validation: do not update the validation more frequently than every 23 milli-seconds. That little bit of delay allowed me to finish typing, and the quality of the feedback did not suffer (maybe even improved), by bringing it into the human scale.

Buffering results until someone actually wants it, takes us back to the concept of Lazy Loading. If nobody is going to read your data, don't bother calculating it. This reduces overhead because you may have 20 updates, but only one view (and therefore calculation).

If you are speaking about push notifications, you should be thinking about lazy-loading. A push notification can be used to notify our system that it needs to update, but we can use Lazy Loading to defer that processing until it is needed. But it is a balancing act. We can defer processing, but also need to balance it with performing it frequently enough.

Conclusion

Most of us are not attempting to detect the oncoming wave of a nuclear blast, nor are we racing ahead of a Solar Flare about to destroy our multi-billion dollar data on our international network infrastructure. Most of us operate on a timescale of minutes or hours, well within the operating tolerances of even the most basic of desktop computers.

Given this we need to remember two key points

Time scales dictate “real-time”
You should always be processing one unit of time smaller than will be consumed

Through a combination of lazy-loading and push notifications, we can scale our response time to an appropriate level. Obviously, an aircraft trying to stay airborne requires a different scale than inventory management in a retail organisation. Inundating humans with data does not improve information uptake.

Having said that, we want to make sure we keep ahead of our audience. We can deliver information faster and more frequently than people need it, there is therefore no reason to have it standing by ready for them when they want it. There is no reason for us to not do our checks and balances well in advance.

The key is balance.

Don't let sales tactics (and your own ego and macho-ism) make you forget that every solution has its own set of drawbacks, and every solution is subject to the law of diminishing returns. At some point we need to recognise that the problem is solved, and that solution is “good enough”.

Once “real-time” moves past “observable”, getting faster is “wasted-time”.

Turn a Shiny Dashboard into a Desktop App

2021-11-06T00:00:00Z

Shiny is popular web publishing service, unfortunately, not every application can be deployed on servers. This tutorial demonstrates a simple means by which to deploy a shiny app to desktop by creating a Site Specific Browser. Mostly to skip the bureaucratic begging for a server.

Available on GitLab: Jeff Cave / shinyapp-desktop · GitLab

An R-Shiny dashboard can be run as a desktop application, from a double click, to give non-technical users a seamless experience.

About two years ago, I had one of the Data Scientists in our organization call me with a problem. They had just spent a significant amount of time putting together a dashboard in R and Shiny and were wondering where they could host the dashboard to share it with clients.

Don't have a Medium Account? Use my friend link!

They wanted to know where our Shiny server was stored and how they could publish to it.

It took everything I had not to laugh at them.

The thing I was working on at the moment he had called was a generic deployment system for exactly that kind of project, but was running into all kinds of negotiations with Security, Finance, Architecture … everybody has to have their say. To get him the server he wanted, I estimated years.

Like any large organization, the bureaucracy must be fed.

This was a huge blow to the Data Scientist, his team had been developing the dashboard for months. The business had invested precious effort in describing their informational needs. The team had demonstrated the value of Shiny. They were now ready to realize all that effort, and the organization's statement was: we can't do that.

That's a lot of wasted effort.

After some discussion, I took serious pity on him and his team (and myself, I'd actually invested a lot of my coffee breaks coaching his junior Data Scientists).

Your customer's need the dashboard now? (yes)
Is the dashboard computationally expensive? (no)
Do you have a shared folder you could publish the application to? (yes)
Are the customer's at all technically savvy? (no)

I told him to give me the weekend and I'd give him a prototype solution on Monday.

Project

The intent is not to teach how to do complex mathematics, or to write Shiny Apps, but rather to demonstrate how to configure a project within the organizational environment. It is hoped that this will act as a spring-board, helping users get setup quickly.

The code itself is a simple demo app exported from RStudio. The real trick is to get it to run on the desktop environment.

The expectation is that the developer wants to deploy a shared application, but is in an environment where there is no Shiny server, and is not likely to be one anytime soon. Rather than wait, the developer can take advantage of a shared folder structure, and an old feature of FireFox, to run the application on individual desktops. While not being as elegant, it represents a solution that is likely suitable for most reporting needs, and can be implemented immediately (aka: use the tools already present).

Pre-Requisites

The demo assumes you have RStudio installed, and will be interacting with the system via PowerShell. There is no reason this will not work on Linux, however it is not what we use at the office, so not what it was tested on.

Checkout the base project

To get started, clone the sample project and open it in RStudio

Navigate: https://gitlab.com/jefferey-cave/shinyapp-desktop
Click: Fork
Open your copy of the project
Get the clone URL
Go to command line and checkout project

cd ~/Project/Folder git clonehttps://gitlab.com/jefferey-cave/shinyapp-desktop.gitcdshinyapp-desktopls -al

You should see a listing of all the files in the project.

Before we go any further, we should probably check to see that the project runs on our computer. This ensures that there are no basic configuration issues before the actual work begins.

Double click on the file desktopshiny.Rproj
Open: app.R
Click Run App

You should see the shiny app open in the in-built browser. There may be some dependency resolution that needs to happen.

Proof that the application is running and any problems we may experience are not with the computer configuration or app code.

Create a Starter Script

While it is great to know that the application works, it is not currently a great user experience. We have been asked to create an app for those less technically inclined, and they should not need an instruction manual to get up and running.

We can ease their experience by creating a starter script that loads their application from a shortcut.

The first thing to note is the output during the run of our Shiny dashboard. When we click on the button Run App we see the exact R command being executed to achieve all of this, and its output:

shiny::runApp()

Listening on http://127.0.0.1:5436

Try copy/pasting your URL into your local browser, you should see the same app.

Knowing that there is an R command that will start our application, we can skip the IDE and run the application using rscript

Stop your app in RStudio
Open a PowerShell terminal
Change to your project folder
Run your project using rscript

rscript.exe -e "shiny::runApp('.')"

Your app should be started, but without having to have the customer load the IDE:

Listening on http://127.0.0.1:3145

Try pointing your browser at that new URL. You should be looking at the app.

NOTE

The port changes every time you start. It is randomly assigned at start. You can specify the port that will be used; however if we put together more than one dashboard, having a random port means less coordination between data scientists (it should just work).

We will not be making assumptions about the start conditions and will not set a static port.

If we check the runApp parameters, there is one extra parameter we can include to make this a little more user friendly:

rscript.exe -e "shiny::runApp('.',launch.browser=TRUE)"

Save that in a text file called start.ps1.

You now have a basic script for users to start your interactive report. Having your customer click on the start script will give them a basically seamless experience.

Creating a new browser instance

Since Shiny advertises the port it is listening on, it is possible for us to capture that information, and then instantiate a special browser instance on behalf of the user.

For our user, it would be nice if we were able to start a browser instance just for the application. We also would like to stop the Shiny instance when the browser stops using it.

There is no data. There is only XUL!

(the XUL platform slogan)

For this example, we will use Firefox as it is based on the XUL platform which has a well documented and modifiable interface. To summarize (in a brutal way), FireFox is itself a webpage that can be dynamically modified (if you know how). We are going to use this feature to create a primitive Site Specific Browser.

Using PowerShell, we can extend our shiny server starting script to listen for the advertised port. We can then use this advertised URL and port to start a Firefox instance.

# Start an instance of the app using `rscript` and # capture `stderr` for the port number & "rscript.exe" -e "shiny::runApp('.')" 2>&1 | % { # look for the `url` line if($_ -like '*Listening on*'){ # Parse the input for the url ("$_" -replace ".*Listening on ",'').Trim(); } } | % { # open FireFox using the discovered URL & "C:\Program Files\Mozilla Firefox\firefox.exe" $_ };

This solution gets us part way there: we are now starting a unique instance of the browser for the shiny app.

The issue is that when we terminate our Firefox instance, our Shiny instance continues to run in the background. We must manually stop it.

We can continue to modify our script to start both the Shiny dashboard and the Firefox instance separately, then allow our script to maintain enough intelligence about them to monitor their independent process states.

# Start the `shiny` "thread" $shiny = Start-Job -Name "shiny" -ArgumentList($pwd) -ScriptBlock{ param($workingdir); cd $workingdir; # Start the shiny app and print the URL & "rscript.exe" -e "shiny::runApp('.')" 2>&1 | % { if($_ -like '*Listening on*'){ ("$_" -replace ".*Listening on ",'').Trim() } } }

At this point Shiny is started as a job, and jobs maintain a reference which can then be used to stop the job at a later time. This does add the problem that we need to read from the output stream slightly differently.

#poll the shiny thread for output
while ($shiny.HasMoreData -or $shiny.State -eq "Running") { $url = $shiny.ChildJobs[0].output.readall(); # when we find the URL ... stop if($url){ break; } }

Now, we have the URL at the script level, and can proceed to start Firefox.

Again, we want to create it as a separate process that we can monitor.

# create an array of arguments $args = @('-profile','./profile','-new-instance',"-url `"$url`""); # start the firefox instance $ff = Start-Process "C:\Program Files\Mozilla Firefox\firefox.exe" -ArgumentList $args -PassThru -Wait

This will block the script until FF stops.

Pay close attention to the arguments passed to Firefox.

profile: This uses a pre-existing profile that is customized to our purposes.
new-instance: Ensures, it does not re-use any instances of FireFox that may already be open
Wait: Ensures that the PowerShell job blocks processing until it ($ff)completes

This forces a “new-instance” and a new “profile” and a “wait” until Firefox completes. This custom profile is used to manipulate the way Firefox appears to the user. For the adventurous, go inspect the included profile to see ways you can manipulate Firefox to make it behave more like we want to.

Our final step is to simply stop the shiny process once the Firefox process has terminated.

Stop-Job $shiny.Id

The completed script looks like:

$shiny = Start-Job -Name "shiny" -ArgumentList($pwd) -ScriptBlock{ param($workingdir); cd $workingdir; & "rscript.exe" -e "shiny::runApp('.')" 2>&1 | % { if($_ -like '*Listening on*'){ ("$_" -replace ".*Listening on ",'').Trim() } } }

while ($shiny.HasMoreData -or $shiny.State -eq "Running") {
    $url = $shiny.ChildJobs[0].output.readall()
    if($url){
        break;
    }
}
$args = @('-profile','./profile','-new-instance',"-url `"$url`"");
$ff = Start-Process "C:\Program Files\Mozilla Firefox\firefox.exe" -ArgumentList $args -PassThru -Wait
Stop-Job $shiny.Id

At this point you should be able to simply run the script

.\run.ps1

and (eventually… it's a little slow) see your app running in a window.

Our dashboard, running as a standalone desktop application. The icon can be changed by modifying the files in the “profile” folder.

Creating a Shortcut

One of the issues with creating a PowerShell script is that regular users can't run it without a bit of “know-how”. The easiest way to get around this is to create an old fashioned BAT file.

cd <working directory> C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe ".\run.ps1"

If you run this as a regular user, you will likely get an error. The problem is that Windows (in their infinite wisdom) makes scripting unavailable to users by default. This is to protect them from malicious scripts.

In order to activate the script, you need to indicate that you “know what you are doing”.

Since we only want our users to see the desktop window, we may as well hide the console window while we are at it.

cd <working directory> C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -windowstyle hidden -executionpolicy bypass ".\run.ps1"

executionpolicy: allows the script to be run
windowstyle: allows us to hide the terminal window

Given this is a desktop application, a shortcut file (lnk) is probably a better option than the above bat file. These allow us to specify all the same parameters, but also an icon file, while removing all the console windows.

The link settings diaglogue, showing it filled in and with the icon set

Navigate to the working directory in Windows Explorer
Right-Click > New ... > Shortcut
Set the properties
- Target: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -windowstyle hidden -executionpolicy bypass ".\run.ps1"- Start in: <working directory>
-Change Icon ...

By setting those three options, your users are a double click away from a reasonably seamless desktop experience.

A Happy(ish) Colleague

The lead for the Data Science team I was working with was (reasonably) happy with the solution. It was a hack, but it got his team up and running in a matter of days.

We both agreed that optimal solution was to get a Shiny Server installed on-prem and link the URL form the internal website, so I put him in touch with the correct procurement experts as well as giving him this solution. I don't know what ever came of the procurement.

Found this useful or interesting? Consider leaving a tip … it helps.

We did some refinements for the team's solution (mostly automating the deploy to the shared filesystem from GitLab's CI/CD features), but for the most part the above solution represents a quick and dirty way for Data Scientists get their work in front of decision makers.

To this day, the internal website maintains a link starting with file:/// that points to the shared network filesystem.

For those paying attention, you may notice that this solution is not constrained to Shiny, but to any served web application: Node, Python, or perhaps something tucked away in a docker instance. This can also be used as a means of constraining a user to a web based application, perhaps for keeping students from cheating in a test, or keeping a temporary labour pool focused on their tasks.

I make no claims that this is the right solution, what I suggest is that it is a feasible solution. Any organization that can build a Shiny application, also has the tools to implement this solution. I share this solution in the hopes that it helps another Data Developer when Bureaucracy gets in the way.

Why won't Staples take my money?

2021-07-19T00:00:00Z

Scroll to the bottom for a list of related goofiness where companies can't take money I'm trying to give them

I spent 6 hours today trying to buy a printer … and failed.

There is money set aside for this purchase in my bank account.
There are printers on the shelves
Websites advertise printers for sale

The reason I was unable to purchase a printer was because a corporation with a near monopoly on the Canadian market, one of the largest privately owned retailers in Canada, could not tell me which printers they had available for sale.

Let that sink in:

A multi-billion dollar retailer could not tell me if they had anything to sell me.

On the surface this is laughable, funny, but I believe this belies a more insidious problem that is pervasive, and down right dangerous.

The Sequence of Events

I recently made a long distance move across almost the entirety of Canada. I'll give credit to the movers, most of my stuff made it in one piece; unfortunately, my printer suffered catastrophic snapping of some hinges. Since we do not deal in paper as much as we used to, we initially just tried live without it; in the short term, this saved us a bunch of money, but it also helped us to evaluate what we really needed in a printer:

Scanner with an auto-feeder
Cheaper ink cartridges
Duplex printing (nice to have)
Ink colours in separate cartridges (nice to have)

After a moment of distress trying to print a government form, and with a fairly clear idea of what I wanted, my Wife and I decided to put an end to the problems and just pick something (anything) up at a nearby retailer: Staples Canada.

Naturally, my first reaction was to look on their website: staples.ca. Reviewing a listing of available products over a cup of coffee, in our pyjamas, was exactly the way to start figuring out what we wanted. It was a quick search:

all available printers
ordered by price

By jumping forward to the first couple of printers that met our needs we would get a reasonably good idea of what our critical price point are. Further, I also saw they had an option for 2 hour curbside pickup, so we should be picking it up in a couple of hours. We came across a couple of printers that met our needs, at around $100. Within 30 minutes, we clicked to order.

None available in store
None available online

What? That's annoying. Let's try the next one.

None available in store
None available online

Hold up. Let's try a filter: “Only show items with 2 hour delivery”. Surely that will filter to items that are available in the store right now.

None available in store
None available online

The more I searched, the more frustrated I became. After an hour of this, my wife and I finally decided to just go into the store and buy whatever they had available.

Upon arriving at the store, it didn't take long for us to narrow in on the products we were looking for. They were more expensive than their equivalents online, but there they were … or were they?

We had it narrowed down to one of three printers, when a clerk came over. We pointed to them and asked a couple of questions, and he stated: “well, we should probably check to see if they are available before we go any further. We may not have them in stock.”

Momentarily slack-jawed, we proceeded for him to check to see if they were in stock, only to find… they weren't.

None of the printers we had expressed an interest in were available for purchase. There they were on the shelf, but none of them were available for sale. Order? Nope, please go online to check availability.

As we walked out of the store, empty handed, the manager stopped us and asked if we had found everything we were looking for (“no”), how could he help, and spent the next two hours explaining it wasn't his fault that they didn't have any stock and trying to upsell me a high performance Laser Printer. My only question at this point is: “Is it available for purchase”.

The manager left me with a customer complaint phone number, and the promise to email me a link to a printer I was interested in. He had verified with the warehouse that it was available, and I could get it shipped to the store. He could not request it, but if I ordered it online via the website there should be no difficulty.

The results of my attempting to order a Brother MFC-J497DW from Staples.ca

On my way home, I had stopped in to the local Home Hardware to pick up a ladder and a shovel. They didn't have any ladders in stock, but pointed me to their website

Clicking on the “similar in-stock items” link had no effect

The Significance

These are fundamental problems in the Software and Information Systems space: inventory systems should be able to count inventory, sales systems should be capable of selling products. That businesses are allowing systems to be released without testing of the fundamental feature that was requested is a massive failing. Software developers are allowing the release of features that do not work… and not in subtle, nuanced, ways.

It's like manufacturing a car and forgetting to attach one of the wheels.

Developers did not even bench test their work
No peer review of any kind took place

That is fairly significant, and I suspect their are two elements at play:

The development was outsourced to a consulting firm that is motivated to appease managers.
It is highly likely that the feature was delivered on schedule.

Software development has been commoditized, and in the process the managers and developers have forgotten a very fundamental truth: software is built to solve real world problems that real people have. If you develop tools that don't work, people are hurt (even if it is just because they have to manually search through a store's catalogue, or because they have to add up numbers in a spreadsheet before entering into the payroll system). While it is up to developers to enact that quality, it is absolutely necessary that managers expect that quality.

This means deadlines may not be met.

… but then again, you have to ask the question: if the feature isn't implemented, has the deadline actually been met.

It's Embarrassing

Since the dawn of software development, testing and validation of the work has been paramount. When you entrust a person to design a process to take care of people, it must take care of people. There is no human judgement involved to cover your mistakes: if your process is flawed the machine will carry out a flawed process.

Margaret Hamilton realized this in the 60s when her team developed the software that landed the Apollo, introducing the concept of Software Engineer. She didn't develop software that worked because “astronauts are trained not to make mistakes” but because she tested the crap out of her software. Later leading to the language `001` and `USL`, languages that made programmers think about errors before they happened … because that's what programmers do.

After 23 days, Home Hardware was unable to tell me anything more than my order is “in progress”. After asking me if I'd like to wait it out, we took another 10 minutes and cancelled the order.

In the late 1990s and early 2000s, unit testing, test driven development, continuous integration, and automated regression testing came into their own, offering means of ensuring that features were guaranteed to be minimally tested for basic user expectation prior to release.

DevOps in 2010 should have made developers even more responsible for basic functionality testing, making them directly answerable to the errors of the system.

The issue is likely two-fold:

Managers believing that the skill lay in the visual elements they can see, rather than the business processes that they cannot
Developers not standing up for the engineering act they are engaged in.

It was a very long time, and a lot of hard work, before Hamilton's peers considered software a type of engineering. Watching software be released that does not meet the most basic of functionality requirements makes me realize that many developers are not living up to her legacy. I'll take it one step further … managers have to stop expecting that software will have the “testing” check-box tick, and start understanding that quality is a mindset of continuous improvement.

An ordered list supplied by HomeHardware.ca, which is out of order.

It Matters

I think this should be a moment for deep reflection within the domain of Information Systems and Software Engineering.

Airplanes fall out of the sky
Small business owners get falsely accused and imprisoned
Young mothers have their children taken from them
Retail organizations are physically unable to sell their products

Programmers need to consider the direct consequences of their actions; Managers need to take a moment and reflect on the value they are adding to their organization; and Executives need to reconsider the actual deliverables they are asking for.

Should any of the afore mentioned organizations like to engage my consulting services to identify gaps in their information systems which are costing them sales, please do feel free to reach out to me.

Also, please leave your examples of bonehead fails in the comments.

2021–07–19: Staples won't take my money

2021–08–20: Pizza 73 won't let me order a pizza due to a curl error.

2021–09–18: Out of 57 monitors listed, with the “for in-store pickup” filter in place, my Wife cannot find any in inventory on Staples website

2022–01–16: After 3 weeks, Aviva Insurance is unable to find a way to tell me how much I owe them.

2022–07–20: My washing machine broke down. Local vendor has none in stock, Whirlpool's vendor portal has been down for a week.

2023–02–28: After 2 weeks of trying to book a flight from London (LGW) to Calgary (YYC), my sister-in-law has learned that WestJet can't accept payment from UK citizens.

2023–03–01: … so she tried Air Canada, same problem.

2024–02–17: Amazon KDP will not let me publish a book with them because my address does not exist (according to their system)

2025–09–16: Telus Health booked my doctor's appointment in a timezone that neither I, nor my doctor, were in. Missed my appointment.

Git and the Intermittent Network

2020-12-28T00:00:00Z

Within my organisation, we have been moving toward modern web-based platforms. This offers many benefits to our users in terms of availability, sustainability, and growth capability, and I have been one of the leading proponents of it.

While modern web-based services are the norm, and desirable, there are risks associated with them that should be considered for mitigation. Primarily, these risks revolve around the centralisation of service and the network availability of clients. These risks are well understood and most tools used by modern development teams were designed with these types of issues in mind, however as the internet becomes more pervasive and stable, it is common for us to lose sight of its limitations.

A Personal Experience

As I write this (2020–11–03), I am experiencing an internet outage.

This event occurred mid-meeting and has resulted in a situation where I cannot connect to the online resources required to complete my corporate objectives:

No connection to a production server to conduct repairs on that server (or even inspect the logs)
No connection to Microsoft Azure to conduct experimental work in our laboratory environment
All forms of meeting communications have been cut-off (MS Teams, Webex, VOIP telephone)
An Outlook plugin cannot connect to an encryption server and is frozen because it is attempting to show me an encrypted email
I can't take any of the corporate training I'm supposed to do, or read that manual for that new tool I'm investigating

I'm completely dead in the water.

According to my provider, a fibre-optic line has been cut “somewhere between Halifax and Montreal” resulting in massive connectivity loss for the region. The only productive task left to me is to write up an assessment of the current failure, on my local device, and upload it to the network when communication is re-established.

Hold on… Did I just describe getting work done and loading it later? That is a caching strategy that is well known and commonly used for resolving network latency issues.

Historic Note

The internet, as we understand it, has not always been as accessible, available, or reliable, as we have come to expect.

Scene from “A Boy and His Dog” (1975), which depicts the kind of conditions the Internet was designed to continue operating under. [Fair Use]

It is worth remembering that The Internet was initially designed as a distributed communication tool to allow the military to continue operating remote computers in the event of massive node loss (dating back to 1966). The assumption of loss of network availability has been an underlying assumption of much of the internet's growth, and is built into the fabric of the internet.

In the early days of general access to internet services, connections to the network were made intermittently. This was performed by dial-up connections which would be initiated for short periods of time.

As networks became more common and robust, much of the shared development of software (Open Source) began to be shared across the network as opposed to letter carrier and print (via the “Share” catalogue). Unfortunately, internationally, not all network connections are created equally, and some users suffered from several disruptions to connectivity.

What was “Share”?

In the mid-1950s, a user organisation for scientific applications … was formed. One of its most important functions was serving as a clearinghouse for contributed software subroutines. The organisation was called Share … and the contributed routines became the first library of reusable software.

— Robert L. Glass, “Facts and Fallacies of Software Engineering”

Out of this sense of sharing evolved several clearing houses such as SourceForge, Tigris, and eventually GitLab and GitHub. Unfortunately, even these clearing houses were subject to disruption.

In the highly competitive days just after Y2K, several organisations rose and fell in rapid succession and crises formed when code hosting platforms were simply turned off due to corporate takeover, sabotage, copyright infringement lawsuit, or even simple bankruptcy. Thousands of hours of work were lost to the simple issue of the centralised servers being turned off. Modern VCS (Version Control System) solutions (such as Git) evolved in this environment, allowing for each node to retain complete histories of work, allowing for projects to be completely retained through a single surviving distributed node.

CAP Theorem (CouchDB: Definitive Guide)

In more recent years, CAP theorem has evolved to explain that high availability comes at certain costs, which, once the costs are accepted, can offer the benefits seen by the BBC newspaper during the Russo-Georgian conflict. During this period, communication lines were severed meaning that correspondents and readers could not communicate across the national boundary. Service continued to be delivered to each side of the boundary, allowing reporters to continue reporting, and commentors to continue to offer feedback during the entire conflict. Automated synchronisation of news reports, and on-the-ground reader comments, occurred during periods when alternate communication paths were established.(ERROR: I could have sworn this case was described in CouchDB: The Definitive Guide … I can't seem to find the reference to this anymore)

Each of these historic scenarios have common elements:

contributors are forced to disconnect from communication and wait
contributors wish to continue to prepare their communications
“caching” is used to overcome communication latency, allowing people to continue working locally until the connection is reestablished.

This “batching” or “caching”, offers means to mitigate connectivity issues with web development platforms.

Web Based Publishing

Regardless of the content being published, content development has many common elements through its progression. Whether this is dynamic content of Software, or the static content of News Videos, there is a common process in creating and distributing the content online.

To use the example of an individual publishing an article to their newspaper (maybe their blog), they (the Contributor) would connect to the internet (NetProviderA), and type their article into Open Journal Service, or Wordpress, or Medium (the Server). They can continue to type into the software on the server, perhaps running some spell and grammar checks, until they hit the “publish” button. At this point it is possible for the message to be retrieved by the consumer whenever the customer wants.

Using web based development tools, the process for software development would be the same. The contributor would connect to the internet, edit their document on the server, and indicate readiness to “publish” which would make the application available to consumers of the application.

Intermittent Connections

Looking at the historic development of the internet, and the current issue at hand, we can see that there is a risk associated with the network not being available. We cannot consider the server in isolation, and must also include the effects of the network.

We can see that if the Contributor's network connection is terminated, they are unable to perform any work. Sticking to the newspaper article example, the author may have a wonderful idea in their mind, or may know of a flaw in the argument, or (frankly) just want to get some work toward the publishing dead-line; unfortunately, they are stopped.

There is another layer that can be considered to overcome this issue, and that is the local computer which can be used to cache work: the Contributor can type their document on their local computer, and save to their local disk.

Looking at the previously discussed internet history, we can look to VCS tools to assist us in solving this problem. SVN and Git (as well as their predecessors and competitors) were developed in an environment where work needed to be buffered against future connections. Specifically, work needs to be performed locally, and stored locally until such time as it is possible to transmit it.

This has been an ongoing evolution, and Linus Torvalds specifically developed Git to resolve buffering issues he saw in SVN.

Tip

Git is not a simple upgrade of SVN, there are trade offs between the two products. Git stores complete copies of the database on every node, SVN stores one copy on each node. Git stores full copies of each state in its database, SVN stores a sequence of state changes in its database.

The end result of these differences is that Git is always recoverable (any single node can rebuild the entire system), but SVN requires less storage space and works better when large binary files are involved.

While Git is now dominant, many users of large binary files (Engineering Diagrams, Cartography) continue to prefer SVN.

Local or Web

Using local development tools and synchronising periodic changes is a common practice to allow us to only communicate changes we are committed to, but this practice also offers the benefit of resolving latency issues. During 2020, lock-downs have resulted in many of us having to work from home, and being remotely positioned to our workspace. We are using networks that were established for scenarios that demand significantly less resilience (binge watching movies) for situations that demand significant resilience (earning income to pay for groceries). For those of us whose livelihoods have become tied to these networks for the first time, this can be surprising. In cases where internet has temporarily failed, and we are left unable to progress, it can be distressing to both our managers and ourselves.

This does not mean I believe that local tools are better than web-based tools.

For many years, my favourite platform was Cloud9, an online web based IDE that allowed for workstations to be stood up on demand. This allowed me to maintain several development environments that met various needs. The ability pick up work from anywhere in the world, allowed me to continue working on projects from a hotel courtyard in Ecuador, from an the old indestructible RCA Cambios. The ability for the vendor to supply me with powerful remote computer meant I could work from a $100 computer. This means I received software upgrades immediately, and could work from any cheap hardware I could scrounge up.

There are trade-offs to be considered, and that is what this has been about. Be aware of the trade-offs before wholly committing to one solution or the other. IDE vendors want you to be tied to their tool, and this looses many of the benefits of distributed VCS platforms. On the other hand, we have computing networks, take advantage of them.

In the end I recommend a balanced approach that takes the lessons from the rich history of information sharing that is the internet.

Use Web based IDEs, but use generic ones. Do not depend on always having access to the vendor's editor. Instead maintain regular local pulls from your VCS repository, and use programming languages and data formats that are based on simple text. This allows you to switch to a local copy during network outages, as well as protecting you from vendor lock-in.

Avoiding “Psychic” Software Development

2020-10-24T00:00:00Z

James Randi died October 20, 2020.

The Amazing Randi was a successful stage magician, famously surpassing many of Harry Houdini's achievements. Later in his career, James Randi took his mastery of magic and used it to turn a critical and skeptical eye to claims of paranormal powers. Claims of divine healing powers, telekinesis, and psychic powers were all put to the test with what grew to a $1,000,000 reward to anyone who could prove their powers.

The “Randi Prize” was never claimed during its 50 years.

Modern Product Development

Edward Deming once said, “In God we trust, all others must bring data”.

Over my career I have become infamous for citing Deming, but if this is a time for confessions, I must confess that my knowledge of Deming came much later than my attitudes toward evidence based decision making. Rather, I can attribute my personal distrust of claims and constant demands for evidence to people like James Randi showing me how, and when, to be critical of my own beliefs.

Both Deming and Randi demand that we approach our observations critically and skeptically manage our own biases.

This has impacted me professionally in two significant ways:

I am suspicious of software developers that claim they have finished a difficult task
I am suspicious of consultants promising to make problems go away quickly and cheaply

I say this as someone who has both development and consulting in his past.

A Personal Confession

In my youth, I was fascinated by psychic powers. I bought all the books on remote viewing, developing my psychic powers, and becoming a medium. I recognised that information was a powerful tool and was interested in any means to acquire more of it.

The problem is that, like Fox Mulder, I've always wanted to believe. Lewellyn publishing can account for much of my allowance.

Scene from the X-Files with Mulder's famous “I want to believe” poster in the background

Unfortunately for my desire to believe, at some point I saw the now classic episode of What's My Line with guests James Hydric and James Randi. In the show, Randi is skeptical and flat out states that Hydric's telekinetic ability to turn book pages amounts to him blowing on the pages, and introduces some light weight Styrofoam around the book. He is firm and calm, and completely unrelenting in his stance, and the scene eventually gets uncomfortably awkward as Hydric begins a convoluted explanation of his ensuing failures.

Hydric is a bit of a tragic character. While he was obviously a fraud, it appears he became so as an attention seeking behaviour. Watching his confessional interviews after the Randi event, I got the impression that he had come to believe his own hype; that he had misled even himself.

Whatever the case, Randi's critical approach to assessing paranormal capabilities has haunted my ability to blindly believe with a shadow of skepticism.

Software Developers

In the process of developing software, developers are expected to produce solutions to problems that have not been solved. That is the nature of the craft. However there are corporate expectations that the problem be solved in a reasonable amount of time, because time is money. This places pressure on developers to “be finished”, and this creates the risk space: scared for our job, eager to please, wanting to appear skilled, we want it to be done too.

So we tell our managers that the job is complete.

We are physically incapable of seeing that it does not completely solve the problem, or that it is too difficult to use in its current state. Like those that are healed by Faith, we want it to be true.

James Randi and Bob Barker expose James Hydric as a fraud. Subsequent interviews with Hydric give the feel that this was the moment he stopped deceiving himself.

This is where processes and philosophies around the SDLC come into play.

Issue boards keep us from reporting more progress than we have actually achieved
Sprints and backlogs keep us focused on the most pressing issues
Automated software build/test/deploy ensures that evaluation is unbiased and reproducible.

While the debates continue about which controls are the best, there is no doubt that we need the controls. Like James Randi placing Styrofoam around a phone book, these controls ensure we are being honest … even to ourselves.

Faith Healing

Even assuming it had been real, Hydric's ability to turn phone book pages was little more than a novelty act. It may have sold a few books, but debunking it was not exactly an earth-shattering revelation. A more significant case can be found in that of Peter Popoff.

Peter Popoff is an evangelical minister whose television broadcast became famous for his claims of divine knowledge and healing abilities. During his shows he would call arbitrary individuals from the audience by name and cite details of their life with no prior knowledge. As he approached them, and they began to stand, he would tell the gathered audience what terrible diseases the person had and that he would heal them. Both the knowledge and the healing were claimed to be directly imparted divine powers.

Rather than being divinely inspired, Randi discovered that Popoff's wife was the true source of knowledge. Prior to the show, she would gather information from attendees and was broadcasting their names and information via radio to an ear-piece he wore.

Here lies a case of fraud that demonstrates true harm.

People that sought out Popoff truly believed God spoke to him, and that his hands could remove illness from them. Believing that they had been cured meant that they would stop seeking treatment for arthritis, or epilepsy, or heart conditions. When one adds the $4 million a year in “donations” people made, Popoff has made it a little more difficult to replace the pills he told them to throw out.

James Randi, did thousands of people a service when he exposed Popoff. Though we do need to ask why Popoff continues his “healing” to this day.

Consultants and Vendors

One way management can reduce costs is to hire outside expert consultants that understand the problem better than the internal staff, and nobody knows the solutions better than the vendors. Naturally, as experts they are to be paid more than internal staff, this is justified by their being more knowledgeable.

Unfortunately, this is often not the case.

Due to the short time frame they are present for, consultants are not actually paid for measurable results. The measurable results of their suggestions and changes come after they have left. Their real rewards are tied to making the manager that hired them feel good about their decision. This does not necessarily mean they were successful at solving the problem.

Like the Faith Healer, consultants can reap huge rewards for making promises of solving problems, and making their audience feel that the problem has been solved through some special conference from an authority. Unfortunately, like the Faith Healer, this can be, and often is, done as an act of faith.

In fact, it is almost impossible for this to be undone because the person that has paid out their life savings to be healed, or the manager that has spent a significant portion of their budget, cannot admit to themselves that they were swindled.

The more we pay, the more we want to believe.

I have worked with some great consultants over the years, but I've spent more working hours with bad ones. The reinforcement cycle is one in which the best rewards go to those who make management feel good. Unfortunately, they are also the ones that consume the most time in fixing and retrofitting good solutions around their popular one. Given they are paid by the hour, this means the feedback mechanism benefits the dishonest.

There is no easy answer to this except to be skeptical of smooth talking salesmen that echo what you already want to believe. Often consultants are brought in because local staff have been asked to solve a problem and have given an undesirable response. Unfortunately, that is often the honest, but hard-to-hear, truth.

Like James Randi upsetting a lot of Peter Popoff's believers, the truth can be hard to hear, but healthier for you in the long run.

Faith Healing in Modern Times

In reminiscing about the impact of James Randi, naturally I turned to Wikipedia to refresh my memory. It has been a long time since I have had need to know about Randi's work: a different time. We now rely on scientific reasoning, and no longer believe in psychics and faith healing. We no longer make business plans based on “gut feelings” but rather “collect data to provide a basis for action”.

That was a different time, a simpler time.

Imagine my shock to learn that frauds Randi has exposed continue to be active as recently as 2015, with terrifying consequences.

The ADE-351 “bomb detector” [Wikipedia]

According to Wikipedia, the ADE-651 is an explosive detection device that is used internationally to keep people safe from terrorism. Naturally, people want to be safe from terrorist bombs and put their faith in technological devices to protect them. Randi first challenged the developers of the device in 2008, and it has since been demonstrated to be ineffective, to the point of containing no operating machinery at all. The FBI has repeatedly issued bulletins to law enforcement to stop using the device. In spite of this, it continues to be used, as a life saving device, by several countries and local law enforcement agencies.

People are so desperate for it to be true, they just will not let it go, and people are dying as a result.

The false sense of security provided by the device had catastrophic effects for many Iraqi people, hundreds of whom were killed in bombings that the ADE 651 failed to prevent

— Wikipedia: ADE-651, Investigations, Iraq

Perhaps people cling to it for hope, perhaps they cling to it for vanity, but in these modern times, people are paying millions of dollars for devices that end up getting them killed.

Reading about the ADE-651 I am reminded that there is no quick cure for superstition. We still want to believe that we are finished, and still want to believe that we are clever, and we are still greedy when it comes to getting that promotion. Like Randi, all we can do is be eternally vigilant against our own fears, hopes, and biases.

Thank-you Mr. Randi

Randi has left a swath of fraudsters in his path: Uri Geller, James Hydric, Peter Popoff, James McCormick and many others. Each of them represents a swindler filling their pockets with millions by feeding on the hopes and fears of thousands of people. He showed the danger of blind faith, and importance of protecting ourselves from our own desires.

From time-to-time I still blame software errors on “planetary alignment”, or “demonic possession”. Other times I amaze people with my psychic ability to know an error without ever having to have seen the problem they are experiencing. But these are done in jest, and always followed by (at least the offer of) a detailed investigation or explanation as to how the discovery was made.

As individuals with a responsibility to achieve goals, and under pressure to deliver, its is sometimes hard to hold ourselves to account. Sometimes we feel tempted to give or accept false hope to preserve our own dignity. Randi's approach to debunking the paranormal did not make him friends with believers, but it cut directly to the heart of the matter. When lives and livelihoods are on the line that's what really counts.

So thank-you, James Randi. You did not make friends among the frauds and charlatans of the world, but you certainly inspired at least one developer to push beyond the illusion of success.

Allow people to make assumptions and they will come away absolutely convinced that assumption was correct and that it represents fact … It's not necessarily so.

James Randi

Using WebGL to Solve a Practical Problem

2019-10-28T00:00:00Z

Some time ago, I was teaching introductory Python, and basic browser programming. During this time, I wrote an application that compares pieces of software code and presents their similarity in a force directed and tornado diagram. I ran this software semi-regularly (weekly) and, very early on, a significant problem appeared with my solution.

It took a really long time to solve. A really long time. A painfully long time.

I needed to find a way to speed up the processing.

The points are computer programs, and the orange line represents the progress of a comparison. With 36 comparisons, this takes … long enough that I get bored.

At the time I wrote the tool, GPU processing was hot and everyone was talking about how this was going to speed up everything. No matter what the question, GPU was the answer. This was an obvious avenue of investigation. However, I had decided to write this tool in the browser (legal and ethical constraints), and browsers do not have direct access to the underlying hardware.

So if all the cool kids are using GPUs, and this is written in the browser, and I'm intensely curious …

it looks like I'm learning WebGL and GLSL.

WebGL Demo
*Because WebGL handles 4 memory registers naturally (RGBA), we use those for (four?) memory locations. This also makes…*jefferey-cave.gitlab.io

Since writing M.I.S.S., Google has released tensorflow.js, also other tools like gpu.js and twgl.js are also available. While I chose to directly write in WebGL, the compute abstraction layer offered by these libraries is often useful.

WebGL is not a Compute language … but when has that stopped the adventurous.

####Pre-Requisites

Before beginning you should be comfortable with programming. The demonstration is written in vanilla Browser JavaScript, so no particularly advanced techniques are used; however, using WebGL requires switching between two languages, and compiling of code. Web programming does not usually involve those things.

The only programmatic technique you should be vaguely familiar with would be “cellular-automaton”: Conway's Game of Life is the classic example of this. GoL has been a staple of programming instructors for 50 years because the problem is relatively simple, the solution is complex enough to exercise student skills, and the output is kind of fun.

In addition, I strongly recommend going to the local office supply store and buying a cheap pencil, eraser, and pad of grid paper. Nothing builds understanding like working through problems yourself.

How GPUs Speed Up Processing

GPUs are a completely distinct mechanism from CPUS. CPUs are designed in such a way as to offer many operations to people and allows you to run them one at a time. GPUs offer fewer operations but sets them up in a way that you can run a bunch of them simultaneously (parallel processing).

This comes at a couple of different costs to us programmers.

It's like working on a different computer.
The instructions we write for one, don't necessarily exist on the other.

That's annoying, but … parallel processing: as long as they all run the same set of instructions, you can run a calculation a couple of thousand times, but simultaneously. Very simply put:

GPUs do parallel processing of a single function.

Technically, the function is called a “kernel”, in my code I referred to it as a “program”.

Consider the following function:

function MultiplicationTable(size=10){
  let table = Allocate2DArray(size);
  for(memLoc.x=0; memLoc.x &lt; list.length; memLoc.x++){
    for(memLoc.y=0; memLoc.y &lt; list.length; memLoc.y++){
      table[memLoc.x][memLoc.y] = memLoc.x * memLoc.y;
    }
  }
  return table;
}

Parallel processing on the GPU is about doing the same action simultaneously. In this case, the multiplication is a process that is consistently the same.

I'm going to just do some basic math: a 10 x 10 array costs us 100 units of processing time.

Now consider the processing using the GPU

function MultiplicationTable(size=10){
    let table = Allocate2DArray(size);
    table = gpu(table)
        .forEach((memLoc)=>{**return memLoc.x * memLoc.y;**});
    return table;
}

That forEach costs one (1) unit of processing time, no matter whether it is 10x10, or 10000x10000.

I made that code up, it won't work, but it does give you some idea of what we are trying to work toward. No matter how big we make table it will take 1 unit of processing time.

Using GPUs

GPUs are mechanically different from CPUs.

Because of this mechanical difference, it is useful to think of GPUs as a completely separate computer that you are attached to. Not only do you have separate processing unit (GPU instead of CPU), it also uses separate memory, and a separate instruction set.

These three elements of separation mean there are three major phases that we need to go through to make use of them:

Send the instructions in the GPU space (compilation)
Exchange memory with the GPU space (transfer — read/write)
Execute the instructions (execution)

Managing these three phases are the most difficult and complex part of using WebGL. There are a significant number of details that need to be handled just to exchange information with the other space.

Part of the reason for this is that the G in GPU and WebGL stands for “Graphics”. We are attempting to use something designed for manipulating images, to do general computation. The details that need to be managed revolve around defining elements of an image; this means we need to go through a process of describing our raw numbers in terms of an image.

This is simplified by creating helper functions that will describe our data for us.

####psGPU

In WebGL.html, just such a helper class was setup, called psGPU [webgl.html:158]. It has a few functions that abstract much of the configuration away:

addProgram: Compiles and sends a block of GLSL code (as a string) to the GPU space. [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L384-456" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L384-456" class="markup--anchor markup--li-anchor" rel="noopener" target="_blank">webgl.html:384</a>]
initMemory: Creates a hidden image that will act as our processing memory [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L289-346" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L289-346" class="markup--anchor markup--li-anchor" rel="noopener" target="_blank">webgl.html:289</a>]
write: Transfers our memory (UInt8Array) over to the GPU space [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L254-264" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L254-264" class="markup--anchor markup--li-anchor" rel="noopener" target="_blank">webgl.html:254</a>]
read: Transfers our memory back from the GPU space [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L241-252" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L241-252" class="markup--anchor markup--li-anchor" rel="noopener" target="_blank">webgl.html:241</a>]
run: executes the program we compiled [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L266-287" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L266-287" class="markup--anchor markup--li-anchor" rel="noopener" target="_blank">webgl.html:266</a>]

As a novice, these functions became critical to setting up the GPU. I was strongly interested in implementing an algorithm, and the complexities of memory management were acting as a significant distraction to the complexity I wanted to focus on.

####pixel

At some point, a second helper class was created called pixel. [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L96-156" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L96-156" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl:96</a>]

The memory that is interchanged through read and write consists of a byte array. The interpretation of an image of as a byte array, calls for a few more helpers. In particular, each pixel of the image is interpreted as 4 bytes, representing the Red, Green, Blue, and Alpha (rgba) values of the pixel. Within the GPU, these values are represented by the type vec4 which is a collection of 4 values (r,g,b,a).

To help maintain consistent naming across the CPU/GPU boundary, the pixel class was created. The pixel class is really just a convenience for mapping the returned UInt8Array to the 4 bytes that represent a given pixel, allowing them to be referred to by the same rgba notation.[<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L102-112" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L102-112" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl:102</a>]

####DrawGrid

Probably the most interesting (maybe “useful” is a better word) utility function is DrawGrid. [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L828-883" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L828-883" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl:828</a>]

Because GPUs are designed to manage images, the only inspection of memory changes that is available is by looking at a picture. Since the purpose of this project has nothing to do with images, colour is not a meaningful representation. This makes debugging … trickier.

To help, DrawGrid does nothing more than render each pixel location as their underlying numeric values. It is roughly equivalent to JavaScript's console.log, allowing the developer to dump a set of values to a visible location for inspection.

It is most effectively used by placing it (and a break-point) immediately after a kernel run. Don't forget to comment it out when measuring speed.

####Actually Running

Once the helpers are in place, it is a simple matter of defining our processing functions and calling them in the correct order.

Send the instructions in the GPU space
Exchange memory with the GPU space
Execute the instructions

Given the amount of helper code in place, there is not a significant number of instructions left to perform in the actual code portion on the CPU side. All of the really interesting logic should be moved to the GPU; all of the interesting processing should reside in the kernel definitions.

On the CPU side, we send the instructions to the GPU (addProgram), and then repeatedly send notifications to execute the function (run).

The General Problem

Comparing code for similarity is a well solved problem. “Comparing sequences of instructions, while taking into account small variations”, sounds very similar to DNA comparisons.

In the world of genetics, Sequence Alignment algorithms have been around a long time (Needleman-Wunsch dating to about 1970). Comparing sequences of DNA for similarity, while taking into account small variations due to mutation or cross-over is a common goal, is the same problem we are trying to solve. Think of this as a paternity-test for software.

In my case, I reached for a Smith-Waterman comparison.

The best way to understand an algorithm is to solve it with pencil and paper. In this case, I spent a great deal of time with a pencil, eraser, and pad of grid paper from the local convenience store.

Take the names of two animals: <a href="http://slideplayer.com/slide/5142106/16/images/21/Smith-Waterman-Algorithm.jpg" data-href="http://slideplayer.com/slide/5142106/16/images/21/Smith-Waterman-Algorithm.jpg" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">coelecanth</a> and <a href="http://slideplayer.com/slide/5142106/16/images/21/Smith-Waterman-Algorithm.jpg" data-href="http://slideplayer.com/slide/5142106/16/images/21/Smith-Waterman-Algorithm.jpg" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">pellican</a>.

On the surface, they do not appear similar, however, closer inspection (surprisingly) shows they do align reasonably well:

c o e . l e c a n t h
p . e l l i c a n

Programatically we can find this alignment by solving a Smith-Waterman matrix:

In order to understand how the above alignment can be solved mechanically, I recommend following the example given on Wikipedia. I don't mean go read the Wikipedia page, I mean pull out that grid paper, and reproduce the example for yourself. Solve each step, and verify it against the example. If you made a mistake, spend some time understanding your error, and start over.

When you can solve a Smith-Waterman for yourself, you have proof that you understand it.

While the implementation is discussed in detail below, it it worth having an intuitive understanding of one of the two processes involved; trying to learn both simultaneously is harder. If you are interested in how to implement algorithms on the GPU, already understanding the example algorithm is useful. If you are already a master of the GPU and are interested in how to implement Smith-Watermans, this very basic example may be a good stepping stone.

Put it together

It isn't easy. Using the GPU requires us to think in parallel, and this requires us think in ways that we aren't usually used to. Things we take for granted in linear processing aren't available to us; things we would normally avoid, we accept for the sake of being able to use the tool.

The idea for how to do this actually came from one of my favourite college assignments: Conway's Game of Life (GoL). In GoL, each life-form location changes its state based on the state of it's nearest neighbours. This is normally solved in a grid represented with a table of values on the screen.

The key to that statement is that that each automata-cell has a state that is resolved independently of all the others, and based on the values of its nearest neighbours. That pretty much describes the the Smith-Waterman as well. The only real change is that Smith-Watermans only consider the neighbouring cells in the upper-left corner (North, West, and North-West).

These “parent” values need to be calculated prior to being able to calculate values, representing one of the challenges of parallelizing the algorithm: you cannot calculate dependant values in parallel. The first break through mentally was an animated GIF I found online that demonstrated a diagonal parallelization of values in an SW matrix. (the original reference is lost to the recesses of memory now … if you know it, leave a comment)

This was compounded by the fact that I had originally optimised my algorithm for low memory consumption. To do this, I had maximised the early release of the memory associated with a given cell if it was not part of a chain. One of the challenges in using a GPU was determining a way to arrange the memory in such a way that calculations were only performed on elements that have a complete parent set.

Working diagonally, allows us to maintain sufficient parent cells (green) to calculate a number of child cells (yellow) in parallel. Processing this on the GPU means that all of the other cells (white) will be calculated for no reason. A reasonable trade-off.

This focus on optimisation blinded me to the fact that the cost of evaluating a matrix with a GPU is 1 regardless of the size of the matrix. The original speed using CPU calculations required that every cell be evaluated one at a time: width * height processing time. Using the GPU, and calculating all of the elements each time felt like wasteful work, but at some point the realisation it was still only width **+** height processing time dawned on my dummy programmer brain.

Who cares about wasting a bunch of processing effort when it saves that much time!?

While I'm sure there are efficiencies to be gained, they are insignificant when compared to the speed increases of just evaluating cells needlessly until the entire matrix is solved.

Once this situation is accepted, it becomes reasonable to create the GPU function (or “kernel”) smithwaterman that can solve for an individual 2-D matrix cell [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L528-586" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L528-586" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl:528</a>]. An initialisation routine was also created which calculates the initial match value [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L687-710" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L687-710" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl:687</a>].

The first thing to note is the entire program is passed to addProgram as a string. This is to allow the GPU to compile the code, but unfortunately means no syntax highlighting. It also leads to very cryptic messages about invalid syntax. It is recommended that as you make changes, make them small to ensure you can identify where a syntax error was made.

The algorithm itself is very small in its implementation…

Initialise the Memory

Before we can start acting on values, we need to transfer the values to the GPU. While addProgram is used to write the code to the GPU, writing memory is performed by the write function. We start by fetching an appropriately sized array, and then fill only the top, and left portions of the array [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L735-744" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L735-744" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:735</a>]. This minimises CPU cycles by leaving the iterative portion to the GPU.

The initial Matrix score is done by comparing the extreme North and West values. Matches get a base score of 2 while everything else is set to 0.

Our first GPU task (initializeSpace) is to compare the intersections of these values for matches [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L687-710" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L687-710" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:687</a>]. For each cell, we lookup the value to the extreme west and north [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L700-701" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L700-701" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:700</a>], and assign 2 points for a match, or 0 points for a mismatch [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L707" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L707" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:707</a>]. This score is stored on the red channel.

It is worth noting that the GPU does its processing in terms of fractional values (float): everything is a portion of 1. So while the scores are intended to be the integer values 0 and 2, these must be consumed as some proportion. The values are actually passed to the function as 0.0/255.0 and 2.0/255.0, making them easily convertible between an UInt8 and float.

This is an important thing to remember. For the purposes of this algorithm:

Everything GPU-side is treating the numbers as floats, but the returned memory is an integer.

Solve the Matrix

Smith-Waterman's construct a chain of values representing the “best” matches. In this case, “best” is defined as the neighbour with the highest running score.

The first step is to look up the nearest neighbours, and this requires us to determine how close those neighbours are. The GPU thinks in terms of fractional values (float), while we are thinking in terms of discrete values (int). We need to calculate the fractional size of a memory location (a pixel) [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L544" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L544" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:544</a>]. Once this is done, we can lookup the value of the current cell (here), and its nearest neighbours (nw, n, w) [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L548-551" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L548-551" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:548</a>].

Once we have identified the critical neighbours, we can begin to assess their values. This is done by checking all three to see which has the highest score, and temporarily storing it on the blue channel [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L562-565" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L562-565" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:562</a>].

Looking at the nearest neighbours (yellow) we can determine the direction of the match. In this case horizontal (127) is the best match. Values are stored as fractions of 255 (255/2 → 127).

Knowing which score was highest, allows us to determine which direction forms the chain; each direction is tested to see if it forms the desired chain. In the event of a tied score, diagonal matches should be favoured due to, horizontal and vertical matches representing a skip; horizontal and vertical ties can be resolved arbitrarily. Directionality is represented by enumerations of 1-north, 2-west, 3-northwest, and is stored on the blue channel [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L568-576" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L568-576" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:568</a>].

Knowing the directional of the chain, we can now tally up the running total. The running score of the chain, is added to the local matching score; logically this is done after direction is recorded, however, due to only having 4 memory locations per cell, it is pulled from the temporary value stored on the blue channel before we finalise direction [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L565" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L565" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:565</a>]. We also apply a “skip” penalty (-1 points) to chains that had to perform a skip operation (non-diagonal direction) [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L579" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L579" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:579</a>].

Having identified the parent, we can add the running score, to the local score, and apply any skip penalties. In this case (`2+2-1`), which then gets stored on the current alpha channel.

Once the calculation is complete, the values are stored permanently to the current cell on the alpha channel [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L584" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L584" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:584</a>]. It is worth reviewing that there are 4 memory locations per cell, and how we have allocated them:

`red`: local matching score
`green`: unused (reserved for future use)
`blue`: chain direction
`alpha`: chain score

This diagram shows the calculation wave as it moves across the matrix. Green values represent cells with sufficient information to solve, yellow values represent values that have settled into their final state, and white values are ones that are indeterminate. Every cell is calculated on every cycle. Using this pattern, we can reduce the number of required cycles from ``x*y`` to ``x+y``.

As noted earlier, it is not sufficient to execute this process once. While we are calculating every cell's chain score, there is insufficient information for the last cell to complete its chain calculation until it's neighbours have completed their calculation. To resolve this, we run the GPU processing multiple times: first calculating the worst-case number of cycles required [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L518" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L518" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:518</a>], and then sending processing signals to the GPU a loop [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L752-755" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L752-755" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:752</a>]. Each iteration of GPU processing moves the “wave” of completed calculations forward one step.

Once this loop completes, and the tip of this wave reaches the bottom-right of our matrix, the first phase of the calculation is complete. Most importantly it was completed in x+y cycles, rather than x*y cycles. While the test samples are too small to take accurate readings (1ms resolution in most browsers), the animals sample went from 10ms to under a millisecond, while the lorem sample went from 5.5 seconds to about 1 millisecond, no initial readings were taken for identical, longchain, or gilbertsulivan [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L477-501" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L477-501" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:477</a>].

Even accounting for significant measurement error, this is a significant improvement!

We can see a significant improvement in speed in just the first 30 seconds of processing.

Conclusion

I can no longer find the source, but one of the best over views of the GPU on the web was from a video of a conference talk that ended with

If this feels like a dirty hack to you, that's because it is.

— if you know the video, please leave a comment

Some of this definitely has the feel of a dirty hack, for example, working around all the references to RGBA feels weird. WebGL was designed for graphics, not a computation. For the adventurous, this lends a sense of excitement and challenge.

I find this takes me back to my (very brief) days of working in C, where memory manipulation is a little closer at hand. Conforming to vec4 memory (rgba) really encourages you to think of new ways of using (and abusing) the way you use memory, or reusing memory, or squeezing that extra bit of information into an incompletely used byte (most of which has been refactored out of the example).

While I have loved Conway's Game of Life for decades, I never thought I would find a practical purpose for cellular automatons. Having noticed the similarity between this problem and GoL, I now want to revisit a personal attempt at a JavaScript implementation of aerosolized particulate dispersion models (mine has gone missing, this one is pretty cool).

Lastly, the most educational part of this for me was the value of pencil and grid paper. A lot of debugging revolved around solving the grid with a pencil and then comparing the resulting manually solved matrix to the program's solution.

Solving Smith-Waterman's by hand is also like doing a giant Sudoku or Crossword… kind of fun.

Next Steps

Open in the browser
Hit `F12`
Insert a Break-point
Start stepping through code

In this case, I'd actually suggest saving a local copy first to allow you to make minor changes to see the effect.

The performance gains I see by implementing the algorithm on the GPU were significant: approximately 5000 times in terms of speed. However, the more I ponder the problem, the more ways I see to improve it.

On the other hand …

This tool was written as a personal utility to meet personal needs. Until such time as there is more interest in M.I.S.S., there is likely little reason to implement performance gains.

Its fast enough… for now.

Looping

I suspect my calling of run in a loop is inefficient. Likely, it would be better to implement the loop as part of the GPU code. However, there are two reasons I did not do this:

The current implementation allows for periodic reading of the data, allowing for progress bars in the visualisation (orange connectors).
Input and Output memory is declared prior to execution. In the helper functions, input memory and output memory are swapped after every run. I don't know how to do this within the context of a single execution.

Neither of these issues seem insurmountable.

Chain Resolution

Smith-Waterman calls for two phases

Build the chains
Resolve the chains

This article intentionally ignores the second phase of a Smith-Waterman; the code that reads the chains back off the memory. The current implementation in M.I.S.S. uses pure CPU JavaScript to resolve the chains. However, a recent discussion in the office got me re-thinking about how that was done, and inspired me to rewrite the chain resolution function to use the GPU more.

I encourage readers to have a look at the GPU function chain [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L588-685" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L588-685" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:588</a>] to see its implementation. It uses very similar techniques to those already discussed [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L763-768" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/webgl.html#L763-768" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">webgl.html:763</a>].

Memory Consumption

Building a 2-D matrix in memory means height times width. That is going to grow quickly depending on your inputs. Also, as the number of token grows, there is a risk that the numbers that represent them will exceed 65535 (2-bytes).

One of the initial validation tests I had for the code was to do a comparison between the genomes of <a href="https://www.ncbi.nlm.nih.gov/nuccore/CU651637.1?report=fasta" data-href="https://www.ncbi.nlm.nih.gov/nuccore/CU651637.1?report=fasta" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">E.Coli</a> and <a href="https://www.ncbi.nlm.nih.gov/nuccore/AE009952.1?report=fasta" data-href="https://www.ncbi.nlm.nih.gov/nuccore/AE009952.1?report=fasta" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">Y.Pestarius</a> [<a href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/samples/bigcompares.zip" data-href="https://gitlab.com/jefferey-cave/miss/blob/medium-practicalwebgl/www/samples/bigcompares.zip" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">bigcompares.zip</a>]. Unfortunately, that eats up memory, and an 8TB array exceeds my laptop's capabilities.

I would love to see an implementation that cuts the giant matrix into a series of smaller ‘tiles' (16000x16000 for about 2GB?). This would ensure that they never consume more memory than is available, and never generate a token identifier greater than 2-bytes.

Tokens could be mapped to an index not exceeding UInt16, for the given tile. The tiles could be solved for independently (in parallel if you have more than one GPU), storing only their internal chains and edges. The edges could then be “stitched” together during chain-resolution.

It's an interesting idea, and I'd love to see someone run with it…

Build a Chromolabe, a Simple Pencil and Paper Computer

2019-10-14T00:00:00Z

This article is based on an activity originally designed as a classroom micro-activity to introduce some concepts of Visualisation in Data Analysis, in a fun and interactive way. The activity is suitable for children aged 12 to 120.

I would like to thank my wife, Sharon Cave, for her invaluable assistance in developing this activity.

Most people rely on their natural human intuition for decision-making, and human intuition is based on interacting with the physical world around us. Computers, on the other hand, deal in abstract ideas; something humans just aren't well equipped to deal with.

What goes on inside a computer's mind is a mystery to most people. In fact, its kind of scary: you can't see, touch, or taste it. This makes it hard to understand what it is that programmers are doing, and therefore can be intimidating for people.

This means that programming is constrained to the realm of people with “a freakish knack for manipulating abstract symbols”. This activity is designed to help bridge that gap. We are going to build a physical computer.

Computer programmers are often faced with the daunting task of taking those abstract concepts and making them concrete for their audience; a tricky and detailed task.

One of the tools used to do this is colour.

Classes from my personal calendar, D3 map chloropleth, force directed graph, gantt style chart, or a knock off of a popular video game.

In each of the example cases, colour is used to visualise the categorisation of ideas: time blocks in a calendar, political parties, characters in a game, we use colour everywhere to group related things together.

As a programmer, we have just come across our first problem: how do we choose the colour palette we will use in our program?

Good Palettes

There are three components to selecting a good colour palette for use in visualisations:

High Differentiation
Sufficient Colours
Aesthetically Pleasing

Every colour palette used in a visualisation should maintain these three basic elements

Sufficient Colours

There should be enough colours to meet the needs of the visualisation.

In the early days of video games, it was often enough to have 4 different colours: one for each player. In cartography, the number of colours needed is dictated by the number of shared borders, the same colour should never touch.

“Sufficient” is different depending on what you are trying to represent.

High Differentiation

Finding food in the wilderness is a matter of survival for hunter/gatherer humans . The differentiation of colour is “built-in” in humans. [image: Wikimedia]

The colours used should be sufficiently different, that people can tell them apart. If you are trying to tell the difference between “stop” and “go” at a traffic intersection, there should be no ambiguity in which colour you are showing your audience.

This problem is complicated by internationalisation (colour means different things in different cultures), biologic issues (colour blindness), or other things.

Aesthetically Pleasing

At the end of the day, humans are the ones that will be looking at this visualisation, they should find it pleasing to look at.

Choosing colours that are “complimentary” can often have bizarre visual consequences. We need to work with the way humans are built. There is also a fashionable element to this, colours that are popular today may not be popular tomorrow.

Complimentary colours have high contrast but, when placed side-by-side, can have negative consequences. Interestingly, the pastel variants do not suffer these consequences.

Infinite Colours to Choose From

Choosing colours is difficult.

There are an infinite number of colours to choose from, but we need to choose just 4–12 colours, and those colours need to meet our three requirements.

Hard decision making is exactly what we build machines to help us with. To start building our information machine, we need to first organise everything we know, and all we know so far is a basic truth of the universe

There are a lot of colours to choose from

To help us sort through the options we need some way of organising the them. That is a key thing programmers do, they organise stuff. By organising things, we can simplify our problem.

Moses Harris, The Natural System of Colours (1776) [Wikipedia: Color Wheel]

Colour wheels have long been a tool for grouping similar colours with one another.

So we first need to narrow the set down, let's only pick from colours humans can see.

This process can be represented with a Colour Wheel.

Colour wheels represent the variations of colours that can be made by mixing different base colours together. The base colours are placed around the outside.

While not representative of “every” colour possible, these colour wheels give us many colours to choose from. They also have the advantage of placing similar colours physically close together.

This helps us satisfy two of our requirements:

Sufficient: If we need many, there are many.
Differentiation: the further they are physically, the more different they are.

Mechanically Making Decisions

Narrowing down an infinite number of colours, to just the 4–6 we need, is difficult, and building a machine to choose for us is the point of the exercise. Unfortunately, describing the machine and how it works mathematically, sounds like a bunch of abstract symbol manipulation.

Rather than trying to explain it, let the demonstration be its own proof. Besides, this machine is really simple...

… like … really simple.

Building the machine is way easier than explaining it. Once you build it, it should become self evident how the colour picker works.

1. Gather your supplies

Get a pencil … sharper is better.

Print the supplementary material: the key pages are the colour-wheel, and the blank circle.

The two template images that will need to be printed from the slides.

2. Draw a Spiral

Using the blank circle, starting anywhere along the outside, begin tracing it gently with your pencil. When you have a feel for the circle, draw an evenly spaced spiral toward the middle.

3. Do the calculation

Place your pencil on the topmost tick-mark of the circle.

Now, move your pencil three steps, and mark the position:

all the way across the circle, to the opposite side
over one tick-mark (either way)
move in one step toward the middle
mark that position

Now, starting from this new position,

One of the reasons we like machines is they do repetitive tasks for us. Describing the repetitive tasks is often the real trick

move all the way across to the opposite side
move over one tick-mark (same direction)
move in two steps toward the middle
mark that position

Did you see it? The small change in the repeating pattern?

Keep moving around marking points on the chart until you work your way to the very middle.

4. Make it permanent

When you think you have all of your dots in the correct location, carefully…

(make sure you are careful, this is both delicate, and dangerous)

place your pencil over the first dot and…

(don't mess it up: we are building computers here)

Ram that pencil through!

Seriously, don't be gentle. Punch a good sized hole in the paper.

(just be careful not to put your hand in the way)

Do that for each of the dots you made.

Reading the Results

Align your spiral graph with the colour wheel, and watch your colour palette reveal itself. Start from the first dot and copy down each of the colours your program chose.

That is your individual colour palette to use in your visualisations.

Remember when we started, we were looking for a colour palette that satisfied three criteria:

High Differentiation
Sufficient Colours
Aesthetically Pleasing

High Differentiation

By starting with a colour wheel, we locate similar colours physically close together. Our algorithm then travels far across to select colours that are physically far apart.

Sufficient Colours

The spiral pattern ensures that we never select the same colour twice. No matter how many colours we select, we will always have different colours.

You may notice that the differentiation of colours decreases as you approach the middle: the more colours you pick, the more similar they get. This is a compromise we need to make. It is a balance between having a lot, and having them be different.

Aesthetically Pleasing

This is always a tough one, what is pleasing to me, may not be pleasing to you.

There is no real way to have our simple computer decide whether the selected colour is pretty or not. That is something best left to humans. So take a good look at your colours, and decide if you like them or not. By luck, you may have gotten a good starting point, but maybe you didn't.

There is one last thing you should try: poke a small hole right in the middle.

Now you can rotate the selector wheel around the colour wheel to fine tune your selection process.

Conclusion

Don't give this as a written assignment, it was designed to be an interactive activity. Instead, deliver it as performance art. A lot of how to manipulate the paper and pencil is very obvious when demonstrated, but is very difficult to describe.

This was originally designed with a target audience of first year programming students in mind, but can easily be scaled to the elementary level (by leaving out all the theory), or scaled up to the undergrad level by asking students to mathematically define the formulas involved, or write software that implements the process.

Don't forget that the “reveal” is part of the “fun”.

Variations

I would recommend explaining a colour wheel and colour mixing as a separate, precursor, exercise. For elementary students, have them mix paint. For high school students, have them physically make their own colour wheel with pencil crayons or water colour. For undergrad programming students, have them create a colour wheel programmatically.

The perfect combination activities (in my fanatasy world) would be for students to build a colour wheel in their art or physics class, construct the selector wheel in maths class, and use the resulting colours in charts constructed for their civics class. All of it should be hand drawn to maximise comprehension.

A natural extension of this activity is to write a function (programmatic or algebraic) to pick the colour. This would require students to define the colour wheel space, as well as traverse the space in a spiral. HTML colour codes based on RGB make a good return value.

If you really wanted a challenge, move into a more complete 3 dimensional colour-space. RGB defines three dimensions: define a function that selects from a colour sphere, with a 3 dimensional spiral search of the space…

hmm…

Even I'm not sure how to do that.

Tips For Teachers

As an introduction, promise students that in the next 30 minutes they are going build their first computer program.
Don't have students cut-out the circles. By keeping them on the page, you imply “no rotation”. This delays discovery long enough for you to suggest it as “intentional thought”.
Students will have different “tightness” of their spirals. This is a hint as to the trade-off between high differentiation, and more colours.
There will be the student that is tentative about punching a hole in their paper. The hole needs to be big enough that the colour shows through. Encouraging the student to be a little rough with it gets a couple of chuckles.
Have the students (as a group) describe how their computer meets the three criteria that were laid out.
Ignore the aesthetics until the very end. Inevitably a student will bring up aesthetics. Act surprised… like you forgot about it. Then hold up your demo and rotate it. If you don't like the palette you got, turn it to select a different palette.
Like any activity, leave this to the end of the class. The excitement caused by comparing colour schemes brings the class to an end. Expect to do no more than have students take a written handout on the way out the door.
At the beginning, you drew a spiral. This was not strictly necessary. Really, you should do this using Radial Coordinate graphs (on paper), or a continuous formula. The act of counting across, and in, was the real spiral algorithm.

Activity Plan

Skills

Algorithmic Thinking

Problem definition

Learning Objectives

Describe the use of colour in categorisation
Describe the components of a good colour palette
Apply the components of a good colour palette

Audience

Grades 7 — Undergraduate
Introductory programming students
Introductory data analysis students

Materials

Colour Wheel: Make one, or print the templates
Radial graph: must be same size as colour wheel
Pencil: must be sharp

Progression

Learn the three components of a good colour palette (~5 min)
Construct a colour selector (~5 min)
Reflect

Steps

Gather Supplies (pencil/printouts)
All the way across, 1 over, step in, Repeat until full
Punch Holes
Select colors, record by colouring in grid

Reflection/Assessment

What changes could you make to get even more colours?
What happens to differentiation as you get more colours?
What decision making inputs can you control on the machine?
How can you change the colours if they aren't aesthetically pleasing?

Facts and Fallacies of Software Engineering

2018-06-02T21:57:37Z

This content is taken from the Table of Contents of the book. I can think of no more important step to professional and product development than to (at minimum) be aware that these are enduring myths of software.

Studying and pondering the implications is the next step.

This listing was created to use as a searchable index. Since I am currently reviewing one a week, it is useful to know there are 65 items total (55 facts, and 10 fallacies).

Facts and Fallacies of Software Engineering, Robert L. Glass (2003)

Facts

Management

People

The most important factor in software work is the quality of the programmers.
The best programmers are up to 28 times better than the worst programmers.
Adding people to a late project makes it later.
The working environment has a profound impact on productivity and quality.

Tools and Techniques

Hype (about tools and techniques) is the plague on the house of software.
New tools and techniques cause an initial loss of productivity/quality.
Software developers talk a lot about tools. They evaluate quite a few, buy a fair number, and use practically none.

Estimation

One of the two most common causes of runaway projects is poor estimation.
Most software estimates are performed at the beginning of the life cycle. This makes sense until we realize that estimates are obtained before theh requirments are defined and thus before the problem is understood. Estimation, therefore, usually occurs at the wrong time.
Software estimation is usually done by the wrong people.
Software estimates are rarely corrected as the project proceeds.
Since estimates are so faulty, there is little reason to be concerned when software projects do not meet estimated targets. But everyone is concerned anyway.
There is a disconnect between software management and their programmers. In one research study of a project that failed to meet its estimate and was seen by its management as a failure, the technical participants saw it as the most successful project they had ever worked on.
The answer to a feasibility study is almost always "yes".

Reuse

Reuse-in-the-small is a solved problem.
Reuse-in-the-large remains a mostly unsolved problem.
Reuse-in-the-large works best in families of related systems.
There are two rules of three in resuse:
1. It is three times as difficult to build reusable components as single use components, and
2. a resuable component should be tried out in three different applications before it will be sufficiently general to accept into a reuse library.
Modification of reused code is particularly error-prone. If more than 20 to 25 percent of a component is to be revised, it is more efficient and effective to rewrite it from scratch.
- Corollary: It is almost always a mistake to modify packaged, vendor-produced software systems.
Design pattern reuse is one solution to the problems inherent in code reuse.
- Corollary: Design patterns emerge from practice, not from theory

Complexity

For every 25 percent increase in problem complexity, there is a 100 percent increase in solution complexity. That's not a condition to try to change (even though reducing complexity is always a desirable thing to do); that just the way it is.
Eighty percent of software work is intellectual. A fair amount of it is creative. Little of it is clerical.

About the Life Cycle

Requirements

One of the two most common causes of runaway projects is unstable requirements.
Requirements errors are the most expensive to fix during production.
Missing requirements are the hardest requirements errors to correct.

Design

Explicit requirements 'explode' as implicit requirements for a solution evolve.
There is seldom one best design solution to a software problem.
Design is a complex, iterative process. Initial design solutions are usually wrong and certainly not optimal.

Coding

Designer 'primitives' rarely match programmer 'primitives'.
COBOL is a very bad language, but all the others are so much worse.

Error removal

Error removal is the most time-consuming phase of the lifecycle.

Testing

Software is usually tested at best to the 55 to 60 percent coverage level.
100 percent test coverage is still far from enough.
Test tools are essential, but rarely used.
Test automation rarely is. Most testing activities cannot be automated.
Programmer-created, built-in debug code is an important supplement to testing tools.

Reviews and Inspections

Rigorous inspections can remove up to 90 percent of errors before the first test case is run.
Rigorous inspections should not replace testing.
Post-delivery reviews, postmortems, and retrospectives are important and seldom performed.
Reviews are both technical and sociological, and both factors must be accommodated.

Maintenance

Maintenance typically consumes 40 to 80 percent (average, 60 percent) of software costs. Therefore, it is probably the most important software lifecycle phase of software.
- Corollary: Old hardware becomes obsolete; old software goes into production every night.
Enhancements represent roughly 60 percent of maintenance costs.
Maintenance is a solution -- not a problem.
Understanding the existing product is the most difficult maintenance task.
Better methods lead to more maintenance, not less.

Quality

Quality is a collection of attributes. (ED NOTE: Glass lists seven)
Quality is not user satisfaction, meeting requirements, achieving cost and schedule, or reliability.

Reliability

There are errors that most programmers tend to make.
Errors tend to cluster.
There is no single best approach to software error removal.
Residual errors will always persist. The goal should be to minimize or eliminate severe errors.

Efficiency

Efficiency stems more from good design than good coding.
High-order language code can be about 90 percent as efficient as comparable assembler code.
There are tradeoffs between optimizing for time and optimizing for space.

Research

Many researchers advocate rather than investigate. As a result,
1. some advocated concepts are worth far less than their advocates believe, and
2. there is a shorage of evaluative resarch to help determine what the value of such concepts really is.

Fallacies

About Management

(x) You can't manage what you can't measure
(x) You can manage quality into a software product

People

(x) Programming can and should be egoless
(x) Tools and techinques: one size fits all
(x) Software needs more methodologies

Estimation

(x) To estimate cost and schedule, first estimate lines of code.

About the Life Cycle

Testing

(x) Random test input is a good way to optimize testing.

Reviews

(x) "Given enough eyeballs, all bugs are shallow".

Maintenance

(x) The way to predict future maintenance costs and to make product replacement descisions is to look at past cost data.

About Education

(x) You teach people how to program by showing them how to write programs.

Shit Disturbers Reinvent the Wheel

2010-08-12T00:00:00Z

[Wikimedia, Fir0002/Flagstaffotos, GFDL v1.2]

Automated testing of systems is my pet peeve: I think every computer system should have a series of tests that get run by another computer that tests every problem ever thought of. My current customer has asked me to start developing a system just like this for their record-keeping and delivery system.

Currently, the customer has purchased a third-party tool for automating control of the software, unfortunately, the tool sucks. It is very difficult for non-programmers to understand its roundabout logic (it's using screen-scraper-triggered events), and has no mechanism for managing large numbers of scripts (each one is managed in and of itself). When building testing systems, the tests themselves tend to be easy to create, it is managing and tracking large numbers of tests becomes the problem.

Being a diligent consultant (alright, a diligent problem solver), I suggested it was possible to build a custom tool that wrapped the objects, and was better able to be understood by non-programmers, allowing us to more easily manage a large number of tests.

Steam Locomotive wheel [Wikimedia, CC-SA]

Reinventing the Wheel

That's when I heard it:

There's no point reinventing the wheel.

I take exception to this, I am encouraged to solve problems in the office and invention is the key to doing this. I recognize that all problems have been solved; we already have wheels. The only problem that ever exists is the need to refine the general solution to the particular instance of the problem; we need wheels suited to the current task.

If we had never reinvented the wheel we would still be driving around on Wagon Wheels. And …

I myself like having soft rubber tires on my car.

In the end, we reinvent the wheel regularly, not every wheel is perfect for every vehicle. Similarly, when solving problems at the workplace designing systems, it is sometimes necessary to build a custom component that suits the needs of the problem.

While not a total reinvention, they are a design better suited to the problem at hand. To work around the foibles of the existing technology, just because the technology already exists, is the kind of short-sightedness that leads to planes falling out of the sky.

Automotive Tire [Wikimedia, Public Domain]

Shit Disturber

Naturally, the moment I suggest all of this, I am accused of being a Shit Disturber. But …

When someone accuses me of being a Shit Disturber, I know I'm on the right track.

Shit Disturber. Let's break that term down; “shit” and “disturber”; or a “disturber of shit”. For this to be true, there must be shit to be disturbed.

That I am being accused of being a Shit Disturber forces my audience to acknowledge that there is (in fact) shit present.

If there is shit present that has been ignored and avoided; it may be more important to ask questions like, “When does somebody intend to do something about this shit?” (This is usually the hardest part of convincing people to change: getting them to acknowledge that there is a problem which requires fixing.)

Being a Disturber of Shit is not a bad thing.

Just because you are disturbing the shit, does not mean you put it there.

If the shit is in the middle of the road, we can either ignore the shit or do something about it.

Naturally, this causes some discomfort: people have got used to their path around the shit; while it is being moved, the shit tends to stink; people have a hard enough time cleaning their own shit (let alone someone else's); and the person that put the shit there probably feels like shit for not cleaning it up in the first place.

The Disturber is just the person willing to do something about the problem. The fact of the matter is, we can ignore problems for a long time, or put up with the temporary discomfort of fixing them.

Michelin's new tire for the Future [Wikimedia, CC-SA]

Conclusion

Shit stinks and The Wheel turns; these are two truths of the world. Ignoring them does not make them go away.

In life, we need to identify problems (shit), find solutions (reinvent the wheel), and make the changes to enact those solutions (disturb the shit).

In the past, I have been both punished and praised for taking drastic action to solve drastic problems (often regarding the same problem and by the same person). While we may find change uncomfortable, we should never turn away from those solutions.

So a “tip of the hat” to all those Shit Disturbers out there; may you always keep finding ways to reinvent the wheel.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

The John Deer Low-Down, wheel-driven, Manure Spreader [Wikimedia, Public Domain]

PlaidSheep

The Evils of Sequential IDs

False sense of utility

Risks

Security

Imbalanced Indexes

Valid Uses

Audits

Replication Checkpoints

Solutions

Big Integers

Random values, Hashes, and UUIDs

Primary Keys are never to be sequential

Non-primary, Unique Identifiers

Conclusion

How To Quantify Privacy in Datasets

Definitions

An Example

Applying the Measure to whole datasets

Using the Thresholds in Automated Checks

Conclusion

Going a Little Further

Further Reading

Maintaining Privacy in Data Shares

The Road to Hell is Paved with Good Intentions

The Problem

A Real Risk

What Can We Do?

Increase the Entropy

Measure the Entropy

Create Versatile Environments

Further Reading

Dangers of the Habit of Ignoring Developer Error Messages

Common Examples of Programmer Error Habituation

Errors and Warnings

Known Software Defects

TODO and Change Comments

Relearning the Lesson (Twice)

Preventing Habituation

Errors that can be ignored

Failing Tests

Comments Calling for Action

Conclusion

Further Reading

The Complexity of a Simple Chart

Aside: Defining the Problem

Overwhelming the Viewer

Starting Over

Order and Limit

States

Iconography

Timeliness

An Aside: The part I'm embarrassed to show

Next Steps

Animation

Summary

Further Reading

I still want my agile back

UPDATE: 2023–03–22

Education, Training, and Indoctrination

Distinguishing Between the Dimensions

Training

Education

Indoctrination

Not Mutually Exclusive

Corporate Training

Failed Corporate Training

Conclusion

The Angry Chatterbot

Social Groups are Complex

Getting on the Team's Nerves

Social Laws

Writing the Code

Conclusion

The Social Result

A Second (Unintended) Lesson

The Hidden Lesson for Manager

Hindsight is 20/20

Footnotes

Storing Data in QR Codes