Can We Learn Anything From Juicero?

At the risk of over-generalizing, I would think that in the security community most have long since fully internalized that

1. it is all about managing risk
2. 1 is hard

Now we have a new, and very public example of a risk management failure: Juicero, which ceased operations yesterday, costing at least 18 investors at least $118.5 million. 

There has been snark flying around about crafty, greedy venture capitalists deploying their collective foot-guns. I am more interested in looked at it as a risk management failure. I have at least one question, and possibly three.

1. Do investment organizations talk to each other?
2. If so, under what circumstances?
3. If so, in this particular case, was groupthink part of the problem?

The Setup

Juicero was based around the idea that a high-margin kitchen appliance would make money in and of itself (especially if businesses were charged as much as $1,200), and sales of produce packs for $5 to $8 would lead to further profits. A useful analogy might be computer printer vendors, which today are basically in the ink and toner business (plug "printer ink costs more than gold" into your favorite search engine) back in the days when the printers themselves also turned a nice profit.

Bloomberg had a good roundup of the problem Juicero was having in April: Silicon Valley’s $400 Juicer May Be Feeling the Squeeze, including 

One of the most lavishly funded gadget startups in Silicon Valley last year was Juicero Inc. It makes a juice machine. The product was an unlikely pick for top technology investors, but they were drawn to the idea of an internet-connected device that transforms single-serving packets of chopped fruits and vegetables into a refreshing and healthy beverage. 

 ...

Juicero has managed to find a niche at high-end hotels and restaurants. Workers from seven businesses that own Juicero machines said they like the product because the disposable packs can be discarded with minimal cleanup. All seven said they didn’t know Juicero packs could be squeezed by hand. In Bloomberg’s squeeze tests, hands did the job quicker, but the device was slightly more thorough. Reporters were able to wring 7.5 ounces of juice in a minute and a half. The machine yielded 8 ounces in about two minutes.

and much more. Including a mention that sales of produce packs would be limited to owners of the machine.

The threat is that there is nothing to prevent the purchase of a single machine, then doing a bit of hand pressing where multiple machines might be required to handle peak loads in a juice bar without hand pressing. Groups could club together to buy a single machine, then all the produce packs they jointly wanted for hand pressing. Resales of produce packs are possible, etc. People are clever.

You can think of it as protocol design failure. Authentication did not provide the security that the creators of the business model and investors thought it did. This was a rather too-obvious failure on the part of Juicero, and we have an indication that it was in fact known by them. Again, from Bloomberg (really, go read that article):

Juicero declined to comment. A person close to the company said Juicero is aware the packs can be squeezed by hand but that most people would prefer to use the machine because the process is more consistent and less messy. The device also reads a QR code printed on the back of each produce pack and checks the source against an online database to ensure the contents haven’t expired or been recalled, the person said. The expiration date is also printed on the pack.

I don't know that there is anything preventing the machine owner from also verifying that it hasn't hasn't been recalled as well, before sharing, reselling, or whatever. Assuming that a re-seller even cared, which seems an unwarranted assumption.

Who Lost Money?

Bloomberg mentions Kleiner Perkins Caufield & Byers, Alphabet, and Doug Chertok (presmbly through Vast Ventures).

According to TechCrunch the funding rounds were

• Apr, 2016 $28M / Series C
• Mar, 2016 $70M / Series B (lead investor Artis Ventures)
• Apr, 2014 $16.5M / Series A
• Oct, 2013 $4M / Seed

That is $118.5 million. In that Series B round were 17 investors

1. Abstract Ventures
2. Acre Venture Partners
3. AGO Partners
4. Artis Ventures (AV)
5. Bryant Stibel Investments
6. Campbell Soup Company
7. Campfire Capital
8. First Beverage Group
9. GV
10. Haas Portman
11. Interplay Ventures
12. Kevin W. Tung
13. Kleiner Perkins Caufield & Byers
14. Melo7 Tech Partners LLC
15. Thrive Capital
16. Two Sigma Ventures
17. Vast Ventures

Two of these were already mentioned by Bloomberg, but Alphabet (investment amount unknown) was not. So at least 18 investors were involved. I would expect Alphabet (as the parent of Google it is more than a VC), and Campbell Soup Company to have strong risk management teams. And any organization which exists purely to return profit on investment obviously should as well.

Yet we see a widespread failure.

A Comparison to Banking

Banks exist to manage risk for their customers: stuffing money under the mattress doesn't scale. And they cannot survive (without government bailouts, if they are large enough, and the circumstances dire enough) without functional internal risk management. They don't compete on the security front. Given the requirements of automated clearing, etc., they mitigate shared risks by sharing information about them.

Having no experience with one, I have no idea if venture capital organizations talk to each other about risk 

• at all, 
• only in the case of a group with a lead investor, 
• occasionally, or
• commonly.

It seems likely that whatever communication exists may be informal, dictated entirely by circumstance. But only by knowing those circumstances will it possible to answer my original question(s):

1. Do investment organizations talk to each other?
2. If so, under what circumstances?
3. If so, in this particular case, was groupthink part of the problem?

It seems quite possible to me that answers might become available.

I hope so, because otherwise the only people who will gain useful knowledge from this are the investors who lost money. People in the general security community gain no insight on when to advocate for new lines of communication, with whatever warning of the risk of groupthink might be indicated by the Juicero example, etc. Tech, as a whole, has a very short memory for what has presented security problems in the past. But security workers still have a professional obligation to at least try. 

Mean-of-Means Under Unequal Cardinality

These days, I'm hearing too much politics, of the weird and/or horrible sort. This is weird and/or horrible only in the sense of really bad statistics. Assuming that a political poll matches reality, anyway.

A former co-worker recently sent me an 'analysis' of a Gallup poll, purporting to show Trump's approval declining in stages. A lot of effort went into this. Each data point was tediously recorded via mousing over the online plot, then averages (arithmetic means) were calculated for each varying period, which were cherry-picked.

It's clear that while these data show declining popularity, and it sorta looks like that's happening in discrete stages, that's about all you can tell. Beyond the cherry-picking, only simple arithmetic means were calculated, and means-of-means were used across different sample sizes. Which is often a cardinality problem. I'm using the mathematical definition of cardinality here: a measure of the number of elements in a set.

In a bog-standard arithmetic mean I'm using the following notation:
where S = sample
and n = sample count
mean = S/n

so n = cardinality.

See the current Gallup Daily: Trump Job Approval plot to see rollover data, etc.

Immediately after the page title lies the critical sentence, "Each result is based on a three-day rolling average." You should also note that "Margin of error is ±3 percentage points." So a lot of the day-to-day variance is indistinguishable from noise. And not plotting even simple error bars trades information for drama. Bad dog, Gallup. But that is no worse than what is to be expected, and I digress.

Long story short: you have to use a weighted mean here.

The scary bit is that this former co-worker influences financial decisions related to capacity planninng for various security systems. When and where more storage, networking, or CPU would be needed. So this was just waiting to bite, in one or more security-critical system(s). That can be career-limiting, and is best avoided.

This is Python3. I use an unweighted mean-of-mean first, just to demonstrate failure under unequal cardinality.

Note that this is truly terrible Python. While it works, it bears little resemblance to how one would actually write Python. Python has been called "executable pseudocode."  I am trying to capitalize on that to make what it does (illustrate my point) decipherable to those who have never seen Python and have no desire to experiment with it, or may not be any sort of coder at all.

# Python is dynamically typed; it would have been easy to do this without 
# using an array. But this way, and specifying floating point (the 'f' in the 
# a_in = array.array statement) may be more explicit for those who don't use 
# Python.
import array # Efficient arrays of numeric values.
from statistics import mean # Arithmetic mean ("average") of data.

def clr_vars():
    """Clear variables, shown in order of appearance."""
    del a_in    # The input array.
    del ms1     # Mean of subset 1.
    del ms2     # Mean of subset 2.    
    del a_ms12  # Array of the means of the two subsets.
    del ma_in   # Mean of input array.
    del ms12    # Mean of both subsets

print('Equal Cardinality Subsets')
a_in = array.array('f', [1, 2, 3, 4])
print('Input:', a_in)
print('Subset1:', a_in[:2])
print('Subset2:', a_in[2:])
ms1 = mean(a_in[:2])
ms2 = mean(a_in[2:])
print('Mean of subset1:', ms1)
print('Mean of subset2:', ms2)
print('Sum of subset means succeeds:', ms1 + ms2)
a_ms12 = array.array('f', [ms1, ms2])
ma_in = mean(iter(a_in))
print('Mean of entire input array:', ma_in)
ms12 = mean(a_ms12)
print('Mean of subset means succeeds:', ms12)

Which returns

Equal Cardinality Subsets
Input: array('f', [1.0, 2.0, 3.0, 4.0])
Subset1: array('f', [1.0, 2.0])
Subset2: array('f', [3.0, 4.0])
Mean of subset1: 1.5
Mean of subset2: 3.5
Sum of subset means succeeds: 5.0
Mean of entire input array: 2.5
Mean of subset means succeeds: 2.5

clr_vars

print('\nUnequal Cardinality Subsets')

a_in = array.array('f', [1, 2, 3, 4, 5])

print('Input:', a_in)

print('Subset1:', a_in[:2])

print('Subset2:', a_in[2:])

ms1 = mean(a_in[:2])

ms2 = mean(a_in[2:])

print('Mean of subset1:', ms1)

print('Mean of subset2:', ms2)

print('Sum of subset means succeeds:', ms1 + ms2)

a_ms12 = array.array('f', [ms1, ms2])

ma_in = mean(iter(a_in))

print('Mean of entire input array:', ma_in)

print('Mean of subset means fails:', ms12)

Which returns

Unequal Cardinality Subsets
Input: array('f', [1.0, 2.0, 3.0, 4.0, 5.0])
Subset1: array('f', [1.0, 2.0])
Subset2: array('f', [3.0, 4.0, 5.0])
Mean of subset1: 1.5
Mean of subset2: 4.0
Sum of subset means succeeds: 5.5
Mean of entire input array: 3.0
Mean of subset means fails: 2.5

Weighted Means

As you can (hopefully, Python is clear enough) see, cardinality didn't matter with addition. But with mean-of-means, it did. There are some subtleties involved with sum- vs mean-of-means. You may be asking two different questions, depending on which you use, so either can be correct. But here we can just use weighted means, which allow for the different number of samples (cardinality).

Differences

Instead of iterating over the entire array, we group by the subsets.

Recall that in a bog-standard arithmetic mean I'm using the following notation: where S = sample and n = sample count mean = S/n

We calculate a weighted mean of two samples by expressing groups of sample counts and sample means:
Weighted mean = (((mean of s1)(n of s1)) + ((mean of s2)(n of s2))) / ((n of s1)(n of s2))

We introduce two new variables s1 = subset1, s2 = subset2, instead of just printing what they

would have contained.

We then use len() to get n of s1 and s2, storing them in two more new variables, ns1 and ns2.

Substituting Python variables, and using Python syntax (still with extra parens for clarity):

w_mean = ((ms1 * ns1) + (ms2 * ns2)) / (ns1 + ns2)

Here's the code.

s1 = a_in[:2] s2 = a_in[2:]
print('s1 is:', s1) print('s2 is:', s2) ns1 = len(s1) ns2 = len(s2)
print('ns1 is:', ns1) print('ns2 is:', ns2) ms1 = mean(s1) ms2 = mean(s2) print('ms2 is:', ms1) print('ms2 is:', ms2) w_mean = ((ms1 * ns1) + (ms2 * ns2)) / (ns1 + ns2) print('Mean of entire input array:', ma_in) print('Weighted mean succeeds:', w_mean)

Which returns

s1 is: array('f', [1.0, 2.0])
s2 is: array('f', [3.0, 4.0, 5.0])
ns1 is: 2
ns2 is: 3
ms2 is: 1.5
ms2 is: 4.0
Mean of entire input array: 3.0
Weighted mean succeeds: 3.0

There is another potential problem here. Not having the original data, we can't see outliers, and there is no methodology discussion beyond "Daily results are based on telephone interviews with approximately 1,500 national adults". So we don't know that the arithmetic mean was safe to use. On the other hand, rolling averages smooth data. It's probably fine, but we would have to do a lot of analysis to establish a confidence level.

We would start by experimenting with harmonic means (the mean of choice when

there are outliers, though we have to go outside the Python standard library. Harmonic means are also commonly required in IT capacity planning or performance analysis, or anywhere else that data are spikey. My former co-worker should certainly have been aware of this too, but somehow wasn't. Another way to get bitten... The thing is that in most ways, this a pretty knowledgeable security person. On the theory that anything that helps get more science and engineering into the security field, at any level, is A Good Thing, I was happy to help.

What's in the Python stdlib is actually pretty weak for this sort of thing, compared to what's available in the science stack. The only mean available in the statistics module is the arithmetic mean, and arrays are limited as well. The solution starts as follows.

import numpy as np # array goodies and vastly more, required by

from scipy importstats # supplies harmonic mean, and lots more

import matplotlib.pyplot as plt

import seaborn as sns

Note that you also have to have Pandas on the system to use seaborn. Which is fine -- 

you'll want it anyway. Or use something else; there are several plotting packages available. 


When the environment is set up, we would use stats.hmean() and lots of plots as our 

exploration tools. That would take more play time than I actually have. Especially since all 

this assumes that I want to fool with anything political, and that Gallup polls are worth 

anything to begin with. I would really start by looking up their results from the 2016 election, 

where we know there were polling failures. 

If you care to push your way into the science stack, the packages I mentioned above are at 
http://www.numpy.org/
https://www.scipy.org/
https://matplotlib.org/
https://seaborn.pydata.org/ 

http://pandas.pydata.org/

Tools for HR Departments

The following is the text that went into gitlab.com/secinfo/hr.

Over the years, it’s become apparent to me that HR departments could use some
help on the security front. The cannonical example is a breakdown in
communications with IT that results of former employees being left on systems.
More subtle are issues with the hiring process: I’ve seen jobs advertised
looking for admins for old, unsupported, public-facing Web applications. And
too much detail about software which, combined with work shift requirements to
supply timing information, can provide useful information to an attacker.

Is it possible to increase precision, without a downside? One possibility
involves standardizing, if only somewhat, titles, responsibilities, and
typical tasks. Here the U.S. federal government may actually be of some help.
Though they don’t make it easy.

If you visit
https://niccs.us-cert.gov/workforce-development/cyber-security-workforce-
framework
(part of DHS National Initiative For Cybersecurity Careers And Studies) you
find tons of information. Seven categories, with a varying number specialty
areas under each, Drilling down further, you find lists of Competencies, Key
Skill Areas, Tasks, and Related Job Titles. And links to education resources.

There’s a lot there: hundreds of descriptors. It occurred to me that it might
be useful to have this in a machine-readable format, but that isn’t available.
Until now, anyway. The vast majority of it, save the education resources,
which seem likely to change frequently, has been (tediously) added to a couple
of JSON files. Warts and all – there are a few instances where they have
truncated lines and whatnot.

The files will change, in that some fields may be added. Under specialty, if
you open, say, Tasks, you find a top line that says something like
"Professionals involved in this Specialty Area perform the following tasks:"
Other times it says "experts".  I’m not sure it would be a useful thing to
include.

The structure will likely change at some point: I’ve committed the folly of
creating a data structure with no idea how it will be used. My hope is that an
HR professional will see it, it will spark an idea, and then an application
will drive the data structure, as it should. Which would make the tedium
worthwhile.

It’s also a serious eye-opener to security generalists such as myself, which
is another reason I did. JSON does make it possible to see it as a whole,
which you can’t do by clicking around in the NICCS  Web site. We are well past
any hope of a single person, no matter how dedicated, being able to  encompass
more than a small fraction of the skills that the field needs.

Hopefully, now that this is out there, I can use some of my scant spare time
to get some work done that should live in various private projects under
gitlab.com/secinfo/hr.

Resource Depletion Attacks: Commonly Performed by Security Industry

I make heavy use of the Linux 'at' and 'batch' facilities, which provide simple but very effective methods of increasing productivity via automation. Essentially, I want machines working to their thermal, IO, etc., limits as much as possible, so that I don't have to work to my thermal, IO, etc., limits. Naturally, I regard unused cores or threads, etc., as Bad Things.

At lunch today, there were three of four jobs in the queue, which is fairly typical. But none finished when expected, which is a bit atypical. The problem turned out to be, as it often is, a random security news service running core-sucking Javascript in an effort to get me to subscribe to their email service.

My bad, in some respects. I know better than to leave browser tabs open, leaving the marketroids an opportunity, but sometimes it just isn't possible to sort through a long list of tabs when you are trying to go to lunch. Sometimes you get burned by running batch jobs on an active workstation instead of shuffling them off to another machine.

On the other, having to even think about this an indictment of the security industry. Of which I am part, and that is FUBAR.

The definition of a resource depletion attack, according to Mitre's Common Attack Pattern Enumeration and Classification (an information-sharing resource worth using) is as follows.

Attack patterns within this category focus on the depletion of a resource to the point that the target's functionality is affected. Virtually any resource necessary for the target's operation can be targeted in this attack. The result of a successful resource depletion attack is usually the degrading or denial of one or more services offered by the target. Resources required will depend on the nature of the resource being depleted, the amount Pof resources the target has access to, and other mitigating circumstances such as the target's ability to shift load or acquire additional resources to deal with the depletion. The more protected the resource and the greater the quantity of it that must be consumed, the more skill the adversary will need to successfully execute attacks in this category.

Note that I had an opportunity to shift the load. But I am using 'at' and 'batch' casually, from the command-line. A better system would also be a much more formal system. I hadn't planned to review and retest real scheduling systems until toward the end of the year. If I had time, but only because I personally regard them as important and interesting. Perhaps I should bring that forward a bit, but it's a balance of annoyance versus available time: I have no current or horizon gig where robust scheduling systems play a central role. So, no immediate business driver. I'd like to flesh that topic out, but it would require another post to cover even the bare essentials.

Please excuse the digression, and it's time to return to the topic of the post. In the limit, almost no marketing can be regarded as ethical in the security industry, given that

1. Your brain is a resource, and by far the most vital one
2. Security news sources are commonly a FUD tsunami
3. 2 is an attack on 1
4. Current PCI-DSS requirements (sections 5.1.2, 6.1) require news monitoring

It's often helpful to look at the worst case. PCI-DSS originated as an effort to avoid governmental regulation by self-regulation. In the earliest iteration, it was as basic as requiring a firewall -- some financial organizations did not, even when firewalls alone were a highly effective defense. It evolved into something as relatively sophisticated as saying, "Read the damned news, people." Because they routinely didn't. At all.

As usual, security people are trapped by the lowest common denominator. Here's how that might map to an all too common day-in-the-life of someone on a PCI team.

1. You must read the news
2. Security news sources are commonly a FUD tsunami
3. FUD tsunamis are a DoS attack on your brain
4. Your brain is your most vital resource
5. FAIL

The largest offenders are the generic security news sources, and I very specifically include security product vendors. These are by far the most likely to burn your physical and mental cores with a FUD tsunami. Vendors of the software you are actually running? Sometimes they offer nothing at all (as in the early days of nginx, when it was basically just some random Russian supplier of an httpd, but still enthusiastically embraced due to efficiency) and not nearly enough offer anything that can be consumed by software. So we have to plod through this stuff manually.

Treasure the suppliers of the software you actually run who also provide up to date vulnerability data, particularly if it can be consumed by software. They free your brain, and that's important.

I assign news sources to tiers, according to a mechanism that works for me. Some random source that announces hacks of a PHP app run by 3 people? Not going to make my list, but it might make yours, if you have been trying to get management to get rid of that circa-1999 POS. Yes, that actually happens.

Develop something at least semi-formal

Know what you are running. A systems and software inventory can be surprisingly difficult to do. In larger enterprises, expensive high-bandwidth Internet connections can fall through the cracks. As can 7-figure database systems.

Know where your most important data live. That can be amazingly hard to do: there's always that one key worker. The one in that cube waaaay over there. With a spreadsheet that turns out to be vital.

There's an ancient dictum that suggests that you must know what you are protecting, and from whom. In many cases, forget it. It's no longer possible to predict who your adversary is. Whether your data is financial, or the secrets of military hardware, or 'just' passwords, (so, really all financial) the evolution of adversarial scanning technologies, exploit kits, etc., enables sophisticated yet entirely opportunistic attacks.

So, read the news. But have some sort of evaluation criteria, and put it under version control, because threats evolve. It has to be customized to your environment. Tier 1 will involve vendor alerts, because that involves patch planning, and (always short) testing windows. You might want a sub-schedule to break it into work shifts. Not all software comes from the same time zone, and a reaction time of less than 24 hours has repeatedly made all the difference. 

Assigning to Tiers 2 and 3 might involve how you evaluate sources as to reliability, frequency, frequency in combination with consequence, etc. Get used to thinking of it as a matrix, because it very much is. I have additional considerations, which include stability of source URLs, because I track reliability and frequency over time. You may or may not need to do that -- my requirements are peculiar.

Common Denominator: Vendor Bogosity is a DoS Attack

Security is hard. There is never enough time, there are never enough resources. 

A principle confounding factor is that much of what passes for news (and research, which is perhaps a greater problem) is delivered via what meets an essential definition of an attack. Background resentment of this state of affairs has long been pervasive within the security community; I'm certainly not the first to carp about it.

What is needed is not carping, but howls of resentment. It took that level of blow-back to get Microsoft to begin to change their ways, back in the day. When it became impossible to deploy a system before it was compromised, and CEOs complained, Microsoft found it impossible to ignore the noise. Much the same thing happened with vendors of proprietary Unix variants, though it wasn't as blatant. That completely changed the vulnerability disclosure game, though the vendors howled in protest. Or first began to howl -- they still do it.

It is only when we begin to call another generation of vendors out that another vast waste of scarce resources will end. This lossage is more difficult to quantify, so it's more difficult to raise a collective voice. Perhaps a start might be made by recognizing that vendor bogosity is a DoS attack, and telling them so. In, let's say extremely vigorous terms. Because history shows that nothing like a subtle approach is going to have an effect. 

Sometimes the best tool for the job really is a hammer.

It's So Easy to Be Taken In

In other exciting news, Social Engineering attacks still work. Duh. But here's an illustrative example of it being done completely innocently. This is from another security worker-bee who was all on about why mobile and Bring Your Own Device (BYOD) was such a corporate threat.

Bogus Vatican Image

That's a complicated topic, as evaluating risk always is, and is wide of the point that I want to make: the most effective possible social engineering attack comes from the innocent and mistakenly trusted. A very human failing, greatly magnified by transitive trust (friend-of-a-friend) issues. Which, make no mistake about it, we are all prone to. I might be particularly susceptible; because I am such an open, trusting sort of person.

Boris Karloff, The Mummy, 1932

The thing about that Saint Peter's Square image is that it was already in my database as bogus. Unlike the above Karloff image, which I only include because it was a cool old movie. Frivolity, thy name is Greg.

So how did I spot this unwitting social engineering attack? Chance. Striking images stick in the mind, and I happened to remember a source that really was in my DB: a Washington Post piece titled About those 2005 and 2013 photos of the crowds in St. Peter’s Square. There is no effective defense against social engineering attacks against a broad workforce, most of whom are just trying to live their lives.

If you do not assume that you will be hacked, you are Doing It Wrong. Worse, you are making that mistake in the face of a vast body of contrary evidence, and "Your security is important to us," PR is becoming widely ridiculed by both the security community, and more importantly, the public. Who are growing rather tired of the charade.

There are obvious things that can be done in beginning to address the problem. Most of them involve policy and standards, and the mechanisms for creating and enforcing them, or even (very doubtfully) convincing the workforce that their perfect performance is necessary. But these are, in the main, only available to larger organizations, where they work no better than they do at smaller scales.

As long as this sorry of affairs persists, the security industry will continue to fail, in an increasingly obvious manner.

Blackhole Crimeware Creator Gets 7 Years

That's a nice law enforcement win. 'Blackhole' is variously known as an exploit-kit or -pack or just straight-up crimeware, as it often came with regular updates,  or even support contracts. I have enough Blackhole references, dating back to 2012, in my database that it became boring to add them.

Brian Krebs reported this on 2016-04-14, at http://krebsonsecurity.com/2016/04/blackhole-exploit-kit-author-gets-8-years/. Note that there is a one year discrepancy between the URL and the the stated sentence.

I've already heard rumbles (possibly from other security worker-bees who hated plugging 'Blackhole' into a database for the nth time) that the sentence wasn't long enough. The line of thought was about scale: that Dmitry “Paunch” Fedotov, whom Krebs reports as having more than 1,000 customers, was earning $50,000 per month, and likely contributed to tens of millions of dollars stolen from small to mid-sized businesses over several years.

I can see the temptation there. Particularly the bit about 'tens of millions', and particularly the 'small to mid-sized businesses'. Organizations that fit that size description have been some of my favorite clients, are often most in need of the help, and I just generally feel better about having helped out an organization of that size, rather than some Fortune 500 behemoth. I would be amazed if I were to discover that that viewpoint is unusual, if could somehow survey the people down in the security trenches.

But was the penalty really light, at seven or eight years? Possibly not. First off, this was a Russian law enforcement win, and the sentence will be served in a penal colony. I don't know about you, but the idea of spending 7-8 years in a Russian penal colony does not take me to my Happy Place. I'm not going to address that further.

Suppose this was a United States thing? A US citizen, in US courts, with a potential for serving a sentence in a US prison?

Krebs refers to the likelihood of 'tens of millions of dollars stolen'. I completely agree. But let's compare this to the physical world. That necessarily involves bank heists, armored car robberies, etc., where people are likely to be injured or killed. Much drama, making it a natural for movies, such as Oceans n, or based on the Lufthansa heist, etc. Wikipedia has a list of large-value US robberies, several of which are in that tens of millions category. The most recent of $10+ million robberies date to 1997. The largest of which was the Dunbar Armored robbery, involving $27.9 million in 2016 dollars. The sentence? 24 years for mastermind Allen Pace, an insider. Under parole guidelines, he will have to serve 18 years, and five others will have to serve 8-17 years.

Bear in mind that this was a record robbery: it seems likely that it was politicized to at least some degree. The Loomis Fargo robbery ($25.5 million today) occurred the same year, yielded sentences from probation to 11 years. I haven't researched possible parole dates.

Differences in criminal justice systems make it difficult to judge whether Fedotov drew a sentence that was appropriate. But it seems to me to be broadly comparable, at minimum. That is a win for law enforcement. Penalties used to be no more than a slap on the wrist, as long as the crime was committed over the network. The extent of the damages didn't seem to matter.

There will be no immediate effect, no matter how much we might wish otherwise.

Sending signals has been less than effective in even the geopolitical realm, where huge numbers of government bureaucrats (State Department, etc.) are employed to keep it all sorted out, and react in something like real-time. Criminals will entirely miss this one, even if it should prove to be the start of a trend toward commensurate sentencing. It seems likely to be a generational thing.

I'm fine with that.

A couple of years ago I posted Law Always Lags, As It Should, "The universal claim seems to be that the law is behind the times. My take is that is better to have law that lags than law that leads. While lagging legal thought will certainly lead to injustice, it is less likely to lead to wholesale injustice. It is the lesser of two evils in an imperfect world."

DitL: writing about files, of all things

Have a Day in the Life post, written on a Sunday night, after a lovely Spring afternoon spent with a text editor. Gack. That is just wrong.

Writing. 121 lines, 965 words, 5836 bytes, and all about writing files of all things. It really did take all afternoon, for not very much usable output. Some days just go like that. I mostly discovered what I should have been writing, which is a piece in three (four?) parts.

1. How badly file creation is currently being done
2. That interstitial bit between writing and reading, which leads to exploitable race conditions
3. Reading is not so much a problem as parsing, which has been a gold mine of exploits over the years
4. Possibly a lead-in bit, which I am attempting to dodge by posting this

An additional problem is how to present the material, as an introduction to the subject, without it being an off-putting wall of text. For instance, introducing hexdump to beginners, as well as a few programs in core-utils, all in text, turns out to be non-trivial. This stuff is a lot easier when you can just get in front of a whiteboard in scribble-yack-enjoy mode. 

NIST Defines Microservices, Application Containers and System Virtual Machines

Commentary::Architecture
Audience::All

I'm back on about microservices again, as I was in my
last post, Microservices and Linters. Because yesterday NIST released a draft of Definition of Microservices, Application Containers and System Virtual Machines, which you can see at http://csrc.nist.gov/publications/PubsDrafts.html#800-180.

I have problems with it. The public comments period runs through 2016-03-18, and a comment template is available at the link above. If you don't read many NIST drafts, note that line numbers are part of the document. While that is useful if you catch a typo, or take exception to one statement, it is less useful if you take exception to the entire thing, to the extent of labeling it FUBAR.

That is, of course, just one opinion. I could make a claim about not wanting to bias any opinions, but then I've just done that, to at least some extent. This is mostly about not even knowing where to start, regarding this specific document.

I still regard microservices as something that is not nearly as widely deployed as one might think, given the trade press hype coverage. But it might be nice if, for once, security considerations might be taken into account early in the game. Yes, that is a forlorn hope, but even long odds will come home, given enough time.

Bur, srsly.  "Microservices are built around capabilities as opposed to services..."? Wut? I am not at all convinced that we have a robust system, Linux or otherwise, based on the capabilities security model. Because

• the capabilities security model is still a current research topic
• implementations will contain flaws or a long time to come
• all microservices architectures I have seen have been composed around the services (function) model

In this case, NIST seems to muddy the water through poor definitions. Particularly as it seems unlikely that wide awareness of capabilities-based security models, even as a research topic, exists at all within the wider software developer community. I've certainly seen little evidence for it, for whatever that might be worth.

Perhaps their usage of 'capabilities' was all about a possibly rapid evolution of APIs in containerized software, and not the capabilities security model. But rapid replacement of containerized software is about far more than API changes. Rather importantly, they are also about bug fixes. A percentage of which have always been exploitable, and as these are inherently remotely-accessible, would tend to have severity numbers reflecting that broad attack surface.

A lack of clarity over what, exactly, 'capabilities' might mean, in a NIST publication with 'Definition' in the title is, in and of itself, a problem.

Of course this is just anecdotal, from some random security worker. Not even remotely real evidence. So please ask around in your own organization, form your own conclusions, and comment on 800-180, if at all, as you see fit.

UUID: 92ce460e-58a6-4fd1-b3ed-d44f2d9c0183

Microservices and Linters

Commentary::Coding

Audience::Entry

Microservices are all the rage at the moment, for good reason. I am of course interested in the security aspects, and I am also on record as loving me some Python. Why, in detail, is probably something I should write up in a future post. For now, I'm just going to mention an intersection between the two.

In Chapter 9 (Security) of Building Microservices (Sam Newman, O'Reilly, 2015) we have exactly one mention of static analysis, under Baking Security In. Please don't misunderstand me – I found Building Microservices a worthwhile read. It is, however, a rather broad overview, and does have an unfortunate tendency to follow fashion. That last is probably not avoidable: the title has to sell, after all. And there is no possible way that all languages, strategies, etc., might be mentioned. So, no points off. Overall, author Sam Newman has done a nice job with this title.

That said, I still have a problem with static testing getting only one mention. Much has been written about developers needing to raise their game. It really is ridiculous that most injection attacks (SQL, LDAP, etc), for example, can exist in 2016. A lot of this is justified, but there are also some bits about QA, where that luxury still exists, that do not seem to get a lot of mention. Test-driven development can only take you so far, and an external QA group is a hugely useful defense against groupthink, deadline pressure, and other sources of problems in the delivery of reliable code.

Real developers could probably provide me with a lengthy list, and are invited to do so. You could further educate me with an ordered list – one can never have too much data.

Circling Back to Python

Back in June Andrew Collette (creator of much Python HDF5 code) wrote an excellent piece:

My Experience Using Static Analysis With Python. He was on Travis CI, and recommended both pyflakes (at minimum) and PyLint.

As it happens, we ended up using PyLint, which found about 100 legitimate issues with the code base, ranging from missing docstrings to calls to functions with the wrong number of arguments.

The takeaway is to use at least some sort of linter. Fine. That's doable, in either a full-on Continous Integration environment, or just using git hooks in a personal repo. Low marginal cost, better code. What's not to like?

Distributed Computing Is Inherently Complicated

Modern computing environments may consist of a single machine, comprised of multiple threads on multiple cores, many nodes in a cluster, or extremely parallel computing via GPU. In some cases (RDMA comes to mind), basic security mechanisms provided by Unix-like kernels are already being bypassed.

The need for reliable user-land code is never going to decrease. If even a minor improvement can be had by using something as widely known as a linter, yet that is not a universally accepted practice, then we are collectively Doing It Wrong.

UUID: d4b72b13-5dd1-46d1-913a-9dc470e0b6d7

If There Were One Feature I Wish Bugzilla Had

Commentary::Performance
Audience::Entry
UUID: ddf3eae9-a84d-4083-987e-a84cf2ec8aec

It would involve track records. Specifically, there is no way to know how many bugs in Open Source software you have reported have never been assigned, and were simply closed due to End of Life (EOL), or what the track record of an assignee (who never addressed it, but simply closed EOL), closed but reopened due to a broken or incomplete fix, no communication whatsover from the assignee, etc.).

I have biases which you should probably be aware of.

1. I weight communications issues more heavily than some do, because I am usually thinking in terms of security, and taking a sane approach to disclosure. This is often about how well communications works, and timeliness.
2. Many bugs can fall into a security context, for reasons that may not be readily apparent. Failure of some Linux update mechanisms in terms of alerting on rpm.new/.save files is a pet peeve, as is maintaining performance.

This is about that second point -- maintaining performance. I have another bias there -- if I didn't do security, I would do perf. It's the next most interesting problem. The high-frequency trading community is running without firewalls. Some turn SELinux off not due to usability concerns, but because characterizing perf impacts is difficult. Some run Remote DMA over converged Ethernet (RoCE) for the performance pop without considering the security implications of bypassing defenses built into the kernel.

There are a lot of interesting behaviors out there, and not all are well-considered. The last thing we should be doing is making it difficult to easily explore the performance impacts of things we might recommend. That's a recipe for losing all credibility, and becoming part of the problem.

I've recently destroyed the lab (again), because 2016 is coming up fast, and I wanted a first cut on what the hardware budget should look like. One of the things I wanted to explore is the overhead of rapidly spawning a lot of processes. Likely the last time I would do such a thing, for a couple of reasons.

1. Amazon AWS probably makes more financial sense than tearing up the lab, though reproducible research is a concern. But Amazon is only one vendor.
2. 'a lot of processes' is subjective, and relevance is entirely dependent on your *aaS (Infrastucture, Platform, etc., as a Service), your existing or contemplated local/cloud/hybrid security posture, etc.

Given the large number of possible deployment scenarios in modern infrastructure, it would be really nice if even the basics of performance tools were reliable. Nicer still if one could have some confidence that 'bleeding edge' Linux distributions could give us a preview of coming attractions, as used to be the case. Sadly, this is much less the case than it used to be.

In the case of Fedora and the rest of the Red Hat family, it goes beyond default file systems. I could go on about that, but this is about performance testing. So, well past time that I circled back to https://bugzilla.redhat.com/process_bug.cgi (login required), and spawning processes. That submission, for those who do not have a Red Hat Bugzilla account, reads as follows.

Description of problem: /usr/bin/free -s fails for floating-point and integers.

Version-Release number of selected component (if applicable):
3.3.10-5

How reproducible: Always.

Steps to Reproduce:
1. /usr/bin/free -s 1
free: seconds argument `1' failed

2. /usr/bin/free -c 1
works

3. /usr/bin/free -s 1 -c 1
free: seconds argument `1' failed

4. /usr/bin/free -c 1 -s 1
works

Actual results: As above. Verified that c > 1 work, when -c works at all.

Expected results: Functional continuous results from /usr/bin/free, and agreement between man page and program output. From man:
-c, --count count
Display the result count times. Requires the -s option.
-s, --seconds seconds
Continuously display the result delay seconds apart. You may actually specify any floating point number for delay, usleep(3) is used for microsecond resolution delay times.

Currently, it is -s that requires -c. Which is perverse if one wants to use free as a rough and ready means of tracking memory usage as several processes are started, and the time required to do that is unknown. Nor should order of specifying -c and -s matter, which would be a usability bug.

Additional info: Discovered this while attempting to use '-s 0.1', and discovering that even integers did not work.

Bad Weather: Must Rant on Scientists

Commentary::Science
Audience::Entry
UUID: 42bea053-2a40-4be6-93d0-dd3e17142907

This the current NOAA weather forecast. Note the absence of blue sky. People whom I respect have fled to San Diego. Why? I'm guessing that it's because they are much smarter than me.

Somewhere between the last week of October and the first week of November, the weather always goes to hell, from Puget Sound to at least the south end of the Willamette Valley. I don't need NOAA to tell me that, which is A Good Thing, as I have already mentioned that NOAA Can't Predict Weather, Can't Secure Their Systems.

I'm still annoyed with NOAA, not least because when they have clearly blown a forecast, as in whatever they predicted is obviously wrong, their updates just ignore it. High temp of 80° F, but it's already 87° by 1130 and climbing fast? Screw it. Keep predicting 80°. Perhaps people won't notice that they are baking.

That's a really bad example, given the time of year, but the frequency of this sort of thing has led me to log at least some of the more egregious examples into daily notes file. So, worth a minor rant-by-example.

I tend to follow the weather fairly closely. I'm out in it a fair amount, though I do tend to back off when it's really rotten. So I follow some weather blogs, and a bit of research. One of the more popular blogs is http://cliffmass.blogspot.com/. He's a University of Washington scientist, and is often spot-on. When he isn't spot-the-FUBAR-off.

Problems? Yeah, I have some. Appeal to Authority flaws, sometimes in the same post that he denigrates those authorities, inconsistency of message, and more than a little hype about certain topics. Again, sometimes in the same post in which he denigrates hype.

Perhaps my biggest problem is that he is a scientist. I am very far from being some sort of anti-science Luddite; I am a heavy consumer of science. But there's a corner case involved, in his field, which seems to be ignored. Mostly, it's about where the money goes. As taxpayers, how are we to judge whether we are receiving value? Why is NOAA not quantifying errors, so that we might judge when forecasts are most likely to be erroneous?

The other axis in that corner case is this.

http://www.probcast.com/ is about probabilities and ensemble forecasting, from UW. It's an experiment, with a useful 'about' page. But still no indication of the circumstances under which it wanders off into left, and it's run by scientists. Who are often the most security-clueless people imaginable. It's somewhat understandable, because they are all about generating new knowledge, and sharing it widely.

Still, having even half a security clue is useful. There is a reason that Linux directories under /home are private in most modern distros. Despite UNIX being historically rooted in research environments.

Now and then (very occasionally) I report a problem to a Web site owner. Not often, because as often as I find them, it would burn too much time. In this case, I got on the phone. The UW person I spoke with had no idea what I was even talking about. She knew of no such site, who to contact, etc. The issue was fixed in fairly short order (do not remember exact timing) but in terms of appropriate response, it was a miserable failure.

Scientists mostly don't do operations. The concept of domain squatting (excuse me,  'cybersquatting') completely escapes them. I'm fine with that, actually. Different fields of endeavor, and I'd rather scientists were paid to do science.

But I get annoyed as hell when some scientist writes about something that is way outside their area of expertise, expecting to be considered an authority.

A Tier 1 Information Source: Ross Anderson

Commentary::Sources
Audience::Intermediate
UUID: 34e6bddc-58a3-47a7-a1e2-7e83981bacc8

On 3/20/14 I published Congratulations to Leslie Lamport, winner of the 2013 Turing AwardCongratulations to Leslie Lamport, winner of the 2013 Turing Award, as announced to the public a bit later, in CACM volume 57, number 6 (June, 2014). That post is a bit dated now -- I don't host ACM logos now, the post contains a link to an Adobe Flash presentation, etc. But Lamport did so much work as the original developer of LaTeX, and that is very close to the heart of my documentation production pipeline.

Fine. Times change, sometimes for the better. Flash is finally becoming recognized as the security nightmare that it has always been. ACM still seems to have no concept of this (scientists often have absolutely no clue about security), as their webinars still require it. Another reason (there are several) that I no longer belong to the ACM.

In that issue, there is an article by Ross Anderson and Ross Murdoch. In CACM, it's behind the paywall, but it's also public, as EMV: Why Payment Systems Fail. An extract:

Now that US banks are deploying credit and debit cards with chips supporting the EMV protocol, our article explores what lessons the US should learn from the UK experience of having chip cards since 2006. We address questions like whether EMV would have prevented the Target data breach (it wouldn’t have), whether Chip and PIN is safer for customers than Chip and Signature (it isn’t), whether EMV cards can be cloned (in some cases, they can) and whether EMV will protect against online fraud (it won’t).

More generally, a summary of Cambridge Computer Laboratory research is available from Anderson, as well a more general overview from the Security Group. Once upon a time (actually twice upon a time) CLCAM used to emit "Three Paper Thursday". But it was dependent on the availability of grad students, current research priorities, etc. Back in July of 2014, I asked if he might bring it back, and it turned out to be impractical. That sucked, as I always had a block of calendar time that was devoted to reading it. Largely on the strength of reading the first edition of Security Engineering. There is a second edition out now, and it is even better. It's also available as chapter-by-chapter series of PDFs, for free. My Wiley first edition (2001) cost about what you would expect for a tech book, but that is available as well, so you can compare the two, and get a notion of how Anderson thinks the security landscape has changed. If you are any sort of security worker-bee, you have little excuse for not having read it.

Why post this now? Because UK banks were proven to not be capable of even recognizing the concepts of ethics or morality, as first widely widely published by Anderson. Does anyone think that the Oct 1 shift in liability to merchants, which will undoubtedly drive EMV adoption in the US, will be any different?

I could go on about this, and the ongoing legal battles between the retail and banking sectors. But the post is becoming very long, as is. I'm going to let it ride, at least for now. Suffice it to say that both sectors are worth trillions of dollars per year. When such enormous sums are involved, neither sector will have your best interests in mind, whether you are subject to PCI-DSS self-regulation, or are simply a consumer shopping on the Internet.

I don't do generic blogroll links, either by request or some weird notion. This is my second. The first was to Brian Krebs, largely because he is the best security blogger I know of, in the consumer space. I hate to go there (the consumer security space, not Krebs on Security), which is why my first (10/2/14) source was titled A Brief Foray Into the Horrible.

How I divide news sources into tiers (currently Vendor, then Tiers 1-3) lies somewhere in the area of proprietary, complex (as opposed to complicated), and just really hard to describe. I may post something about that in future (sorry, 'going forward' is the current corporate bullshit fashion) but no promises.

However, I do promise to talk about their take on mobile security. You can read up on it the new link to Light Blue Touchpaper, now appearing under "(Some) Blogs I Read".

Intellectual Property: A Useless Term

Commentary::Marketing
Audience::Entry
UUID: 3a061855-fa21-4efc-a0cc-494418698118

I mostly hate the term Intellectual Property (IP), because it is mostly useless. Copyright, patent, trademark, etc., law has little in common, anywhere in the world.  Personal and societal impacts of those laws are similarly disparate, as one would expect.

Now and then, something that seems to absolutely fly in the face of common sense (granting that sense does not seem to be common) is particularly grating. Such was the case with Definitive Guide (TM) to Cyber Threat Intelligence.

There are various regulatory (and attempts to self-regulate, in order to avoid actual regulation, such as PCI) regimes that require a lot of reading related to possible threats. Fine. Been doing it for years, because I regard it as necessary, and so I applaud that. But it is extremely time-consuming: the doc mentioned above is a 74-page PDF, and is only part of today's reading list. The madness that allows a common phrase to be trademarked is annoying as hell.

Still, one of the contexts that I'll be reading this in (and marking it up for future reference) is DevOps, where there might possibly be some insight to be had from Table 1-1. The first table, so I will have to read this entire pile of marketing nonsense, in case there is support for it later in the doc. There might be something relevant to more recent DevOps concepts (and other marketing nonsense that have been about it) as opposed to older security-related divisions (network, infrastructure, and security operations) within an organization, and their contributions at the 'Tactical, Operational and Strategic' levels. Never mind that tactical and operational divisions would seem very artificial.

So, 74 pages that is mostly marketing noise. Some of which is about banks, etc., being clients. I covered that above, writing about regulatory (and avoidance of same) requirements.

That DevOps reference? DevOps is also rife with marketing noise. But much of DevOps does promote ideas related to getting beyond some long-standing (and foolish) IT practices, such as throwing code over the wall. Which makes me far more tolerant of that group.

In the unlikely event that you have some twisted urge to read this doc too, it can be had from https://cryptome.org/2015/09/cti-guide.pdf. I'm not supplying a direct link to these people directly, for two reasons:

1. Zero desire to provide them any Google-juice.
2. I expect that some sort of registration would be required, hence you would be bombarded with email marketing for, roughly, forever.

If you never see another post on fubarnorthwest, it may be because reading

Chapter 7, "Selecting the Right Cyber Threat Intelligence Partner," enumerates criteria for evaluating cyber threat intelligence providers.

caused immediate brain-death.

Sometimes I Just Have to Rant

That is a failure on my part. I've been working on a set of three posts that would likely have been more helpful, that do not involve Intellectual Property, and would point readers toward things that are more useful, such as why I am about to recommend the Computer Laboratory, University of Cambridge.  Sorry about that. Coming soon.

Long Odds: 7,975 to 1 Against

Commentary::Statistics
Audience::Entry
UUID: df96273e-2da9-4d92-9b6d-cbdfdfb9b5c8

What went wrong by noon. So that you can say, "I am doing way better than that guy."

Waking up at 0430

That's 4:30AM to an unfortunate majority of my fellow citizens. Side note: we are one of a very few countries to use this system, and I hate it. It's harder to parse in software, etc. But leaving the side note firmly aside, waking up at 0430 sucks, unless you have some good reason for it. Which I did not. It just happened, so it sucked.

My month-old coffee maker decided to not make coffee

If I am up at 0430, a cup of coffee will usually sound like a truly fine idea. Grrr, and dig out the drip pot I use when I'm camping. It makes very good coffee, though not good enough to wade through warranty hassles. Because nothing is that good.

Discovered that it had not rained

You might think that no rain is a Good Thing, but then you might not be living in western Oregon, in a year that has set records for number of days above 90° F (I don't mind the Fahrenheit scale, because of better resolution). and an ongoing drought. There was rain in the forecast, but NOAA, as usual, blew it. I had been hoping to prep the ground for a new herb garden this evening, but that ground remains hard as stone.

The clothes dryer does not dry clothes

Forgot to toss wet laundry in last night, when I was working late (I work at least 90% from home), so I did it this morning. Motor runs, and it heats up. But the drum doesn't spin, unless you spin it by hand, which you should not be able to do. Great. Probably a belt has broken, or somehow come off of a pulley. Deal with it tomorrow, since I'm driving into town anyway, for another coffee maker. Because screw waiting for some random warranty process to complete: I'll donate the replacement to some worthy group.

Retreat to data analysis fails

The work projects were all well ahead of schedule, except for one new project that I didn't know the extent of. Fine. The sample is a series of small (all under 4 MB) flat files, and the usual approach for this sort of thing, under Linux, is to poke them with bash shell tools, before loading them into an iPython notebook, and doing the real analysis with the Pandas data analysis library. I went a bit too far in the bash stage, binning some things out with regular expressions in GNU grep. How many results are in a single-digit bin, versus a 10-19 bin?

It turned out that in the first file I examined the numbers were equal, which was, to put it mildly, unlikely. Of course, the now you have two problems RegEx issue came to mind, so I did a bit of staring at code, which revealed no bug. I checked the result three different ways, and found it to be accurate.

Then I did what I should have done to start with, given the way the day had gone far. I checked the other files, and found that this was the only file that generated equal numbers. Given the way the day had gone so far, I was biased toward looking for something else that had gone Badly Wrong, however unlikely, and I wasted time.

7,975 to 1 against

Those were the odds (calculated after the fact) against running into that particular scenario. What are the odds of the day, overall, having gone so incalculably wrong? I have no idea. Because incalculable. Duh. We lack crucial numbers on coffee maker (by manufacturer and model number) failure rates...

There's a well-known example of unlikely things happening on a daily basis. If you have a data source, look at how many cars are registered in your state/province/whatever. Then, next time you are doing some random drive, pick a license plate number, and calculate the odds of having seen it.

There are probably very long odds against it. If not, congratulations may be in order: you may have just had an opportunity to discover, in day-to-day life, how difficult randomness (the source of much cryptography) really is. Consider sources of error e.g. picking a plate close to home, at a time many of your neighbors leave for work, is going to skew your result.

A double-plus-good would arise from seeing such a failure of randomness, and gaining an insight into the power, for good or ill, of data mining.

Risk Transfer Failure: a Possible Example

Commentary::Risk
Audience::Intermediate
UUID: 2f7d104e-1066-4735-a1a7-c1c0f39882f4

In classic risk analysis, if such a thing can be said to exist, there are four categories of risk response.

1. Mitigate

2. Avoid

3. Transfer

4. Accept

where Transfer means transfer part or all of the risk by e.g. insurance, hedging, outsourcing, or partnering. This category is sometimes split into Transfer and Share, giving us five categories.

There are many things that can go wrong in assigning the probability/consequence numbers that drive which response category is chosen. But let us make the completely unwarranted assumption that those are all solved problems, and apply our imperfect solution to the risk of the loss of medical records. The choice to transfer risk is usually made when that risk is evaluated as low probability, but high consequence.

One assumes that this what Cottage Healthcare System did when they chose transfer the risk, via insurance, to Columbia Casualty Company in the form of a "NetProtect360" policy, as they are subject to HIPPA regulation, which amongst other things, requires a risk analysis.

Then, in 2013 32,500 records were stolen, and in a class action, they were sued by their customers for $4.125 million, which Columbia paid. Risk transferred, job done, mission accomplished, right? Well, no.

It turns out that when Columbia paid off on the policy, they reserved all rights, including the right to seek reimbursement. There was an ongoing Department of Justice investigation, which was entirely appropriate, as HIPPA is federal law, and it turns out that Cottage had done some things that were perhaps unwise, to put it mildly, such as placing those records on an Internet-facing anonymous FTP server. Given the current state of network scanning, which is best described as 'constant', this made the actual risk high probability, not low. It also became visible to Google, which made the breach a virtual certainty.

In applying for the policy, Columbia required a "Risk Control Self Assessment", which included such questions as "Do you have a way to detect unauthorized access or attempts to access sensitive information?" and "Do you control and track all changes to your network to ensure it remains secure?", to which Cottage ticked "Yes", and which then became part of the terms of the policy. Access via anonymous FTP is obviously not compatible with either of these.

HIPPA Bites Again

And now we are back to the DoJ investigation. The risk analysis required by HIPAA contains questions such as the following.

• §164.312(a)(1) Standard Does your practice identify the security settings for each of its information systems and electronic devices that control access?
• §164.312(a)(2)(i) Required Does your practice have policies and procedures for the assignment of a unique identifier for each authorized user?
• §164.312(a)(2)(i) Required Does your practice require that each user enter a unique user identifier prior to obtaining access to ePHI?

where ePHI means electronic Protected Health Information. Access via anonymous FTP is obviously a complete non-starter, and Cottage would seem to have violated Code of Federal Regulations Title 45. See 45 CFR 164.312 -Technical safeguards.

We Now Have a Partial Risk Transfer Failure

At the very least, legal fees are mounting. Columbia sought reimbursement in United States District Court, Central District of California. The case was dismissed without prejudice (meaning the issue can reappear) in July, because the policy also allows for resolution by mediation, and both parties have decided to pursue that approach first.

We may never know the degree to which risk transfer failed. Cottage is a not-for-profit, so some information may turn up in financials - a Form 990, or it may fail, and reappear in the courts. But mediation might drag on for quite a while, and I could miss the resolution.

I'm curious because at the moment public information on examples of risk transfer failure is rare. I expect that to change: insurance companies moved into this market some time ago, and they are growing tired of paying out potentially large (Target had $50 million in coverage, which may be insufficient) settlements, particularly where failures on the part of the insured seem provably fundamental and complete.

Takeaways

It seems possible, even likely, that was a rogue server installation. It happens: I've found an Internet-facing rogue myself, which contained sensitive information on a new product. The cause seems to generally be that there was some sort of perceived business emergency, it was supposed to be temporary, and there was a disconnect between operations and security. 

1- Compliance with the terms of an insurance policy must be included in security operations.

2- Whether you have transferred risk or not, pay attention to the fundamentals. Scan your network, and analyze traffic logs.

Also, while it isn't a takeaway from this specific example, be aware of the possible existence of so-called 'Shadow IT' within your organization. The server that bites may exist not in your DMZ, but in the cloud, funded by credit card. Administrative controls fail at least as often as technical controls.