If There Were One Feature I Wish Bugzilla Had

Commentary::Performance
Audience::Entry
UUID: ddf3eae9-a84d-4083-987e-a84cf2ec8aec

It would involve track records. Specifically, there is no way to know how many bugs in Open Source software you have reported have never been assigned, and were simply closed due to End of Life (EOL), or what the track record of an assignee (who never addressed it, but simply closed EOL), closed but reopened due to a broken or incomplete fix, no communication whatsover from the assignee, etc.).

I have biases which you should probably be aware of.

1. I weight communications issues more heavily than some do, because I am usually thinking in terms of security, and taking a sane approach to disclosure. This is often about how well communications works, and timeliness.
2. Many bugs can fall into a security context, for reasons that may not be readily apparent. Failure of some Linux update mechanisms in terms of alerting on rpm.new/.save files is a pet peeve, as is maintaining performance.

This is about that second point -- maintaining performance. I have another bias there -- if I didn't do security, I would do perf. It's the next most interesting problem. The high-frequency trading community is running without firewalls. Some turn SELinux off not due to usability concerns, but because characterizing perf impacts is difficult. Some run Remote DMA over converged Ethernet (RoCE) for the performance pop without considering the security implications of bypassing defenses built into the kernel.

There are a lot of interesting behaviors out there, and not all are well-considered. The last thing we should be doing is making it difficult to easily explore the performance impacts of things we might recommend. That's a recipe for losing all credibility, and becoming part of the problem.

I've recently destroyed the lab (again), because 2016 is coming up fast, and I wanted a first cut on what the hardware budget should look like. One of the things I wanted to explore is the overhead of rapidly spawning a lot of processes. Likely the last time I would do such a thing, for a couple of reasons.

1. Amazon AWS probably makes more financial sense than tearing up the lab, though reproducible research is a concern. But Amazon is only one vendor.
2. 'a lot of processes' is subjective, and relevance is entirely dependent on your *aaS (Infrastucture, Platform, etc., as a Service), your existing or contemplated local/cloud/hybrid security posture, etc.

Given the large number of possible deployment scenarios in modern infrastructure, it would be really nice if even the basics of performance tools were reliable. Nicer still if one could have some confidence that 'bleeding edge' Linux distributions could give us a preview of coming attractions, as used to be the case. Sadly, this is much less the case than it used to be.

In the case of Fedora and the rest of the Red Hat family, it goes beyond default file systems. I could go on about that, but this is about performance testing. So, well past time that I circled back to https://bugzilla.redhat.com/process_bug.cgi (login required), and spawning processes. That submission, for those who do not have a Red Hat Bugzilla account, reads as follows.

Description of problem: /usr/bin/free -s fails for floating-point and integers.

Version-Release number of selected component (if applicable):
3.3.10-5

How reproducible: Always.

Steps to Reproduce:
1. /usr/bin/free -s 1
free: seconds argument `1' failed

2. /usr/bin/free -c 1
works

3. /usr/bin/free -s 1 -c 1
free: seconds argument `1' failed

4. /usr/bin/free -c 1 -s 1
works

Actual results: As above. Verified that c > 1 work, when -c works at all.

Expected results: Functional continuous results from /usr/bin/free, and agreement between man page and program output. From man:
-c, --count count
Display the result count times. Requires the -s option.
-s, --seconds seconds
Continuously display the result delay seconds apart. You may actually specify any floating point number for delay, usleep(3) is used for microsecond resolution delay times.

Currently, it is -s that requires -c. Which is perverse if one wants to use free as a rough and ready means of tracking memory usage as several processes are started, and the time required to do that is unknown. Nor should order of specifying -c and -s matter, which would be a usability bug.

Additional info: Discovered this while attempting to use '-s 0.1', and discovering that even integers did not work.

Bad Weather: Must Rant on Scientists

Commentary::Science
Audience::Entry
UUID: 42bea053-2a40-4be6-93d0-dd3e17142907

This the current NOAA weather forecast. Note the absence of blue sky. People whom I respect have fled to San Diego. Why? I'm guessing that it's because they are much smarter than me.

Somewhere between the last week of October and the first week of November, the weather always goes to hell, from Puget Sound to at least the south end of the Willamette Valley. I don't need NOAA to tell me that, which is A Good Thing, as I have already mentioned that NOAA Can't Predict Weather, Can't Secure Their Systems.

I'm still annoyed with NOAA, not least because when they have clearly blown a forecast, as in whatever they predicted is obviously wrong, their updates just ignore it. High temp of 80° F, but it's already 87° by 1130 and climbing fast? Screw it. Keep predicting 80°. Perhaps people won't notice that they are baking.

That's a really bad example, given the time of year, but the frequency of this sort of thing has led me to log at least some of the more egregious examples into daily notes file. So, worth a minor rant-by-example.

I tend to follow the weather fairly closely. I'm out in it a fair amount, though I do tend to back off when it's really rotten. So I follow some weather blogs, and a bit of research. One of the more popular blogs is http://cliffmass.blogspot.com/. He's a University of Washington scientist, and is often spot-on. When he isn't spot-the-FUBAR-off.

Problems? Yeah, I have some. Appeal to Authority flaws, sometimes in the same post that he denigrates those authorities, inconsistency of message, and more than a little hype about certain topics. Again, sometimes in the same post in which he denigrates hype.

Perhaps my biggest problem is that he is a scientist. I am very far from being some sort of anti-science Luddite; I am a heavy consumer of science. But there's a corner case involved, in his field, which seems to be ignored. Mostly, it's about where the money goes. As taxpayers, how are we to judge whether we are receiving value? Why is NOAA not quantifying errors, so that we might judge when forecasts are most likely to be erroneous?

The other axis in that corner case is this.

http://www.probcast.com/ is about probabilities and ensemble forecasting, from UW. It's an experiment, with a useful 'about' page. But still no indication of the circumstances under which it wanders off into left, and it's run by scientists. Who are often the most security-clueless people imaginable. It's somewhat understandable, because they are all about generating new knowledge, and sharing it widely.

Still, having even half a security clue is useful. There is a reason that Linux directories under /home are private in most modern distros. Despite UNIX being historically rooted in research environments.

Now and then (very occasionally) I report a problem to a Web site owner. Not often, because as often as I find them, it would burn too much time. In this case, I got on the phone. The UW person I spoke with had no idea what I was even talking about. She knew of no such site, who to contact, etc. The issue was fixed in fairly short order (do not remember exact timing) but in terms of appropriate response, it was a miserable failure.

Scientists mostly don't do operations. The concept of domain squatting (excuse me,  'cybersquatting') completely escapes them. I'm fine with that, actually. Different fields of endeavor, and I'd rather scientists were paid to do science.

But I get annoyed as hell when some scientist writes about something that is way outside their area of expertise, expecting to be considered an authority.

A Tier 1 Information Source: Ross Anderson

Commentary::Sources
Audience::Intermediate
UUID: 34e6bddc-58a3-47a7-a1e2-7e83981bacc8

On 3/20/14 I published Congratulations to Leslie Lamport, winner of the 2013 Turing AwardCongratulations to Leslie Lamport, winner of the 2013 Turing Award, as announced to the public a bit later, in CACM volume 57, number 6 (June, 2014). That post is a bit dated now -- I don't host ACM logos now, the post contains a link to an Adobe Flash presentation, etc. But Lamport did so much work as the original developer of LaTeX, and that is very close to the heart of my documentation production pipeline.

Fine. Times change, sometimes for the better. Flash is finally becoming recognized as the security nightmare that it has always been. ACM still seems to have no concept of this (scientists often have absolutely no clue about security), as their webinars still require it. Another reason (there are several) that I no longer belong to the ACM.

In that issue, there is an article by Ross Anderson and Ross Murdoch. In CACM, it's behind the paywall, but it's also public, as EMV: Why Payment Systems Fail. An extract:

Now that US banks are deploying credit and debit cards with chips supporting the EMV protocol, our article explores what lessons the US should learn from the UK experience of having chip cards since 2006. We address questions like whether EMV would have prevented the Target data breach (it wouldn’t have), whether Chip and PIN is safer for customers than Chip and Signature (it isn’t), whether EMV cards can be cloned (in some cases, they can) and whether EMV will protect against online fraud (it won’t).

More generally, a summary of Cambridge Computer Laboratory research is available from Anderson, as well a more general overview from the Security Group. Once upon a time (actually twice upon a time) CLCAM used to emit "Three Paper Thursday". But it was dependent on the availability of grad students, current research priorities, etc. Back in July of 2014, I asked if he might bring it back, and it turned out to be impractical. That sucked, as I always had a block of calendar time that was devoted to reading it. Largely on the strength of reading the first edition of Security Engineering. There is a second edition out now, and it is even better. It's also available as chapter-by-chapter series of PDFs, for free. My Wiley first edition (2001) cost about what you would expect for a tech book, but that is available as well, so you can compare the two, and get a notion of how Anderson thinks the security landscape has changed. If you are any sort of security worker-bee, you have little excuse for not having read it.

Why post this now? Because UK banks were proven to not be capable of even recognizing the concepts of ethics or morality, as first widely widely published by Anderson. Does anyone think that the Oct 1 shift in liability to merchants, which will undoubtedly drive EMV adoption in the US, will be any different?

I could go on about this, and the ongoing legal battles between the retail and banking sectors. But the post is becoming very long, as is. I'm going to let it ride, at least for now. Suffice it to say that both sectors are worth trillions of dollars per year. When such enormous sums are involved, neither sector will have your best interests in mind, whether you are subject to PCI-DSS self-regulation, or are simply a consumer shopping on the Internet.

I don't do generic blogroll links, either by request or some weird notion. This is my second. The first was to Brian Krebs, largely because he is the best security blogger I know of, in the consumer space. I hate to go there (the consumer security space, not Krebs on Security), which is why my first (10/2/14) source was titled A Brief Foray Into the Horrible.

How I divide news sources into tiers (currently Vendor, then Tiers 1-3) lies somewhere in the area of proprietary, complex (as opposed to complicated), and just really hard to describe. I may post something about that in future (sorry, 'going forward' is the current corporate bullshit fashion) but no promises.

However, I do promise to talk about their take on mobile security. You can read up on it the new link to Light Blue Touchpaper, now appearing under "(Some) Blogs I Read".

Intellectual Property: A Useless Term

Commentary::Marketing
Audience::Entry
UUID: 3a061855-fa21-4efc-a0cc-494418698118

I mostly hate the term Intellectual Property (IP), because it is mostly useless. Copyright, patent, trademark, etc., law has little in common, anywhere in the world.  Personal and societal impacts of those laws are similarly disparate, as one would expect.

Now and then, something that seems to absolutely fly in the face of common sense (granting that sense does not seem to be common) is particularly grating. Such was the case with Definitive Guide (TM) to Cyber Threat Intelligence.

There are various regulatory (and attempts to self-regulate, in order to avoid actual regulation, such as PCI) regimes that require a lot of reading related to possible threats. Fine. Been doing it for years, because I regard it as necessary, and so I applaud that. But it is extremely time-consuming: the doc mentioned above is a 74-page PDF, and is only part of today's reading list. The madness that allows a common phrase to be trademarked is annoying as hell.

Still, one of the contexts that I'll be reading this in (and marking it up for future reference) is DevOps, where there might possibly be some insight to be had from Table 1-1. The first table, so I will have to read this entire pile of marketing nonsense, in case there is support for it later in the doc. There might be something relevant to more recent DevOps concepts (and other marketing nonsense that have been about it) as opposed to older security-related divisions (network, infrastructure, and security operations) within an organization, and their contributions at the 'Tactical, Operational and Strategic' levels. Never mind that tactical and operational divisions would seem very artificial.

So, 74 pages that is mostly marketing noise. Some of which is about banks, etc., being clients. I covered that above, writing about regulatory (and avoidance of same) requirements.

That DevOps reference? DevOps is also rife with marketing noise. But much of DevOps does promote ideas related to getting beyond some long-standing (and foolish) IT practices, such as throwing code over the wall. Which makes me far more tolerant of that group.

In the unlikely event that you have some twisted urge to read this doc too, it can be had from https://cryptome.org/2015/09/cti-guide.pdf. I'm not supplying a direct link to these people directly, for two reasons:

1. Zero desire to provide them any Google-juice.
2. I expect that some sort of registration would be required, hence you would be bombarded with email marketing for, roughly, forever.

If you never see another post on fubarnorthwest, it may be because reading

Chapter 7, "Selecting the Right Cyber Threat Intelligence Partner," enumerates criteria for evaluating cyber threat intelligence providers.

caused immediate brain-death.

Sometimes I Just Have to Rant

That is a failure on my part. I've been working on a set of three posts that would likely have been more helpful, that do not involve Intellectual Property, and would point readers toward things that are more useful, such as why I am about to recommend the Computer Laboratory, University of Cambridge.  Sorry about that. Coming soon.

Long Odds: 7,975 to 1 Against

Commentary::Statistics
Audience::Entry
UUID: df96273e-2da9-4d92-9b6d-cbdfdfb9b5c8

What went wrong by noon. So that you can say, "I am doing way better than that guy."

Waking up at 0430

That's 4:30AM to an unfortunate majority of my fellow citizens. Side note: we are one of a very few countries to use this system, and I hate it. It's harder to parse in software, etc. But leaving the side note firmly aside, waking up at 0430 sucks, unless you have some good reason for it. Which I did not. It just happened, so it sucked.

My month-old coffee maker decided to not make coffee

If I am up at 0430, a cup of coffee will usually sound like a truly fine idea. Grrr, and dig out the drip pot I use when I'm camping. It makes very good coffee, though not good enough to wade through warranty hassles. Because nothing is that good.

Discovered that it had not rained

You might think that no rain is a Good Thing, but then you might not be living in western Oregon, in a year that has set records for number of days above 90° F (I don't mind the Fahrenheit scale, because of better resolution). and an ongoing drought. There was rain in the forecast, but NOAA, as usual, blew it. I had been hoping to prep the ground for a new herb garden this evening, but that ground remains hard as stone.

The clothes dryer does not dry clothes

Forgot to toss wet laundry in last night, when I was working late (I work at least 90% from home), so I did it this morning. Motor runs, and it heats up. But the drum doesn't spin, unless you spin it by hand, which you should not be able to do. Great. Probably a belt has broken, or somehow come off of a pulley. Deal with it tomorrow, since I'm driving into town anyway, for another coffee maker. Because screw waiting for some random warranty process to complete: I'll donate the replacement to some worthy group.

Retreat to data analysis fails

The work projects were all well ahead of schedule, except for one new project that I didn't know the extent of. Fine. The sample is a series of small (all under 4 MB) flat files, and the usual approach for this sort of thing, under Linux, is to poke them with bash shell tools, before loading them into an iPython notebook, and doing the real analysis with the Pandas data analysis library. I went a bit too far in the bash stage, binning some things out with regular expressions in GNU grep. How many results are in a single-digit bin, versus a 10-19 bin?

It turned out that in the first file I examined the numbers were equal, which was, to put it mildly, unlikely. Of course, the now you have two problems RegEx issue came to mind, so I did a bit of staring at code, which revealed no bug. I checked the result three different ways, and found it to be accurate.

Then I did what I should have done to start with, given the way the day had gone far. I checked the other files, and found that this was the only file that generated equal numbers. Given the way the day had gone so far, I was biased toward looking for something else that had gone Badly Wrong, however unlikely, and I wasted time.

7,975 to 1 against

Those were the odds (calculated after the fact) against running into that particular scenario. What are the odds of the day, overall, having gone so incalculably wrong? I have no idea. Because incalculable. Duh. We lack crucial numbers on coffee maker (by manufacturer and model number) failure rates...

There's a well-known example of unlikely things happening on a daily basis. If you have a data source, look at how many cars are registered in your state/province/whatever. Then, next time you are doing some random drive, pick a license plate number, and calculate the odds of having seen it.

There are probably very long odds against it. If not, congratulations may be in order: you may have just had an opportunity to discover, in day-to-day life, how difficult randomness (the source of much cryptography) really is. Consider sources of error e.g. picking a plate close to home, at a time many of your neighbors leave for work, is going to skew your result.

A double-plus-good would arise from seeing such a failure of randomness, and gaining an insight into the power, for good or ill, of data mining.

Risk Transfer Failure: a Possible Example

Commentary::Risk
Audience::Intermediate
UUID: 2f7d104e-1066-4735-a1a7-c1c0f39882f4

In classic risk analysis, if such a thing can be said to exist, there are four categories of risk response.

1. Mitigate

2. Avoid

3. Transfer

4. Accept

where Transfer means transfer part or all of the risk by e.g. insurance, hedging, outsourcing, or partnering. This category is sometimes split into Transfer and Share, giving us five categories.

There are many things that can go wrong in assigning the probability/consequence numbers that drive which response category is chosen. But let us make the completely unwarranted assumption that those are all solved problems, and apply our imperfect solution to the risk of the loss of medical records. The choice to transfer risk is usually made when that risk is evaluated as low probability, but high consequence.

One assumes that this what Cottage Healthcare System did when they chose transfer the risk, via insurance, to Columbia Casualty Company in the form of a "NetProtect360" policy, as they are subject to HIPPA regulation, which amongst other things, requires a risk analysis.

Then, in 2013 32,500 records were stolen, and in a class action, they were sued by their customers for $4.125 million, which Columbia paid. Risk transferred, job done, mission accomplished, right? Well, no.

It turns out that when Columbia paid off on the policy, they reserved all rights, including the right to seek reimbursement. There was an ongoing Department of Justice investigation, which was entirely appropriate, as HIPPA is federal law, and it turns out that Cottage had done some things that were perhaps unwise, to put it mildly, such as placing those records on an Internet-facing anonymous FTP server. Given the current state of network scanning, which is best described as 'constant', this made the actual risk high probability, not low. It also became visible to Google, which made the breach a virtual certainty.

In applying for the policy, Columbia required a "Risk Control Self Assessment", which included such questions as "Do you have a way to detect unauthorized access or attempts to access sensitive information?" and "Do you control and track all changes to your network to ensure it remains secure?", to which Cottage ticked "Yes", and which then became part of the terms of the policy. Access via anonymous FTP is obviously not compatible with either of these.

HIPPA Bites Again

And now we are back to the DoJ investigation. The risk analysis required by HIPAA contains questions such as the following.

• §164.312(a)(1) Standard Does your practice identify the security settings for each of its information systems and electronic devices that control access?
• §164.312(a)(2)(i) Required Does your practice have policies and procedures for the assignment of a unique identifier for each authorized user?
• §164.312(a)(2)(i) Required Does your practice require that each user enter a unique user identifier prior to obtaining access to ePHI?

where ePHI means electronic Protected Health Information. Access via anonymous FTP is obviously a complete non-starter, and Cottage would seem to have violated Code of Federal Regulations Title 45. See 45 CFR 164.312 -Technical safeguards.

We Now Have a Partial Risk Transfer Failure

At the very least, legal fees are mounting. Columbia sought reimbursement in United States District Court, Central District of California. The case was dismissed without prejudice (meaning the issue can reappear) in July, because the policy also allows for resolution by mediation, and both parties have decided to pursue that approach first.

We may never know the degree to which risk transfer failed. Cottage is a not-for-profit, so some information may turn up in financials - a Form 990, or it may fail, and reappear in the courts. But mediation might drag on for quite a while, and I could miss the resolution.

I'm curious because at the moment public information on examples of risk transfer failure is rare. I expect that to change: insurance companies moved into this market some time ago, and they are growing tired of paying out potentially large (Target had $50 million in coverage, which may be insufficient) settlements, particularly where failures on the part of the insured seem provably fundamental and complete.

Takeaways

It seems possible, even likely, that was a rogue server installation. It happens: I've found an Internet-facing rogue myself, which contained sensitive information on a new product. The cause seems to generally be that there was some sort of perceived business emergency, it was supposed to be temporary, and there was a disconnect between operations and security. 

1- Compliance with the terms of an insurance policy must be included in security operations.

2- Whether you have transferred risk or not, pay attention to the fundamentals. Scan your network, and analyze traffic logs.

Also, while it isn't a takeaway from this specific example, be aware of the possible existence of so-called 'Shadow IT' within your organization. The server that bites may exist not in your DMZ, but in the cloud, funded by credit card. Administrative controls fail at least as often as technical controls.

Don't Roll Your Own Crypto: Examples

Commentary::Crypto
Audience::Entry
UUID: bb828445-8bfc-4ccf-835b-5fdfa181ffc6

The usual reason given for asking software developers to not roll their own crypto is that anyone can build a cryptosystem that they cannot themselves break. That is perfectly true, but there don't seem to be many concrete examples that might convince the unwary. I'd like to provide a couple.

What most people see are references to Secretary of Defense Donald Rumsfeld's there are known knowns response to a question (much laughter from media pundits), and more scholarly approaches, such as calling it the Dunning–Kruger effect (a cognitive bias) after their widely-quoted Unskilled and Unaware paper.

Lots of people in the security trenches think of it simply as some_random_person who doesn't know that they have no freaking clue. That's a bit unfair, because we all do this, to some degree – that's the nature of a cognitive bias. For example, many people who can actually write would consider my blog posts a prime example. A more appropriate example would be the (several) people I have met who do networking, including quite competent jobs of network architecture, who have never read an RFC, and have only a vague idea of what the Internet Engineering Task Force (IETF) is. In many areas of IT, one can get by perfectly well without deep knowledge. There are fewer such areas in security, and none at all in cryptographic design and implementation.

The IETF are a great example for me to use, because I generally like to make more people aware of their existence, and, in this specific case because I recommend the IETF Crypto Forum Research Group Discussion Archive - Date Index as the first of those concrete examples. Read a few threads, and you will hopefully realize that this is something best left to specialists – those who have a made a career of it.

If that doesn't do it, have a look at Introduction to Modern Cryptography, by Katz and Lindell. You can look inside the second edition on Amazon. The first edition was a game-changer because it widely introduced provable security: the notion that rigorous mathematical proofs of the properties of a cryptosystem were the way forward. Provable security is one of the few means by which cryptography may get past the invent-break-fix stage. Here's a search onBirthday Attack, in the first edition at Google Books. Katz maintains a page on the book linking to reviews, courses where the book is used as a text, and errata, as well as a page on the page on the second edition.

On my shelves, I have only the first edition of Katz and Lindell, from 2007. It impressed me enormously, and now there is a second edition. So why aren't I talking about that edition? Because I am in no way qualified to roll my own crypto, and can't take years to learn the subject. My opinion would be irrelevant, even if I had one. Again, it's a very specialized thing, and unless you are both extremely clever, and willing to devote the better part of a career to it, you really shouldn't go there.

Also, please be aware that provable security is about algorithms. Implementations of those algorithms is a whole different thing, and we as yet have no complete solution to the implement-break-fix cycle. Crypto code is often complex, not least because it needs to defend against things like timing attacks. That famous line about "all bugs being shallow, given enough eyeballs," certainly does not apply here, where the essential requirement are those very few highly qualified eyeballs. Hopefully auditing well-commented code. Let me remind everyone that Heartbleed was an implementation bug, in one of the most widely used protocols (TLS, though also DTLS) crypto libraries (openssl), yet it remained undiscovered for two years.

So, for the third time, don't go there. Because there is already enough suffering in the world. Your time will likely be far better spent along the following lines.

• understanding what you are trying to secure, and from whom
• the strengths and weaknesses of various crypto suites and algorithms
• their applicability to the problem you are currently facing
• their suitability on your intended deployment platform

About this lack of recent posts

Sorry about that. Well, not really. It's been due to a combination of several things. Some professional stuff that I can't talk about, but mostly because Oregon in the summer. I am pretty outdoors-oriented in my off time, and needed to take advantage of the weather, which will be turning Pacific Northwest bad, which is famously bad, soon enough. 

I expect that now that the long Labor Day weekend is over, the pace will pick up a bit. Standing on the deck sipping coffee at 0530 this morning, with a half moon in the east, and Orion and the Pleiades almost at zenith was great, though I was listening for owls, and didn't hear any. And it wasn't so long ago that sunrise was at 0530, and none of this would have been seen. At the Winter Solstice, when there are less than 9 hours of any light in the sky, and it's likely to be pouring rain during those brief hours, I'll probably be posting a lot more often.