Exploring System Data: Use Anything but bash.

Recommendation::Language
Audience::Intermediate
UUID: 4e163e7c-ec63-430e-83e2-605e9df95526

In a gmail conversation related to changes to the Linux kernel, I asked whether anyone still used gnuplot, which was used in the example. Because one of the first things you do when exploring data is to look at the distribution. Duh.

Of course, I am sure that gnuplot is still in constant use. People don't scrap production systems simply because something is more fashionable. Or they shouldn't, anyway. The math is not favorable.

As a side note, I really need to take a decision on how I want to display math on this blog.

I started a project related to data analysis using some old-school techniques, all based around shells. Shells can be a win, for answering questions such as, "How is this new application changing my system." That can be important. I've seen Web application servers deployed before the location and content of log files was known, much less characterized at a level of, "What sensitive information might be written if the log level is DEBUG?"

Shells are fine for that sort of fast initial cut. The problem is people don't want to throw that code away. They keep writing one more grep statement, or whatever. My personal alarms tend to ring at arrays. If the system becomes complex enough that I need arrays, I am going to question the wisdom of doing it in a shell.

• They aren't POSIX, so you become wedded to one particular shell. Want to use dash instead of bash? Sorry, but you can't.
• You can't pass arrays to functions, if you need to do something more complex than loop over them. Even for that, you are probably going to use a reference. Modify them? Sorry, but you can't.
• You can't even take a slice of an array. Sorry, but you can't.
• What stop-gaps exist for dealing with arrays, or even faking them if they aren't available in your shell, tend to use 'eval'. Which adds a whole new layer of potential security issues. Sorry, but you really shouldn't.

Shell arrays don't do anything more complex than map strings to integers. Except in the case of bash associative arrays, which are a newer, shinier, and deeper can of worms. The point is that the most advanced data structures available in shells are not really suitable for building a software with any sort of complexity.

I pretty much won't start down that slippery path, any more, and I hope that you won't either. I tend to use Python. If you prefer Ruby, have fun. It's just too slow for me, but it's also widely used in the security community, including in the Metasploit framework.

There is value in knowing pretty much any language, especially in the security field, if for no other reason than to know how problems with them can be exploited. That is not an argument for falling into the same trap of eval'ing something, wedding yourself to sanitization problems, because you pushed the language too far.

2015-04-23 Addendum

The power of the shell is seductive. I still use it all the time. Moments ago, on a Linux machine:

# ls -lh /var/log/secure

-rw-------. 1 root root 8.3M Apr 23 13:24 /var/log/secure

# wc -l /var/log/secure

69397 /var/log/secure

# grep hddtemp /var/log/secure | wc -l

69319

#

But this is not a mechanism for monitoring log growth. I can immediately see log file size, that (non-SELinux) permissions are correct, and that this one is mostly about monitoring a drive temperature. 

The problems will surface when I try to reuse these commands: add -Z to /usr/bin/ls will show me the SELinux context, find lines that aren't about hddtemp, etc. But in scripts, to start with, you should not parse the output of /usr/bin/ls. stat(1) is your friend (and don't forget to supply, and appropriately quote, a format string).

The shell gives you a lot of power to spot-check things. Leave it at that, and save yourself some grief.

There Are No 'Safe' Languages

In posts like http://fubarnorthwest.blogspot.com/2013/10/choosing-python-over-r.html I did a bit of Python advocacy. I didn't cover half of it; R has some very strange, by modern standards, naming conventions, which makes learning R harder than it should be. But what I really wanted to mention is that a few people who know me and my focus on security, ran into the Googles. There are a few claims out there about Python being a 'safe' language, and they somehow assumed that I had blessed Python.

No. Whatever 'safety' posts they read were likely about buffer overflows, which Python will indeed protect you from, until the language implementation itself has a problem with buffer overflows. See http://bugs.python.org/issue20246. They are rare, but when they do occur, the problem reduces to keeping your Python system patched, with the attendant patch verification, propagation, etc., problems that you would expect, somewhat mitigated by Python having nice testing frameworks. But this is mostly just a bog-standard IT patch-mangement problem.

Nor is that the end of the story. The native Python data persistence mechanism (pickle) carries the same risk as any persistence mechanism: how is it protected between write and read? There are also the usual tribe of cut-and-paste folk who have no idea what they are doing, but have an immediate need to make something work. Mistakenly fumbling uuid1 instead of uuid4 into your code, because you did not read the docs, is going to be a problem if leaking a network address is a serious flaw.

There are no circumstances that would allow me to 'bless' a language, in a security context. The best that can be said is that some are safer than others.

Choosing Python Over R

I feel the need for speed. If you are messing about with decision-support in a security context, you probably do too. It turns out that for most of what I have needed to do in the last couple of weeks, Python has been taking closer me to my targets than R.

It will be a while (probably a long while) before Python tooling can match the comprehensiveness of R, which has > 4k packages available. http://r4stats.com/2013/03/19/r-2012-growth-exceeds-sas-all-time-total/

For pure statisticians, R is still the win, and I don't mean to trash the tool or the field in any way. If I hadn't found R, way back when, I would probably have thought MS Excel was an acceptable program for stats. Leading to FAIL.

But, Python tooling looks like being faster, in both execution and development speed for my needs of the moment. R may still be the winner in creating interactive doc. I still need to take a weekend and compare the two. But free weekends are in short supply right now, and it would have to be an awfully big win to make much of a difference. I am huge believer in 'go get knowledge, then teach it' but I am not primarily an educator, and tools such as the iPython HTML Notebook seem adequate.

Regarding iPython: don't use anything else. Seriously. The only time I ever enter 'python' instead of 'ipython' on the command line is if I need a quick basic calculator, or if all I want to do is import numpy, do a couple couple of quick array operations, and leave.

Don't laugh at the idea of using Python as a basic calculator (snarking on KDE Kcalc):
import math
math.sqrt(4)
2
Seriously, kcalc, WTF is your problem? People have been able to take square roots on hand-held calculators for 40 years, but your software, running on this comparative pile-driver of awesome, cannot.

With iPython,  you get log files and other huge advantages. Mess around with it for a couple of evenings, and you will never go back. I wished for something like this for ten years before we got it, so now I have been enjoying it. You will too.

Java Security Revisited--Part 1

Java Security Revisited--Part 1

I have already thrashed on Oracle Java security failings once, but it doesn't hurt to go on with it. There is a human cost to this, in stolen identities, banking details, etc.

At
https://blogs.oracle.com/security/entry/maintaining_the_security_worthiness_of
there is a post titled Maintaining the security-worthiness of Java is Oracle’s
priority.

This is as FUBAR as anything needs to be.

It is perhaps more appropriate to subsitute 'Establishing' for 'Maintaining' in
that title. Java has a long history of problems. You might want to go read that
5/17/13 post first. The one where I claimed that they had more or less fallen
on their sword, and admitted complete security FAIL.

Back now? Great. Let us start with a quote from that blog post. "Hi my name is
Nandini Ramani, I lead the software development team building the Java
platform. My responsibilities span across the entire Java platform and include
platform security."

This is The Man. The guy with the ultimate responsibility (save Larry Ellison,
who will probably fire him if results are not forthcoming, pretty damned
quickly).

"Over the past year, there have been several reports of security
vulnerabilities in Java, primarily affecting Java running in Web browsers."

Over the past year? I don't want to accuse Mr. Ramani of being dissengenuos on
a corporate blog; perhaps he has not been keeping up with even not-so-current
events. But let us look at open source intelligence.

I regard Brian Krebs as one of the foremost researchers of the business
mechanisms behind underground fora, exploit pack proliferation, and botnets,
and related matters. He is focused, talented, and has sources I will never
have. Read
http://krebsonsecurity.com/2010/10/java-a-gift-to-exploit-pack-makers/ in which
Mr. Krebs says, "I also found Java flaws to be the leading exploit vectors for
both the Crimepack and Eleonore exploit packs." This was in October, 2010, not
"Over the past year" and the rabbit-hole goes deeper than that.

At this point, I have no good reason to trust Mr. Ramani. I do, however, have
good reason to question his integrity, competence, or both.

Further down in the post, you will find that Java will be updated four times per
year, in sync with other Oracle Critical Patch Updates. This is obviously going
to increase the testing load for those who have to deploy this stuff. But my
focus is on security, and my take is that this should already be built into
your deployment workflow. If hasn't been, in the past, take it up-stream. Your
workload has certainly increased, and you need an increased budget. It's just
part of the ever-increasing cost of doing business with Oracle.

Mr. Ramani even provides helpful reminders. Summarizing:

February 2012 Critical Patch Update for Java SE provided 14 security fixes
June 2012 release 14
October 2012 release 30
(thus the total number of new security fixes provided through Critical Patch
Updates for Java in 2012 was 58)

February 2013 security releases provided 55 new security fixes
April 2013 Critical Patch Update for Java SE provided 42 new security fixes
(bringing the total number of security fixes released through the Critical
Patch Update for Java in the first half of 2013 to 97.)

The way Mr. Ramani probably does not want you to interpret these data is that
these numbers are a measure of how horribly broken Oracle Java security has
been, and how exposed you have been. Increased pressure on Oracle is indicated.
What else is out there, that you do not yet know about?

"The Java team has engaged with Oracle’s primary source code analysis provider
to enhance the ability of the tool to work in the Java environment. The team
has also developed sophisticated analysis tools to weed out certain types of
vulnerabilities (e.g., fuzzing tools)."

Sounds great, and while fuzzing is hugely useful, that usefulness has only one
domain. Gary McGraw, a recognized authority, pointed out in
Software Security: Building Security In, published by Addison-Wesley, that less
than 50% of vulnerabilities come from implementation flaws. Code analysis tools
do not find architecture flaws.

This is getting way too long, and I am not even remotely done. I am going to
have to continue this in a second part.

The current Java is update 7u21. There will never be a 7u22.

Write once, run everywhere. It's secure because it runs in a sandbox.

Eventually, even the largest IT marketing machines cannot overcome IT reality--social issues are a whole different thing, but (mostly) out of scope for this post.

"Write once, debug everywhere," is commonly heard. And we really don't understand how to reliably and easily build secure sandboxes, from chroot jails on up.

But I believe a page titled  Java SE - Change in Version Numbering Scheme to be a landmark. So far as I know, this is the first time that a major vendor of 'Enterprise' software has pretty much admitted complete defeat on the security front. Microsoft came close, several years ago (they have gotten much better), but this is an admission of  abject defeat.

Oracle has never had a good rep, on several fronts. Hardball business tactics and vendor lock-in don't really concern me here. That is an easy Web search away, if you aren't already affected by it. Being widely regarded as the least secure of the 'Enterprise' DB vendors does, but only insofar as it is symptomatic of the current problem. Which is that Oracle has not only consistently produced horrible products, but they have been very slow to fix their problems.

The way Java updates worked in the past was

1. Limited Update patches (feature adds and and non-security bugfixes) got even update numbers.
2. Critical Patch Updates (CPUs) got odd update numbers.

They are done with that. There have been so many security flaws that they have had to renumber releases, which causes huge problems for users in risk analysis, patch scheduling, etc. The majority of the problems involved Java running in Web browsers, not server-side Java.

That was the one ray of sunshine, but security teams had to take very careful looks at exactly what was being run, and where. Multiple versions, installed on the same PC, were a problem. Common exploit code can call the vulnerable version if it is available. And some organizations did not prevent Java installation, of any version, regardless of whether it was really necessary on that employee machine. To make matters worse, a common installation scenario involved developer and QA machines. These users could often mount a specious argument that they needed it to do their jobs, and would be given blanket permissions. 

It was, and is, completely fubar. It is a really great way to make an adversary's job easier.

So, Oracle has at last bagged it. How have they admitted complete defeat (though not in so many words, of course)? Like 1970 BASIC programmers, leaving space between line numbers. 

Limited Update patch numbers are assigned in increments of 20. That leaves fill-in-the-blanks room for CPUs that will be issued as odd numbers that are increments of 5. Leaving still more fill-in-the-blanks room toad more fixes, and/or fix their previous fixes. And they reserve the right to use even numbers in those CPU intervals. I would guess that they are expecting more than a few.

Buy Oracle products. Because Larry Ellison needs a bigger yacht, more CA beachfront property, or another Hawaiian island. It's the right thing to do, because Larry is a caring kind of guy.

6/4/13 Update. This just gets gets better and better. See Java Security Revisited--Part 1.