Insecurity Blues

It hasn't been a good month for my code. Samba, the project I'm responsible for, has had to announce several security flaws. Unfortunately some of them were in code I wrote. I always do a large amount of soul-searching whenever that happens. There's nothing worse than finding out something you were responsible for is the cause of many thousands of people having to waste their time rolling out patches. It always makes me wonder if the time has come to give up this programming lark and end my days peacefully in management, messing up other peoples code instead of creating my own.

It's very educational however to look at the causes of the flaws, and try and learn what we can from the bugs and also our reactions to them. Samba is an old program. The initial code was originally written fifteen years ago. At that time modern security problems such as integer wrap (where adding two numbers together can end up with a number smaller than both of them, due to the fixed sizes of integers that processors deal with), or heap overflow vulnerabilities (where overwriting unallocated memory on the program heap can allow a clever attacker to seize control of a program) were unheard of. We knew about buffer overruns (where reading more data into a buffer than was originally allocated for it can cause a security breach) and denial of service attacks, but 1992 was a simpler, less hostile time for network software development. Most initial deployments of Samba were on networks isolated from the main Internet, by technically advanced administrators who knew how to download the source code from the Internet.

The kind of security bugs we get now are much more sophisticated than those that were originally present in the first releases of Samba. We also treat them much more seriously than we used to. In the early days, we (or someone else examining the code) would discover a security problem, and almost immediately we made an announcement of a fixed version. Those were also the days where binary releases of Open Source/Free Software were unknown, as everyone knew how to compile from source code. At least everyone who we were interested in working with did. Our response time for security bugs was usually measured in hours (depending on the timezone of the person reporting the bug). I still remember the first attempt by someone to sabotage Samba with a zero-day exploit posted directly to our mailing list without any warning. Andrew Tridgell (the original author of Samba) was so incensed that he immediately raced to ensure the very next message on the list was the patch for the bug. He succeeded. These days we rarely get zero-day exploits posted openly to lists. The bad guys who discover exploits keep them secret for their own illicit use, and the good guys go through the currently agreed-upon best practice of sending notice to the Samba developers directly (at the mailing address security@samba.org) and work with us in coordinating the announcement of the problem and its fix.

Our response time appears to be much slower these days, even though internally we still usually create an initial fix for the problem within a few hours of seeing the report of the bug. Having Samba Team members in almost all timezones helps with this speed of response. What slows us down these days is our own success. In order to responsibly release a security fix version of Samba we must coordinate with a large number of vendors who are reselling and shipping our code. Most of them have their own patch release schedule and internal security teams who validate the suggested patch and agree on a time to announce the vulnerability to everyone.

Who discovers our security bugs? These days, it's about fifty fifty between internal audits by the Samba Team and other Open Source security people and external security researchers, who make their money winning contracts by proving how smart they are by discovering vulnerabilities in Open Source/Free Software projects. The way we respond has changed also. What we do now on seeing a security bug is immediately audit the entire code-base to discover if there are any similar problems, or even similar coding practices that might cause future problems, and re-write or remove all such code. It takes longer, but is much safer in the long run. If you examine parts of the Samba code you'll find common functions that are known to be insecure simply won't compile if added to our code. A set of automated macros warns of any use of known bad functions. Still, we can't depend on automation for protection. Our current security vulnerabilities involve exploiting an assumption in one part of the code to cause a vulnerability in what appears to be a completely separate part of the code, which is actually interdependent. Attackers and security researchers are very clever these days.

Is it easier to find and fix security bugs in Open Source/Free Software code rather than proprietary code? My personal feeling on that is “yes”. Being able to examine the intent of the author by looking at the code and comments it makes it much easier to see where the intent and the code don't match. That's where the security bugs lie. Is Open Source/Free Software code safer than proprietary code? I'd say “yes” to that also. Looking at the number of vulnerabilities is misleading. A good fifty percent of our security bugs are found by our own internal code audits, or someone just looking over some code they're working on and saying to themselves “that looks funny...” Whenever we find something like that we have to report it to the our vendor community and make a security announcement. It is painful doing this but as all our processes are open we don't have the luxury of silently fixing something and hoping no one will ever notice. Proprietary vendors do have that luxury, they would be insane not to take it, and I'm sure they do. Proprietary code vendors need all the help they can get competing with good Open Source/Free Software.

Samba is still written in C. I've examined the possibility of moving to C++, but to be honest that won't really help for the kind of security bugs that we seem to suffer from. In fact by hiding interdependencies in object classes it may make them more difficult to find. Also I'm not the final decision maker on that and many of the Samba Team hate C++ (Luddites that they are :-)). Re-writing in a managed code language like Java or C#.NET would definitely help reduce our security problems but would reduce the performance that people find to be one of the compelling reasons to use Samba. It's no accident that none of the competing CIFS servers are written in managed code either. It would cost us six months to a year to re-write everything and get back to the same level of features, probably with reduced performance. We simply don't have the people or time to do that. For the moment we're stuck with C and we have to make the best of it. If someone is creating a competing Open Source/Free Software CIFS server I'd encourage you to do so in a managed code language as an experiment. If it works as well as I'd hope we may throw our code away and join you :-).

In order to keep Samba relevant and current with modern practice, we've had to keep re-writing and re-designing core parts of the code. Our upcoming release, Samba 3.2, bears little relation to the original 1992 code and uses modern memory management techniques to avoid heap overflow, buffer overruns and memory leaks, much of which was invented by the Samba Team for internal use. The goal is to keep ourselves secure, as the Samba Team uses Samba for file services ourselves. This is all about scratching our own itch, and helping out everyone else as we do so.

I still feel bad about the mistakes I've made, but I try and redeem myself by using them as examples of how not to write code when I do interviews here at Google. If someone doesn't see the bug it always makes me feel better, but acts as a black mark against them in the interview score. After all, everyone we hire should be able to write code better than me :-). The code is out there to be studied; so learn from my mistakes and don't end up having to coordinate a security announcement. That's one learning experience it's better not to have.

Jeremy Allison
Samba Team.
San Jose, California.
24th November 2007



Comments

The only way to have no bugs is not to write software at all

Keep up the good work: there are definitely much worse bugs in proprietary software, but we only don't know about them until it's too late, whereas you fix bugs _before_ they get exploited!!!
I think you rock!

obj-c

It's a shame everybody forgets obj-c and stop modernization of C codebases just because c++ is ugly!

Bugs

It has been said that any large piece of software will have bugs and if the software is big enough, it will have bugs that will never be found.  Windows is the classic example of this.

Samba is of a size and complexity that it is inevitable that bugs will creep in.  Don't let that bother you.  A few weeks ago, I found a bug in a program I wrote over a decade ago.  There is no such thing as perfection in human endeavor. However, there's always some room for improvement.

Programming languages don't write code.

And neither do functions call themselves. And there is no
reason to feel 'bad' about having made mistakes, because
the only people who don't make mistakes are people which
don't do anything at all.
.
It is more important to be able to repair defective
code than chasing the chimera of somehow not
writing code with defects which need to be fixed.
.
That this fact tends to make people who (at the
core of their soul) believe that programmers are
actually strange wizards and programming inexplainable
magic is the problem of these people.

Offensive Generalistaion of Independent Security Researchers

I find your generalisation of independent security researchers offensive, whenever I publish a vulnerability it's because I care about and genuinely want to contribute to open source security. I'm a security geek, thats where my expertise lies and where I can best help, I really just want to help by contributing patches and research before the bad guys do the same. Yes, we expect credit, just as you put your name in comments or AUTHORS/ChangeLog/etc, I dont think this is too much to ask.

From the wording of your comment, it sounds like you're unhappy that we want to help keep your users secure. I personally have not made a penny from open source security research (i'm a student), and certainly wont be looking at samba in future, as I'm not interested in contributing to projects where my work is not welcome.

No need to take offense

I think you are taking offence when none is intended. He listed 3 categories of people: "the Samba Team and other Open Source security people and external security researchers".

Perhaps if you put yourself in the 'Open Source security people' category then you'd feel better?

And whilst his categorisation of the business model of external security researchers is a generalisation, I don't think it's too far from the truth, and nor is it perjorative. That's just how the business works.

I'm sure the Samba people appreciate the efforts of people like yourself - the article says as much. No need to get all offended.

Wookey

Not intended to offend

I wasn't meaning to give offence. I have nothing against independent security researchers making money by publishing flaws. This is part of the economic ecosystem around FLOSS code, I get paid to write Samba after all :-).

The best way to get revenge is to find more Samba security bugs please :-). That'll teach me a lesson (especially if it's in my code :-).

Best wishes,

Jeremy.

Security is hard

It's good to hear that the Samba team is keeping on top of the state of security. It's amazing when you think about how much you really need to know to write secure C/C++. I suppose that's a price you pay for the power such languages provide.

I have a few references that I personally like for secure *nix coding. The CERT C/C++ Secure Coding Standards are very current. I'd also recommend two books from contributors to the project: "Secure Coding in C and C++" (Seacord) and "The Art of Software Security Assessment" (Dowd).

Back to top