Guest

SCADA Security in Community Owned Utilities

Mon, 09 Jan 2012 16:48:37 -0800

Municipalities (also known as Community Owned Utilities or COUs) face a unique set of security challenges. They are typically smaller organizations, rarely reaching "enterprise" size. COUs face a unique set of challenges that affect the security of our critical infrastructure.

COUs are owned by ratepayers. This means that they have deep public scrutiny of everything they do. Spending on security can be difficult if they are unable to sufficiently justify the expense to the ratepayer. Further, since many have elected officials at the helm, politics is a major factor in everything they do. There are cases where politicians may be unwilling to spend money on equipment and staff unless those initiatives are directly tied to getting/maintaining future votes.

Industrial control systems at many utilities including COUs are generally legacy systems that were never designed to include to security. They are not easy to upgrade and many staff at COUs are forced to operate with aging infrastructure that has been extended far beyond its normal lifespan.

As bad as infrastructure funding is, insufficient staffing is far worse because it's far easier to get money for capital expenses than for operations or headcount.

One of the biggest concerns with COUs is that they underestimate cyber security threats and their relative vulnerability. They think "we're too small for anyone to attack" or "we're too small to be a target" or "we don't really have anything of value to a hacker, terrorist or organized crime ring."

These assumptions are wrong in many ways. Beyond the many issues connected with managing infrastructure critical to local population centers, most COUs are connected to larger utility networks and any breach that affects them has the potential to impact the surrounding energy grid.

It might be surprising, but the staff at municipal utilities are actually remarkably dedicated and resourceful people. They have to be, given the circumstances they face everyday.

The #1 PCI Compliance Issue Today

Thu, 19 Jul 2007 09:53:17 -0800

There is an ancient proverb (largely believed to be Persian in origin) that goes a bit like this:

He who knows not and knows not that he knows not is a fool; avoid him.
He who knows not and knows that he knows not is a student; teach him.
He who knows and knows not that he knows is asleep; wake him.
He who knows and knows that he knows is a wise man; follow him.

In today's world of PCI compliance, the biggest problem many organizations have is very similar to that held by the individual in the first line - they don't know that they don't know. Let me explain my thinking here.

I've consulted with and audited a number of organizations for PCI compliance, both large and small. On the surface, the PCI standard is well-written and generally more explicit in terms of describing what you need to do to achieve compliance. However, no compliance mandate or information security guideline can help organizations fix what they don't know is broken. Particularly in large or more distributed organizations, there are some "gaps" that just don't get addressed. By and large, these aren't the "big things" - organizations know when they have undertaken a massive storage or encryption effort. Likewise, organizations know what brand of enterprise-class antivirus software they have deployed. No, the biggest headache for many organizations is not a particular technical control or product. It's the lack of a truly proactive attitude. This alone can significantly affect the overall security posture of an enterprise, and the state of PCI compliance efforts as a result

Most organizations are doing something about vulnerabilities. Patches are being monitored and deployed, some internal scans are probably run every now and then, and some degree of log monitoring is probably going on. Host-based firewalls or IDS/IPS might be deployed, well-configured images might be the standard, and so on. However, things change. People miss that one box when patching. The new Windows co-op might have screwed up the configuration. Would you know? When's the last time you performed an assessment

I'm a firm believer in the notion of "continuous assessment" for a few reasons. First, over a period of time, this mentality offers companies the best chance to develop a sound and measurable baseline of activity in their environments. This baseline is then monitored constantly - you know those kids' puzzles with the two identical pictures that ask you to "spot what's wrong" in the second one? Right, of course you do. Well, that would be an impossible puzzle without the first picture, wouldn't it? Yep - that would be one seriously frustrating puzzle, alright.

The second major reason I believe in the notion of continual assessment is straightforward - based on my experience I can vouch for it because it works. There, it's that simple. By being proactive, and learning a) what you have, b) how it's configured, and c) when something changes, you can create a truly effective security regimen that is much easier to monitor and maintain. So many people think that running a vulnerability scanner means clicking a button on a scanner, coming back 10 hours later and printing out the 478-page PDF file that now tells you exactly what is wrong in every nook and cranny of your infrastructure. That's a bit old-school: the new breed of tools can assess a LOT of things with a more automated approach, all of which can tie to a solid security program and a sound PCI compliance strategy. Here's a few:

Determining whether your patch management program is effective
Determining whether your hardening standards and guidelines are effective and being followed
Determining whether you already have an intrusion that needs to be dealt with
Determining whether corporate-wide security policies are being adhered to
Learning quickly when new systems come online, or when existing systems change in some way
Learning whether unencrypted protocols and services are in use
And on and on...

Continually assessing risks and exposures and discovering vulnerabilities is a program worth establishing. By learning what your issues are, fixing them, then continually assessing your own environment, you will quickly find that you are not a fool at all - you might just be on your way to being a wise person.

View from the other side

Fri, 22 Jun 2007 09:54:29 -0800

The genesis of this article started with the observation that the time lag between the discovery of a security issue in a product and the (hopefully) eventual release by the vendor of a fix is highly variable. Some reasons for that variability, and in particular, why that lag may seem long is the topic of this discussion. I'll try to stay on track, but may occasionally veer off to observe interesting stuff (at least to me) along the way.

I should point out that my perspective is that of the vendor of a product and not that of a security researcher.

Not too long ago an email made its way to many of the security mailing lists with this topic:

Multiple Vendor libexif Integer Overflow Heap Corruption Vulnerability iDefense Security Advisory 06.13.07 http://labs.idefense.com/intelligence/vulnerabilities/

It discusses a problem with parsing Exif data, used in most digital cameras, and in most software shipped by manufacturers of software with those cameras. In this case the library is open source. Now here's the relevant part:

VIII. DISCLOSURE TIMELINE
08/16/2006 Initial vendor notification
06/05/2007 Second vendor notification
06/11/2007 Initial vendor response
06/13/2007 Coordinated public disclosure

Ten months went by between the first notice to the vendor and the second notice. The second notice seems to have gotten some attention, and only a few days later the public disclosure occurred. But TEN MONTHS? Why didn't the security researcher ping the vendor sooner than ten months after the first disclosure? Did the vendor miss the first notification? If not, what were they doing in the interval?

Speaking of intervals, remember the news regarding bike locks and pens back in 2004?
http://www.wired.com/culture/lifestyle/news/2004/09/64987
Note this part of the article:

"The lock's flaw was apparently first publicized in 1992 in the United Kingdom... The BBC even covered it, but the news apparently didn't resurface until a dozen years later."

Talk about lag time. You can still buy locks that are way too easy to open, but that's a different topic for another time.

Back to software. Now, I was not a party to any of this, but let's posit a hypothetical that instead of an open source vendor of this library, the vendor was a commercial supplier. They supplied this library to many vendors who used it in software that eventually made its way to consumers. Let's look at the various channels that software might follow to get to the consumer, and consider what time lags are built into each channel. We may find where those ten months went.

When you buy a digital camera most often you find a CD in the box that contains some sort of image processing software. You may find a version of Adobe Photoshop Elements, you may find a proprietary software package, and you may find both. That CD was made in bulk months before the camera was purchased, inserted into the box on an assembly line overseas, and shipped to a warehouse somewhere in the world. It could easily have sat on a shelf, either in the warehouse or in a retail store, for a year or two before being purchased. That's one channel, and we can see a few issues already.

The CD needs to be remastered, at least for new production runs. The original developer of the software needs to integrate the fixed library into the software, do their QA, provide a new master copy of the software to the camera company, which then would presumably also need to go through QA or acceptance testing, a build cycle for the new CD, and so on. They might need to decide on what info they supply to registered users of the software regarding this security issue. Do they host a patch or update on their web site (more QA)? How many languages does the notice need to be translated into? What training, if any, needs to be given to call center employees (and yes, there will be calls if they post or mail or email a notice). This leaves out the question of product recalls and such.

It gets better. Let's consider another channel. In this case the image software is pre-installed on the hard drives of a major retailer of consumer computers. Yup, another set of QA cycles, new builds of a gold master hard drive image to install on computers at the factory or factories, and the same set of questions regarding disclosure/notice to consumers who purchased a computer with this software pre-installed. Plus all the issues of FAQs on a web site, call center training, notice translations, product recalls, and so on.

One of the interesting things about the libexif advisory above is that none of the final products that incorporate the library are disclosed. The closest to that degree of disclosure is this statement:

Exploitation requires that a targeted user process a malicious image using one of several available tools that utilize libexif for Exif tag parsing. These tools include, but are not limited to, several applications included in the GNOME and KDE desktops.

Now imagine the discussions, negotiations, politics and such that would have gone on in our hypothetical case above with a commercial supplier and a security researcher who had found the vulnerability in just one of the many commercial products that incorporated it. Should the supplier of the library disclose their full customer list? In fact, they may be contractually prohibited from doing so. Clearly they need to contact all their customers, but during that TEN MONTH lag perhaps the researcher finds other applications that have incorporate the library. The list of affected application shifts over time, and getting agreement from each final customer on the wording of disclosures becomes a major effort. What's that you say, why should the wording be negotiated? Why should the end customer have input, be involved? Remember that the end customer may need to create web FAQs, emails, or paper mail that discusses the issue, and consistency of messaging is important. Remember, this stuff is being translated into a dozen or more languages, approved by product line managers in many countries, and coordinated with PR agencies.

Who pays for the costs of the mitigations listed above? New software builds, testing, new masters, translations, call center training. The costs rapidly exceed the revenue the library supplier generates from sales of the library. What do the contracts say with respect to responsibility? Who considered this scenario when the contracts were drafted?

Now, some of the issues above could be addressed by smart programming. Producing software that can be updated auto magically via the Internet would help. Having a feature that periodically checks with the mothership for those updates would help (do it right or you've got another security risk). None of this changes the issues of coordinated release, call center training, and so on. Nor does it remove the issues of remastering CDs and gold hard drive images.

Now I'm not advocating that vendors go dark for months at a time to deal with their internal and customer issues. Far better to communicate quickly, directly, and honestly with security researchers about the time needed to resolve vulnerabilities, not just from the code perspective but from the downstream customer perspective as well. This communication needs to be persistent and continuous, setting appropriate expectations on both sides. This may not change the time between discovery and disclosure but it will improve the relationships between security researchers and the vendors they research.

RoHS and WEEE only the tip of the iceberg

Wed, 13 Jun 2007 09:12:23 -0800

As nCircle's Director of Operations, I thought I would share some of my experiences working with the new environmental regulations that today's nCircle press announcement described.

Anyone hoping for a break after achieving RoHS (European Directive 2002/95/EC, also known as the Restriction of Hazardous Substances or "RoHS" directive) and WEEE (European Directive 2002/96/EC, also known as Waste Electrical and Electronic Equipment directive) compliance will be disappointed as the amount of proposed new environmental programs has only accelerated in 2007. No sooner did the dust begin to settle on the RoHS front than the European Commission began the process of reviewing and amending the list of restricted substances. There is no guidance at this time as to which additional substances may be added to the current list of six or the timeframe for elimination, but it's only a matter of time.

On the other side of the world, China officially launched its RoHS program on March 1st of this year. Though modeled on the European Union RoHS legislation, China RoHS has its own unique requirements that must be followed. More recently, South Korea submitted a WTO notification for their version of RoHS/WEEE with a comment deadline of July 21, 2007.

Closer to home, virtually every state in the Union is in the process of considering or passing some form of environmental legislation. Most states are targeting the large consumer items - TV's, personal computers, and large appliances at this time, but computer servers like nCircle appliances will soon follow.

What's next? In addition to more countries adopting their own RoHS/WEEE type programs, there are moves afoot to improve energy usage and efficiency in data centers - project "Green Grid" and IBM's "Green Team" are two of the higher profile efforts in this area that will impact how servers are designed, deployed and utilized in the future. Finally, more and more organizations are moving toward eco-friendly purchasing. The University of California recently announced a new far-reaching "Environmental Sustainability policy" under which they will only buy products registered under EPEAT (Electronic Product Environmental Assessment Tool). Similar to "Energy Star", EPEAT measures products against a set of environmental standards including reduction in harmful chemicals and designs that are more easily recycled.

It's been a busy few years for those of us responsible for RoHS/WEEE compliance and that was only the start. The next few years promise to be even more exciting and I, for one, can't wait.

Got redundancy?

Fri, 04 May 2007 12:18:23 -0800

Right now, there's an unpleasant bug going around. No, not a computer virus, a human one. The first symptom is a blinding headache - literally blinding, people who get it say they weren't able to read for a few days their heads hurt so bad, or they can't move their head without excruciating pain. I've got the bug too, so far I can still focus my eyes most of the time, but I'm not getting any real work done. I did get some stuff done yesterday because it needed to be finished, but I went through it slowly and double- and triple- checked everything I did. I'd have vastly preferred to given the tasks to someone else to finish and gone back to bed. Well, really I'd vastly prefer to not have the cold/flu, but that plan isn't working out so well.
When we build mission critical systems, we build in redundancy. A lot of it. We build systems so that they can handle multiple points of failure. We pay for vendor support so that if a critical component goes down, it is fixed or replaced within a certain amount of time. All of this costs real money, and we spend it to manage the risks inherent in the components of the overall system.
What about the people who run the systems? How good is the redundancy in the people involved? Are they cross trained to handle different failure types? If you have someone out sick (or worse, to use the standard quote, if they get hit by a bus), how well can your organization handle that type of failure?
With the pace of modern business, especially in the technology sector, business tends to not to have quality human redundancy. People are expensive, and so businesses tend to give them tasks that aim for the bottom line, not ones that don't directly or obviously have a return. The problem (as you know far to well if you're in the infosec business) is that a failure in the human side of the business can be far more damaging that a failure in a piece of technology.

Redundancy is a type of insurance to mitigate risk, do you have it?

Communicating outside your (security) culture

Tue, 01 May 2007 09:28:39 -0800

A little while back I was talking with my six year old, and said six year old asked me "What is risk?". I realized I didn't have an answer that was one or two sentences. In fact, I didn't have an answer that I thought would really get the idea across, though after going through several tries I think I got the idea across. The hardest part was finding a common frame of reference to build on. And yes, I was a bit dismayed that I didn't have one or two sentences to communicate an idea that is a basic part of Information Security to someone who didn't know anything about it.

What this episode made me realize (again) is that we, as security professionals, usually have a very different way of looking at things than the people we work with, both "business" people and IT people. This difference can be hard to detect - for example, as adults we all know what 'risk' means. Or do we? Overall, I think the majority of people will have roughly the same idea for the word 'risk', though I believe each person's meaning will be colored by their own experiences and observations. When it comes to clearly understanding a particular set of "risks", this is where I think our different viewpoints (security, IT, "business") result in very different understandings of the "risks".

I've seen this disparity of cultural viewpoint a number of times, and seen people struggle with it as they try to understand each other and move forward. Sometimes it takes awhile for them understanding they have a communication failure (vs. "so and so is just {"an idiot", "paranoid", "YOUR_FAVORITE_LABEL_HERE"}"), sometimes they don't. When they do make the realization, and I mean really make it, when they get that the other people are looking at the situation in a completely different way, I see them go through what I did in trying to explain risk to a person who has no real experience in it - trying to find a common frame of reference so they can build up a real understanding. Over and over again I've seen this be quite challenging, and where its been successful two of the common threads have been people listening to each other and trying to see the situation from the other person's point of view. Its not easy to twist our brains around to a different way of thinking, yet every time I see people do it, I see success. What has been (or was devolving into) an acrimonious relationship becomes one of trust and mutual respect, which then becomes a highly productive one. As well, both sides usually learn from the experience, both what and how the other people do and they learn a bit more about themselves and what they do and how they do it.

Overall lesson? Listen when the uninitiated ask questions, you might just find something useful and interesting in them.

Our next guest blogger is...

Wed, 25 Apr 2007 16:25:25 -0800

...Eric Hall! Eric is a security architect and consultant who specializes in the design, implementation, and in-depth troubleshooting of complex information systems with a focus on security. I look forward to reading Eric's upcoming posts on this blog, and I hope you do too. Enjoy!

Inaugural post: Fear, Uncertainty and Doubt

Fri, 06 Apr 2007 16:23:45 -0800

It is somewhat daunting to post my first blog entry to a collection of forums that claims Tim Keanini as one of its participants. As much as I love to engage in cerebral discussions with TK I rarely traverse the same plane as he does in my blog postings. Anyone who has read my threatchaos security blog knows I like to focus on threats, the security industry, and the response, or lack of response, of enterprise.

If there is one theme I keep harping on it is the complete lack of preparation that most organizations exhibit when it comes to cyber security. Thus, I think it appropriate to raise that point here in my inaugural post to nCircle's guest blog.

If security investments are done properly they are done in the context of a risk management program. But risk management analysis invariably underestimates cyber risks because it relies on past experience which is not relevant in a threat environment that is growing exponentially.

I was speaking at the FDIC a couple of years ago and heard about one audit they had done at a LARGE data center on the Gulf Coast of Florida. The FDIC auditor was going through the list of risk factors they tracked and he came to "Hurricane" which was ranked 1 out of 10, the least chance of occurrence. When challenged the response was "We have not had a major hurricane come ashore here more than once every 100 years, so we rank that low". The auditor said, "Yes, but there is a category 5 hurricane in the Gulf right now heading your way, doesn't that impact your risk?"

The moral of the story is that risk is dynamic and risk management programs must take that in to account. Right now it is becoming painfully obvious that bad guys are making concerted efforts to steal identities, particularly credit cards, in any way they can. From the wireless attacks against BJ Wholesale and DSW to the physical attacks against Stop and Shop, and TJX the warnings have been sounded. If there is a similar attack against any retailer in the next six months they will not be able to plead ignorance of the threat level.

And operators of critical web sites that account for significant revenue are also on notice. The bad guys have identified you and your web assets. If they cannot steal directly from you they will launch denial of service attacks against your site and attempt to extort money from you. Not being prepared is not an excuse. The cost of recovery, after an attack, will exceed the cost of being prepared by a factor of ten.

First up on the Guest blog - Richard Stiennon!

Fri, 30 Mar 2007 09:00:43 -0800

We are excited to announce that our nCircle Guest Blog series is about to kick off to an amazing start. We are working on a killer line up of guest bloggers for you, and the first one up is Richard Stiennon.

Richard is currently the Chief Marketing Officer at Fortinet, and has more than 25 years of experience in the security industry. You may already know Richard from his tenure as Vice President of Research for Gartner's Security and Privacy group. Most recently, Richard was the Founder and Chief Research Analyst of IT-Harvest, Inc., an independent IT research firm. Richard also writes for the ZDNet Threat Chaos blog. While at IT Harvest, Richard participated in one of nCircle's webinars, titled Doomsday Scenarios: Understanding the variables driving doomsday scenarios and strategies to protect your organization.

Check back on this blog soon - Richard should be posting here shortly!
Richard's archived web seminar can be viewed here: www.ncircle.com/index.php?s=registration_registernew&src=doomsday

And the winners are...

Fri, 09 Feb 2007 15:46:21 -0800

Thanks to all the entrants in my "Overheard at RSA" contest - I had a good laugh reading through my inbox over the past couple of days. Here are my favorite three submissions (all of the authors will receive an nCircle remote control car, as promised!) - at least the top three that were fit to print!

1. "Well, we're not the ones who drop our pants first..." - said by an exec at a security vendor, discussing the ongoing pricing wars between security companies.

2. "Wow, she's tall." - said by everyone passing the booth that employed a seven-foot tall model to attract attendees.

3. "Everyone claims to have a compliance product this year. Even if it was a toaster last year, it's a compliance product this year." - said by a member of the press as he walked the show floor.

Thanks for playing! Until next time...

Overheard at RSA

Tue, 06 Feb 2007 09:02:22 -0800

Well, I’m excited to be the first official guest blogger for the new nCircle Guest Blog. I must admit though, it’s a little intimidating to post alongside people like Storms, TK, Terlin and the VERT crew. My original thought was to post a play-by-play report from the RSA show floor, but those guys will do a better job covering the show content than I ever could. So instead, I’ll just fall back on my strengths and giveaway FREE STUFF! I am in marketing, after all.

So – here’s the deal. Anyone who attends RSA always comes back afterwards with tons of “I overheard the funniest thing today on the show floor” stories. Well, send me your best “Overheard at RSA” comment by noon PST on Friday (2/9) and I’ll send the top three comments (in my own personal opinion) one of those super cool remote control cars that we are giving away at our booth. If you haven’t seen them yet, get yourself over to the nCircle booth (#931) and take one for a spin. Email your submissions to jerale@ncircle.com and I will post the winning submissions on Friday afternoon.

Welcome to the nCircle Guest Blog!

Thu, 01 Feb 2007 12:34:00 -0800

We created this space to host commentary by nCircle customers, members of the nCircle executive team, and industry thought leaders. Stay tuned - we have an exciting line up coming your way.

If you have suggestions on authors and/or topics for this blog, please post them in the comments section or send an email to info@ncircle.com.