The things being counted must be named, defined, and ultimately understood by a community of practice.  The very act of naming is an act of mapping or classification; it comes with a certain level of precision and consequences. A useful classification standard for one community may be useless for another. To the degree that this mapping or classification is common with others in your community of practice, you achieve a mutual semantic coherence (some call this objectivity but I reject that term).  The durability of a set of metrics is challenged when multiple communities of practices are asked to engage in a common objective for the business.  Such is the case when one proposes a standard terminology and metrics that apply across a large enterprise consisting of multiple communities of practice and diverse personas.  To be useful one must know what these metrics mean and to be able to draw inferences from experience.

A measurement system must be judged on the notion of "usefulness to a community of practice" and this scoping must be made explicit.  The utility is a function of the audience's ability to draw inference from the counts and things counted.  Let me share with you an example I experienced with my Toronto team.  I said to one of my Canadian coworkers "Dude, it was in the 90's in San Francisco today".  A blank face appeared as I saw him think and convert this implicit 90 degrees Fahrenheit to Celsius ((F - 32) x 5/9) because he could not draw an inference from Fahrenheit.  Inferences like it being weather for shorts, no jacket required, that it is odd for San Francisco to have a high of 32 Celsius, that homes in San Francisco don't have AC because it is never that hot and so on and so on.

When you look at the notion of temperature, you can see that the different communities have chosen different standards because of the way they have come to know those units and it is more about the semantics than the mathematics.  This becomes exponentially more difficult when the syntax is the same but the semantics vary.  Take terms like 'asset' or 'platform' and you can fill a page with what it means in certain context with certain communities even within the same enterprise.  Each community of practice has come to know the term 'asset' in very different ways; this person has encoded work and meaning in ways that are different than others.  While mathematics remains important, we must turn our focus to formal ways to share semantics. Only then can we share both the numbers (the count) within their intended context (the things counted); semantics that can only be seen through a keen ethnographic eye that respects heterogeneous sense-making and the diverse viewpoints of an enterprise.

The event as you can see for yourself from the link above was broken up into case studies, panels, metric frameworks, measurement of real data, and last but not least modeling R&D.  The material was very high quality and for the most part, there were no surprises.  I took notes and from here on out you will get my humble opinion. 

In the Enterprise Case Studies, it was interesting to hear eBay, Kaiser, and Google speak about their measurement systems.  I have a very sensitive ear toward the community of practice for these systems and while eBay and Kaiser was your traditional start at the top with these measurements, Google was more of a bottom up which is great to see.  The role of the designer of these systems is to put data in terms that the audience can understand, not to dictate the way in which the audience should understand it. This required both a ethnographical evaluation as well as a mathmatical evaluation.

In the Metrics from Real Data, Jeremiah Grossman from Whitehat always has good stuff and it was followed up with Wade Baker from Verizon on their breach investigations.  In the framework section, I found Fred Cohen’s work on legal matters very educational.  This community of practice, judges and layers, have a very well established method to understanding information and it was great to hear the challenges for measurement in that space.  Essentially, a bag of bits is real if and only if it has an intersection with other bags of bits and event that support the claims.  It is like a n-dimensional crossword puzzle where just being correct up and down is not sufficient.  One has to be right across and in some cases many other vectors.

Its about 8am in SF and I begin another crazy day at RSA.  In closing, I want to make an observation about all of these experts who claim to have the ultimate measurement system.  Your challenge is not in the numbers or mathematically consistency.  It is in the semantics and the classifications of the objects within the domain.  The reality is that a large enterprise will have nothing short of 5 very discreet personae who on a good day can’t even agree on what to order for lunch.  Getting them all to come to common terms on the meaning of ‘x’ is much more difficult than getting them to understand that 5 is one more than 4.  This standardization of object within a domain is a prerequisite to measurement and must be addressed before one can impose a metric system across multiple communities of interest.

Research in this area [Star 2009] shows that standards are:

• Nested inside one another
• Distributed unevenly across the socio-culture landscape
• relative to communities of practice; one persons ideal standard can be another's nightmare
• increasingly interwoven in ways that are not always hierarchical
• consequential on the value systems of the community

The measurement is not in the numbers but in the understanding of the numbers. 

—tk

For what it is worth, I’m going to blog as much as I can this week.  Tomorrow, it all begins with Metricon 3.5. This year our host will be Google and the day goes from 8am to 6pm.  Yikes. 

For those of you not familiar with Metricon, it is the product of securitymetrics.org.  While I go to these Metricon events, it is awkward because I’m not on the mailing list.  I have been waiting to get on the securitymetrics mailing list now for 3 years.  I wonder if they still have my subscription request. Oh well.

Tuesday through Friday will be all about RSA mayhem.  If you will be there, stop by the nCircle booth and say hi.

—tk

I wonder if I am the only one that feels this way.  Major application and OS’s do a great job at offering this review before a user approves the update but such is not the case in the land of the Xbox 360 game console.  Sure you could argue that console gamer is not going to know a DLL from LSD but nonetheless, offering optional information about what the update is going to do for them is good form.   In Xbox360 land, you get a screen that looks something like this

and it would be great if the X or Y button gave you information on what was about to change on your system.  And while your taking down my feature request wonderful product manager of the xbox360, it would be nice to see the update history of the machine. 

Does the information exist?  Sure it does but you have to really hunt for it and I’m not sure all the updates have made it to the web.  For example, http://blogs.msdn.com/xboxteam/archive/2007/11/30/december-2007-system-update.aspx

http://www.xbox.com/en-US/community/news/2006/1030-novemberupdate-completelist.htm

From a security stand point, it just spooks me out when I approve an update to my system and have no idea what has downloaded or what has been modified.  The number of independent game developers for Xbox360/Xbox-live are taking off and Microsoft has a solid program.  Lets just say that things will start to get very interesting.

—tk

I bought another LCD monitor and with it was a mail-in rebate for 30.00.  Like all of these, you spend time to gather the required information, sent it in, and after a good 6 weeks time, you get a check.  Done?  Not quite because the “Pay To the Order of” has misspelled my last name.  If this was the first time this happened, it would not be an issue but 3 times in the last 6 months, something seems wrong.

Could it be that there is a strategy out there to raise the cost of accounting on the payee so that they at some point think it is not even worth it to pursue?  I wish we could see the statistics of all the people who go through with the mail-in but because of the run around, end up ultimately not redeeming their rebate. 

This information is not available so all we have to go on are patterns and paranoia.  Is 30 minutes of sitting on hold and filing more paperwork worth $30.00?  At some point, everything come to a cost/benefit decision.

—tk

When I first started going to RSA, there were more vendors than there were customers.  It was a huge vendor boondoggle and while the business development people were having a great time, I was looking for customers to speak with and have a great conversation about what they were looking for at the show and what type of problems they were trying to solve. 

This year was great in terms of customers-to-vendor ratio.  We had a great turnout at our booth and I’ve almost lost my voice from non-stop conversations.  What does this change mean for future RSA shows?  I remember one year being at the show and having a customer tell me “You know what TK, this is a show of car parts, and frankly, I need transportation.”.  I’ll never forget this statement and I have a working theory. 

In the early days of the RSA show, the exhibitors sold all kinds of parts that when put together by a skilled craftsmen, created a powerful solution.  Composability was more important than Usability.  As the attendees change to more of a business level buyer persona, consumers that are not security subject matter experts, we move toward deeper solutions where Usability trumps Composability. 

When I hear those words “…this is a show of car parts, and frankly, I need transportation.”, I imagine a trend on the exhibit floor dominated by much more complete solutions.  Product designed for a persona that does not know how to fire up a debugger, does not know how to read a set of ACLs, but knows how to read market results and can use Excel to model any financial system you can imagine.  That might be a little extreme but nonetheless, the customers out number the vendors by a larger and larger margin. 

I predict that RSA next year will have less small highly technical one-trick-pony companies and more multi-product solutions and managed services companies.  To use that great quote, there will be more vendors selling cars and transportation services than there will be vendors selling parts. 

—tk

Huh?  With all that virtualization, load balancing, and other service abstraction strategies we have today, why do we still have to deal with scheduled downtime?

I understand that we cannot plan on ever getting rid of an unscheduled outage because “stuff happens” but we certainly have at our fingertips methods that can avoid scheduled downtime once and for all.

I’m just bitter because it may take a bite out of my Halo3 Team Slayer.  The Master Chief would never allow for scheduled downtime!  It must be the work of the Covenant.  The fight continues…

—tk

Marcus Vitruvius Pollio was born ~80 BC and died 25 BC.  Regardless of his abilities as an architect, he lives today because he was the person who wrote about the craft and documented the essence the architecture of his time.  If being referenced some 2000+ years later is not enough of a value proposition to get you to document your contribution,  I don’t know what is.

Jennifer referenced De architectura (Latin: “On architecture”) which consisted of 10 scrolls and “The Ten Books on Architecture” which is the translation and available on books.google.com.  Vitruvius said that well-designed buildings must exhibit three qualities: firmitas, utilitas, and venustas.  Respectively, utility, attractiveness, stability.

Looking at information system design, these qualities are also beneficial.  What is interesting in Jennifer’s presentation is that applications at some moment in time can be mapped to a vector in a firmitas, utilitas, and venustas space.

For example, an application can be at position ‘X’ when it is in demo format and ultimately its goal is to move to position ‘Z’.  There are times when an application would be not as attractive or has low utility but is ultra stable like ‘Y’; its goal over time is to get to position ‘Z’. 

Another thing that was clear was how Vitruvius understood his users.  He had an intimate understanding of who would occupy the dwelling and what tasks they would perform on a daily basis.  Up front in the design was a serious considerations for private and public spaces.  I can see how this has a parallel with information system. 

Vitruvius is quoted as saying “The eye is always in search of beauty” and who can argue that.  We should set our design goals high and demand beauty, utility, and stable system. 

—tk

It is South by Southwest time again and Austin Texas is completely consumed by inventors, designers, artists, gamers, authors, and any other category that describes a creative class.  The beauty of this conference is that it brings together many creative disciplines and everyone shares their passion.  It is the intersection of software, film, and music. 

Today was registration and as you can see by my badge, I’ll just attend the technical sessions this year.  If I sound a little bummed about that it is because there are a few bands this year that I really wanted to see but things are just too busy at work.  Oh well.  This afternoon, I sat in on a good talk about Javascript patterns and tomorrow I’m looking forward to some great design sessions.  I’ll blog some of my thoughts. 

—tk

Will you be at RSA?  I’m giving a talk on Game Theory and how these patterns can be applied to IT Security.  Check your program and don’t be shy – come over and say hello.

—tk