nCircle Patterns Blog: April 2009 Archives

In my independent study of Gregory Bateson and Alfred Korzybski I truly understood for myself that the name is not the things named or as some would say the map is not the territory.  I call your attention to this manner of thinking because we have a problem with metrics in that the count is not the things counted.  Many metrics for risk and compliance describe beautiful mathematical formulas but only see a limited success because the classification of the things being counted is narrowly understood.  This blog posting makes the assertion that our problem with effective metrics is not one of numbers but one of semantics; not of the counts but of the things counted.

The things being counted must be named, defined, and ultimately understood by a community of practice.  The very act of naming is an act of mapping or classification; it comes with a certain level of precision and consequences. A useful classification standard for one community may be useless for another. To the degree that this mapping or classification is common with others in your community of practice, you achieve a mutual semantic coherence (some call this objectivity but I reject that term).  The durability of a set of metrics is challenged when multiple communities of practices are asked to engage in a common objective for the business.  Such is the case when one proposes a standard terminology and metrics that apply across a large enterprise consisting of multiple communities of practice and diverse personas.  To be useful one must know what these metrics mean and to be able to draw inferences from experience.

A measurement system must be judged on the notion of "usefulness to a community of practice" and this scoping must be made explicit.  The utility is a function of the audience's ability to draw inference from the counts and things counted.  Let me share with you an example I experienced with my Toronto team.  I said to one of my Canadian coworkers "Dude, it was in the 90's in San Francisco today".  A blank face appeared as I saw him think and convert this implicit 90 degrees Fahrenheit to Celsius ((F - 32) x 5/9) because he could not draw an inference from Fahrenheit.  Inferences like it being weather for shorts, no jacket required, that it is odd for San Francisco to have a high of 32 Celsius, that homes in San Francisco don't have AC because it is never that hot and so on and so on.

When you look at the notion of temperature, you can see that the different communities have chosen different standards because of the way they have come to know those units and it is more about the semantics than the mathematics.  This becomes exponentially more difficult when the syntax is the same but the semantics vary.  Take terms like 'asset' or 'platform' and you can fill a page with what it means in certain context with certain communities even within the same enterprise.  Each community of practice has come to know the term 'asset' in very different ways; this person has encoded work and meaning in ways that are different than others.  While mathematics remains important, we must turn our focus to formal ways to share semantics. Only then can we share both the numbers (the count) within their intended context (the things counted); semantics that can only be seen through a keen ethnographic eye that respects heterogeneous sense-making and the diverse viewpoints of an enterprise.

Yesterday (Monday) was all about Metricon 3.5 in San Francisco.  It was a long day beginning at 8am and concluding around 5pm.  The event was at the San Francisco Google office and a special thanks to John Flynn and the Google team for hosting this event.  I can’t even tell you how impressive the lunch buffet was at this place.  If I worked at Google I would be 400 lbs in a few weeks.

The event as you can see for yourself from the link above was broken up into case studies, panels, metric frameworks, measurement of real data, and last but not least modeling R&D.  The material was very high quality and for the most part, there were no surprises.  I took notes and from here on out you will get my humble opinion. 

In the Enterprise Case Studies, it was interesting to hear eBay, Kaiser, and Google speak about their measurement systems.  I have a very sensitive ear toward the community of practice for these systems and while eBay and Kaiser was your traditional start at the top with these measurements, Google was more of a bottom up which is great to see.  The role of the designer of these systems is to put data in terms that the audience can understand, not to dictate the way in which the audience should understand it. This required both a ethnographical evaluation as well as a mathmatical evaluation.

In the Metrics from Real Data, Jeremiah Grossman from Whitehat always has good stuff and it was followed up with Wade Baker from Verizon on their breach investigations.  In the framework section, I found Fred Cohen’s work on legal matters very educational.  This community of practice, judges and layers, have a very well established method to understanding information and it was great to hear the challenges for measurement in that space.  Essentially, a bag of bits is real if and only if it has an intersection with other bags of bits and event that support the claims.  It is like a n-dimensional crossword puzzle where just being correct up and down is not sufficient.  One has to be right across and in some cases many other vectors.

Its about 8am in SF and I begin another crazy day at RSA.  In closing, I want to make an observation about all of these experts who claim to have the ultimate measurement system.  Your challenge is not in the numbers or mathematically consistency.  It is in the semantics and the classifications of the objects within the domain.  The reality is that a large enterprise will have nothing short of 5 very discreet personae who on a good day can’t even agree on what to order for lunch.  Getting them all to come to common terms on the meaning of ‘x’ is much more difficult than getting them to understand that 5 is one more than 4.  This standardization of object within a domain is a prerequisite to measurement and must be addressed before one can impose a metric system across multiple communities of interest.

Research in this area [Star 2009] shows that standards are:

• Nested inside one another
• Distributed unevenly across the socio-culture landscape
• relative to communities of practice; one persons ideal standard can be another's nightmare
• increasingly interwoven in ways that are not always hierarchical
• consequential on the value systems of the community

The measurement is not in the numbers but in the understanding of the numbers. 

—tk

Well, here we are again.  This years RSA show will be interesting given all the changes in the world.

For what it is worth, I’m going to blog as much as I can this week.  Tomorrow, it all begins with Metricon 3.5. This year our host will be Google and the day goes from 8am to 6pm.  Yikes. 

For those of you not familiar with Metricon, it is the product of securitymetrics.org.  While I go to these Metricon events, it is awkward because I’m not on the mailing list.  I have been waiting to get on the securitymetrics mailing list now for 3 years.  I wonder if they still have my subscription request. Oh well.

Tuesday through Friday will be all about RSA mayhem.  If you will be there, stop by the nCircle booth and say hi.

—tk