nCircle Patterns Blog

Yesterday (Monday) was all about Metricon 3.5 in San Francisco.  It was a long day beginning at 8am and concluding around 5pm.  The event was at the San Francisco Google office and a special thanks to John Flynn and the Google team for hosting this event.  I can’t even tell you how impressive the lunch buffet was at this place.  If I worked at Google I would be 400 lbs in a few weeks.

The event as you can see for yourself from the link above was broken up into case studies, panels, metric frameworks, measurement of real data, and last but not least modeling R&D.  The material was very high quality and for the most part, there were no surprises.  I took notes and from here on out you will get my humble opinion. 

In the Enterprise Case Studies, it was interesting to hear eBay, Kaiser, and Google speak about their measurement systems.  I have a very sensitive ear toward the community of practice for these systems and while eBay and Kaiser was your traditional start at the top with these measurements, Google was more of a bottom up which is great to see.  The role of the designer of these systems is to put data in terms that the audience can understand, not to dictate the way in which the audience should understand it. This required both a ethnographical evaluation as well as a mathmatical evaluation.

In the Metrics from Real Data, Jeremiah Grossman from Whitehat always has good stuff and it was followed up with Wade Baker from Verizon on their breach investigations.  In the framework section, I found Fred Cohen’s work on legal matters very educational.  This community of practice, judges and layers, have a very well established method to understanding information and it was great to hear the challenges for measurement in that space.  Essentially, a bag of bits is real if and only if it has an intersection with other bags of bits and event that support the claims.  It is like a n-dimensional crossword puzzle where just being correct up and down is not sufficient.  One has to be right across and in some cases many other vectors.

Its about 8am in SF and I begin another crazy day at RSA.  In closing, I want to make an observation about all of these experts who claim to have the ultimate measurement system.  Your challenge is not in the numbers or mathematically consistency.  It is in the semantics and the classifications of the objects within the domain.  The reality is that a large enterprise will have nothing short of 5 very discreet personae who on a good day can’t even agree on what to order for lunch.  Getting them all to come to common terms on the meaning of ‘x’ is much more difficult than getting them to understand that 5 is one more than 4.  This standardization of object within a domain is a prerequisite to measurement and must be addressed before one can impose a metric system across multiple communities of interest.

Research in this area [Star 2009] shows that standards are:

• Nested inside one another
• Distributed unevenly across the socio-culture landscape
• relative to communities of practice; one persons ideal standard can be another's nightmare
• increasingly interwoven in ways that are not always hierarchical
• consequential on the value systems of the community

The measurement is not in the numbers but in the understanding of the numbers. 

—tk