Renaud Bidou from RADWare
Talk: How to test an IPS

The basic story here was ‘IPS buyer beware’. How can you properly evaluate an IPS? You must first and foremost define your success criteria and this criteria must be measurable and reasonable. No your IPS is not going to cure all your ills.

Can IPS’s be evaded? YES, get over it! Anyone who will argue otherwise should take some of their security budget and find a good therapist.

The buyer is at a tremendous disadvantage because of his or her lack of expertise in this domain. I heard this so many times from customers that I’m sick of hearing it. Where can the buyer find the facts? Dude, there is no Consumer Reports for this stuff. And even if there were, the success of this type of device is so dependant on YOUR environment, I am not sure they could do anything but validate the vendors claims. Validating claims and it being success in your environment are completely different problems.

Without getting in to the details, MS03-026 which is an oldie-but-goodie was used in his tests. He looked at three vendors and all of them had problems once active evasion techniques were applied. There also was issues with throughput.

Most of these tests can be found at:
http://www.iv2-technologies.com/~rbidou
Thanks dude for posting facts.
Here we are, many code releases later, and still accuracy issues and performance plague most IPS vendors and customers need to at the very least know the facts.

http://www.iv2-technologies.com/~rbidou/HowToTestAnIPS.pdf

In closing, I would like to say something that bugs the living daylights out of me. During the detailed section of the talk on ‘how to test’, he kept on highlighting principles and guidelines that are fundamental scientific methods. This is the same stuff you learned in 6th grade. Don’t you find it odd that pointing these things out is seen as valuable information? The sad thing is that most people cut corners and do not have the discipline to carry out a test that has integrity. I commend Renaud for his attention to detail in testing. Look, if you can’t trust your test environment and methods, the results are just worthless. Wait, they are worth something: they inform you that you need to improve your testing methodology.

--tk