One Reason Security Is So Hard? Really Bad Reports.

Security is hard. Everyone knows that. The question is: why?

After all, our understanding of cyber threats improves with each day. The tools we use to secure our systems have also improved over time – antivirus software, firewalls, application firewalls, intrusion detection, data leak prevention, and so on. And yet, when we look at the data, there’s not much evidence that better understanding and better tools are leading to better security.

Painting of Flowers
Jon Grier says reports should be ‘security paintings’ that help users see the whole picture.

According to Jonathan Grier, an independent security consultant, the answer to the question ‘Why aren’t we getting better at stopping attacks and protecting data?’ is that we’re not doing a good job of learning from the data we have.

In a conversation with The Security Ledger, Grier, the founder of Grier Forensics,  said that, despite a wealth of security data, the security industry’s approach to analyzing it is immature.

Grier likes working on the cutting edge of computer forensics and application security. At last year’s Black Hat Briefings, he gave a talk on using stochastic forensics to spot insider data theft. His latest project is developing a notion he calls “security paintings:” a way to distill disparate security metrics into easy to digest information.

Grier told The Security Ledger he came to the idea while working as a consultant for the firm Web Shadow, which makes a parental monitoring software. Like many security firms that need to make sense of voluminous data, Web Shadow relied heavily on data-heavy reports and graphs to distill the information it collected from teens’ web sessions for its users – the parents.

Jonathan Grier
Jonathan Grier, Grier Forensics

“I mean, I’m a security professional and I had to really squint at them to figure out what was going on. Now let’s imagine that I’m not a security professional, I’m a parent, and that I’m only spending a few minutes a day. I’m not going to see anything looking at these reports.”

Web Shadow, he said, was making a mistake common to many security software firms when it designed its activity reports.”These vendors are either losing me in an ocean of low-level detail or dumbing me down in high level detail,” he said. “And neither are very useful for parents, or for security professionals in other areas, too.” Both failed users.

Grier set out to look at reports and how people use them. He consulted experts in usability engineering and cognitive science. His conclusion: more than anything else, “people want patterns.”

But most security products had difficulty discerning patterns from low-level data. Instead, he found the same problems with presentation that he encountered at Web Shadow time and again. “Either you’ll see this audit trail, which is this data dump of an entire table with record-field, record-field, record field with no sense of higher patterns, or you get a pie chart.”

Webshadow Screens-Report
Grier worked with Web Shadow to distill data from their parental monitoring software into easy to digest messages.

Why is that? After all, writing an engine to spot malware or suspicious behavior is much harder than creating a report to present the data it has generated, right?

Grier says that may be the problem: software developers, rather than their users, are tasked with designing reports for their products. “It’s like the inmates running the asylum,” Grier said.

“The developers are the people who are the most familiar with the tables and data and database schema are developing the reports, and they develop reports based on what they’re thinking about. Their perspective is ‘I’m a developer. I’m storing the data in this table. Let me show this table to you!’ They’re not thinking about it from the perspective of a security professional which is ‘I want to know X. What is the best way for me to present it to you?’

Grier says he’s identified some common problems with typical reports:

“High level data is interesting, low-level data is credible,” Grier said. “We need both.” At a minimum, reports should draw the reader’s attention to the information that’s the most important to them.  Humans have evolved to look for anomalies in their environment and to focus on them, but reports often fail to identify patterns in the data flag anomalies, he said.

Software designers often view the problem as one of having data and trying to figure out how to display it. But Grier rejects that premise. “I want to turn the question around and ask ‘As a security administrator, what am I trying to do?’ ‘What conclusions do I want to form?’ ”

Grier uses the image of a flower bouquet to drive home his point. “Nobody looks at (the bouquet) and says ‘I see seven carnations and five daffodils and two tulips,'” he argues. Your eye takes the data in and assembles it into a whole.”

Good security reports should do the same thing – taking low-level events and assembling them into mid- and then high level groupings that make sense.

If nothing else, reports should do the job of making data understandable and easy to act on. “The whole point of a report is to get a human involved,” he said. “One of the basic observations I made is that, for any kind of security control, if its obvious enough for a computer to make a decision about it, you don’t need a report,” Grier said. “Right now, our technology isn’t nearly good enough to always classify things correctly. But I found that if you give people enough background in the report, they’d end up making the right decision anyway.”

Grier said the problem of how best to present security data isn’t much different from that faced in other fields, such as finance, or magic, for that matter.

“When I was a kid, I very into magic tricks,” Grier says. “I had a magic teacher who told me ‘If you have 100 ways of finding a card, and one way of presenting it, you have one trick. But if you have one way of finding a card and 100 ways of presenting it, you have 100 tricks.”