RE: searching logs for key phrases

Vos, Arjan (Vos.Arjan@kpmg.nl)
Mon, 2 Dec 1996 11:48:00 +0100 

>
Guido said:

>> It doesn't matter how much log files you process. It is a general principle
>> that you should drop all non-interesting ones after which the interesting
>> ones remain.

Sorry, I must have misunderstood you in the first place. I thought you
were explicitly filtering out non-interesting events. 

>> It's like defensive programming. When you write programs only accepting
>> certain strings, you should only allow the valid ones. If you forget
>>one of the valid strings, it does never break security. On the other hand,
>       >when you disallow the non-valid ones and you forget one you do break
>security.

true, but sometimes it gets more complicated that that.You'd also want
to explicitly check on the validity of strings. It is possible to break
security while passing a valid string, althoughthat's a matter of how
you define "valid strings" regarding your security policy (if you have
one:-)). But hey, now I'm nitpicking, I do agree with you....

>>
>>In you case, when you forget a few loglines indicating a hacker you will not
>>find him. As you already said, you have large logfiles. This means you
>>will never inspect them line by line.
>>
>>I do admit that making the set of regexp's filtering out the non-interesting
>>ones is a hard process but it's worth it.
>>
>>Here at Origin, We have like 160Meg per day of log files on only a
>>small amount of firewalls and we do it the way I described with lots of
>>success.
>>
>>Of course you need the correct tools to efficiently filter the logfiles
>>but that's another story.

yep, and now there's where Intrusion Detection Systems come in.....
There are a lot of interesting discussions going on in this fiels of
research, but still nothing really gets done. I've been dabbling with
some things as well (expert systems, PROLOG) trying to "intelligently"
process logging in order to detect attacks. Also refer to
http://doe-is.llnl.gov/nitb/ids.html.
>>
>>-Guido

Arjan Vos
KPMG EDP Auditors

p.s. Guido: ik heb begrepen dat we een gemeenschappelijke "kennis"
hebben?
>