Web log analysis software

Abstract: Logs, which record valuable system runtime information, have been widely employed in Web service management by service providers and users. A typical log analysis based Web service management procedure is to first parse raw log messages because of their unstructured format, and then apply data mining models to extract critical system behavior information, which can assist Web service management. Most of the existing log parsing methods focus on offline, batch processing of logs. However, as the volume of logs increases rapidly, model training of offline log parsing methods, which employs all existing logs after log collection, becomes time consuming. To address this problem, we propose an online log parsing method, namely Drain, that can parse logs in a streaming and timely manner. To accelerate the parsing process, Drain uses a fixed depth parse tree, which encodes specially designed rules for parsing. We evaluate Drain on five real-world log data sets with more than 10 million raw log messages. The experimental results show that Drain has the highest accuracy on four data sets, and comparable accuracy on the remaining one. Besides, Drain obtains 51.85%~81.47% improvement in running time compared with the state-of-the-art online parser. We also conduct a case study on an anomaly detection task using Drain in the parsing step, which determines the effectiveness of Drain in log analysis.

Abstract: A method of authenticating a Web client to a Web server connectable to a distributed file system of a distributed computing environment. The distributed computing environment includes a security service for returning a credential to a user authenticated to access the distributed file system. In response to receipt by the Web server of a user id and password from the Web client, a login protocol is executed with the security service. If the user can be authenticated, a credential is stored in a database of credentials associated with authenticated users. The Web server then returns to the Web client a persistent client state object having a unique identifier therein. This object, sometimes referred to as a cookie, is then used to enable the Web client to browse Web documents in the distributed file system. In particular, when the Web client desires to make a subsequest request to the distributed file system, the persistent client state object including the identifier is used in lieu of the user's id and password, which makes the session much more secure. In this operation, the cookie identifier is used as a pointer into the credential storage table, and the credential is then retrieved and used to facilitate multiple file accessess from the distributed file system. At the same time, the Web client may obtain access to Web server (as opposed to distributed file system) documents via conventional user id and password in an HTTP request.

Abstract: The present invention generally relates to log message processing such that events can be detected and alarms can be generated. For example, log messages are generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). Often, relatively large numbers of logs are generated from these platforms in different formats. A log manager described herein collects such log data using various protocols (e.g., Syslog, SNMP, SMTP, etc.) to determine events. That is, the log manager may communicate with the network platforms using appropriate protocols to collect log messages therefrom. The log manager may then determine events (e.g., unauthorized access, logins, etc.) from the log data and transfer the events to an event manager. The event manager may analyze the events and determine whether alarms should be generated therefrom.

Abstract: A method for tracking usage patterns of users of hyper-media systems, such as the World-Wide Web, that creates a usage log on a user's client computer and periodically transmits the usage log from the user's client machine to a usage tracking server computer to be incorporated in an overall usage log for a given information server computer. Alternatively, proxy server may be connected between a client computer and an information server with the proxy server acting as a client to the information server and creating a usage log of the user's client computer access to the information server computer to be sent to usage tracking computer. Each time a user connects to a Web site, the client computer or proxy server creates a usage log that records all objects retrieved from that particular Web site and other attributes of user processing such as time spent viewing an object, amount of an object viewed, etc. Periodically, upon the occurrence of one of more predefined events that signify the end of a session, computer or proxy server saves the usage log into a single, compressed file and transmits the resulting file to the server. Events that signify the end of a session and trigger the transmission of a usage log from the user's client computer or proxy server computer to the usage tracking server computer may include a predetermined time interval, accessing a predetermined number of objects, application program or operating system shut down, connecting to a different Web site, and/or modem shut down.