Log management has become one of the biggest use cases for big data solutions and ubiquitous across organizations as departments tap into log data to supply them with supplemental to mission critical information. The following represent some of the more common and popular examples of how people are using log data in organizations today.
Log data is invaluable for managing, maintaining, and troubleshooting IT systems. Common applications for log data in the IT organization includes:
- Monitoring across systems to detect particular log events and patterns in log data
- Monitoring in real-time for anomalies or inactivity to gauge system health
- Identifying performance or configuration issues
- Drilling down on data to gain insight and perform root cause analysis when failures occur
- Meet operational objectives and SLAs
Log data helps technical personnel quickly drill down on application related issues including:
- Pinpointing areas of poor performance
- Assessing application health and troubleshooting
- Diagnosing and identifying the root cause of application installation and run-time errors
Regulator Compliance and SIEM
Automating the gathering, analysis, and correlation of data from multiple security systems and devices across the organization streamlines the IT department, making them more efficient and effective.
Security Information and Event Management (SIEM) products provide many of the features required for log management but provide an additional unique layer of functionality for correlation, alerting, real-time analysis and workflow. SIEM also allows the import of non-event-driven information to provide a more comprehensive view of security and vulnerabilities. SIEM only becomes more vital as IT continues to rely more on the public cloud.
Typical sources of SIEM data are:
- AWS Config & CloudTrail data
- Azure equivalents
- HIDS/NIDS systems like OSSec, AlertLogic
Typical SIEM features include:
- Log and contextual data collection
- Normalization and tagging or categorization
- Data correlation
- Notifications and alerts
- Event prioritization
- Workflow Engine
HIPAA, PCI, MiFID, SOX, FISMA, and others regulatory requirements require the tracking of access to the various systems that contain regulated data. Logs serve as an economical way to meet key regulatory compliance mandates
Typical events tracked include:
- Account logon attempts
- Account management events
- Directory service access
- System events
- Changes in policy
Both SIEM and log management play an important role in maintaining compliance with industry regulations.
“Data are becoming the new raw material of business.” – Craig Mundie, Head of Research & Strategy, Microsoft
Log data typically contains a treasure trove of business insights. Critical business information can be derived from log data and alerts configured when specific business goals are achieved or criteria are met.
- Business process health
- Transactional data (transactions/sec, etc.)
- Customer SLAs
- Revenue per hour
Log file analysis can be used as a strategic advantage for digital marketers to gather insights and analyze their campaign’s impact on visibility, traffic, conversions, and sales. In addition, log file analysis can help reveal new areas for SEO improvements as they can reveal exactly how bots are crawling your website.
- Understand which pages are useful and useless
- Reveal which pages Google crawls and which pages they don’t
- Make better decisions and create better forecasting
- Get alerts on important events or patterns
- Monitor websites more effectively
Various departments in organizations are finding log data to be invaluable to their work and are implementing strategies to use that data advantageously. If you’re not using log data in your department today, using the information in this post is a great place to start. To get more information about how to take log data to the next level be sure to download our Essential Guide to Log Management and stay tuned for our next post on centralized log management!
Join Our AWS Community
Bringing the AWS computing community together to connect and learn from AWS professionals.