An innovative data collection method to eliminate the preprocessing phase in web usage mining

• URL: http://arxiv.org/abs/2501.04364v1
• Date: Wed, 08 Jan 2025 09:03:16 GMT
• Title: An innovative data collection method to eliminate the preprocessing phase in web usage mining
• Authors: Ozkan Canay, Umit Kocabicak,
• Abstract summary: The underlying data source for web usage mining (WUM) is commonly thought to be server logs.<n>This study proposes an innovative method for user tracking, session management, and collecting web usage data.<n>An application-based API has been developed with a different strategy from conventional client-side methods to obtain and process log data.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The underlying data source for web usage mining (WUM) is commonly thought to be server logs. However, access log files ensure quite limited data about the clients. Identifying sessions from this messy data takes a considerable effort, and operations performed for this purpose do not always yield excellent results. Also, this data cannot be used for web analytics efficiently. This study proposes an innovative method for user tracking, session management, and collecting web usage data. The method is mainly based on a new approach for using collected data for web analytics extraction as the data source in web usage mining. An application-based API has been developed with a different strategy from conventional client-side methods to obtain and process log data. The log data has been successfully gathered by integrating the technique into an enterprise web application. The results reveal that the homogeneous structured data collected and stored with this method is more convenient to browse, filter, and process than web server logs. This data stored on a relational database can be used effortlessly as a reliable data source for high-performance web usage mining activity, real-time web analytics, or a functional recommendation system.

Related papers

• Development of an Automated Web Application for Efficient Web Scraping: Design and Implementation [0.0]
  This paper presents the design and implementation of a user-friendly, automated web application that simplifies and optimize the web scraping process for non-technical users.<n>The application breaks down the complex task of web scraping into three main stages: fetching, extraction, and execution.<n>This automated tool not only enhances the efficiency of web scraping but also democratizes access to data extraction by empowering users of all technical levels to gather and manage data tailored to their needs.
  arXiv  Detail & Related papers  (2025-10-22T04:56:00Z)
• WebDS: An End-to-End Benchmark for Web-based Data Science [59.270670758607494]
  WebDS is the first end-to-end web-based data science benchmark.<n>It comprises 870 web-based data science tasks across 29 diverse websites.<n>WebDS sets the stage for significant advances in the development of practically useful LLM-based data science.
  arXiv  Detail & Related papers  (2025-08-02T06:39:59Z)
• Predictive modeling and anomaly detection in large-scale web portals through the CAWAL framework [0.0]
  This study presents an approach that uses session and page view data collected through the CAWAL framework, enriched through specialized processes, for advanced predictive modeling and anomaly detection in web usage mining applications. The results show that this approach offers detailed insights into user behavior and system performance metrics, making it a reliable solution for improving large-scale web portals' efficiency, reliability, and scalability.
  arXiv  Detail & Related papers  (2025-02-01T12:21:59Z)
• Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models [64.28420991770382]
  Data-Juicer 2.0 is a data processing system backed by data processing operators spanning text, image, video, and audio modalities.<n>It supports more critical tasks including data analysis, annotation, and foundation model post-training.<n>It has been widely adopted in diverse research fields and real-world products such as Alibaba Cloud PAI.
  arXiv  Detail & Related papers  (2024-12-23T08:29:57Z)
• Capturing and Anticipating User Intents in Data Analytics via Knowledge Graphs [0.061446808540639365]
  This work explores the usage of Knowledge Graphs (KG) as a basic framework for capturing a human-centered manner complex analytics. The data stored in the generated KG can then be exploited to provide assistance (e.g., recommendations) to the users interacting with these systems.
  arXiv  Detail & Related papers  (2024-11-01T20:45:23Z)
• Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning [3.623224034411137]
  offline multi-agent reinforcement learning (MARL) is an exciting direction of research that uses static datasets to find optimal control policies for multi-agent systems. Though the field is by definition data-driven, efforts have thus far neglected data in their drive to achieve state-of-the-art results. We show how the majority of works generate their own datasets without consistent methodology and provide sparse information about the characteristics of these datasets.
  arXiv  Detail & Related papers  (2024-09-18T14:13:24Z)
• An Integrated Data Processing Framework for Pretraining Foundation Models [57.47845148721817]
  Researchers and practitioners often have to manually curate datasets from difference sources. We propose a data processing framework that integrates a Processing Module and an Analyzing Module. The proposed framework is easy to use and highly flexible.
  arXiv  Detail & Related papers  (2024-02-26T07:22:51Z)
• LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection [73.69399219776315]
  We propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters.
  arXiv  Detail & Related papers  (2024-01-09T12:55:21Z)
• Log Parsing Evaluation in the Era of Modern Software Systems [47.370291246632114]
  We focus on one integral part of automated log analysis, log parsing, which is the prerequisite to deriving any insights from logs. Our investigation reveals problematic aspects within the log parsing field, particularly its inefficiency in handling heterogeneous real-world logs. We propose a tool, Logchimera, that enables estimating log parsing performance in industry contexts.
  arXiv  Detail & Related papers  (2023-08-17T14:19:22Z)
• Demonstration of InsightPilot: An LLM-Empowered Automated Data Exploration System [48.62158108517576]
  We introduce InsightPilot, an automated data exploration system designed to simplify the data exploration process. InsightPilot automatically selects appropriate analysis intents, such as understanding, summarizing, and explaining. In brief, an IQuery is an abstraction and automation of data analysis operations, which mimics the approach of data analysts.
  arXiv  Detail & Related papers  (2023-04-02T07:27:49Z)
• Better Generative Replay for Continual Federated Learning [20.57194599280318]
  Federated learning is a technique that enables a centralized server to learn from distributed clients via communications. In this paper, we introduce the problem of continual federated learning, where clients incrementally learn new tasks and history data cannot be stored. We propose our FedCIL model with two simple but effective solutions: model consolidation and consistency enforcement.
  arXiv  Detail & Related papers  (2023-02-25T06:26:56Z)
• On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds [0.0]
  We propose a collaborative approach for sharing anonymized workload execution traces among users. We mining them for general patterns, and exploiting clusters of historical workloads for future optimizations.
  arXiv  Detail & Related papers  (2021-11-16T20:11:36Z)
• Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts [52.9168275057997]
  This paper presents Bellamy, a novel modeling approach that combines scale-outs, dataset sizes, and runtimes with additional descriptive properties of a dataflow job. We evaluate our approach on two publicly available datasets consisting of execution data from various dataflow jobs carried out in different environments.
  arXiv  Detail & Related papers  (2021-07-29T11:57:38Z)
• DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier [58.979104709647295]
  We bridge the gap between the abundance of available data and lack of relevant data, for the future learning tasks of a trained network. We use the available data, that may be an imbalanced subset of the original training dataset, or a related domain dataset, to retrieve representative samples. We demonstrate that data from a related domain can be leveraged to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2019-12-27T02:05:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.