REVERSIM: A Game-Based Environment to Study Human Aspects in Hardware Reverse Engineering
- URL: http://arxiv.org/abs/2309.05740v2
- Date: Sun, 24 Mar 2024 07:49:17 GMT
- Title: REVERSIM: A Game-Based Environment to Study Human Aspects in Hardware Reverse Engineering
- Authors: Steffen Becker, René Walendy, Markus Weber, Carina Wiesen, Nikol Rummel, Christof Paar,
- Abstract summary: Hardware Reverse Engineering (HRE) is a technique for analyzing Integrated Circuits (ICs)
We have developed REVERSIM, a game-based environment that mimics realistic HRE subprocesses and can integrate standardized cognitive tests.
REVERSIM enables quantitative studies with easier-to-recruit non-experts to uncover cognitive factors relevant to HRE.
- Score: 5.468342362048975
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Hardware Reverse Engineering (HRE) is a technique for analyzing Integrated Circuits (ICs). Experts employ HRE for security-critical tasks, such as detecting Trojans or intellectual property violations. They rely not only on their experience and customized tools but also on their cognitive abilities. Conducting controlled experiments to assess the cognitive processes involved in HRE can open new avenues for hardware protection. However, HRE experts are largely unavailable for empirical research in real-world settings. To address this challenge, we have developed REVERSIM, a game-based environment that mimics realistic HRE subprocesses and can integrate standardized cognitive tests. REVERSIM enables quantitative studies with easier-to-recruit non-experts to uncover cognitive factors relevant to HRE, which can subsequently be validated with small expert samples. To evaluate the design of REVERSIM, the minimum requirements for successful participation, and its measurement capabilities, we conducted two studies: First, we performed semi-structured interviews with 14 professionals and researchers from the HRE domain, who attested to the comparability of REVERSIM to real-world HRE problems. Second, we conducted an online user study with 109 participants, demonstrating that they could engage in REVERSIM with low domain-specific prior knowledge. We provide refined screening criteria, derive fine-grained performance metrics, and successfully perform a cognitive test for mental speed in REVERSIM, thus contributing an important piece of the puzzle for the development of innovative hardware protection mechanisms.
Related papers
- Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [85.51252685938564]
Uncertainty quantification (UQ) is becoming increasingly recognized as a critical component of applications that rely on machine learning (ML)
As with other ML models, large language models (LLMs) are prone to make incorrect predictions, hallucinate'' by fabricating claims, or simply generate low-quality output for a given input.
We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines, and provides an environment for controllable and consistent evaluation of novel techniques.
arXiv Detail & Related papers (2024-06-21T20:06:31Z) - DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents [49.74065769505137]
We introduce DISCOVERYWORLD, the first virtual environment for developing and benchmarking an agent's ability to perform complete cycles of novel scientific discovery.
It includes 120 different challenge tasks spanning eight topics each with three levels of difficulty and several parametric variations.
We find that strong baseline agents, that perform well in prior published environments, struggle on most DISCOVERYWORLD tasks.
arXiv Detail & Related papers (2024-06-10T20:08:44Z) - Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning [53.3760591018817]
We propose a new benchmarking environment for aquatic navigation using recent advances in the integration between game engines and Deep Reinforcement Learning.
Specifically, we focus on PPO, one of the most widely accepted algorithms, and we propose advanced training techniques.
Our empirical evaluation shows that a well-designed combination of these ingredients can achieve promising results.
arXiv Detail & Related papers (2024-05-30T23:20:23Z) - Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science [65.77763092833348]
Intelligent agents powered by large language models (LLMs) have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines.
While their capabilities are promising, these agents also introduce novel vulnerabilities that demand careful consideration for safety.
This paper conducts a thorough examination of vulnerabilities in LLM-based agents within scientific domains, shedding light on potential risks associated with their misuse and emphasizing the need for safety measures.
arXiv Detail & Related papers (2024-02-06T18:54:07Z) - PentestGPT: An LLM-empowered Automatic Penetration Testing Tool [20.449761406790415]
Large Language Models (LLMs) have shown significant advancements in various domains.
We evaluate the performance of LLMs on real-world penetration testing tasks using a robust benchmark created from test machines with platforms.
We introduce PentestGPT, an LLM-empowered automatic penetration testing tool.
arXiv Detail & Related papers (2023-08-13T14:35:50Z) - Exploring Emerging Technologies for Requirements Elicitation Interview
Training: Empirical Assessment of Robotic and Virtual Tutors [0.0]
We propose an architecture for Requirements Elicitation Interview Training system based on emerging educational technologies.
We demonstrate the applicability of REIT through two implementations: Ro with a physical robotic agent and Vo with a virtual voice-only agent.
arXiv Detail & Related papers (2023-04-28T20:03:48Z) - Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement
Learning [55.2080971216584]
We present AIRS: Automatic Intrinsic Reward Shaping that intelligently and adaptively provides high-quality intrinsic rewards to enhance exploration in reinforcement learning (RL)
We develop an intrinsic reward toolkit to provide efficient and reliable implementations of diverse intrinsic reward approaches.
arXiv Detail & Related papers (2023-01-26T01:06:46Z) - Zero-Shot Retrieval with Search Agents and Hybrid Environments [8.017306481455778]
Current language models can learn symbolic query reformulation policies, in combination with traditional term-based retrieval, but fall short of outperforming neural retrievers.
We extend the previous learning to search setup to a hybrid environment, which accepts discrete query refinement operations, after a first-pass retrieval step via a dual encoder.
Experiments on the BEIR task show that search agents, trained via behavioral cloning, outperform the underlying search system based on a combined dual encoder retriever and cross encoder reranker.
arXiv Detail & Related papers (2022-09-30T13:50:25Z) - An Uncommon Task: Participatory Design in Legal AI [64.54460979588075]
We examine a notable yet understudied AI design process in the legal domain that took place over a decade ago.
We show how an interactive simulation methodology allowed computer scientists and lawyers to become co-designers.
arXiv Detail & Related papers (2022-03-08T15:46:52Z) - Multi-Agent Simulation for AI Behaviour Discovery in Operations Research [0.9137554315375919]
ACE0 is a platform for evaluating the suitability and viability of AI methods for behaviour discovery in multiagent simulations.
We provide a technical overview of the system architecture, describe a case study of behaviour discovery in the aerospace domain, and provide a qualitative evaluation of the system.
arXiv Detail & Related papers (2021-08-30T15:14:06Z) - Promoting the Acquisition of Hardware Reverse Engineering Skills [0.7487407411063094]
This research paper focuses on skill acquisition in Hardware Reverse Engineering (HRE)
Even though the scientific community and industry have a high demand for HRE experts, there is a lack of educational courses.
To investigate how novices acquire HRE skills in our course, we conducted two studies with students on different levels of prior knowledge.
arXiv Detail & Related papers (2021-05-28T10:45:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.