That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design

• URL: http://arxiv.org/abs/2411.10053v1
• Date: Fri, 15 Nov 2024 09:11:10 GMT
• Title: That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design
• Abstract summary: In 2020, we introduced a deep reinforcement learning method capable of generating superhuman chip layouts. A non-peer-reviewed paper at ISPD 2023 questioned its performance claims, despite failing to run our method as described in Nature. We publish this response to ensure that no one is wrongly discouraged from innovating in this impactful area.
• Score: 6.383127282050037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In 2020, we introduced a deep reinforcement learning method capable of generating superhuman chip layouts, which we then published in Nature and open-sourced on GitHub. AlphaChip has inspired an explosion of work on AI for chip design, and has been deployed in state-of-the-art chips across Alphabet and extended by external chipmakers. Even so, a non-peer-reviewed invited paper at ISPD 2023 questioned its performance claims, despite failing to run our method as described in Nature. For example, it did not pre-train the RL method (removing its ability to learn from prior experience), used substantially fewer compute resources (20x fewer RL experience collectors and half as many GPUs), did not train to convergence (standard practice in machine learning), and evaluated on test cases that are not representative of modern chips. Recently, Igor Markov published a meta-analysis of three papers: our peer-reviewed Nature paper, the non-peer-reviewed ISPD paper, and Markov's own unpublished paper (though he does not disclose that he co-authored it). Although AlphaChip has already achieved widespread adoption and impact, we publish this response to ensure that no one is wrongly discouraged from innovating in this impactful area.

Related papers

• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• The Theorems of Dr. David Blackwell and Their Contributions to Artificial Intelligence [0.0]
  This survey examines three of his most consequential theoretical results the Rao Blackwell theorem, the Blackwell Approachability theorem, and the Blackwell Informativeness theorem.<n>We show that these results, developed primarily in the 1940s and 1950s, remain technically live across modern subfields including Markov Chain Monte Carlo inference, autonomous mobile robot navigation (SLAM), generative model training, no-regret online learning, reinforcement learning from human feedback (RLHF), large language model alignment, and information design.
  arXiv  Detail & Related papers  (2026-04-08T03:01:58Z)
• Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs [75.62489398666644]
  We show that an emphautoresearch-style pipeline citepkarpathy2026autoresearch powered by Claude Code discovers novel white-box adversarial attack.<n>textbfsignificantly outperform all existing (30+ methods) in jailbreaking and prompt injection evaluations.
  arXiv  Detail & Related papers  (2026-03-25T16:50:56Z)
• SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization [52.635237306338574]
  We introduce SE-Bench, a diagnostic environment that obfuscates the NumPy library and its API doc into a pseudo-novel package with randomized identifiers.<n>Agents are trained to internalize this package and evaluated on simple coding tasks without access to documentation.<n>Our investigation reveals three insights: (1) the Open-Book Paradox, where training with reference documentation inhibits retention, requiring "Closed-Book Training" to force knowledge compression into weights; (2) the RL Gap, where standard RL fails to internalize new knowledge completely due to PPO clipping and negative gradients; and (3) the viability of Self-Play for internalization, proving models
  arXiv  Detail & Related papers  (2026-02-04T17:58:32Z)
• AlphaEvolve: A coding agent for scientific and algorithmic discovery [63.13852052551106]
  We present AlphaEvolve, an evolutionary coding agent that substantially enhances capabilities of state-of-the-art LLMs.<n>AlphaEvolve orchestrates an autonomous pipeline of LLMs, whose task is to improve an algorithm by making direct changes to the code.<n>We demonstrate the broad applicability of this approach by applying it to a number of important computational problems.
  arXiv  Detail & Related papers  (2025-06-16T06:37:18Z)
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage [62.049868205196425]
  AutoReproduce is a framework capable of automatically reproducing experiments described in research papers in an end-to-end manner.<n>Results show that AutoReproduce achieves an average performance gap of $22.1%$ on $89.74%$ of the executable experiment runs.
  arXiv  Detail & Related papers  (2025-05-27T03:15:21Z)
• PaperBench: Evaluating AI's Ability to Replicate AI Research [3.4567792239799133]
  PaperBench is a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch. PaperBench contains 8,316 individually gradable tasks.
  arXiv  Detail & Related papers  (2025-04-02T15:55:24Z)
• START: Self-taught Reasoner with Tools [51.38785489790888]
  We introduce START (Self-Taught Reasoner with Tools), a tool-integrated long Chain-of-thought (CoT) reasoning LLM. START is capable of performing complex computations, self-checking, exploring diverse methods, and self-ging. It significantly outperforms the base QwQ-32B and achieves performance comparable to the state-of-the-art open-weight model R1-Distill-Qwen-32B.
  arXiv  Detail & Related papers  (2025-03-06T17:11:51Z)
• Benchmarking End-To-End Performance of AI-Based Chip Placement Algorithms [77.71341200638416]
  ChiPBench is a benchmark designed to evaluate the effectiveness of AI-based chip placement algorithms. We have gathered 20 circuits from various domains (e.g., CPU, GPU, and microcontrollers) for evaluation. Results show that even if intermediate metric of a single-point algorithm is dominant, the final PPA results are unsatisfactory.
  arXiv  Detail & Related papers  (2024-07-03T03:29:23Z)
• Theoretically Achieving Continuous Representation of Oriented Bounding Boxes [64.15627958879053]
  This paper endeavors to completely solve the issue of discontinuity in Oriented Bounding Box representation. We propose a novel representation method called Continuous OBB (COBB) which can be readily integrated into existing detectors. For fairness and transparency of experiments, we have developed a modularized benchmark based on the open-source deep learning framework Jittor's detection toolbox JDet for OOD evaluation.
  arXiv  Detail & Related papers  (2024-02-29T09:27:40Z)
• Simulation of IBM's kicked Ising experiment with Projected Entangled Pair Operator [71.10376783074766]
  We perform classical simulations of the 127-qubit kicked Ising model, which was recently emulated using a quantum circuit with error mitigation. Our approach is based on the projected entangled pair operator (PEPO) in the Heisenberg picture. We develop a Clifford expansion theory to compute exact expectation values and use them to evaluate algorithms.
  arXiv  Detail & Related papers  (2023-08-06T10:24:23Z)
• A LLM Assisted Exploitation of AI-Guardian [57.572998144258705]
  We evaluate the robustness of AI-Guardian, a recent defense to adversarial examples published at IEEE S&P 2023. We write none of the code to attack this model, and instead prompt GPT-4 to implement all attack algorithms following our instructions and guidance. This process was surprisingly effective and efficient, with the language model at times producing code from ambiguous instructions faster than the author of this paper could have done.
  arXiv  Detail & Related papers  (2023-07-20T17:33:25Z)
• ChiPFormer: Transferable Chip Placement via Offline Decision Transformer [35.69382855465161]
  reinforcement learning can improve human performance in chip placement. ChiPFormer enables learning a transferable placement policy from fixed offline data. ChiPFormer achieves significantly better placement quality while reducing the runtime by 10x.
  arXiv  Detail & Related papers  (2023-06-26T14:59:56Z)
• The False Dawn: Reevaluating Google's Reinforcement Learning for Chip Macro Placement [1.4803764446062861]
  Reinforcement learning for physical design of silicon chips in a Google 2021 Nature paper stirred controversy due to poorly documented claims. We show that Google RL lags behind (i) human designers, (ii) a well-known algorithm (Simulated Annealing), and (iii) generally-available commercial software, while being slower. Crosschecked data indicate that the integrity of the Nature paper is substantially undermined owing to errors in conduct, analysis and reporting.
  arXiv  Detail & Related papers  (2023-06-16T05:32:24Z)
• Tricking AI chips into Simulating the Human Brain: A Detailed Performance Analysis [0.5354801701968198]
  We evaluate multiple, cutting-edge AI-chips (Graphcore IPU, GroqChip, Nvidia GPU with inferior Cores and Google TPU) for brain simulation. Our performance analysis reveals that the simulation problem maps extremely well onto the GPU and TPU architectures. The GroqChip outperforms both platforms for small networks but, due to implementing some floating-point operations at reduced accuracy, is found not yet usable for brain simulation.
  arXiv  Detail & Related papers  (2023-01-31T13:51:37Z)
• Bilevel Continual Learning [18.293397644865454]
  This paper presents Bilevel Continual Learning (BiCL), a general framework for continual learning. BiCL fuses bilevel optimization and recent advances in meta-learning for deep neural networks. Experimental results show BiCL provides competitive performance in terms of accuracy for the current task.
  arXiv  Detail & Related papers  (2020-11-02T18:06:42Z)
• Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training [118.10946662410639]
  We propose a novel Self-PU learning framework, which seamlessly integrates PU learning and self-training. Self-PU highlights three "self"-oriented building blocks: a self-paced training algorithm that adaptively discovers and augments confident examples as the training proceeds. We study a real-world application of PU learning, i.e., classifying brain images of Alzheimer's Disease.
  arXiv  Detail & Related papers  (2020-06-22T17:53:59Z)
• Chip Placement with Deep Reinforcement Learning [40.952111701288125]
  We present a learning-based approach to chip placement. Unlike prior methods, our approach has the ability to learn from past experience and improve over time. In under 6 hours, our method can generate placements that are superhuman or comparable on modern accelerator netlists.
  arXiv  Detail & Related papers  (2020-04-22T17:56:07Z)
• Robust Pruning at Initialization [61.30574156442608]
  A growing need for smaller, energy-efficient, neural networks to be able to use machine learning applications on devices with limited computational resources. For Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned.
  arXiv  Detail & Related papers  (2020-02-19T17:09:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.