Interpretable Company Similarity with Sparse Autoencoders

• URL: http://arxiv.org/abs/2412.02605v3
• Date: Sat, 24 May 2025 15:21:24 GMT
• Title: Interpretable Company Similarity with Sparse Autoencoders
• Authors: Marco Molinari, Victor Shao, Luca Imeneo, Mateusz Mikolajczak, Vladimir Tregubiak, Abhimanyu Pandey, Sebastian Kuznetsov Ryder Torres Pereira,
• Abstract summary: Sparse Autoencoders (SAEs) have shown promise in enhancing the interpretability of Large Language Models (LLMs)<n>We benchmark SAE features against SIC-codes, Industry codes, and Embeddings.<n>Our results demonstrate SAE features surpass sector classifications and embeddings in capturing fundamental company characteristics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Determining company similarity is a vital task in finance, underpinning risk management, hedging, and portfolio diversification. Practitioners often rely on sector and industry classifications such as SIC and GICS codes to gauge similarity, the former being used by the U.S. Securities and Exchange Commission (SEC), and the latter widely used by the investment community. Since these classifications lack granularity and need regular updating, using clusters of embeddings of company descriptions has been proposed as a potential alternative, but the lack of interpretability in token embeddings poses a significant barrier to adoption in high-stakes contexts. Sparse Autoencoders (SAEs) have shown promise in enhancing the interpretability of Large Language Models (LLMs) by decomposing Large Language Model (LLM) activations into interpretable features. Moreover, SAEs capture an LLM's internal representation of a company description, as opposed to semantic similarity alone, as is the case with embeddings. We apply SAEs to company descriptions, and obtain meaningful clusters of equities. We benchmark SAE features against SIC-codes, Industry codes, and Embeddings. Our results demonstrate that SAE features surpass sector classifications and embeddings in capturing fundamental company characteristics. This is evidenced by their superior performance in correlating logged monthly returns - a proxy for similarity - and generating higher Sharpe ratios in co-integration trading strategies, which underscores deeper fundamental similarities among companies. Finally, we verify the interpretability of our clusters, and demonstrate that sparse features form simple and interpretable explanations for our clusters.

Related papers

• Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models [64.58262227709842]
  ARISE (Attention-weighted Representation with Integrated Semantic Embeddings) is presented.<n>It builds semantic-aware representations that complement the metric space of categorical data for accurate clustering.<n>Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts.
  arXiv  Detail & Related papers  (2026-01-03T11:37:46Z)
• STRAPSim: A Portfolio Similarity Metric for ETF Alignment and Portfolio Trades [0.5847369405576658]
  STRAPSim is a novel method that computes portfolio similarity by matching constituents based on semantic similarity.<n>We benchmark our approach against Jaccard, weighted Jaccard, as well as BERTScore-inspired variants across public classification, regression, and recommendation tasks.<n> Empirical results show that our method consistently outperforms baselines in predictive accuracy and ranking alignment.
  arXiv  Detail & Related papers  (2025-09-29T00:57:41Z)
• Mapping Overlaps in Benchmarks through Perplexity in the Wild [8.321258152814986]
  We develop signatures of capacity familiarity to characterize large language model (LLM) benchmarks and their meaningful overlaps.<n>Our analysis situates signatures in relation to both the semantic similarity of benchmark questions and the correlation of model performance.<n>Ultimately, we identify cross-functional overlaps across logic, math, language, instruction following, and world modeling, with coding emerging as the least overlapping domain.
  arXiv  Detail & Related papers  (2025-09-27T20:23:13Z)
• Analysis of Variational Sparse Autoencoders [1.675385127117872]
  We investigate whether incorporating variational methods into SAE architectures can improve feature organization and interpretability.<n>We introduce the Variational Sparse Autoencoder (vSAE), which replaces deterministic ReLU gating with sampling from learned Gaussian posteriors.<n>Our findings suggest that naive application of variational methods to SAEs does not improve feature organization or interpretability.
  arXiv  Detail & Related papers  (2025-09-26T23:09:56Z)
• NDCG-Consistent Softmax Approximation with Accelerated Convergence [67.10365329542365]
  We propose novel loss formulations that align directly with ranking metrics.<n>We integrate the proposed RG losses with the highly efficient Alternating Least Squares (ALS) optimization method.<n> Empirical evaluations on real-world datasets demonstrate that our approach achieves comparable or superior ranking performance.
  arXiv  Detail & Related papers  (2025-06-11T06:59:17Z)
• Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval [13.31210969917096]
  We propose a novel interpretability framework for Dense Passage Retrieval (DPR) models.<n>We generate natural language descriptions for each latent concept, enabling human interpretations of both the dense embeddings and the query-document similarity scores of DPR models.<n>We show that Concept-Level Sparse Retrieval (CL-SR) achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches.
  arXiv  Detail & Related papers  (2025-05-28T02:50:17Z)
• Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs [34.52554840674882]
  This paper argues that mechanistic interpretability should prioritize feature consistency in SAEs.<n>We propose using the Pairwise Dictionary Mean Correlation Coefficient as a practical metric to operationalize consistency.
  arXiv  Detail & Related papers  (2025-05-26T17:31:36Z)
• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing [6.836374436707495]
  Sparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large language model (LLM) activations. One alternative source of evidence would be demonstrating that SAEs improve performance on downstream tasks beyond existing baselines. We test this by applying SAEs to the real-world task of LLM activation probing in four regimes.
  arXiv  Detail & Related papers  (2025-02-23T18:54:15Z)
• Estimating Commonsense Plausibility through Semantic Shifts [66.06254418551737]
  We propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts.<n> Evaluations on two types of fine-grained commonsense plausibility estimation tasks show that ComPaSS consistently outperforms baselines.
  arXiv  Detail & Related papers  (2025-02-19T06:31:06Z)
• Sparse Autoencoder Features for Classifications and Transferability [11.2185030332009]
  We analyze Sparse Autoencoders (SAEs) for interpretable feature extraction from Large Language Models (LLMs)<n>Our framework evaluates (1) model-layer selection and scaling properties, (2) SAE architectural configurations, including width and pooling strategies, and (3) the effect of binarizing continuous SAE activations.
  arXiv  Detail & Related papers  (2025-02-17T02:30:45Z)
• Supervised Similarity for High-Yield Corporate Bonds with Quantum Cognition Machine Learning [0.8706730566331037]
  We investigate the application of quantum cognition machine learning (QCML) to distance metric learning in corporate bond markets. We show that QCML outperforms classical tree-based models in high-yield (HY) markets, while giving comparable or better performance in investment grade (IG) markets.
  arXiv  Detail & Related papers  (2025-02-03T16:28:44Z)
• Post-hoc Probabilistic Vision-Language Models [51.12284891724463]
  Vision-language models (VLMs) have found remarkable success in classification, retrieval, and generative tasks. We propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our results show promise for safety-critical applications of large-scale models.
  arXiv  Detail & Related papers  (2024-12-08T18:16:13Z)
• Addressing Uncertainty in LLMs to Enhance Reliability in Generative AI [47.64301863399763]
  We present a dynamic semantic clustering approach inspired by the Chinese Restaurant Process. We quantify uncertainty of Large Language Models (LLMs) on a given query by calculating entropy of the generated semantic clusters. We propose leveraging the (negative) likelihood of these clusters as the (non)conformity score within Conformal Prediction framework.
  arXiv  Detail & Related papers  (2024-11-04T18:49:46Z)
• Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models [26.748765050034876]
  Specialized Sparse Autoencoders (SSAEs) illuminate elusive dark matter features by focusing on specific. We show that SSAEs effectively capture subdomain tail concepts, exceeding the capabilities of general-purpose SAEs. We showcase the practical utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs achieve a 12.5% increase in worst-group classification accuracy when applied to remove spurious gender information.
  arXiv  Detail & Related papers  (2024-11-01T17:09:34Z)
• FRAG: Toward Federated Vector Database Management for Collaborative and Secure Retrieval-Augmented Generation [1.3824176915623292]
  This paper introduces textitFederated Retrieval-Augmented Generation (FRAG), a novel database management paradigm tailored for the growing needs of retrieval-augmented generation (RAG) systems. FRAG enables mutually-distrusted parties to collaboratively perform Approximate $k$-Nearest Neighbor (ANN) searches on encrypted query vectors and encrypted data stored in distributed vector databases.
  arXiv  Detail & Related papers  (2024-10-17T06:57:29Z)
• SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI [58.29510889419971]
  Existing benchmarks for evaluating the security risks and capabilities of code-generating large language models (LLMs) face several key limitations.<n>We introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations.<n>Applying this framework to Python, C/C++, and Java, we build SeCodePLT, a dataset of more than 5.9k samples spanning 44 CWE-based risk categories and three security capabilities.
  arXiv  Detail & Related papers  (2024-10-14T21:17:22Z)
• Federated Contrastive Learning for Personalized Semantic Communication [55.46383524190467]
  We design a federated contrastive learning framework aimed at supporting personalized semantic communication. FedCL enables collaborative training of local semantic encoders across multiple clients and a global semantic decoder owned by the base station. To tackle the semantic imbalance issue arising from heterogeneous datasets across distributed clients, we employ contrastive learning to train a semantic centroid generator.
  arXiv  Detail & Related papers  (2024-06-13T14:45:35Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens [57.37893387775829]
  We introduce a fast and balanced clustering method, named textbfSemantic textbfEquitable textbfClustering (SEC) SEC clusters tokens based on their global semantic relevance in an efficient, straightforward manner. We propose a versatile vision backbone, SECViT, to serve as a vision language connector.
  arXiv  Detail & Related papers  (2024-05-22T04:49:00Z)
• Learnable Item Tokenization for Generative Recommendation [78.30417863309061]
  We propose LETTER (a LEarnable Tokenizer for generaTivE Recommendation), which integrates hierarchical semantics, collaborative signals, and code assignment diversity. LETTER incorporates Residual Quantized VAE for semantic regularization, a contrastive alignment loss for collaborative regularization, and a diversity loss to mitigate code assignment bias.
  arXiv  Detail & Related papers  (2024-05-12T15:49:38Z)
• CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation [85.13209973293229]
  Class incremental semantic segmentation aims to strike a balance between the model's stability and plasticity. We propose Contrast inter- and intra-class representations for Incremental (CoinSeg)
  arXiv  Detail & Related papers  (2023-10-10T07:08:49Z)
• Prompt Tuned Embedding Classification for Multi-Label Industry Sector Allocation [2.024620791810963]
  This study benchmarks the performance of Prompt Tuning and baselines for multi-label text classification. It is applied to classifying companies into an investment firm's proprietary industry taxonomy. We confirm that the model's performance is consistent across both well-known and less-known companies.
  arXiv  Detail & Related papers  (2023-09-21T13:45:32Z)
• Company2Vec -- German Company Embeddings based on Corporate Websites [0.0]
  The paper proposes a novel application in representation learning with Company2Vec. The model analyzes business activities from unstructured company website data using Word2Vec and dimensionality reduction. Company2Vec maintains semantic language structures and thus creates efficient company embeddings in fine-granular industries.
  arXiv  Detail & Related papers  (2023-07-18T15:14:09Z)
• CompanyKG: A Large-Scale Heterogeneous Graph for Company Similarity Quantification [1.7156312157033258]
  We propose and publish a knowledge graph, named CompanyKG, to represent and learn diverse company features and relations. Specifically, 1.17 million companies are represented as nodes enriched with company description embeddings. 15 different inter-company relations result in 51.06 million weighted edges.
  arXiv  Detail & Related papers  (2023-06-18T23:45:15Z)
• Open-Set Recognition: A Good Closed-Set Classifier is All You Need [146.6814176602689]
  We show that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes. We use this correlation to boost the performance of the cross-entropy OSR 'baseline' by improving its closed-set accuracy. We also construct new benchmarks which better respect the task of detecting semantic novelty.
  arXiv  Detail & Related papers  (2021-10-12T17:58:59Z)
• The Benchmark Lottery [114.43978017484893]
  "A benchmark lottery" describes the overall fragility of the machine learning benchmarking process. We show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks.
  arXiv  Detail & Related papers  (2021-07-14T21:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.