Kotlin ML Pack: Technical Report

• URL: http://arxiv.org/abs/2405.19250v1
• Date: Wed, 29 May 2024 16:33:50 GMT
• Title: Kotlin ML Pack: Technical Report
• Abstract summary: We present three novel datasets of Kotlin code: KStack, KStack-clean, and KExercises. We also present a version of the HumanEval benchmark rewritten by human experts into Kotlin.
• Score: 5.691600667043724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this technical report, we present three novel datasets of Kotlin code: KStack, KStack-clean, and KExercises. We also describe the results of fine-tuning CodeLlama and DeepSeek models on this data. Additionally, we present a version of the HumanEval benchmark rewritten by human experts into Kotlin - both the solutions and the tests. Our results demonstrate that small, high-quality datasets (KStack-clean and KExercises) can significantly improve model performance on code generation tasks, achieving up to a 16-point increase in pass rate on the HumanEval benchmark. Lastly, we discuss potential future work in the field of improving language modeling for Kotlin, including the use of static analysis tools in the learning process and the introduction of more intricate and realistic benchmarks.

Related papers

• SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization [64.95852289011385]
  Large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved.<n> evaluating their capabilities has become increasingly expensive, as performing inference on a large number of benchmark samples incurs high computational costs.<n>We propose SparseEval, a method that, for the first time, adopts gradient descent to optimize anchor weights and employs an iterative refinement strategy for anchor selection.
  arXiv  Detail & Related papers  (2026-02-08T11:12:45Z)
• CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models [0.0]
  Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities.<n>Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses.<n>We introduce CodeEval, a multi-dimensional benchmark dataset designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming.
  arXiv  Detail & Related papers  (2026-01-06T21:42:01Z)
• Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation [69.62857948698436]
  Recent advances in large language models (LLMs) have improved their performance on coding benchmarks. However, improvement is plateauing due to the exhaustion of readily available high-quality data. We propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity.
  arXiv  Detail & Related papers  (2025-02-20T18:32:19Z)
• UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance [65.01483640267885]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet code generation remains a major challenge. We introduce UnitCoder, a systematic pipeline leveraging model-generated unit tests to guide and validate the code generation process. Our work presents a scalable approach that leverages model-generated unit tests to guide the synthesis of high-quality code data from pre-training corpora.
  arXiv  Detail & Related papers  (2025-02-17T05:37:02Z)
• CoCoNUT: Structural Code Understanding does not fall out of a tree [4.081759833067852]
  Large Language Models (LLMs) have shown impressive performance across a wide array of tasks involving both structured and unstructured textual data. Recent results on various benchmarks for code generation, repair, or completion suggest that certain models have programming abilities comparable to or even surpass humans. We demonstrate that high performance on such benchmarks does not correlate to humans' innate ability to understand structural control flow in code.
  arXiv  Detail & Related papers  (2025-01-27T19:29:11Z)
• KAE: Kolmogorov-Arnold Auto-Encoder for Representation Learning [2.6713407440802253]
  The Kolmogorov-Arnold Network (KAN) has recently gained attention as an alternative to traditional multi-layer perceptrons (MLPs) In this paper, we introduce the Kolmogorov-Arnold AutoEncoder (KAE), which integrates KAN with autoencoders (AEs) Experiments on benchmark datasets demonstrate that KAE improves latent representation quality, reduces reconstruction errors, and achieves superior performance in downstream tasks.
  arXiv  Detail & Related papers  (2024-12-31T12:49:03Z)
• Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists [41.94295877935867]
  We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science. We demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-10-30T17:59:01Z)
• How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data [26.836532205017104]
  We find that many datasets suffer from severe data leakage. This discovery reveals a new challenge: identifying which dataset genuinely qualify as high-quality code instruction data. We present XCoder, a family of models finetuned from LLaMA3.
  arXiv  Detail & Related papers  (2024-09-05T17:46:30Z)
• SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models [54.78329741186446]
  We propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. Experiments across both in-domain and out-of-domain benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
  arXiv  Detail & Related papers  (2024-08-28T06:33:03Z)
• Numerical Literals in Link Prediction: A Critical Examination of Models and Datasets [2.5999037208435705]
  Link Prediction models that incorporate numerical literals have shown minor improvements on existing benchmark datasets. It is unclear whether a model is actually better in using numerical literals, or better capable of utilizing the graph structure. We propose a methodology to evaluate LP models that incorporate numerical literals.
  arXiv  Detail & Related papers  (2024-07-25T17:55:33Z)
• PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods [0.552480439325792]
  We present Polish Information Retrieval Benchmark (PIRB), a comprehensive evaluation framework encompassing 41 text information retrieval tasks for Polish. The benchmark incorporates existing datasets as well as 10 new, previously unpublished datasets covering diverse topics such as medicine, law, business, physics, and linguistics. We conduct an extensive evaluation of over 20 dense and sparse retrieval models, including the baseline models trained by us.
  arXiv  Detail & Related papers  (2024-02-20T19:53:36Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets [69.91340332545094]
  We introduce FLASK, a fine-grained evaluation protocol for both human-based and model-based evaluation. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance.
  arXiv  Detail & Related papers  (2023-07-20T14:56:35Z)
• Better Language Models of Code through Self-Improvement [18.75015225501755]
  We propose a simple data augmentation framework for pre-trained language models for code (PLMCs) Our framework utilizes knowledge gained during the pre-training and fine-tuning stage to generate pseudo data, which is then used as training data for the next step. The results show that our framework significantly improves PLMCs' performance in code-related sequence generation tasks.
  arXiv  Detail & Related papers  (2023-04-02T10:59:19Z)
• CodeExp: Explanatory Code Document Generation [94.43677536210465]
  Existing code-to-text generation models produce only high-level summaries of code. We conduct a human study to identify the criteria for high-quality explanatory docstring for code. We present a multi-stage fine-tuning strategy and baseline models for the task.
  arXiv  Detail & Related papers  (2022-11-25T18:05:44Z)
• GEMv2: Multilingual NLG Benchmarking in a Single Line of Code [161.1761414080574]
  Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
  arXiv  Detail & Related papers  (2022-06-22T17:52:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.