Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation

• URL: http://arxiv.org/abs/2608.30114v1
• Date: Mon, 31 Aug 2026 01:00:14 GMT
• Title: Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation
• Abstract summary: We release Manac-1B, an open decoder-only model of 1.72 billion parameters trained from scratch for Brazilian Portuguese.<n>We evaluate the model against nine open baselines on four Portuguese benchmarks under a single harness.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Brazilian Portuguese remains under-served by open language models, and the few that exist are difficult to reproduce and are often compared without measures of uncertainty. We release Manacá-1B, an open decoder-only model of 1.72 billion parameters trained from scratch for Brazilian Portuguese with a fully containerized, reproducible pipeline. The pretraining is stable, with zero skipped or NaN steps and self-recovering loss spikes, and we release its full log and dynamics. We evaluate the model against nine open baselines on four Portuguese benchmarks under a single harness. Every comparison reports a standard error and a paired significance test, and the harness is validated against previously published numbers. On last-word prediction Manacá-1B is the strongest model below the 7B scale, exceeding both Tucano-1b1 and Tucano-2b4 on LAMBADA-PT with large paired margins; it is competitive on commonsense completion and near chance on multiple-choice reasoning, as are all small base models. Along the way we document a concrete evaluation pitfall: converting a SentencePiece tokenizer with case-folding normalization to the HuggingFace fast format silently drops the normalizer, routing every capitalized token to byte-fallback and depressing scores in a way that is invisible in aggregate metrics. The uncorrected tokenizer lowered LAMBADA-PT accuracy from 45.3 to 25.0; we quantify the effect and provide a one-line fix that reproduces the training tokenizer exactly. Code, raw training and evaluation logs, per-example prediction vectors, the model weights, and the corrected tokenizer are released so that every number in this paper can be recomputed.

Related papers

• Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models [67.72077328700088]
  We present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously.<n>We find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau.<n>By operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-kruncation during logit dumping.
  arXiv  Detail & Related papers  (2026-09-11T00:17:22Z)
• Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer [0.0]
  We present Arkios, a dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text.<n>Arkios exceeds three comparably sized open models (Pythia-1.4B, TinyLlama-1.1B, OLMo-1B) despite an order of magnitude fewer training tokens.
  arXiv  Detail & Related papers  (2026-08-30T23:39:44Z)
• Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents [22.30468215041597]
  A tool-using agent is given the same task in a different language, does it still take the same steps?<n>We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages.
  arXiv  Detail & Related papers  (2026-08-11T16:18:34Z)
• DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search [2.0669978569487384]
  We show how English supervision transfers to multilingual retrieval through translate-train.<n>We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources.<n>We translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples.
  arXiv  Detail & Related papers  (2026-07-29T17:50:51Z)
• MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese [0.0]
  MTEB-BR is a benchmark of 22 native Brazilian-Portuguese tasks across seven categories.<n>We evaluate 93 models spanning 23M to 27B parameters.<n>We release every task, our code, and a public leaderboard, so practitioners can choose Portuguese embedding models on native evidence.
  arXiv  Detail & Related papers  (2026-07-06T01:19:30Z)
• Pretraining Language Models to Ponder in Continuous Space [50.52734567589996]
  We introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step.<n>We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations.
  arXiv  Detail & Related papers  (2025-05-27T03:47:33Z)
• Rho-1: Not All Tokens Are What You Need [132.31428897792114]
  Previous language model pre-training methods uniformly applied a next-token prediction loss to all training tokens.<n>Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that align with desired distribution.<n>When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks.
  arXiv  Detail & Related papers  (2024-04-11T17:52:01Z)
• Language models scale reliably with over-training and on downstream tasks [121.69867718185125]
  Scaling laws are useful guides for derisking expensive training runs. However, there remain gaps between current studies and how language models are trained. In contrast, scaling laws mostly predict loss on inference, but models are usually compared on downstream task performance.
  arXiv  Detail & Related papers  (2024-03-13T13:54:00Z)
• Paloma: A Benchmark for Evaluating Language Model Fit [112.481957296585]
  Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.<n>We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains.
  arXiv  Detail & Related papers  (2023-12-16T19:12:45Z)
• An Analysis and Mitigation of the Reversal Curse [70.13419502543915]
  Recent research observed a noteworthy phenomenon in large language models (LLMs) The reversal curse is that when dealing with two entities, $a$ and $b$, LLMs excel in handling sequences in the form of $aRb$,'' but encounter challenges when processing $bR-1a$''
  arXiv  Detail & Related papers  (2023-11-13T17:01:12Z)
• Nonparametric Masked Language Modeling [113.71921977520864]
  Existing language models (LMs) predict tokens with a softmax over a finite vocabulary. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval.
  arXiv  Detail & Related papers  (2022-12-02T18:10:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.