論文の概要: Best Practices for Distilling Large Language Models into BERT for Web Search Ranking

• arxiv url: http://arxiv.org/abs/2411.04539v1
• Date: Thu, 07 Nov 2024 08:54:46 GMT
• ステータス: 翻訳完了
• システム内更新日: 2024-11-08 19:38:11.926121
• Title: Best Practices for Distilling Large Language Models into BERT for Web Search Ranking
• Title（参考訳）: Web検索ランキングのための大言語モデルをBERTに蒸留するベストプラクティス
• Authors: Dezhi Ye, Junwei Hu, Jiabin Fan, Bowen Tian, Jie Liu, Haijin Liang, Jin Ma,
• Abstract要約: LLM(Large Language Models)は、潜在的文書のランク付けリストを生成する。 LLMのランキングの専門知識をBERTのようなよりコンパクトなモデルに移行し、ランキングの損失を利用して、リソース集約の少ないモデルのデプロイを可能にします。 2024年2月現在,我々のモデルは商用ウェブ検索エンジンに統合されている。
• 参考スコア（独自算出の注目度）: 14.550458167328497
• License:
• Abstract: Recent studies have highlighted the significant potential of Large Language Models (LLMs) as zero-shot relevance rankers. These methods predominantly utilize prompt learning to assess the relevance between queries and documents by generating a ranked list of potential documents. Despite their promise, the substantial costs associated with LLMs pose a significant challenge for their direct implementation in commercial search systems. To overcome this barrier and fully exploit the capabilities of LLMs for text ranking, we explore techniques to transfer the ranking expertise of LLMs to a more compact model similar to BERT, using a ranking loss to enable the deployment of less resource-intensive models. Specifically, we enhance the training of LLMs through Continued Pre-Training, taking the query as input and the clicked title and summary as output. We then proceed with supervised fine-tuning of the LLM using a rank loss, assigning the final token as a representative of the entire sentence. Given the inherent characteristics of autoregressive language models, only the final token </s> can encapsulate all preceding tokens. Additionally, we introduce a hybrid point-wise and margin MSE loss to transfer the ranking knowledge from LLMs to smaller models like BERT. This method creates a viable solution for environments with strict resource constraints. Both offline and online evaluations have confirmed the efficacy of our approach, and our model has been successfully integrated into a commercial web search engine as of February 2024.
• Abstract（参考訳）: 近年の研究では、Large Language Models (LLMs) がゼロショット関連ランキングとして有意義な可能性を強調している。 これらの手法は主に、潜在的文書のランク付けリストを生成することにより、クエリとドキュメント間の関連性を評価するために、即時学習を利用する。 それらの約束にもかかわらず、LLMに関連するかなりのコストは、商用検索システムで直接実装する上で大きな課題となる。 この障壁を克服し、テキストランキングのためのLLMの能力をフル活用するために、低リソース集約型モデルのデプロイを可能にするため、BERTのようなよりコンパクトなモデルにLLMのランキングの専門知識を移す手法を探索する。 具体的には、継続事前学習によるLCMのトレーニングを強化し、クエリを入力とし、クリックしたタイトルと要約を出力とする。 次に、ランク損失を用いてLLMの教師付き微調整を行い、最終トークンを文全体の代表として割り当てる。 自己回帰型言語モデルの本質的な特性を考えると、前のトークンをカプセル化できるのは最終トークン </s> のみである。 さらに, LLM から BERT などの小型モデルにランキング知識を移すために, ポイントワイドとマージンのハイブリッド MSE 損失を導入する。 この手法は、厳密な資源制約のある環境に対して実行可能なソリューションを作成する。 2024年2月時点では,オンラインとオフラインの双方で本手法の有効性を確認し,商用ウェブ検索エンジンへの統合に成功している。

関連論文リスト

• RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine [2.0379810233726126]
  大規模言語モデル(LLM)は、クエリとドキュメント間の意味的関係を理解するのに優れている。 これらのクエリは、少ないユーザエンゲージメントと限られたフィードバックのため、フィードバックベースのランキングでは難しい。 本稿では,エンコーダモデルとデコーダモデルの両方に対して,効率的なラベル生成パイプラインと新しいsLLMトレーニング手法を提案する。
  論文  参考訳（メタデータ） (2024-10-08T11:28:06Z)
• Soft Prompting for Unlearning in Large Language Models [11.504012974208466]
  この研究は、データ保護規制を動機とした大規模言語モデルのための機械学習の研究に焦点をあてる。 我々はtextbfUntextbflearning (SPUL) のための textbfSoft textbfPrompting フレームワークを提案する。 本研究では,提案手法の厳密な評価を行い,SPULが実用性と忘れとのトレードオフを大幅に改善できることを示す。
  論文  参考訳（メタデータ） (2024-06-17T19:11:40Z)
• Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization [12.885866125783618]
  大規模言語モデル(LLM)は、特定のクエリに対する不正確な応答を生成する傾向がある。 我々は, LLMのトークン化に挑戦するために, $textbfADT (TokenizerのAdrial dataset)$という逆データセットを構築した。 GPT-4o, Llama-3, Qwen2.5-maxなど, 先進LLMのトークン化に挑戦する上で, 当社のADTは極めて有効であることが明らかとなった。
  論文  参考訳（メタデータ） (2024-05-27T11:39:59Z)
• Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning [35.03338699349037]
  本稿では,機能エンジニアとして大規模言語モデルを用いる新しい文脈内学習フレームワークFeatLLMを提案する。 FeatLLMは高品質なルールを生成し、TabLLMやSTUNTなどよりも大幅に(平均で10%)優れている。
  論文  参考訳（メタデータ） (2024-04-15T06:26:08Z)
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models [52.98743860365194]
  本稿では,SPIN(Self-Play fIne-tuNing)と呼ばれるファインチューニング手法を提案する。 SPINの中心には自己再生機構があり、LLMは自身のインスタンスと対戦することでその能力を洗練させる。 このことは、自己プレイの約束に光を当て、熟練した相手を必要とせずに、LSMにおける人間レベルのパフォーマンスの達成を可能にする。
  論文  参考訳（メタデータ） (2024-01-02T18:53:13Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  大規模言語モデル(LLM)は、素早い工学を通して、文脈内学習能力の出現を示す。 自然言語理解と質問応答におけるLLMの一般化性と事実性の向上という課題は、まだ未解決のままである。 本研究では, LLM の信頼性を高める枠組みを提案する。1) 分布外データの一般化,2) 差別モデルによる LLM のメリットの解明,3) 生成タスクにおける幻覚の最小化。
  論文  参考訳（メタデータ） (2023-12-26T07:24:46Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  調整された大規模言語モデル(LLM)は、タスク解決、指示に従うこと、安全性を確保することにおいて、例外的な能力を示す。 既存の連続学習ベンチマークでは、LLMをリードする上で十分な課題が欠如している。 LLMにおける継続学習を評価するための新しいベンチマークであるTRACEを紹介する。
  論文  参考訳（メタデータ） (2023-10-10T16:38:49Z)
• Label Supervised LLaMA Finetuning [13.939718306233617]
  本稿では,Large Language Models (LLM) のラベル管理型適応について紹介する。 最終LLaMA層から潜在表現を抽出し、ラベル空間に投影し、クロスエントロピー損失を計算する。 LS-LLaMAは、複雑な技術や外部知識がなければ、LS-LLaMAの10倍の規模でLLMを著しく上回ります。
  論文  参考訳（メタデータ） (2023-10-02T13:53:03Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  本稿では,事前学習した報酬モデルを診断ツールとして活用する,新たな合理的評価手法を提案する。 より長い会話は、質問を理解する能力の観点から言語モデルの包括的把握を示す。 この結果から,LLMは日常言語でよく使われる単語レベルの摂動に対する脆弱性をしばしば示している。
  論文  参考訳（メタデータ） (2023-09-20T09:23:46Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [73.86954509967416]
  マルチモーダル言語モデル(MLLM)は、マルチモーダルタスクを実行するために強力なLLMに依存している。 本稿では,MLLM 評価ベンチマーク MME について述べる。 知覚能力と認知能力の両方を合計14のサブタスクで測定する。
  論文  参考訳（メタデータ） (2023-06-23T09:22:36Z)
• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents [56.104476412839944]
  大規模言語モデル(LLM)は、様々な言語関連タスクにまたがる顕著なゼロショットの一般化を実証している。 本稿では、情報検索(IR)における関連性ランキングのためのジェネレーティブLLMについて検討する。 LLMのデータ汚染に関する懸念に対処するため,我々はNovereEvalという新しいテストセットを収集した。 実世界のアプリケーションの効率を向上させるため、ChatGPTのランキング能力を小さな特殊モデルに蒸留する可能性を探る。
  論文  参考訳（メタデータ） (2023-04-19T10:16:03Z)

関連論文リストは本サイト内にある論文のタイトル・アブストラクトから自動的に作成しています。

指定された論文の情報です。
本サイトの運営者は本サイト（すべての情報・翻訳含む）の品質を保証せず、本サイト（すべての情報・翻訳含む）を使用して発生したあらゆる結果について一切の責任を負いません。