LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs

Erik Schultheis; S. T. John

doi:10.1007/978-3-031-70371-3_33

LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs

Erik Schultheis^*
, S. T. John

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

Abstract

LLaMA-Annotate is a tool that allows visually inspecting the confidences that a large language model assigns to individual tokens, and the alternative tokens considered for that position. We provide both a simple, non-interactive command-line interface, as well as a more elaborate web application. Besides generally helping to form an intuition about the “thinking” of the LLM, our tool can be used for context-aware spellchecking, or to see how a different prompt or a differently trained LLM can impact the interpretation of a piece of text. The tool can be tried online at https://huggingface.co/spaces/s-t-j/llama-annotate.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings
Editors	Albert Bifet, Povilas Daniušis, Jesse Davis, Tomas Krilavičius, Meelis Kull, Eirini Ntoutsi, Kai Puolamäki, Indrė Žliobaitė
Publisher	Springer
Pages	424-428
Number of pages	5
ISBN (Electronic)	978-3-031-70371-3
ISBN (Print)	978-3-031-70370-6
DOIs	https://doi.org/10.1007/978-3-031-70371-3_33
Publication status	Published - 1 Sept 2024
MoE publication type	A4 Conference publication
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Vilnius, Lithuania Duration: 9 Sept 2024 → 13 Sept 2024 Conference number: 24

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Publisher	Springer
Volume	14948 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Abbreviated title	ECML PKDD
Country/Territory	Lithuania
City	Vilnius
Period	09/09/2024 → 13/09/2024

Keywords

Large language model
Token-level confidence
Visualization

Access to Document

10.1007/978-3-031-70371-3_33

Cite this

Schultheis, E., & John, S. T. (2024). LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs. In A. Bifet, P. Daniušis, J. Davis, T. Krilavičius, M. Kull, E. Ntoutsi, K. Puolamäki, & I. Žliobaitė (Eds.), Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings (pp. 424-428). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14948 LNAI). Springer. https://doi.org/10.1007/978-3-031-70371-3_33

Schultheis, Erik ; John, S. T. / LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs. Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings. editor / Albert Bifet ; Povilas Daniušis ; Jesse Davis ; Tomas Krilavičius ; Meelis Kull ; Eirini Ntoutsi ; Kai Puolamäki ; Indrė Žliobaitė. Springer, 2024. pp. 424-428 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{f8b1e0b341ed469ebe95488edfd6444f,

title = "LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs",

abstract = "LLaMA-Annotate is a tool that allows visually inspecting the confidences that a large language model assigns to individual tokens, and the alternative tokens considered for that position. We provide both a simple, non-interactive command-line interface, as well as a more elaborate web application. Besides generally helping to form an intuition about the “thinking” of the LLM, our tool can be used for context-aware spellchecking, or to see how a different prompt or a differently trained LLM can impact the interpretation of a piece of text. The tool can be tried online at https://huggingface.co/spaces/s-t-j/llama-annotate.",

keywords = "Large language model, Token-level confidence, Visualization",

author = "Erik Schultheis and John, \{S. T.\}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD ; Conference date: 09-09-2024 Through 13-09-2024",

year = "2024",

month = sep,

day = "1",

doi = "10.1007/978-3-031-70371-3\_33",

language = "English",

isbn = "978-3-031-70370-6",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "424--428",

editor = "Albert Bifet and Povilas Daniu{\v s}is and Jesse Davis and Tomas Krilavi{\v c}ius and Meelis Kull and Eirini Ntoutsi and Kai Puolam{\"a}ki and Indrė {\v Z}liobaitė",

booktitle = "Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings",

address = "Germany",

}

Schultheis, E & John, ST 2024, LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs. in A Bifet, P Daniušis, J Davis, T Krilavičius, M Kull, E Ntoutsi, K Puolamäki & I Žliobaitė (eds), Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14948 LNAI, Springer, pp. 424-428, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Vilnius, Lithuania, 09/09/2024. https://doi.org/10.1007/978-3-031-70371-3_33

LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs. / Schultheis, Erik; John, S. T.
Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings. ed. / Albert Bifet; Povilas Daniušis; Jesse Davis; Tomas Krilavičius; Meelis Kull; Eirini Ntoutsi; Kai Puolamäki; Indrė Žliobaitė. Springer, 2024. p. 424-428 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14948 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs

AU - Schultheis, Erik

AU - John, S. T.

N1 - Conference code: 24

PY - 2024/9/1

Y1 - 2024/9/1

N2 - LLaMA-Annotate is a tool that allows visually inspecting the confidences that a large language model assigns to individual tokens, and the alternative tokens considered for that position. We provide both a simple, non-interactive command-line interface, as well as a more elaborate web application. Besides generally helping to form an intuition about the “thinking” of the LLM, our tool can be used for context-aware spellchecking, or to see how a different prompt or a differently trained LLM can impact the interpretation of a piece of text. The tool can be tried online at https://huggingface.co/spaces/s-t-j/llama-annotate.

AB - LLaMA-Annotate is a tool that allows visually inspecting the confidences that a large language model assigns to individual tokens, and the alternative tokens considered for that position. We provide both a simple, non-interactive command-line interface, as well as a more elaborate web application. Besides generally helping to form an intuition about the “thinking” of the LLM, our tool can be used for context-aware spellchecking, or to see how a different prompt or a differently trained LLM can impact the interpretation of a piece of text. The tool can be tried online at https://huggingface.co/spaces/s-t-j/llama-annotate.

KW - Large language model

KW - Token-level confidence

KW - Visualization

UR - https://www.scopus.com/pages/publications/85203873664

U2 - 10.1007/978-3-031-70371-3_33

DO - 10.1007/978-3-031-70371-3_33

M3 - Conference article in proceedings

AN - SCOPUS:85203873664

SN - 978-3-031-70370-6

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 424

EP - 428

BT - Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings

A2 - Bifet, Albert

A2 - Daniušis, Povilas

A2 - Davis, Jesse

A2 - Krilavičius, Tomas

A2 - Kull, Meelis

A2 - Ntoutsi, Eirini

A2 - Puolamäki, Kai

A2 - Žliobaitė, Indrė

PB - Springer

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases

Y2 - 9 September 2024 through 13 September 2024

ER -

Schultheis E, John ST. LLaMA-Annotate—Visualizing Token-Level Confidences for LLMs. In Bifet A, Daniušis P, Davis J, Krilavičius T, Kull M, Ntoutsi E, Puolamäki K, Žliobaitė I, editors, Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Proceedings. Springer. 2024. p. 424-428. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-70371-3_33