Literal Input & Runtime Vocabulary

ARTI’s literal runtime is an alpha API built on a simple constraint: vocabulary identity is supplied as rigid tensor input at runtime. Learning happens in neural readers and scorers after that tensor—not in a permanent embedding row indexed by a global token ID.

The contract

literal strings
  → glyph / control / coordinate tensors
  → runtime vocabulary [K, ...]
  → learned keys [K, D]
  → local logits [..., K]

The output index is a local slot in the vocabulary view supplied for that call. Reordering the view reorders the logits; it does not change the represented items.

Literal tensors

TextTensorRenderer provides Unicode-aware layout with three visible fields:

• glyph appearance;
• explicit control channels for spaces, tabs, line/page breaks, zero-width characters, joiners, BOMs, and combining marks;
• coordinates across character, line, and page position.

Normalization can be raw, NFC, or NFD. Identity modes range from glyph-first representations to optional auxiliary codepoint channels for controls and fallback glyphs. This surface is alpha and should not be treated as a frozen tokenizer replacement.

For fixed-size visual literals, render_text_bitmap() and render_text_vocab() produce bitmap tensors and provide collision, distance, and entropy checks.

import arti

layout = arti.render_text_layout("r\u200br")
sequence = layout.to_sequence_tensor()

vocab = arti.render_text_vocab(["strawbery", "strawberry", "strawberrry"])
report = arti.bitmap_vocab_report(vocab)
assert report.collision_count == 0

Independent input and output vocabularies

LiteralVocabModel deliberately decouples the input view from the output view.

import arti
import torch

input_vocab = arti.render_text_vocab(["a", "b", "→"])
output_vocab = arti.render_text_vocab(["yes", "no", "?"])
ids = torch.tensor([[0, 1, 2]])

model = arti.LiteralVocabModel(
    input_vocab_tensor_dim=input_vocab[0].numel(),
    output_vocab_tensor_dim=output_vocab[0].numel(),
    hidden_dim=64,
)

logits = model(ids, input_vocab, output_vocab)
assert logits.shape == (1, 3, 3)

LiteralInput reads local IDs under the supplied input view. OutputLexiconContext lets hidden states inspect the available output range. LiteralOutputHead performs the final dynamic softmax over its K supplied literals.

Per-sample views and caching

Output vocabularies may be shared as [K, ...] or batched as [B, K, ...]. A boolean mask supports padded per-sample views. LiteralVocabCache reuses encoded keys across decoding steps; detach it for inference, but rebuild after optimizer updates while training the encoder.

Literal sequence decoding

LiteralSequenceDecoder bridges a context state—such as a frozen model’s BPE-level representation—to a supplied literal output vocabulary. It supports teacher forcing, greedy generation, per-sample EOS slots, masked vocabularies, and a small tensor-native fit recipe.

What ARTI does not claim

• It does not define the application’s tokenizer, semantics, task head, or global schema.
• Runtime vocabulary hashes or tensor identity are not publisher signatures.
• Literal and text-tensor APIs are alpha; the stable 1.x promise applies to the core layer surface documented in Stability & Security.