IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges

Overview

Motivation & Contributions

Our Approach

IROH processes a natural-language query against a corpus of candidate documents enriched with generated rationales. Scores are carried forward across stages and fused via weighted linear interpolation of min-max normalised signals, so each stage refines rather than discards the previous one.

Rationale Distillation

Because LLMs struggle to learn humor from labels alone, Gemma 4 generates a one-sentence, query-aware rationale for each query-document pair explaining why a text is or is not a relevant joke. For example, for the Tom Swifty “I bought myself fifty hamburgers, and I’ve only ten left,” said Tom with fortitude., the model explains the pun arises from misapplying “fortitude” (courage) to a quantitative statement.

Both pipelines mirror positive and negative rationales so the judge sees a symmetric supervision signal.

Data Analysis

The JOKER Task 1 corpus consists of short English texts with binary relevance judgments. We combine the 2025 and 2026 editions, deduplicate recurring documents, and build a balanced set of jokes (label 1) and non-jokes (label 0). Surface features are useful but insufficient: punctuation density is the strongest single correlate of the label (r = 0.47), followed by quote presence and word count. The remaining difficulty lies in subtle, context-dependent wordplay no single feature captures.

Table 1. Linguistic feature statistics by label. Jokes are markedly shorter and far more dialogue-heavy (more quotes and punctuation) than non-jokes, but no surface feature alone separates the two classes.

Results

IROH reaches 0.6347 MAP, a substantial advance over prior editions of the task. The corpus expanded across editions, so these figures are indicative of progress rather than a strictly controlled benchmark.

Table 2. Progress on JOKER Task 1 English across editions.

Our Findings

Ablation Studies

We report official CodaBench MAP throughout, computed with pytrec_eval over the full corpus. A held-out 20% query split is used only for training-time model selection.

Cross-Encoder Comparison

Table 3. Augmentation degrades the two strongest backbones. We adopt GTE-Reranker-ModernBERT-Base without augmentation as the Stage 2 cross-encoder.

Dense Embedder Comparison

Table 4. bge-en-icl wins in isolation but at far higher cost (a 7B-parameter embedder); the smaller bge-base-en-v1.5 achieves higher MAP once paired with the full judge pipeline, so we select it.

Judge Comparison

Table 5. Augmentation consistently degrades performance and generic rationales beat typed for both judges. The smaller Qwen2.5-7B on generic rationales achieves the best single-judge MAP (0.6055), ahead of every Gemma-4-31B configuration, though Gemma is the stronger model per-variant on typed rationales.

Final Submission Configuration

Limitations

Contact & Acknowledgements

Citation (Placeholder until the official proceedings entry is available)

@InProceedings{Ana_Maria_Luisa_Mocanu_2026_IROH_CLEF,
    author    = {Mocanu, Ana-Maria Luisa and Mocanu, Sebastian and Truică, Ciprian-Octavian and Apostol, Elena-Simona},
    title     = {IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English},
    booktitle = {Conference and Labs of the Evaluation Forum (CLEF), Joker 2026 Track Task 1 English},
    month     = {June},
    year      = {2026}
}