Rating Plausibility of Word Senses in Ambiguous Sentences through Narrative Understanding

Introduction

Problem

The AmbiStory task asks a model to score how plausible a given continuation of an ambiguous short story is, based on human judgments. Unlike a typical classification setup, the model has to pick up on subtle contextual and semantic cues to tell competing interpretations apart — the core difficulty being that a story's "correct" reading often depends entirely on context. The dataset itself is compact: 2,280 annotated samples built from only 380 unique story setups, which makes overfitting a real risk for larger neural models.

The Data

Disagreement peaks at mid-level plausibility,indicating that ambiguity is strongest when neither interpretation is clearly dominant.

This gives us an important insight into the nature of the dataset: the most challenging cases for modeling are likely those with moderate plausibility scores, where human judgments are more variable. This suggests that models will need to capture subtle contextual cues and narrative nuances to eectively predict plausibility in these ambiguous scenarios.

Each point is one sample's mean plausibility score plotted against annotator disagreement (std. dev.); the red line shows the binned trend. Disagreement is lowest at the extremes and rises through the low-to-mid range, staying elevated across the middle before dropping off sharply at the highest plausibility scores.

To prepare the data, text was tokenized using the tokenizer corresponding to the selected transformer architecture. K-fold cross validation was employed to maximize the use of the available data, while a validation split within each fold was used for model selection and early stopping.

Given the relatively small dataset size and the language understanding nature of the task, encoder-based transformer models were chosen over decoder-based architectures. We first establish a RoBERTa baseline, introduce a ranking-based loss to better align training with the evaluation metric, and nally evaluate a DeBERTa-based model. The goal is to examine the impact of both architectural improve ments and ranking-aware optimization on task performance.

Technical Solution

Baseline: TF-IDF Regression Model

As a classical baseline, we train a Ridge Regression model using TF-IDF features extracted from the story context. Ridge Regression is preferred over standard Linear Regression as the high-dimensional and sparse nature of TF-IDF features can increase the risk of overtting. Despite capturing some lexical information from the stories, the model achieves only a weak correlation with the ground-truth plausibility scores. This suggests that TF-IDF representations are insucient for modeling the contextual and semantic reasoning required by the AmbiStory task.

Scatter plot of predicted plausibility scores from the TF-IDF regression model versus actual average plausibility scores. Each point represents one dataset sample.

Initial Model: RoBERTa

To better capture contextual information, we adopt RoBERTa as our rst encoder-based transformer model. RoBERTa generates contextualized token representations that allow the model to understand words based on their surrounding context rather than relying solely on frequency statistics. The initial RoBERTa model is trained using the task-specic regression objective. Although this model signicantly improves upon the TF-IDF baseline, the optimization objective does not directly align with the evaluation metric used in the task. To address this limitation, we introduce a ranking-based loss component. Since the primary evaluation metric is Spearman correlation, preserving the relative ordering of plausibility scores is often more important than minimizing individual prediction errors.

Transition from RoBERTa to DeBERTa

Building on the improvements obtained from the ranking objective, we replace RoBERTa with DeBERTa while retaining the ranking-based loss. DeBERTa introduces disentangled attention mechanisms that separately model content and positional information, enabling richer contextual representations. The overall architecture remains unchanged,with the encoder output being passed through a regression head to predict plausibility scores. The final model combines DeBERTa's stronger contextual understanding with a ranking aware optimization objective, resulting in the best overall performance among the investigated approaches.

Training Conguration

Training Results

Training loss over epochs for the RoBERTa and DeBERTa models.

Validation loss over epochs for the RoBERTa and DeBERTa models.

Figures above illustrate the training and validation loss curves for the three transformer-based congurations. All models exhibit a rapid decrease in training loss during the initial epochs, indicating successful learning of the underlying task. The addition of the ranking loss leads to faster convergence and consistently lower training loss compared to the RoBERTa baseline.

The validation loss curves further highlight the benets of the ranking-based objective and the DeBERTa encoder. While the RoBERTa baseline shows relatively stable validation performance, both ranking-loss variants achieve lower validation losses throughout training. Among the evaluated models, DeBERTa with ranking loss achieves the lowest validation loss, suggesting superior generalization to unseen data. The gap between training and validation loss remains small across all models, indicating limited overtting despite the relatively small dataset size. Early stopping was employed to prevent unnecessary training once validation performance plateaued. The optimal stopping points were reached at epoch 10 for DeBERTa with ranking loss and epoch 11 for RoBERTa with ranking loss. Overall, the training curves demonstrate that both the ranking-based loss and the stronger DeBERTa encoder contribute to improved optimization and generalization performance, motivating the selection of DeBERTa with ranking loss as the final model.

Optimization Strategy

All transformer-based models were optimized using AdamW. AdamW is a variant of the Adam optimizer that decouples weight decay from the gradient update process, leading to improved regularization and more stable training. This optimizer has become a standard choice for ne-tuning transformer architectures due to its ability to achieve fast convergence while reducing the risk of overtting. Given the relatively small size of the AmbiStory dataset, eective regularization was particularly important. AdamW, combined with dropout and early stopping, helped improve the generalization performance of themodels while maintaining stable optimization throughout training.

Why Encoder-Based Models?

Encoder-based models were chosen over decoder-based architectures due to both the nature of the task and the limited dataset size. The AmbiStory dataset contains only 2,280 annotated samples from 380 unique story setups, making it a relatively low-resource setting. Since the objective is to predict plausibility scores rather than generate text, encoder models such as RoBERTa and DeBERTa are more suitable as they are designed for language understanding tasks and are generally more data-ecient. In addition, they require fewer computational resources while providing strong contextual representations for regression and ranking objectives.

Evaluation

Evaluation Setup

To obtain a robust estimate of model performance, all transformer-based models were evaluated using K-fold cross-validation.Within each fold, a validation split was used for model selection, early stopping, and monitoring training progress. This approach maximizes the use of the available data while reducing the variance associated with a single train-test split.

Evaluation Metrics

The primary evaluation metric for this task is Spearman correlation (ρ), which measures the agreement between the ranking of predicted plausibility scores and the ranking of ground-truth scores. Spearman correlation was chosen because the task focuses on preserving the relative ordering of plausibility judgments rather than predicting exact numerical values. In addition to Spearman correlation, Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE) and Pearson correlation were used to evaluate prediction accuracy. These metrics provide complementary information regarding the magnitude of prediction errors.

Results

Spearman correlation over epochs for the RoBERTa and DeBERTa models.

Figure shows the validation Spearman correlation across training epochs for the transformer-based models. All models exhibit steady improvements during the early stages of training, indicating successful learning of the ranking task. The introduction of the ranking-based loss consistently improves correlation compared to the RoBERTa baseline, demonstrating the benets of aligning the training objective with the evaluation metric. Among the evaluated models, DeBERTa with ranking loss achieves the highest validation Spearman correlation and converges to a better solution than the RoBERTa variants. The performance gains are particularly noticeable during later epochs, suggesting that the stronger contextual representations provided by DeBERTa allow the model to better capture semantic relationships between story contexts and plausibility scores. The plateau observed near the nal epochs motivated the use of early stopping, preventing unnecessary training once validation performance ceased to improve.

Table above summarizes the performance of the evaluated models. The TF-IDF baseline achieves the lowest performance across all metrics, demonstrating the limitations of sparse lexical representations for modeling narrative plausibility. While TF-IDF captures word frequency information, it lacks the contextual understanding required to interpret ambiguous story continuations.

Replacing TF-IDF with RoBERTa leads to a substantial improvement in performance, highlighting the benets of contextualized language representations. The introduction of the ranking-based loss further improves Spearman correlation, indicating that aligning the training objective with the evaluation metric helps the model better preserve the relative ordering of plausibility scores.

The best overall performance is obtained using DeBERTa with ranking loss. Although the improvement over RoBERTa with ranking loss is moderate, the consistently lower validation loss and higher correlation scores suggest that DeBERTa produces more informative contextual representations for this task.

Prediction heatmap for the nal DeBERTa model.

Figure above shows the distribution of rounded predictions produced by the nal DeBERTa model. Most predictions are concentrated around the diagonal, indicating strong agreement between predicted and ground-truth plausibility scores. The majority of errors occur between adjacent score levels rather than at the extremes, suggesting that the model generally preserves the relative ordering of plausibility judgments. Severe mispredictions are relatively rare, supporting the strong Spearman correlation achieved by the model.

Error Analysis

Table lists the ten worst predictions on the development set, ranked by absolute error.Scores range from 1 (clear mismatch) to 5 (clear match). Rows shaded in light red have zero annotator standard deviation, indicating unanimous human agreement.

Reflection & Future Work

Given limited compute, the team ran a manual grid search over a handful of configurations rather than a full hyperparameter sweep — flagged as the most promising avenue for further gains. Other directions considered include data augmentation (back-translation or paraphrasing of low-resource homonym contexts), scaling up to larger backbones like DeBERTa-v3-large, and ensembling RoBERTa and DeBERTa given their differing error profiles across homonym categories.

Two recurring failure modes emerge. First, lexical sense confusion: the model predicts near the opposite end of the scale from the gold label in all ten cases. Second, missing ending context: four cases lack an ending sentence,and their errors average 0.3 points higher than full-context examples

The two worst cases illustrate the rst failure mode clearly:

• Sample 290 (tips, err=3.93): judged sense is monetary gratuity; sentence: he shared some valued tips with the waiter; ending claries feedback was unwelcome. Gold: 1.0, Pred: 4.93 the restaurant setting activates the gratuity sense despite the disambiguating ending.
• Sample 271 (tipped, err=3.35): judged sense is monetary gratuity; sentence: he accidentally tipped the table on his way out; ending mentions rst aid. Gold: 1.0, Pred: 4.35 the café context overrides the physical-action cue in the sentence itself.

Among other notable cases, sample 446 (rare, σ=1.73) is the only instance with high annotator disagreement the prediction of 4.52 may reect a legitimate alternative reading. Sample 78 (stars, σ=0) is the clearest comprehension failure: strong astronomical cues (telescope, binoculars, sleeping bag) yield a prediction of only 2.19 against a gold label of 5.0.

Summary

Performance improves consistently as both the model architecture and training objective are enhanced. The baseline RoBERTa model achieves a Spearman correlation of 0.334. Incorporating the ranking-based loss increases performance to 0.450, demonstrating that optimizing for relative ordering better aligns with the evaluation metric than pointwise regression alone. Replacing RoBERTa with De-BERTa further improves the Spearman correlation to 0.524, indicating that DeBERTa's stronger contextual representations are better suited for capturing semantic ambiguity. Error analysis reveals two primary failure modes. First, the model tends to over-predict plausibility in contexts where strong situational cues activate an incorrect interpretation of a homonym. Second, performance degrades when story endings are absent, suggesting that the model relies on complete narrative context to resolve ambiguity eectively.

Annotator disagreement contributes minimally to the largest prediction errors, as most mispredicted examples exhibit little variation in human judgments. This suggests that many errors arise from limitations in contextual reasoning rather than inherent ambiguity in the data. Overall, the results indicate that both ranking-aware optimization and stronger encoder architectures are benecial for ambiguity-aware plausibility prediction.

Reflection

RoBERTa → DeBERTa. The original plan used RoBERTa as the sole backbone. After it plateaued at ρ=0.450, we switched to DeBERTa-v3-base, whose disentangled attention encodes token content and relative position separately better suited to sense disambiguation where the homonym's position carries meaningful signal. The switch yielded a +16.4% relative gain (0.450 → 0.524) at the cost of a modest increase in training time (125M → 184M parameters).

Ranking loss adoption. The initial objective was MSE alone. Since MSE ignores the ordinal structure of the task, we added a soft Spearman ranking loss (weight 0.3) to directly optimise the evaluation metric. This produced the largest single gain in the project: +34.9% relative over MSE-only RoBERTa.

No decoder-based models. Generative models were excluded for three reasons: (1) they do not natively produce scalar regression outputs; (2) ne-tuning under cross-validation was computationally prohibitive; and (3) encoder-only models consistently outperform decoder models on sentence-level scoring tasks.

No full hyperparameter optimisation. A broad HPO sweep was replaced by a manual grid search over four congurations per model, covering learning rate, dropout, batch size, and early stopping patience. Full HPO would have required an order of magnitude more GPU hours than available, and DeBERTa is known to be robust to hyperparameter choice within reasonable ranges. We acknowledge HPO as the most promising avenue for further gains.

Dataset challenges. The dataset presented two key challenges. First, a subset of examples lacked an ending sentence, reducing the contextual information available for disambiguation and increasing prediction diculty. Second, annotator agreement varied considerably across samples, introducing label noise into the plausibility scores. These challenges motivated the use of a ranking-based loss, which is less sensitive to noisy labels and better aligned with the relative ordering of human judgments.

Future work.Several directions could extend this work. First, hyperparameter optimisation via Optuna or Ray Tune remains the most immediate gain, given our grid search covered only four congurations per model. Second, data augmentation for instance, back-translation of low-resource homonym contexts or paraphrasing the pre-context while preserving the target sentence could alleviate the class imbalance between high- and low-agreement examples. Third, larger backbones such as DeBERTa-v3-large (400M parameters) or domain-adapted variants may close the remaining gap, particularly on cases where ne-grained world knowledge is required (e.g. rare beef vs. rarity). Finally, an ensemble of RoBERTa and DeBERTa regressors could combine their complementary strengths, as the two models exhibit dierent error proles across homonym categories.