Emanuele Bugliarello, Yova Kementchedjhieva, Anders Søgaard, Constanza Fierro, Nicolas Garneau
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Facts are subject to contingencies and can be true or false in different circumstances. One such contingency is time, wherein some facts mutate over a given period, e.g., the president of a country or the winner of a championship. Trustworthy language models ideally identify mutable facts as such and process them accordingly. We create MULAN , a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. We hypothesize that mutable facts are encoded differently than immutable ones, hence being easier to update. In a detailed evaluation of six popular large language models, we consistently find differences in the LLMs' confidence, representations, and update behavior, depending on the mutability of a fact. Our findings should inform future work on the injection of and induction of timecontingent knowledge to/from LLMs. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2404.03036")
get_code_for_paper("2404.03036")
have("2404.03036")
Connect an agent — have() is free.