Jason Eisner, Hao Fang, Sam Thomson, Adam Pauls, Benjamin Van Durme, Yu Su, Elias Stengel-Eskin, Emmanouil Antonios Platanios
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
In natural language understanding (NLU) production systems, users' evolving needs necessitate the addition of new features over time, indexed by new symbols added to the meaning representation space. This requires additional training data and results in ever-growing datasets. We present the first systematic investigation of this incremental symbol learning scenario. Our analysis reveals a troubling quirk in building broad-coverage NLU systems: as the training dataset grows, performance on the new symbol often decreases if we do not accordingly increase its training data. This suggests that it becomes more difficult to learn new symbols with a larger training dataset. We show that this trend holds for multiple mainstream models on two common NLU tasks: intent recognition and semantic parsing. Rejecting class imbalance as the sole culprit, we reveal that the trend is closely associated with an effect we call source signal dilution, where strong lexical cues for the new symbol become diluted as the training dataset grows. Selectively dropping training examples to prevent dilution often reverses the trend, showing the over-reliance of mainstream neural NLU models on simple lexical cues. 1 * Work done as an intern at Microsoft Semantic Machines. 1 Code, models, and data are available at https://aka. ms/nlu-incremental-symbol-learning.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2205.12228")
get_code_for_paper("2205.12228")
have("2205.12228")
Connect an agent — have() is free.