We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| ringoreality/uniblock | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| get_num_lines | Ran | ringoreality/uniblock/uniblock/uniblock.py code served (permissive licence) · get_code("2265f41ad4ea9315") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The preprocessing pipelines in Natural Language Processing usually involve a step of removing sentences consisted of illegal characters. The definition of illegal characters and the specific removal strategy depend on the task, language, domain, etc, which often lead to tiresome and repetitive scripting of rules. In this paper, we introduce a simple statistical method, uniblock, to overcome this problem. For each sentence, uniblock generates a fixed-size feature vector using Unicode block information of the characters. A Gaussian mixture model is then estimated on some clean corpus using variational inference. The learned model can then be used to score sentences and filter corpus. We present experimental results on Sentiment Analysis, Language Modeling and Machine Translation, and show the simplicity and effectiveness of our method.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1908.09716")
get_code_for_paper("1908.09716")
have("1908.09716")
Connect an agent — have() is free.