Corpus and benchmark work on how Chinese expresses and encodes spatial relations, at the Center for Chinese Linguistics, Peking University.
Before moving to Utrecht I worked on Chinese spatial semantics with Prof. Weidong Zhan at Peking University, from 2021 to 2024. The question behind the work is what it takes for a system to understand spatial language: not just to recognize that a sentence describes a location, but to track the relations it commits to and to notice when a description is incoherent.
SpaCE
The project built the SpaCE corpus and the STEP annotation schema for spatial semantics in Chinese, and ran a series of shared-task evaluation campaigns on them at the Chinese National Conference on Computational Linguistics between 2021 and 2024. I contributed to the corpus and co-authored the overview papers for the third and fourth campaigns (Xiao et al., 2023)(Xiao et al., 2024), as well as a quality assessment of the benchmark itself (Xiao et al., 2023).
References
2024
Overview of CCL24-Eval Task 3: The Fourth Evaluation on Chinese Spatial Cognition
Liming Xiao, Nan Hu, Weidong Zhan, Yuhang Qin, Sirui Deng, Chunhui Sun, Qixu Cai, and Nan Li
In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations), Jul 2024
A five-task Chinese spatial understanding benchmark exposes substantial room for improvement in LLMs, with the best participating system achieving 60.24% accuracy.
BibTeX
@inproceedings{xiao2024fourth,title={Overview of CCL24-Eval Task 3: The Fourth Evaluation on Chinese Spatial Cognition},author={Xiao, Liming and Hu, Nan and Zhan, Weidong and Qin, Yuhang and Deng, Sirui and Sun, Chunhui and Cai, Qixu and Li, Nan},booktitle={Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations)},pages={122--134},month=jul,year={2024},address={Taiyuan, China},publisher={Chinese Information Processing Society of China},}
2023
Overview of CCL23-Eval Task 4: The 3rd Chinese Spatial Cognition Evaluation
Liming Xiao, Weidong Zhan, Zhifang Sui, Yuhang Qin, Chunhui Sun, Dan Xing, Nan Li, Fangwei Zhu, and Peiyi Wang
In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations), Aug 2023
SpaCE2023 adds scene-equivalence judgments to Chinese spatial understanding evaluation, alongside anomaly detection and spatial relation labeling, and examines ChatGPT’s performance on these tasks.
BibTeX
@inproceedings{xiao2023ccl23,title={Overview of CCL23-Eval Task 4: The 3rd Chinese Spatial Cognition Evaluation},author={Xiao, Liming and Zhan, Weidong and Sui, Zhifang and Qin, Yuhang and Sun, Chunhui and Xing, Dan and Li, Nan and Zhu, Fangwei and Wang, Peiyi},booktitle={Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations)},pages={150--158},month=aug,year={2023},address={Harbin, China},publisher={Chinese Information Processing Society of China},}
A Quality Assessment Report of the Chinese Spatial Cognition Evaluation Benchmark
Liming Xiao, Chunhui Sun, Weidong Zhan, Dan Xing, Nan Li, Chengwen Wang, and Fangwei Zhu
In Proceedings of the 22nd Chinese National Conference on Computational Linguistics, Aug 2023
We document how SpaCE2022 was constructed and quality-checked, identifying substantial label imbalance and low annotator agreement on spatial correctness and anomaly attribution as priorities for future benchmark revisions.
BibTeX
@inproceedings{xiao2023space2022,title={A Quality Assessment Report of the Chinese Spatial Cognition Evaluation Benchmark},author={Xiao, Liming and Sun, Chunhui and Zhan, Weidong and Xing, Dan and Li, Nan and Wang, Chengwen and Zhu, Fangwei},booktitle={Proceedings of the 22nd Chinese National Conference on Computational Linguistics},pages={547--558},month=aug,year={2023},address={Harbin, China},publisher={Chinese Information Processing Society of China},}
Overview of CCL24-Eval Task 3: The Fourth Evaluation on Chinese Spatial CognitionIn Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations), Jul 2024
Overview of CCL23-Eval Task 4: The 3rd Chinese Spatial Cognition EvaluationIn Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations), Aug 2023
A Quality Assessment Report of the Chinese Spatial Cognition Evaluation BenchmarkIn Proceedings of the 22nd Chinese National Conference on Computational Linguistics, Aug 2023