PEZEGO: A precision agriculture system based on large language models and internet of things for pest management

Pests significantly threaten global agricultural production, which causes severe yield losses through feeding and virus transmission. To mitigate yield losses caused by pests, timely and precise pest management practices are critical. Although previous efforts have advanced automated solutions for real-time environmental monitoring in agriculture, implementing precise pest management decision-making and suggestion generation remains challenging due to complex reasoning processes in practice. In response, an enhanced pest management system, PEZEGO, is proposed to provide precise management suggestions through multi-modal environmental data, a fine-tuned open vocabulary detector (OVD),and large language models (LLMs). Specifically, a mobile application and low-cost IoT devices are developed to capture images and environmental information. A hybrid convolutional low-rank adaptation method (HCLoRA) is proposed to finetune pretrained OVDs, enabling zero-shot pest detection for converting images to pest species and quantity information. In addition, a structured data-based retrieval augmented generation workflow for LLMs is proposed to provide precise pest management suggestions through automatically extracted agriculture management knowledge and chain-of-thought. The effectiveness of PEZEGO is validated in a case study of pest management in the UK, including pest detection infield scenarios and management suggestion generation. Compared to advanced model fine-tuning methods, HCLoRA achieves the highest detection performance with 0.1759APh for YOLO World on pest detection. Additionally, the proposed structured data based retrieval augmented generation work flow obtains 68.7% 28 average Entity-level F1 score for knowledge extraction and 77.33%accuracy for pest management suggestion generation. Eventually, auser-friendly mobile application demonstrates the practical effectiveness of the proposed PEZEGO system.