Abstract:
rapid expansion of data volume. Since simply fine-tuning a large model can easily lead to illusion and affect the actual application effect of the model,a knowledge base for the shipping field is often constructed based on the RAG method to assist the model. However, RAG has the problem of information loss in the block embedding stage,which makes it difficult to fully represent the original information and limits its application effect. To address this issue,this paper proposes a perplexity-boundary type delayed partition loss reduction strategy. Specifically,for the problem of information embedding deviation caused by the lack of context in the text embedding of RAG,the delayed partition strategy is used to integrate context information to achieve precise embedding;at the same time,considering that the delayed partition strategy lacks clear boundary conditions and there are problems of ambiguous boundaries and incomplete information during the text partition process,the text block boundaries are divided by the perplexity-boundary partition method to improve the rationality of partitioning. Finally,the results of dataset verification show that the proposed method can reduce the information loss in the RAG block embedding process and improve the effectiveness of RAG.