A self-assessment framework based on large language models and knowledge graphs for extracting, validating, and analyzing organizational tasks from Persian legal texts

A self-assessment framework based on large language models and knowledge graphs for extracting, validating, and analyzing organizational tasks from Persian legal texts

Sana Moradi Chegeni1 Nasim Ahmadi2 Arezou Azizi3 Atefeh Mirzaei4

1) Associate student of software, National Girls Skills College, Khorramabad, Lorestan, Iran
2) PhD in Artificial Intelligence, University of Tehran Campus, Tehran, Iran
3) Associate student of software, National Girls Skills College, Khorramabad, Lorestan, Iran
4) Bachelor s degree student in Networking, National Girls Skills College, Khorramabad, Lorestan, Iran

IOI Code : AAHD-ABGZGZ
Publication : National Conference on Information Technology, Nanotechnology, Artificial Intelligence and Technological Futures Studies(itnaf.ir)
Abstract :
    With the expansion of e-government and the increasing volume of laws, regulations, and legal documents, extracting organizational knowledge from legal texts has become one of the fundamental challenges of knowledge management in government organizations. Traditional methods of legal document analysis, which are primarily based on manual review, face limitations such as time consumption, high costs, and the risk of human error. In recent years, Large Language Models (LLMs) have demonstrated significant capability in extracting information from unstructured texts. However, the direct use of these models in legal environments presents serious challenges, including the generation of false information (hallucination) and the lack of validation mechanisms, which reduce the reliability of the outputs. In this research, a hybrid framework based on large language models, knowledge graphs, and a self-assessment module is proposed, aimed at extracting and analyzing organizational tasks from Persian legal texts. In the proposed framework, legal and organizational entities are first identified using the ParsBERT model. Then, organizational tasks are extracted using large language models. Subsequently, an evaluator module based on a language model verifies and corrects the accuracy of the extracted relations. The generated knowledge is stored in the form of an organizational knowledge graph and is used to analyze inter-organizational relationships, identify task overlaps, and discover potential conflicts. The performance evaluation of the proposed method is conducted on Persian administrative law texts and compared with recent studies. The results show that the hybrid architecture of Llama 3.1-8B and GPT-4o achieves an F1 score of 0.8387 in extracting organizational tasks. By adding a validation layer and a knowledge graph, the proposed framework provides a suitable foundation for developing intelligent legal knowledge management systems and organizational decision support systems.

Keywords : large language models knowledge graph parsbert organizational task extraction persian natural language processing