AI_NovelGenerator/novel_generator/__pycache__/knowledge.cpython-312.pyc

Ë
·åÐhþ
ãó"
—dZd
dl
Z
d
dlZd
dlZd
dlZd
dlZd
dlZd
dlmZ
d
dl	m
Z
mZ
d
dlm
Z

ejdd¬«
d	e
jd
<ej ddej"d
d¬«
ddedededefd„Zdedededededefd„Zy)uT
知识文件导入至��库（advanced_split_content�import_knowledge_file）
éN)
Ú	read_file)Úload_vector_storeÚinit_vector_store)
ÚDocumentÚignorez/.*Torch was not compiled with flash attention.*)
ÚmessageÚfalseÚTOKENIZERS_PARALLELISMzapp.logÚ
az4%(asctime)s - %(name)s - %(levelname)s - %(message)sz%Y-%m-%d %H:%M:%S)ÚfilenameÚfilemodeÚlevelÚformatÚdatefmtÚcontentÚsimilarity_thresholdÚ
max_lengthÚreturncó8
—t
j|«
}|sgSg}g}d
}|D]S}t|«
}||z|kDr(|r |jdj	|«
«

|g
}|}Œ>|j|«

||z
}ŒU|r |jdj	|«
«

|S)u使用基本分段策略rÚ
 )ÚnltkÚ
sent_tokenizeÚlenÚappendÚjoin)	rrrÚ	sentencesÚfinal_segmentsÚcurrent_segmentÚcurrent_lengthÚsentenceÚsentence_lengths	         ú5D:\rep\AI_NovelGenerator\novel_generator\knowledge.pyÚadvanced_split_contentr#s«€ô×"Ñ" 7Ó+€IÙ؈	à€NØ€OØ€NãˆÜ˜h›-ˆØ˜OÑ+¨jÒ8ÙØ×%Ñ% c§h¡h¨Ó&?Ô@Ø'˜jˆOØ,‰Nà×"Ñ" 8Ô,Ø˜oÑ-‰NðñØ×јcŸh™h Ó7Ô8àÐóÚembedding_api_keyÚ
embedding_urlÚembedding_interface_formatÚembedding_model_nameÚ	file_pathÚfilepathcó.—t
jd
|›d|›d|›�«

tjj	|«
st
j
d|›�«

yt
|«
}|j«st
j
d«

yt|«
}ddl	m
}
||||
r|
n
d|«}	t|	|«}
|
sPt
jd	«

t|	||«}
|
rt
jd
«

yt
j
d«

y	|D�cgc]}tt|«
¬«
‘Œ}}|
j|«

t
jd