o =DisH@sdZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZmZmZmZmZmZmZmZmZmZdd l m!Z!d Z"d Z#ed Z$edZ%edZ&edZ'edZ(dZ)GdddeZ*dS)zMySQL-backed vector store for embeddings and semantic document retrieval. Provides a VectorStore implementation persisting documents, metadata, and embeddings in MySQL, plus similarity search utilities. N)AnyIterableListOptionalSequenceUnionDocument) Embeddings) VectorStore) PrivateAttr) MyEmbeddings) VAR_NAME_SPACEatomic_transactiondelete_sql_table execute_sqlextend_sql_tableformat_value_sqlget_random_nameis_table_empty source_schema table_exists)MySQLConnectionAbstractz Hello world!external_sourcez .embeddingz.contextz .context_mapz.retrieval_infoz.optionsinternal_ai_id_c seZdZUdZeZeed<eZe ed<eZ e ed<eZ e e ed<eZeed<eZeed< d0d ed e e d dffd d Zded ee fddZd1ddZd0de ee ded dfddZd1ddZ  d2dee de eede ee ded ee f ddZe  d2dee d e de eed ed ef ddZ d0dee dee d ee fd d!Z! "d3d#e d$ed%ed ee fd&d'Z"d4d)d*Z#d+e$e%dfd,e$e&dfd-e$e'dfd dfd.d/Z(Z)S)5 MyVectorStorea MySQL-backed vector store for handling embeddings and semantic document retrieval. Supports adding, deleting, and searching high-dimensional vector representations of documents using efficient storage and HeatWave ML similarity search procedures. Supports use as a context manager: when used in a `with` statement, all backing tables/data are deleted automatically when the block exits (even on exception). Attributes: db_connection (MySQLConnectionAbstract): Active MySQL database connection. embedder (Embeddings): Embeddings generator for computing vector representations. schema_name (str): SQL schema for table storage. table_name (Optional[str]): Name of the active table backing the store (or None until created). embedding_dimension (int): Size of embedding vectors stored. next_id (int): Internal counter for unique document ID generation. _db_connection _embedder _schema_name _table_name_embedding_dimension_next_idN db_connectionembedderreturncsJtd|_t||_|pt||_||_d|_t |j t |_ dS)a Initialize a MyVectorStore with a database connection and embedding generator. Args: db_connection: MySQL database connection for all vector operations. embedder: Embeddings generator used for creating and querying embeddings. Raises: ValueError: If the schema name is not valid DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. rN) super__init__r!rrr rrrlen embed_queryBASIC_EMBEDDING_QUERYr )selfr"r# __class__T/var/www/Datamplify/venv/lib/python3.10/site-packages/mysql/ai/genai/vector_store.pyr&ds    zMyVectorStore.__init__num_idscCs.ddt|j|j|D}|j|7_|S)z Generate a batch of unique internal document IDs for vector storage. Args: num_ids: Number of IDs to create. Returns: List of sequentially numbered internal string IDs. cSsg|]}d|qS)rr-).0ir-r-r. s z*MyVectorStore._get_ids..)ranger!)r*r/idsr-r-r._get_idss  zMyVectorStore._get_idscs|jdursz2MyVectorStore._make_vector_store..z CREATE TABLE .a  ( `id` VARCHAR(128) NOT NULL, `content` TEXT, `metadata` JSON DEFAULT NULL, `embed` vector(%s), PRIMARY KEY (`id`) ) ENGINE=InnoDB; params)rrrrrrr )r*r7create_table_stmtr-r8r._make_vector_stores    "z MyVectorStore._make_vector_storer4_c Kst|j5}|r|D]}t|d|jd|jd|fdq t||j|jr2|WddSWddS1s=wYdS)a# Delete documents by ID. Optionally deletes the vector table if empty after deletions. Args: ids: Optional sequence of document IDs to delete. If None, no action is taken. Returns: None Raises: DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. Notes: If the backing table is empty after deletions, the table is dropped and table_name is set to None. z DELETE FROM r; WHERE id = %sr<N)rrrrrr delete_all)r*r4r@r9_idr-r-r.deletes   "zMyVectorStore.deletecCsT|jdur(t|j}t||j|jd|_WddS1s!wYdSdS)zc Delete and drop the entire vector store table. Returns: None N)rrrrr)r*r9r-r-r.rBs  "zMyVectorStore.delete_alltexts metadatascKs8t|}ddt||pigt|D}|j||dS)ap Add a batch of text strings and corresponding metadata to the vector store. Args: texts: List of strings to embed and store. metadatas: Optional list of metadata dicts (one per text). ids: Optional custom document IDs. Returns: List of document IDs corresponding to the added texts. Raises: DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. Notes: If metadatas is None, an empty dict is assigned to each document. cSsg|] \}}t||dqS)) page_contentmetadatar)r0textmetar-r-r.r2s z+MyVectorStore.add_texts..)r4)listzipr' add_documents)r*rErFr4r@ documentsr-r-r. add_textss zMyVectorStore.add_textscCs6|durtdt|}|||d}|j||d|S)a Construct and populate a MyVectorStore instance from raw texts and metadata. Args: texts: List of strings to vectorize and store. embedder: Embeddings generator to use. metadatas: Optional list of metadata dicts per text. db_connection: Active MySQL connection. Returns: Instance of MyVectorStore containing the added texts. Raises: ValueError: If db_connection is not provided. DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. Nz@db_connection must be specified to create a MyVectorStore object)r"r#)rF) ValueErrorrKrO)clsrEr#rFr"instancer-r-r. from_textss zMyVectorStore.from_textsrNcCs|rt|t|krdt|dt|d}t|t|dkr&|ngS|dur3|t|}dd|D}|j|}t}||d<||d <||d <d d|D|d <t|j }t ||j |j |d d idWd|S1swwY|S)a Embed and store Document objects as high-dimensional vectors with metadata. Args: documents: List of Document objects (each with 'page_content' and 'metadata'). ids: Optional list of explicit document IDs. Must match the length of documents. Returns: List of document IDs stored. Raises: ValueError: If provided IDs do not match the number of documents. DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. Notes: Automatically creates the backing table if it does not exist. z.ids must be the same length as documents. Got z ids and z documents.rNcSg|]}|jqSr-)rGr0docr-r-r.r2Sz/MyVectorStore.add_documents..idcontentembedcSrTr-)rHrUr-r-r.r2ZrWrHzstring_to_vector(%s))col_name_to_placeholder_string) r'rPr?r5rembed_documentspd DataFramerrrrr)r*rNr4msgrYvectorsdfr9r-r-r.rM.s@       zMyVectorStore.add_documentsquerykkwargscKs|jdurgS|j|}t|j}t|dtdt|gd|dd}|dd|d d |d d d }t |\}} dtd|j d|jd|d|dt dt dt d} t|| |g| dt|dt g} t|d } | D]8} t|d|j d|jd| dfd|\}}}||d}|durt||d<tdi|}| |q{| WdS1swYdS)a Search for and return the most similar documents in the store to the given query. Args: query: String query to embed and use for similarity search. k: Number of top documents to return. kwargs: options to pass to ML_SIMILARITY_SEARCH. Currently supports distance_metric, max_distance, percentage_distance, and segment_overlap Returns: List of Document objects, ordered from most to least similar. Raises: DatabaseError: If provided kwargs are invalid or unsupported. If a database connection issue occurs. If an operational error occurs during execution. Implementation Notes: - Calls ML similarity search within MySQL using stored procedures. - Retrieves IDs, content, and metadata for search matches. - Parsing and retrieval for context results are handled via intermediate JSONs. NzSET @z = string_to_vector(%s)r<distance_metricCOSINE max_distanceg333333?percentage_distanceg4@segment_overlapr)rhrirjz= CALL sys.ML_SIMILARITY_SEARCH( @z3, JSON_ARRAY( 'r;z' ), JSON_OBJECT( "segment", "content", "segment_embedding", "embed", "document_name", "id" ), zR, %s, NULL, NULL, z, @z ) zSELECT @z"SELECT id, content, metadata FROM rA document_name)rXrGrHr-)rrr(rrr VAR_EMBEDDINGstrgetrr VAR_CONTEXTVAR_CONTEXT_MAPVAR_RETRIEVAL_INFOjsonloadsfetchoner append)r*rcrdre embeddingr9rfretrieval_optionsretrieval_options_placeholderretrieval_options_paramssimilarity_search_queryresults context_mapscontextdoc_idrYrHdoc_argsrVr-r-r.similarity_searchgs|           $zMyVectorStore.similarity_searchr cCs|S)a Enter the runtime context related to this vector store instance. Returns: The current MyVectorStore object, allowing use within a `with` statement block. Usage Notes: - Intended for use in a `with` statement to ensure automatic cleanup of resources. - No special initialization occurs during context entry, but enables proper context-managed lifecycle. Example: with MyVectorStore(db_connection, embedder) as vectorstore: vectorstore.add_texts([...]) # Vector store is active within this block. # All storage and resources are now cleaned up. r-)r*r-r-r. __enter__szMyVectorStore.__enter__exc_typeexc_valexc_tbcCs |dS)a Exit the runtime context for the vector store, ensuring all storage resources are cleaned up. Args: exc_type: The exception type, if any exception occurred in the context block. exc_val: The exception value, if any exception occurred in the context block. exc_tb: The traceback object, if any exception occurred in the context block. Returns: None: Indicates that exceptions are never suppressed; they will propagate as normal. Implementation Notes: - Automatically deletes all vector store data and backing tables via `delete_all()` upon exiting the context. - This cleanup occurs whether the block exits normally or due to an exception. - Does not suppress exceptions; errors in the context block will continue to propagate. - Use when the vector store lifecycle is intended to be temporary or scoped. Example: with MyVectorStore(db_connection, embedder) as vectorstore: vectorstore.add_texts([...]) # Vector store is active within this block. # All storage and resources are now cleaned up. N)rB)r*rrrr-r-r.__exit__s zMyVectorStore.__exit__r6)r$N)NN)rb)r$r )*__name__ __module__ __qualname____doc__r rr__annotations__rr rrmrrr intr!r&rKr5r?rrrDrBrdictrrO classmethodr rSr rMrrrtype BaseExceptionobjectr __classcell__r-r-r+r.rIs   '    "  & < l   r)+rrrtypingrrrrrrpandasr]langchain_core.documentsr langchain_core.embeddingsr langchain_core.vectorstoresr pydanticr mysql.ai.genai.embeddingr mysql.ai.utilsrrrrrrrrrrmysql.connector.abstractsrr)EMBEDDING_SOURCErlrorprq VAR_OPTIONSID_SPACErr-r-r-r.s(      0