o ADiSw@s"UdZddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl m Z dd l mZdd l mZdd lmZdd lmZdd lmZgdZeeZiZded<GdddZGdddeZGdddeZGdddeZGdddeZGdddeZ GdddeZ!Gdd d eZ"Gd!d"d"eZ#Gd#d$d$eZ$Gd%d&d&eZ%Gd'd(d(eZ&Gd)d*d*eZ'Gd+d,d,eZ(Gd-d.d.eZ)Gd/d0d0eZ*Gd1d2d2eZ+Gd3d4d4eZ,Gd5d6d6eZ-Gd7d8d8eZ.Gd9d:d:eZ/Gd;d<dd>eZ1Gd?d@d@eZ2GdAdBdBeZ3GdCdDdDeZ4GdEdFdFeZ5GdGdHdHeZ6GdIdJdJeZ7GdKdLdLeZ8GdMdNdNeZ9GdOdPdPeZ:GdQdRdReZ;GdSdTdTeZGdYdZdZeZ?Gd[d\d\eZ@dS)]a upath._flavour_sources Warning ------- Do not modify this file manually! It is generated by `dev/generate_flavours.py` To be able to parse the different filesystem uri schemes, we need the string parsing functionality each of the filesystem implementations. In an attempt to support parsing uris without having to import the specific filesystems, we extract the necessary subset of the AbstractFileSystem classes and generate a new "flavour" class for each of the known filesystems. This will allow us to provide a `PurePath` equivalent `PureUPath` for each protocol in the future without a direct dependency on the underlying filesystem package. ) annotationsN)PurePath)PureWindowsPath)Any)Literal)castparse_qs)urlsplit)make_path_posixinfer_storage_options)stringify_path)AbstractFileSystemFlavourFileSystemFlavourBaseflavour_registryz&dict[str, type[FileSystemFlavourBase]]rc@sXeZdZUdZded<ded<ded<edd Zed d Zed d Z dddZ dS)rz"base class for the fsspec flavoursstr | tuple[str, ...]protocolzLiteral['/', ''] root_marker Literal['/']sepcCtNNotImplementedErrorclspathrO/var/www/Datamplify/venv/lib/python3.10/site-packages/upath/_flavour_sources.py_strip_protocolAz%FileSystemFlavourBase._strip_protocolcCrrrrrrr_get_kwargs_from_urlsEr!z+FileSystemFlavourBase._get_kwargs_from_urlscCrrrrrrr_parentIr!zFileSystemFlavourBase._parentrrcKsNt|jtr |jf}nt|j}|D]}|tvr td|d|t|<qdS)Nz protocol z already registered) isinstancerstrtupler ValueError)rkwargs protocolsrrrr__init_subclass__Ms    z'FileSystemFlavourBase.__init_subclass__N)rr) __name__ __module__ __qualname____doc____annotations__ classmethodr staticmethodr#r$r+rrrrr:s    rc@s^eZdZUdZdZdZded<dZded<d Zd ed <e d d Z e ddZ e ddZ dS)rzfsspec.spec.AbstractFileSystem 2024.10.0abstractrrzLiteral['', '/']r/rrcst|trfdd|DSt|}tjtrjfnj}|D]%}||dr5|t|dd}q!||drF|t|dd}q!|d}|pPjS) zTurn path from fully-qualified to file-system-specific May require FS-specific handling, e.g., for relative paths or links. cg|]}|qSrr .0prrr fz=AbstractFileSystemFlavour._strip_protocol..://N::r6) r%listrrr& startswithlenrstriprrrprotosrrr<rr _s   z)AbstractFileSystemFlavour._strip_protocolcCsiS)afIf kwargs can be encoded in the paths, extract them here This should happen before instantiation of the class; incoming paths then should be amended to strip the options in methods. Examples may look like an sftp path "sftp://user@host:/my/path", where the user and host should become kwargs and later get stripped. rr"rrrr#rs z/AbstractFileSystemFlavour._get_kwargs_from_urlscCs:||}d|vr|ddd|j}|j|S|jS)Nr6r)r rsplitlstripr)rrparentrrrr$s  z!AbstractFileSystemFlavour._parentN)r,r-r.__orig_class____orig_version__rr0rrr1r r2r#r$rrrrrXs       rc@<eZdZdZdZdZdZdZdZe ddZ e dd Z d S) AsyncLocalFileSystemFlavourz&morefs.asyn_local.AsyncLocalFileSystem0.2.2rr6TcC4t|}|dr|dd}n#|dr|dd}n|dr(|dd}n |dr3|dd}t|}tjd kr|d d d krR|dd |d d}}n4|dd d kr|d d }dksp|d |d }dkrv|d}}n|d|||d}}nd}|d p|j}||S|d p|jSNzfile://zfile:zlocal://zlocal:r6rIrB://r5rrDr osrfindrFrrrdriveindex1index2rrrr 0       z+AsyncLocalFileSystemFlavour._strip_protocolcCd||}tjdkr|dddpdS|ddd}t|dkr0|dddkr0|ddS|SNr6rIrr@rBrXz:/r r\rrJrErrpath_rrrr$    z#AsyncLocalFileSystemFlavour._parentN r,r-r.rMrNrrr local_filer1r r$rrrrrP &rPc@s:eZdZdZdZdZdZdZed dd Z e d d Z d S)AzureBlobFileSystemFlavourzadlfs.spec.AzureBlobFileSystem2024.7.0)abfsazabfssr5r6rr&cst|trfdd|DSd}td||ds&|d}d|}t|}d|vr?|dd r>|d|d |d <n|d d rX|d |d krX|d |d |d <|d }|d url|d d||d <td|d||d d}|S)z Remove the protocol from the input path Parameters ---------- path: str Path to remove the protocol from Returns ------- str Returns a path without the protocol cr7rr8r9r<rrr=r>z>AzureBlobFileSystemFlavour._strip_protocol..z.dfs.core.windows.netz_strip_protocol for )abfs://zaz://zabfss://r6rqusernameNrhostr url_query?z_strip_protocol(z) = ) r%rCloggerdebugrDrKr getcount)rr STORE_SUFFIXopsrt stripped_pathrr<rr s*      z*AzureBlobFileSystemFlavour._strip_protocolc Cszt|}i}|dd}|r td|}|r |d}||d<|d}|dur;ddlm}||}d|vr;d |d <|S) zAGet the account_name from the urlpath and pass to storage_optionsrsNz4(?P.+)\.(dfs|blob)\.core\.windows\.net account_namertrr versionidT version_aware)r rxrematch groupdict urllib.parser ) urlpathr{outrsrr}rtr parsedrrrr#s"    z0AzureBlobFileSystemFlavour._get_kwargs_from_urlsN)rr& r,r-r.rMrNrrrr1r r2r#rrrrrls *rlc@8eZdZdZdZdZdZdZeddZ e dd Z d S) AzureDatalakeFileSystemFlavourz"adlfs.gen1.AzureDatalakeFileSystemrm)adlr5r6cCst|}|dSNrr rrr{rrrr sz.AzureDatalakeFileSystemFlavour._strip_protocolcCs(t|}i}|ddr|d|d<|S)z?Get the store_name from the urlpath and pass to storage_optionsrsN store_name)r rx)pathsr{rrrrr#s   z4AzureDatalakeFileSystemFlavour._get_kwargs_from_urlsNrrrrrrs rcs6eZdZdZdZdZdZdZed fdd Z Z S) BoxFileSystemFlavourzboxfs.boxfs.BoxFileSystemz0.3.0)boxr6returnr&cs2t|}|dd}||js|j|}|S)N\r6)superr replacerDrr __class__rrr /s    z$BoxFileSystemFlavour._strip_protocol)rr& r,r-r.rMrNrrrr1r __classcell__rrrrr(rc@,eZdZdZdZdZdZdZeddZ dS) DaskWorkerFileSystemFlavourz0fsspec.implementations.dask.DaskWorkerFileSystemr3)daskr5r6cCs6t|}d|vrd|vrd|dd|diSiS)NrsportclientrXr )rsorrrr#@sz1DaskWorkerFileSystemFlavour._get_kwargs_from_urlsN r,r-r.rMrNrrrr2r#rrrrr9rc@ eZdZdZdZdZdZdZdS)DataFileSystemFlavourz*fsspec.implementations.data.DataFileSystemr3)datar5r6Nr,r-r.rMrNrrrrrrrrI rc@r)DatabricksFileSystemFlavourz0fsspec.implementations.dbfs.DatabricksFileSystemr3)dbfsr5r6NrrrrrrQrrc@s.eZdZdZdZdZdZdZed d d Z d S) DictFSFlavourzmorefs.dict.DictFSrQ)dictfsr5r6rr&rcCsV|dr |tdd}d|vsd|vr|dS|dd}|r(d|S|jS)Nz dictfs://rAr?r6)rDrErFrKrrrrrr `s  zDictFSFlavour._strip_protocolNrr&rr& r,r-r.rMrNrrrr1r rrrrrYsrc@r)DropboxDriveFileSystemFlavourz*dropboxdrivefs.core.DropboxDriveFileSystemz1.4.1)dropboxr5r6Nrrrrrrjrrc@8eZdZdZdZdZdZdZeddZ e ddZ d S) FTPFileSystemFlavourz(fsspec.implementations.ftp.FTPFileSystemr3)ftpr6cCsdt|dddS)Nr6r)r rKrFrrrrr ysz$FTPFileSystemFlavour._strip_protocolcC$t|}|dd|dd|SNrrr poprrrrrr#}  z*FTPFileSystemFlavour._get_kwargs_from_urlsNrrrrrrr rc@sFeZdZdZdZdZdZdZeddZ edd Z edd d Z d S)GCSFileSystemFlavourzgcsfs.core.GCSFileSystemr3)gsgcsr5r6cst|trfdd|DSt|}tjtrjfnj}|D]%}||dr5|t|dd}q!||drF|t|dd}q!|pKjS)Ncr7rr8r9r<rrr=r>z8GCSFileSystemFlavour._strip_protocol..r?r@rArB)r%rCrrr&rDrErrGrr<rr s  z$GCSFileSystemFlavour._strip_protocolcCs(|j|dd\}}}|durddiSiS)NT)rr) _split_path)rr_ generationrrrr#sz*GCSFileSystemFlavour._get_kwargs_from_urlsFc Cs||d}d|vr|ddfS|dd\}}|}d}|rUt|}z&|jr+|j}n|jr=t|j}d|vr=|dd}|durHt||j}Wn t yTd}Ynw|||fS)aH Normalise GCS path string into bucket and key. Parameters ---------- path : string Input path, like `gcs://mybucket/path/to/file`. Path is of the form: '[gs|gcs://]bucket[/key][?querystring][#fragment]' GCS allows object generation (object version) to be specified in either the URL fragment or the `generation` query parameter. When provided, the fragment will take priority over the `generation` query paramenter. Returns ------- (bucket, key, generation) tuple r6r5NrIrr) r rKsplitr fragmentqueryr intrr() rrrbucketkeypartkeyrpartsrrrrrs4    z GCSFileSystemFlavour._split_pathN)F) r,r-r.rMrNrrrr1r r#rrrrrrs  rc@eZdZdZdZdZdZdZefddZ e dd Z Z S) GitFileSystemFlavourz(fsspec.implementations.git.GitFileSystemr3)gitr5r6csLt|d}d|vr|ddd}d|vr!|ddd}|dS)Nr6rXrI@)rr rKrrrrrr s  z$GitFileSystemFlavour._strip_protocolcCsV|dr |dd}i}d|vr|dd\|d<}d|vr)|dd\|d<}|S)Nzgit://rWrXrIrrref)rDrrrrrrr#s  z*GitFileSystemFlavour._get_kwargs_from_urls r,r-r.rMrNrrrr1r r2r#rrrrrrsrcr) GithubFileSystemFlavourz.fsspec.implementations.github.GithubFileSystemr3)githubr5r6cs*t|}d|vrt|S|ddS)Nrrrr6)r rr rK)rroptsrrrr s z'GithubFileSystemFlavour._strip_protocolcCs>t|}d|vr iS|d|dd}|dr|d|d<|S)Nrrpassword)orgreporsshar )rrrrrrr#s z-GithubFileSystemFlavour._get_kwargs_from_urlsrrrrrrsrcs@eZdZdZdZdZdZdZeddZ efdd Z Z S) HTTPFileSystemFlavourz*fsspec.implementations.http.HTTPFileSystemr3)httphttpsr5r6cCs|S)z-For HTTP, we always want to keep the full URLrrrrrr sz%HTTPFileSystemFlavour._strip_protocolcs t|}t|dkr|SdS)NrTr5)rr$rE)rrparrrrr$s  zHTTPFileSystemFlavour._parent r,r-r.rMrNrrrr1r r$rrrrrrs rc@r) HadoopFileSystemFlavourz-fsspec.implementations.arrow.HadoopFileSystemr3)hdfs arrow_hdfsr6cCs*t|}|d}|dr|dd}|S)NrrYrI)r rDrrrrr #s   z'HadoopFileSystemFlavour._strip_protocolcCst|}i}|ddr|d|d<|ddr|d|d<|ddr*|d|d<|ddrFt|d}|ddrFt|dd|d<|S)Nrsrruserrrt replicationr)r rxr r)rr{rqueriesrrrr#,s         z-HadoopFileSystemFlavour._get_kwargs_from_urlsNrrrrrrs rc@r)HfFileSystemFlavourz+huggingface_hub.hf_file_system.HfFileSystemz0.26.5)hfr5r6Nrrrrrr=rrc@r)JupyterFileSystemFlavourz0fsspec.implementations.jupyter.JupyterFileSystemr3)jupyterjlabr5r6NrrrrrrErrc4eZdZdZdZdZdZdZefddZ Z S)LakeFSFileSystemFlavourz!lakefs_spec.spec.LakeFSFileSystemz0.11.0)lakefsr5r6csBt|trfdd|DSt|}t|dr|dS|S)z?Copied verbatim from the base class, save for the slash rstrip.cr7rr8r9r<rrr=Xr>z;LakeFSFileSystemFlavour._strip_protocol..r6)r%rCrr rendswith)rrspathrr<rr Ts  z'LakeFSFileSystemFlavour._strip_protocolrrrrrrMrcr)LibArchiveFileSystemFlavourz6fsspec.implementations.libarchive.LibArchiveFileSystemr3) libarchiver5r6ct|dSNr6rr rKrrrrr fz+LibArchiveFileSystemFlavour._strip_protocolrrrrrr_rrc@rO) LocalFileSystemFlavourz,fsspec.implementations.local.LocalFileSystemr3)filelocalr6TcCrRrSr[r^rrrr trbz&LocalFileSystemFlavour._strip_protocolcCrcrdrerfrrrr$rhzLocalFileSystemFlavour._parentNrirrrrrlrkrc@r) MemFSFlavourzmorefs.memory.MemFSrQ)memfsr5r6cCs$|dr |tdd}t|S)Nzmemfs://)rDrEMemoryFileSystemFlavourr rrrrr s  zMemFSFlavour._strip_protocolNrrrrrrrrc@s,eZdZdZdZdZdZdZeddZ dS)rz.fsspec.implementations.memory.MemoryFileSystemr3)memoryr6cCszt|trt|trt|St|}|dr |tdd}d|vs(d|vr-|dS| dd}|r;d|SdS)Nz memory://rAr?r6r5) r%rrrr rrDrErFrKrrrrr s     z'MemoryFileSystemFlavour._strip_protocolNrrrrrrrrcs@eZdZdZdZdZdZdZefddZ edd Z Z S) OCIFileSystemFlavourzocifs.core.OCIFileSystemz1.3.1)ociocilaker5r6cs`t|trfdd|DSt|}t|}|jkr.d|vr.d|ddddS|S)Ncr7rr8r9r<rrr=r>z8OCIFileSystemFlavour._strip_protocol..rr6rI)r%rCrrr rrFr)rrr|rr<rr s  z$OCIFileSystemFlavour._strip_protocolcCs^||d}d|vr|j|dddSd|vr(|jd|dddStd|)Nr6rIrrz1the following path does not specify a namespace: )r rFrrJrr(rrrrr$s zOCIFileSystemFlavour._parentrrrrrrs rc@r) OSSFileSystemFlavourzossfs.core.OSSFileSystemz 2023.12.0)ossr5r6csdt|trfdd|DSt|}|dr|dd}d}t||}|r-|d}|p1jS)a0Turn path from fully-qualified to file-system-specifi Parameters ---------- path : Union[str, List[str]] Input path, like `http://oss-cn-hangzhou.aliyuncs.com/mybucket/myobject` `oss://mybucket/myobject` Examples -------- >>> _strip_protocol( "http://oss-cn-hangzhou.aliyuncs.com/mybucket/myobject" ) ('/mybucket/myobject') >>> _strip_protocol( "oss://mybucket/myobject" ) ('/mybucket/myobject') cr7rr8r9r<rrr= r>z8OSSFileSystemFlavour._strip_protocol..zoss://rUNz6https?://(?Poss.+aliyuncs\.com)(?P/.+)r)r%rCrrDrcompilerr)rr path_string parser_rematcherrr<rr s    z$OSSFileSystemFlavour._strip_protocolNrrrrrrrrc@r)OverlayFileSystemFlavourz morefs.overlay.OverlayFileSystemrQ) overlayfsr5r6Nrrrrrrrrc@r)ReferenceFileSystemFlavourz4fsspec.implementations.reference.ReferenceFileSystemr3) referencer5r6Nrrrrrrrrc@r) S3FileSystemFlavourzs3fs.core.S3FileSystemr3)s3s3ar5r6cCsFt|}|d}i}|dur!ddlm}||}d|vr!d|d<|S)z When we have a urlpath that contains a ?versionId= Assume that we want to use version_aware mode for the filesystem. rtNrr versionIdTr)r rxrr )rurl_storage_optsrtrr rrrrr#,s  z)S3FileSystemFlavour._get_kwargs_from_urlsNrrrrrr%rrc@r) SFTPFileSystemFlavourz*fsspec.implementations.sftp.SFTPFileSystemr3)sftpsshr5r6cC t|dSrr rrrrr G z%SFTPFileSystemFlavour._strip_protocolcCrrrrrrrr#Krz+SFTPFileSystemFlavour._get_kwargs_from_urlsNrrrrrr @rr c@r) SMBFileSystemFlavourz(fsspec.implementations.smb.SMBFileSystemr3)smbr5r6cCr rr rrrrr Zrz$SMBFileSystemFlavour._strip_protocolcCrrrrrrrr#^s  z*SMBFileSystemFlavour._get_kwargs_from_urlsNrrrrrrSrrc@r)TarFileSystemFlavourz(fsspec.implementations.tar.TarFileSystemr3)tarr5r6Nrrrrrrgrrc@r)WandbFSFlavourzwandbfs._wandbfs.WandbFSz0.0.2)wandbr5r6Nrrrrrrorrc@r) WebHDFSFlavourz&fsspec.implementations.webhdfs.WebHDFSr3)webhdfswebHDFSr5r6cCr rr rrrrr ~rzWebHDFSFlavour._strip_protocolcCs:t|}|dd|ddd|vr|d|d<|S)Nrrrrrrrrrrr#s   z$WebHDFSFlavour._get_kwargs_from_urlsNrrrrrrwrrcs6eZdZdZdZdZdZdZed fd d Z Z S) WebdavFileSystemFlavourzwebdav4.fsspec.WebdavFileSystemz0.10.0)webdavdavr5r6rr&rcst|}tt|S)zAStrips protocol from the given path, overriding for type-casting.)rr rr&)rrstrippedrrrr s  z'WebdavFileSystemFlavour._strip_protocolrrrrrrrrrc@s<eZdZdZdZdZdZdZedd d Z e dddZ dS)XRootDFileSystemFlavourz%fsspec_xrootd.xrootd.XRootDFileSystemz0.4.0)rootr6rstr | list[str]rrcszt|tr+|jr#t|}|jr|jd|jnddp"jS|dp*jSt|t r9fdd|DSt d)Nrur5r6cr7rr8)r:itemr<rrr=r>z;XRootDFileSystemFlavour._strip_protocol..z'Strip protocol not given string or list) r%r&rDrr rrrFrrCr()rrxrr<rr s  0 z'XRootDFileSystemFlavour._strip_protocolur&dict[Any, Any]cCst|}d|jiS)Nhostid)r netloc)r!urlrrrr#s z-XRootDFileSystemFlavour._get_kwargs_from_urlsN)rrrr)r!r&rr"rrrrrrs  rcr)ZipFileSystemFlavourz(fsspec.implementations.zip.ZipFileSystemr3)zipr5r6crrrrrrrr rz$ZipFileSystemFlavour._strip_protocolrrrrrr&rr&c@s eZdZdZdZdZdZdZdS)_DVCFileSystemFlavourzdvc.fs.dvc._DVCFileSystemz3.58.0)dvcr6Nrrrrrr(rr()Ar/ __future__rloggingr\rpathlibrrtypingrrrrr r fsspec.implementations.localr fsspec.utilsr r__all__ getLoggerr,rvrr0rrrPrlrrrrrrrrrrrrrrrrrrrrrrrrrr rrrrrrr&r(rrrrsp             1@IM! @(