Files
witness-fracture/.venv/lib/python3.12/site-packages/internetarchive/__pycache__/utils.cpython-312.pyc
T

131 lines
19 KiB
Plaintext
Raw Normal View History

2025-06-23 17:55:02 -05:00
Ë
“»YhV9ãó —dZddlmZddlZddlZddlZddlZddlmZddl m
Z
ddl m Z  ddl
ZeZd"d„ZGd„d e«Zd#d
„Zd$d „Zd%d „Zd&d
„Zd'd„Zd(d„ZGd„d«ZGd„d«Zd)d„Zd*d„Zd+d„Zd,d„Z d„Z!d-d„Z"d-d„Z# d. d/d„Z$d„Z%d0d„Z&d1d„Z' d2 d3d„Z(d4d „Z)d!„Z*y#e$rddlZej ZYŒšwxYw)5zÖ
internetarchive.utils
~~~~~~~~~~~~~~~~~~~~~
This module provides utility functions for the internetarchive library.
:copyright: (C) 2012-2024 by Internet Archive.
:license: AGPL 3, see LICENSE for more details.
é)Ú annotationsN)ÚMapping)ÚIterable)Ú parseStringcóª—|j«D]?\}}t|t«r"t|j |i«|«}|||<Œ8||||<ŒA|S©N)ÚitemsÚ
isinstancerÚ deep_updateÚget)ÚdÚuÚkÚvÚrs ú^/mnt/c/fieldcraft/witness-fracture/.venv/lib/python3.12/site-packages/internetarchive/utils.pyr r 1sX€Ø—‘“ ò‰ˆˆ1Ü �aœÔ !ܘAŸE™E ! R›L¨!Ó,ˆA؈AˆaŠDà�Q‘4ˆAˆaŠDð ð
€Hócó —eZdZy)ÚInvalidIdentifierExceptionN)Ú__name__Ú
__module__Ú __qualname__©rrrr;s„Ørrc󇇗dŠtˆfd„dD««r td«‚t‰«dkDst‰«dkr td«‚‰jd«r‰d dŠtˆfd
„‰D««r td «‚y )
NzAabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._-c3óD•K—|]}‰j|«du–—Œy­w)TN)Ú
startswith)Ú.0ÚcÚstrings €rú <genexpr>z)validate_s3_identifier.<locals>.<genexpr>Cs!øèø€Ò
A¨Aˆ6× Ñ ˜QÓ  4Ô 'Ñ
Aùsƒ )ú.Ú_ú-zIIdentifier cannot begin with periods ".", underscores "_", or dashes "-".édéz;Identifier should be between 3 and 80 characters in length.ú@éc3ó&•K—|]}|‰v–—Œ
y­wrr)rrÚ legal_charss €rr z)validate_s3_identifier.<locals>.<genexpr>Osøèø€Ò
0 Aˆ1�KÔ Ñ
0ùsƒz¨Identifier can only contain alphanumeric characters, periods ".", underscores "_", or dashes "-". However, identifier cannot begin with periods, underscores, or dashes.T)ÚanyrÚlenr)rr)s`@rÚvalidate_s3_identifierr,?s”ù€ØU€Kô Ó
A°Ô
AÔAÜ(ð*>ó?ð ?ô ˆ6ƒ{�SÒœC ›K¨!šOÜ(ð*2ó3ð 3ð×Ñ˜ÔØ˜˜�ˆä
Ó
0¨Ô
0Ô0Ü(ð*2ó3ð 3ð
rcó€— |jd«tjd|«duS#ttf$rYywxYw)NÚasciiTz\s)ÚencodeÚUnicodeDecodeErrorÚUnicodeEncodeErrorÚreÚsearch)Úss rÚ needs_quoter5XsE€ðØ �‰�Ôô �9‰9�U˜AÓ  dÐ *Ð*øô
Ô 2Ð 3òÙðús ‚+«=¼=cóÈ—t|t«r|jd«}|jtj
j d«}|jd«sd|›�}|S)Nzutf-8ú/)r
ÚbytesÚdecodeÚreplaceÚosÚpathÚsepr)Úfps rÚ
norm_filepathr?`sN€Ü�"”eÔØ
�Y‰Y�wÓ
ˆØ �‰”B—G‘G—K‘K Ó %€BØ
�=‰=˜Ô Ø��ˆXˆØ
€IrcóØ—tj«} |jd«}|sn|j|«Œ&|j dt
j «|j«S)Ni r)ÚhashlibÚmd5ÚreadÚupdateÚseekr;ÚSEEK_SETÚ hexdigest)Ú file_objectÚmÚdatas rÚget_md5rKisX€Ü� ‰ ‹
€AØ
Ø×Ñ Ó%ˆÙØ Ø �‰�Œð ð
×Ñ�QœŸ ™ Ô$Ø �;‰;‹=Ðrc#ó>K— |j|«}|sy|–—Œ­wr)rC)r>Ú
chunk_sizeÚchunks rÚchunk_generatorrOts'èø€Ø
Ø—‘˜
Ó#ˆÙØ ØŠ ð ùs‚cóF‡—tjŠˆfd„}|t_y)ztRegister a new excepthook to suppress KeyboardInterrupt
exception messages, and exit with status code 130.
cóT•—|turtjd«‰|||«y)Né‚)ÚKeyboardInterruptÚsysÚexit)ÚtypeÚvalueÚ tracebackÚold_excepthooks €rÚnew_hookz5suppress_keyboard_interrupt_message.<locals>.new_hookƒs#ø€Ø Ô$Ñ $Ü �H‰H�SŒMÙ�t˜U IÕ.rN)rTÚ
excepthook)rZrYs @rÚ#suppress_keyboard_interrupt_messager\|sø€ô
—^‘^€Nô/ð
„C…Nrcó(—eZdZddd„Zddd„Zd d„Zy)
ÚIterableToFileAdaptercó@—t|«|_||_||_yr)ÚiterÚiteratorÚlengthÚ
pre_encode)ÚselfÚiterableÚsizercs rÚ__init__zIterableToFileAdapter.__init__Œs€Ü˜X›ˆŒ
؈Œ ð
%ˆ�rcó�—|jr%t|jd«jd«St|jd«S)NÚz
iso-8859-1r)rcÚnextrar/)rdrfs rrCzIterableToFileAdapter.read•s8€Ø �?Š?ô ˜Ÿ
™
 rÓ*×1Ñ1°,Ó?Ð ?Ü�D—M‘M 3Ó'Ð'rcó—|jSr)rb©rds rÚ__len__zIterableToFileAdapter.__len__Ÿs €Ø�{‰{ÐrN©F)rfÚintrcÚbool)éÿÿÿÿ)rfro©Úreturnro)rrrrgrCrmrrrr^r^‹s„ô%ô(ôrr^có2—eZdZdZd„Zdd„Zd„Zd„Zd d„Zy)
ÚIdentifierListAsItemszTThis class is a lazily-loaded list of Items, accessible by index or identifier.
có‚—t|t«r|n|g|_dgt|j«z|_||_yr)r
ÚlistÚidsr+Ú_itemsÚsession)rdÚid_list_or_single_idrzs rrgzIdentifierListAsItems.__init__§s>€ä!Ð"6¼Ô=ñ)à.Ð/ð
Œð�fœs 4§8¡8›}Ñ,ˆŒ ؈� rcó,—t|j«Sr)r+rxrls rrmzIdentifierListAsItems.__len__®s€Ü�4—8‘8‹}Ðrcó—t|t«r t|jt |««Žn|gD]G}|j
|�Œ|j j|j|«|j
|<ŒI|j
|Sr) r
ÚsliceÚrangeÚindicesr+ryrzÚget_itemrx)rdÚidxÚis rÚ __getitem__z!IdentifierListAsItems.__getitem__±sw€Ü4>¸sÄEÔ4J”%˜Ÿ™¤S¨£YÓ/Ñ0ÐQTÐPUò DˆAØ�{‰{˜1‰~Ñ%Ø!%§¡×!6Ñ!6°t·x±xÀ±{Ó!C�— ‘ ˜A’ð Dð�{‰{˜3ÑÐrcóf— ||jj|«S#t$rt‚wxYwr)rxÚindexÚ
ValueErrorÚAttributeError)rdÚnames rÚ __getattr__z!IdentifierListAsItems.__getattr__·s4€ð !ؘŸ™Ÿ™ tÓ,Ñ-Ð -øÜò !Ü Ð ð !ús‚  0cóN—|jj›d|j›d�S)Nú(ú))Ú __class__rrxrls rÚ__repr__zIdentifierListAsItems.__repr__½s$€Ø—.‘.×)Ñ)Ð*¨!¨D¯H©H¨<°qÐ9Ð9rNrr©rsÚstr) rrrÚ__doc__rgrmr„rŠr�rrrruru£s „ñòóò ò !ô :rrucóº—d„}ddg} t|«}|d|«}|d|«}|rd|vr|›d|j«›�S|S#t$rt|«cYSwxYw)Ncó®—d}|j|«}|D]<}|jD]+}|j|jk(sŒ||jz
}Œ-Œ>|S)Nri)ÚgetElementsByTagNameÚ
childNodesÚnodeTypeÚ TEXT_NODErJ)Útag_nameÚxml_objÚtextÚelementsÚeÚnodes rÚ
_get_tag_textz&get_s3_xml_text.<locals>._get_tag_textÂs^€ØˆØ×/Ñ/°Ó9ˆØò &ˆAØŸ ™ ò
&�Ø—=‘= D§N¡NÓ2ؘDŸI™IÑ%‘Dñ
&ð &ðˆ rÚMessageÚResourcez'PUTz - )rÚstripÚ Exceptionr‘)Úxml_strrŸÚ tag_namesÚpÚ_msgÚ _resources rÚget_s3_xml_textr©Ás{€òð˜JÐ'€Ið
Ü ˜Ó ˆÙ˜Y¨Ó*ˆÙ! *¨aÓ0ˆ á ˜ yÑ0Ø�V˜3˜yŸ™Ó0Ð1Ð2Ð 2àˆKøÜ òÜ�7‹|Òðús‰7AÁAÁAÁAcóR—t|«rh |jdtj«|j «}|t
j kDrd}|jdtj«|Stj|«}|j}|S#t$rd}Y|SwxYw)Nr) Úis_filelike_objrEr;ÚSEEK_ENDÚtellrTÚmaxsizerFÚOSErrorÚstatÚst_size)Úfile_objrfÚsts rÚ
get_file_sizer´Ùs‘€Ü�xÔ ð Ø �M‰M˜!œRŸ[™[Ô )Ø—=‘=“?ˆDà”c—k‘kÒ!Ø�Ø �M‰M˜!œRŸ[™[Ô )ð €Kô�W‰W�XÓ
ˆØ�z‰zˆØ €Køô ò ؉Dð €Kð  ús�A%B B&Â%B&c#óâK—tj|«D]S\}}}|D]H}tjj||«}tjj ||«}||f–—ŒJŒUy­w)zOGiven a directory, yield all files recursively as a two-tuple (filepath, s3key)N)r;Úwalkr<ÚjoinÚrelpath)Ú directoryr<Ú_dirÚfilesÚfÚfilepathÚkeys rÚiter_directoryr¿êsfèø€äŸW™W YÓ/ò"шˆd�EØò "ˆAÜ—w‘w—|‘| D¨!Ó,ˆHÜ—'‘'—/‘/ (¨IÓ6ˆCؘS�/Ó !ñ "ñ"ùs‚A-A/có2—t|ttf«s|g}d}d}|dur)|jD�cgc]}|j d«‘Œ}}ng}t|t
«r|j
«}n/t|dt«rt |«j
«}n|}|D]|} tjj|«}|r t|«} n|dfg} | D]B\}
} |dur& t|
d«5} t| «}
ddd«
|vrŒ0|t!|
«z
}|dz
}ŒDŒ~||fScc}w#t$r@ |d}tjj|«}n#ttf$rd}YnwxYwYŒ­wxYw#1swYŒzxYw#t$rt|
«}
YŒ“wxYw)z²Given a filepath or list of filepaths, return the total number and size of files.
If `checksum` is `True`, skip over files whose MD5 hash matches any file in the `item`.
rTrBFNÚrbr')r
rwÚsetr»r ÚdictÚvaluesÚtupler;r<ÚisdirÚ TypeErrorrˆr¿ÚopenrKr´)r»ÚitemÚchecksumÚ total_filesÚ
total_sizer¼Úmd5sÚ_filesÚis_dirÚitÚxr"ÚfhÚlmd5s rÚrecursive_file_count_and_sizerÔós«€ô �eœd¤C˜[Ô )Ø�ˆØ€KØ€JØ�4ÑØ&*§j¡jÖ1 �—‘�e• Ð1ˆÑ1àˆÜ�%œÔà—‘“‰ä �e˜A‘h¤Ô &ܘ%“[×'Ñ'Ó)‰FàˆFØ
òˆð Ü—W‘W—]‘] 1Ó%ˆFñ Ü Ó"‰Bà�d�)�ˆBØò ‰DˆAˆqؘ4Ñð&ܘa ›ð+¨"Ü& r›{˜÷+ð
˜4‘<ØØ œ-¨Ó*Ñ *ˆJØ ˜1Ñ ‰Kñ ðð2 ˜
Ð "Ð"ùòI2øôò ð
Ø�a‘D�ÜŸ™Ÿ™ qÓ)‘øÜ"¤IÐ.ò
Ø’ð
üð  ú÷+ð+ûä ò&ä" 1›:’Dð&úse°D"Â#D'Ã" E?Ã. E3Ã:E?Ä' E0Ä1$EÅE0ÅE*Å'E0Å)E*Å*E0Å/E0Å3E< Å8E?Å?FÆFcó"—t|i|¤Ž\}}|S)zFLike `recursive_file_count_and_size`, but returns only the file count.)rÔ)ÚargsÚkwargsrËr"s rÚrecursive_file_countrØ#s€ä2°DÐC¸FÑC�N€K�Ø Ðrcón— tjj|«S#t$r
}Yd}~yd}~wwxYw)zRSpecial is_dir function to handle file-like object cases that
cannot be stat'dNF)r;r<rÆrÇ)ÚobjÚexcs rrÏrÏ)s-€ðÜ�w‰w�}‰}˜SÓ!Ð!øÜ òÜûðús ‚!¡ 4¯4cóN— tj|«y#t$rYywxYw)z,Distinguish file-like from path-like objectsFT)r;ÚfspathrÇ)rÚs rr«r«2s+€ðÜ
� ‰ �#Œðøô òÙðús ‚˜ $£$có—|js|g}njt|jdd«}|jd}t|t«r,|r|j ||z«n|j ||z«n|||gz
}t |«|_‚)aeAppend message to exception while preserving attributes.
Preserves exception class, and exception traceback.
Note:
This function needs to be called inside an except because an exception
must be active in the current scope.
Args:
caught_exc(Exception): The caught exception object
append_msg(str): The message to append to the caught exception
prepend(bool): If True prepend the message to args instead of appending
Returns:
None
Side Effects:
Re-raises the exception with the preserved data / trace but
modified message
Nrq)rÖrwr
r‘ÚappendrÅ)Ú
caught_excÚ
append_msgÚprependÚarg_listÚlast_args rÚreraise_modifyrå<sƒ€ð2 �?Š?à�<‰ô˜
Ÿ™¨¨Ð,Ó-ˆØ—?‘? 2Ñ&ˆÜ �h¤Ô $ÙØ—‘ 
¨XÑ 5Õ6à—‘ ¨:Ñ 5Õ6à ˜ :Ð.Ñ .ˆHܘH“o€J„OØ rc óª—t|tttf«r`t |«d„|D««} |D�chc]$}tt |j
«««’Œ&c}D�cgc]
}t|«‘Œc}St|t«r&t |«d„|j
«D««S|Scc}wcc}w#ttf$r|cYSwxYw)Nc3ó8K—|]}|sŒt|«–—Œy­wr©Ú remove_none)rrÑs rr zremove_none.<locals>.<genexpr>msèø€Ò9¨1²qœ  AŸÑ9ùs‚Šc3óVK—|]!\}}|�|�t|«t|«f–—Œ#y­wrrè)rrrs rr zremove_none.<locals>.<genexpr>ss3èø€òUÙ!˜Q °A°MÀaÀmô& a›.¬+°a«.Ô9ñUùs‚'))