# !rm -rf '/AIKING_HOME/data/spooky.zip'
External
API details: Module for defining config and downloading data from various sources like kaggle, fastai, github
Extend Configuration
aiking_cfg
aiking_cfg ()
Config
object for fastai’s config.ini
aiking_path
aiking_path (folder)
Path to folder
in aiking_cfg
Extend URL
URLs
URLs ()
Global constants for dataset and model URLs.
Kaggle Dataset Utilities
This is going to be a helper class for downloading Kaggle dataset in required structure
KAGGLEs
KAGGLEs ()
Initialize self. See help(type(self)) for accurate signature.
download_kaggle2
download_kaggle2 (url, dest, overwrite=False)
Download url
to dest
unless it exists and not overwrite
# download_kaggle2("kaggle_competitions::spaceship-titanic", cfg.path('data')/"spaceship-titanic")
# !ls -l {cfg.path('data')/"spaceship-titanic"}
download_kaggle
download_kaggle (url, dest, overwrite=False)
Download url
to dest
unless it exists and not overwrite
Utilities for archive extraction
unzip_file
unzip_file (dest, arch_path)
untar_data
untar_data (url, archive=None, data=None, c_key='data', force_download=False)
Download url
to fname
if dest
doesn’t exist, and extract to folder dest
Type | Default | Details | |
---|---|---|---|
url | |||
archive | NoneType | None | |
data | NoneType | None | |
c_key | str | data | |
force_download | bool | False | , extract_func=file_extract, timeout=4): |
list_checked_data
list_checked_data ()
Image Dataset Utilities(Bing / ddg)
Duck Duck Go Downloader
search_images_ddg
search_images_ddg (key, max_n=150)
Bing Downloader
search_images_bing
search_images_bing (key, term, min_sz=128, max_images=150)
Dataset constructor
construct_image_dataset
construct_image_dataset (clstypes, dest, key=None, loc=None, count=150, engine='bing')
Utilities to review datasets folder
list_ds
list_ds (loc=None)
get_ds
get_ds (name, loc=None)
Create or Update Dataset folder
push_ds
push_ds (url, dsname, name=None, subfolder='.')
Download url
to dsname
dataset as name
. Creates dsname
if doesnot exists