asi_core.basics =============== .. py:module:: asi_core.basics .. autoapi-nested-parse:: This module provides basic helper functions. Classes ------- .. autoapisummary:: asi_core.basics.DailyH5 asi_core.basics.DailyH5Writer asi_core.basics.DailyH5Reader Functions --------- .. autoapisummary:: asi_core.basics.ifnone asi_core.basics.get_absolute_path asi_core.basics.replace_double_backslashes_with_slashes_in_path asi_core.basics._get_files asi_core.basics.get_files asi_core.basics.get_image_files asi_core.basics.copy_file asi_core.basics.copy_file_relative_to_directories asi_core.basics.parallel_copy_files asi_core.basics.get_number_of_nans asi_core.basics.get_ETC_GMT_timezone asi_core.basics.parse_datetime asi_core.basics.get_temporal_resolution_from_timeseries asi_core.basics.assemble_path asi_core.basics.fstring_to_re asi_core.basics.adjust_gamma Module Contents --------------- .. py:function:: ifnone(var, otherwise) If a variable's original value is None, replace it with an alternative value .. py:function:: get_absolute_path(filepath, root=None, as_string=False) Combine root and relative path and resolve to absolute path. .. py:function:: replace_double_backslashes_with_slashes_in_path(str_path, root_dir=None) Replace double backslashes from windows paths with slashes. .. py:function:: _get_files(p, fs, extensions=None, substring=None) Get all files in path with 'extensions' and a name containing a 'substring'. .. py:function:: get_files(path, extensions=None, substring=None, recursive=True, folders=None, followlinks=True) Get all files in `path` with optional `extensions` or `substring`, optionally `recursive`, only in `folders`, if specified. .. py:function:: get_image_files(path, recursive=True, folders=None, extensions=IMAGE_EXTENSIONS, substring=None) Get image files in `path` recursively, only in `folders` and with `substring`, if specified. .. py:function:: copy_file(src_file, tgt_file=None, tgt_dir=None, create_parents=False) Copies a file to a specified target file or directory. :param src_file: Path to the source file. :param tgt_file: Path to the target file (optional if tgt_dir is provided). :param tgt_dir: Path to the target directory (optional if tgt_file is provided). :param create_parents: Whether to create parent directories if they do not exist (default: False). .. py:function:: copy_file_relative_to_directories(rel_filepath, src_dir, tgt_dir) Copies a file from the source directory to the target directory while preserving its relative path. :param rel_filepath: File path relative to the source directory. :param src_dir: Root source directory. :param tgt_dir: Root target directory. .. py:function:: parallel_copy_files(filepaths, src_dir, tgt_dir, keep_dir_structure=False, num_workers=0) Copies multiple files in parallel, optionally preserving directory structure. :param filepaths: List of file paths to copy. :param src_dir: Source directory (used to determine relative paths). :param tgt_dir: Target directory where files will be copied. :param keep_dir_structure: Whether to maintain the original directory structure (default: False). :param num_workers: Number of parallel workers for file copying (default: 0, meaning sequential execution). .. py:function:: get_number_of_nans(df) Count number of nans in a dataframe columns-wise and total number of rows with a NaN value in at least one column. .. py:function:: get_ETC_GMT_timezone(desired_timezone='GMT') Return a `pytz` timezone object corresponding to the desired Etc/GMT timezone. Example: UTC+2: get_ETC_GMT_timezone('GMT+2') returns pytz timezone object for "Etc/GMT-2" :param desired_timezone: (str) A string specifying the desired GMT timezone. It should be in the format 'GMT+1' (in the case of UTC+1). If a '+' sign is used, it will be replaced with '-' to follow the Etc/GMT convention (https://en.wikipedia.org/wiki/Tz_database#Area). :return: pytz.timezone, Etc/GMT timezone .. py:function:: parse_datetime(dt_string, datetime_format='%Y%m%d%H%M%S') Extracts timestamp from filename. .. py:function:: get_temporal_resolution_from_timeseries(data) Get temporal resolution from a pandas DataFrame with a DatetimeIndex. .. py:function:: assemble_path(path_structure, camera_name, timestamp, set_subday_to_wildcard=False, exposure_time=None) Assemble path to images or other files and replace timestamp and camera name 'tags' with actual values :param path_structure: (str) path to each image, containing {camera_name} where the camera name should be inserted and {timestamp:...} (e.g. {timestamp:%Y%m%d%H%M%S%f}) where the evaluated timestamp should be inserted :param camera_name: (str) Name of the camera as specified in config file and used in (image) folder structure :param timestamp: (datetime, tz-aware) Timestamp for which an (image) file is requested :param set_subday_to_wildcard: (bool) If True, replace formatters indicating hours, minutes etc. with wildcards :param exposure_time: (int) exposure time of images has to be set if set_subday_to_wildcard is False :return assemble_path: (str) assembled path .. py:function:: fstring_to_re(string) Convert from f-string syntax to regular expression syntax Only a limited set of formatters supported so far, FSTRING_RE should be extended as needed. .. py:class:: DailyH5(products_path, meta_infos={}) Base class for DailyH5 file manipulation .. py:attribute:: products_path .. py:attribute:: meta_infos .. py:attribute:: daily_h5 .. py:method:: get_file(timestamp) Get the path to the current daily h5 file. Initialize if not done yet. :param timestamp: Timestamp of the data to be stored :return: Path to the current daily h5 file .. py:method:: init_h5file(timestamp) Initialize daily h5 file for reading or writing. :param timestamp: Timestamp of the current data to be stored .. py:method:: process_entry(timestamp, mode, data=None, timestamp_forecasted=None) Stores a dataset of one timestamp to the daily h5 file :param timestamp: Timestamp based on which dataset was created :param mode: character, r/w/a, i.e. read, write or append :param data: Dataset to be saved either dataset which can be stored by h5py or dict of such datasets :param timestamp_forecasted: (Optional) timestamp forecasted by the dataset .. py:method:: process_sub_entry(label, data=None) Defines the read/ write operation to be applied recursively :param label: Label of the current data to be stored/ read :param data: Data to be processed .. py:class:: DailyH5Writer(products_path, meta_infos={}) Bases: :py:obj:`DailyH5` Base class for DailyH5 file manipulation .. py:method:: store_entry(timestamp, data, timestamp_forecasted=None) Stores a dataset of one timestamp to the daily h5 file :param timestamp: Timestamp based on which dataset was created :param timestamp_forecasted: (Optional) timestamp forecasted by the dataset :param data: Dataset to be saved either dataset which can be stored by h5py or dict of such datasets .. py:method:: init_h5file(timestamp, do_not_overwrite=True) Initialize daily h5 file, create folders if required, store meta infos to a new h5 file. :param timestamp: Timestamp of the current data to be stored :param do_not_overwrite: If called and a daily file already exists, create additional file instead of overwriting the previous one. .. py:method:: process_sub_entry(handle, label, data) Recursively store all datasets in data :param handle: Handle to an h5file or a group in an h5 file :param label: Label under which data will be stored :param data: dataset or dict of datasets .. py:class:: DailyH5Reader(products_path, meta_infos={}) Bases: :py:obj:`DailyH5` Base class for DailyH5 file manipulation .. py:method:: list_entries(h5_path) :staticmethod: Generate a dataframe of the keys and corresponding timestamps and forecasted timestamps in the h5 file. :param h5_path: Path of the h5 file, the ke :return: Dataframe with columns key, timestamp, forecasted_timestamp .. py:method:: init_from_path(timestamp, h5_path) :staticmethod: Create a DailyH5Reader instance and initializes it from a specific h5 file :param timestamp: Timestamp based on which dataset was created :param h5_path: Path of the h5 file to be read (naming can deviate from convention of DailyH5Writer) :return: DailyH5Reader instance .. py:method:: get_entry(timestamp, timestamp_forecasted=None) Stores a dataset of one timestamp to the daily h5 file :param timestamp: Timestamp based on which dataset was created :param timestamp_forecasted: (Optional) timestamp forecasted by the dataset :return: Dataset to be saved either dataset which can be stored by h5py or dict of such datasets .. py:method:: process_sub_entry(handle, label, data=None) Recursively get all datasets in handle :param handle: Handle to an h5file or a group in an h5 file :param label: Label under which data will be stored :param data: dataset or dict of datasets .. py:method:: init_h5file(timestamp, file_counter=0) Initialize daily h5 file, load meta data. :param timestamp: Timestamp of the current data to be stored :param file_counter: Appends counter suffix to file name. May be useful if multiple files created for a day. .. py:function:: adjust_gamma(image, gamma=1.0) Only for improved visibility, for radiometric evaluations reconsider Taken from https://pyimagesearch.com/2015/10/05/opencv-gamma-correction/ Build a lookup table mapping the pixel values [0, 255] to their adjusted gamma values and apply it. :param image: Input RGB image :param gamma: Gamma scalar parameter :return: Gamma-corrected image