poets.io package

Submodules

poets.io.download module

Provides download functions for FTP/SFTP, HTTP and local data sources.

poets.io.download.download_ftp(download_path, host, directory, port, username, password, filedate, dirstruct, begin, end=None)[source]

Download data via SFTP

Parameters:

download_path : str, optional

Path where to save the downloaded files.

host : str

Link to host.

directory : str

Path to data on host.

port : int

Port to host.

username : str

Username for source.

password : str

Passwor for source.

filedate : dict

Dict which points to the date fields in the filename

dirstruct : list of str

Folder structure on host, each list element represents a subdirectory

begin : datetime.datetime

Set either to first date of remote repository or date of last file in local repository

end : datetime.datetime, optional

Entered in [years]. End year is not downloaded anymore, defaults to datetime.date.today()

Returns:

bool

true if data is available, false if not

poets.io.download.download_http(download_path, host, directory, filename, filedate, dirstruct, begin, end=None)[source]

Download data via HTTP

Parameters:

download_path : str, optional

Path where to save the downloaded files.

host : str

Link to host.

directory : str

Path to data on host.

filename : str

Structure/convention of the file name.

filedate : dict

Dict which points to the date fields in the filename.

dirstruct : list of str

Folder structure on host, each list element represents a subdirectory.

begin : datetime.date

Set either to first date of remote repository or date of last file in local repository.

end : datetime.date, optional

Set to today if none given.

Returns:

bool

true if data is available, false if not

poets.io.download.download_sftp(download_path, host, directory, port, username, password, filedate, dirstruct, begin, end=None)[source]

Download data via SFTP

Parameters:

download_path : str, optional

Path where to save the downloaded files.

host : str

Link to host.

directory : str

Path to data on host.

port : int

Port to host.

username : str

Username for source.

password : str

Passwor for source.

filedate : dict

Dict which points to the date fields in the filename.

dirstruct : list of str

Folder structure on host, each list element represents a subdirectory.

begin : datetime.datetime

Set either to first date of remote repository or date of last file in local repository.

end : datetime.datetime, optional

Entered in [years]. End year is not downloaded anymore, defaults to datetime.datetime.now()

Returns:

bool

True if data is available, false if not.

poets.io.download.get_file_date(fname, fdate)[source]

Gets the date from a file name.

Parameters:

fname : str

Filename

fdate : str

Structure of the date in filename, dict which points to the date fields in the filename

Returns:

datetime.datetime

Date and, if given, time from filename

poets.io.source_base module

class poets.io.source_base.BasicSource(name, filename, filedate, temp_res, rootpath, host, protocol, username=None, password=None, port=22, directory=None, dirstruct=None, begin_date=datetime.datetime(2000, 1, 1, 0, 0), variables=['dataset'], nan_value=None, dest_nan_value=-99, dest_regions=None, dest_sp_res=0.25, dest_temp_res='dekad', dest_start_date=datetime.datetime(2000, 1, 1, 0, 0))[source]

Bases: object

Base Class for data sources.

Parameters:

name : str

Name of the data source.

filename : str

Structure/convention of the file name.

filedate : dict

Position of date fields in filename, given as tuple.

temp_res : str

Temporal resolution of the source.

rootpath : str

Root path where all data will be stored.

host : str

Link to data host.

protocol : str

Protocol for data transfer.

username : str, optional

Username for data access.

password : str, optional

Password for data access.

port : int, optional

Port to data host, defaults to 22.

directory : str, optional

Path to data on host.

dirstruct : list of strings

Structure of source directory, each list item represents a subdirectory.

begin_date : datetime.date, optional

Date from which on data is available, defaults to 2000-01-01.

variables : list of strings, optional

Variables used from data source, defaults to [‘dataset’].

nan_value : int, float, optional

Nan value of the original data as given by the data provider.

dest_nan_value : int, float, optional

NaN value in the final NetCDF file.

dest_regions : list of str, optional

Regions of interest where data should be resampled to.

dest_sp_res : int, float, optional

Spatial resolution of the destination NetCDF file, defaults to 0.25 degree.

dest_temp_res : string, optional

Temporal resolution of the destination NetCDF file, possible values: (‘month’, ‘dekad’), defaults to dekad.

dest_start_date : datetime.datetime, optional

Start date of the destination NetCDF file, defaults to 2000-01-01.

Attributes

name (str) Name of the data source.
filename (str) Structure/convention of the file name.
filedate (dict) Position of date fields in filename, given as tuple.
temp_res (str) Temporal resolution of the source.
host (str) Link to data host.
protocol (str) Protocol for data transfer.
username (str) Username for data access.
password (str) Password for data access.
port (int) Port to data host.
directory (str) Path to data on host.
dirstruct (list of strings) Structure of source directory, each list item represents a subdirectory.
begin_date (datetime.date) Date from which on data is available.
variables (list of strings) Variables used from data source.
nan_value (int, float) N a number value of the original data as given by the data provider.
dest_nan_value (int, float, optional) NaN value in the final NetCDF file.
tmp_path (str) Path where temporary files and original files are stored and downloaded.
data_path (str) Path where resampled NetCDF file is stored.
dest_regions (list of str) Regions of interest where data is resampled to.
dest_sp_res (int, float) Spatial resolution of the destination NetCDF file.
dest_temp_res (string) Temporal resolution of the destination NetCDF file.

Methods

download(download_path=None, begin=None, end=None)[source]

“Download data

Parameters:

begin : datetime.datetime, optional

start date of download, default to None

end : datetime.datetime, optional

start date of download, default to None

download_and_resample(download_path=None, begin=None, end=None, delete_rawdata=False, shapefile=None)[source]

Downloads and resamples data.

Parameters:

download_path : str

Path where to save the downloaded files.

begin : datetime.date, optional

set either to first date of remote repository or date of last file in local repository

end : datetime.date, optional

set to today if none given

delete_rawdata : bool, optional

Original files will be deleted from tmp_path if set True

shapefile : str, optional

Path to shape file, uses “world country admin boundary shapefile” by default.

read_img(date, region=None, variable=None)[source]

Gets images from netCDF file for certain date

Parameters:

date : datetime.datetime

Date of the image.

region : str, optional

Region of interest, set to first defined region if not set.

variable : str, optional

Variable to display, selects first available variables if None.

Returns:

img : numpy.ndarray

Image of selected date.

lon : numpy.array

Array with longitudes.

lat : numpy.array

Array with latitudes.

read_ts(gp, region=None, variable=None)[source]

Gets timeseries from netCDF file for a gridpoint.

Parameters:

gp : int

Grid point index.

region : str, optional

Region of interest, set to first defined region if not set.

variable : str, optional

Variable to display, selects all available variables if None.

Returns:

df : pd.DataFrame

Timeseries for selected variables.

resample(begin=None, end=None, delete_rawdata=False, shapefile=None)[source]

Resamples source data to given spatial and temporal resolution.

Writes resampled images into a netCDF data file. Deletes original files if flag delete_rawdata is set True.

Parameters:

begin : datetime.datetime

Start date of resampling.

end : datetime.datetime

End date of resampling.

delete_rawdata : bool

Original files will be deleted from tmp_path if set ‘True’.

shapefile : str, optional

Path to shape file, uses “world country admin boundary shapefile” by default.

Module contents

Table Of Contents

Previous topic

poets.image package

Next topic

poets.shape package

This Page