Outlier Detection
This notebook aims to provide users with practical experience using the toolbox. We will demonstrate the capabilities of the notebook by using an unsupervised outlier detection algorithm in a binary drought detection task.¶
To gain a comprehensive understanding of the toolbox's structure and detailed usage instructions, we highly recommend referring to the user guide available in the "Read the Docs" reference. Familiarising yourself with the user guide will ensure you have a solid foundation to make the most out of the tutorial and leverage the toolbox effectively.
1. PREREQUISITES OF THE TOOLBOX
2. USER'S SET-UP
2.1 DATASET
2.2 CHOOSING A MODEL
2.2 CONFIGURATION FILE
3. TRAINING
4. EVALUATION AND VISUALIZATIONPlease, start running the following cells to check out the contents of the toolbox, install its dependencies and import the required libraries to make use of it.
a) Check out the contents of the AIDE toolbox¶
In [1]:
# Uncomment to run the notebook in Google Colab
# from google.colab import drive
# drive.mount('/content/drive/')
In [2]:
# %cd /content/drive/MyDrive/AIDE/AIDE/notebooks
In [3]:
a = %pwd
if a.split("/")[-1] != "AIDE":
%cd ../
%ls -h
/home/maria/Documents/AIDE_private/AIDE backbones/ databases/ experiments/ tutorials/ utils/ configs/ evaluators/ main.py* user_defined/
b) Install dependencies¶
In [4]:
import sys
!{sys.executable} -m pip install -r ../requirements.txt
Requirement already satisfied: dask~=2022.7.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 1)) (2022.7.1)
Requirement already satisfied: matplotlib==3.5.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 2)) (3.5.2)
Requirement already satisfied: numpy~=1.23.5 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 3)) (1.23.5)
Requirement already satisfied: pandas==1.4.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 4)) (1.4.3)
Requirement already satisfied: PyYAML==6.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 5)) (6.0)
Requirement already satisfied: scipy==1.8.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 6)) (1.8.1)
Requirement already satisfied: seaborn==0.12.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 7)) (0.12.2)
Requirement already satisfied: torch==2.0.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 8)) (2.0.1)
Requirement already satisfied: torchmetrics==0.11.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 9)) (0.11.1)
Requirement already satisfied: tqdm==4.64.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 10)) (4.64.0)
Requirement already satisfied: xarray==2022.6.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 11)) (2022.6.0)
Requirement already satisfied: pyod~=1.0.5 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 12)) (1.0.5)
Requirement already satisfied: cartopy==0.21.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 13)) (0.21.0)
Requirement already satisfied: scikit-image~=0.20.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 14)) (0.20.0)
Requirement already satisfied: scikit-learn~=1.1.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 15)) (1.1.2)
Requirement already satisfied: pytorch_lightning~=1.7.6 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 16)) (1.7.7)
Requirement already satisfied: segmentation-models-pytorch~=0.3.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 17)) (0.3.2)
Requirement already satisfied: zarr~=2.12.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 18)) (2.12.0)
Collecting shapely
Using cached shapely-2.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
Requirement already satisfied: imageio~=2.22.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 20)) (2.22.4)
Requirement already satisfied: tsai~=0.3.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 21)) (0.3.6)
Requirement already satisfied: captum==0.6.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 22)) (0.6.0)
Collecting sphinx_rtd_theme
Using cached sphinx_rtd_theme-1.3.0-py2.py3-none-any.whl (2.8 MB)
Collecting python_docs_theme
Using cached python_docs_theme-2023.7-py3-none-any.whl (17 kB)
Requirement already satisfied: ipywidgets in /home/maria/anaconda3/lib/python3.9/site-packages (from -r ../requirements.txt (line 25)) (7.6.5)
Collecting notebook~=7.0.0
Using cached notebook-7.0.3-py3-none-any.whl (4.0 MB)
Requirement already satisfied: packaging>=20.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from matplotlib==3.5.2->-r ../requirements.txt (line 2)) (21.3)
Requirement already satisfied: fonttools>=4.22.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from matplotlib==3.5.2->-r ../requirements.txt (line 2)) (4.25.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from matplotlib==3.5.2->-r ../requirements.txt (line 2)) (1.3.2)
Requirement already satisfied: pyparsing>=2.2.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from matplotlib==3.5.2->-r ../requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: python-dateutil>=2.7 in /home/maria/anaconda3/lib/python3.9/site-packages (from matplotlib==3.5.2->-r ../requirements.txt (line 2)) (2.8.2)
Requirement already satisfied: cycler>=0.10 in /home/maria/anaconda3/lib/python3.9/site-packages (from matplotlib==3.5.2->-r ../requirements.txt (line 2)) (0.11.0)
Requirement already satisfied: pillow>=6.2.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from matplotlib==3.5.2->-r ../requirements.txt (line 2)) (9.0.1)
Requirement already satisfied: pytz>=2020.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from pandas==1.4.3->-r ../requirements.txt (line 4)) (2021.3)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (11.7.99)
Requirement already satisfied: triton==2.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (2.0.0)
Requirement already satisfied: networkx in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (3.1)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (10.2.10.91)
Requirement already satisfied: jinja2 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (2.11.3)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (10.9.0.58)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (11.7.99)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (11.10.3.66)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (11.7.101)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (11.7.4.91)
Requirement already satisfied: filelock in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (3.6.0)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (2.14.3)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (11.7.91)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (11.4.0.1)
Requirement already satisfied: typing-extensions in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (4.5.0)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (8.5.0.96)
Requirement already satisfied: sympy in /home/maria/anaconda3/lib/python3.9/site-packages (from torch==2.0.1->-r ../requirements.txt (line 8)) (1.10.1)
Requirement already satisfied: pyshp>=2.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from cartopy==0.21.0->-r ../requirements.txt (line 13)) (2.3.1)
Requirement already satisfied: pyproj>=3.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from cartopy==0.21.0->-r ../requirements.txt (line 13)) (3.4.0)
Collecting shapely
Using cached Shapely-1.8.5.post1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)
Requirement already satisfied: wheel in /home/maria/anaconda3/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->-r ../requirements.txt (line 8)) (0.37.1)
Requirement already satisfied: setuptools in /home/maria/anaconda3/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->-r ../requirements.txt (line 8)) (61.2.0)
Requirement already satisfied: lit in /home/maria/anaconda3/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1->-r ../requirements.txt (line 8)) (16.0.6)
Requirement already satisfied: cmake in /home/maria/anaconda3/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1->-r ../requirements.txt (line 8)) (3.26.4)
Requirement already satisfied: cloudpickle>=1.1.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from dask~=2022.7.1->-r ../requirements.txt (line 1)) (2.0.0)
Requirement already satisfied: partd>=0.3.10 in /home/maria/anaconda3/lib/python3.9/site-packages (from dask~=2022.7.1->-r ../requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from dask~=2022.7.1->-r ../requirements.txt (line 1)) (0.11.2)
Requirement already satisfied: fsspec>=0.6.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from dask~=2022.7.1->-r ../requirements.txt (line 1)) (2023.4.0)
Requirement already satisfied: numba>=0.51 in /home/maria/anaconda3/lib/python3.9/site-packages (from pyod~=1.0.5->-r ../requirements.txt (line 12)) (0.57.1)
Requirement already satisfied: statsmodels in /home/maria/anaconda3/lib/python3.9/site-packages (from pyod~=1.0.5->-r ../requirements.txt (line 12)) (0.13.2)
Requirement already satisfied: joblib in /home/maria/anaconda3/lib/python3.9/site-packages (from pyod~=1.0.5->-r ../requirements.txt (line 12)) (1.2.0)
Requirement already satisfied: six in /home/maria/anaconda3/lib/python3.9/site-packages (from pyod~=1.0.5->-r ../requirements.txt (line 12)) (1.16.0)
Requirement already satisfied: tifffile>=2019.7.26 in /home/maria/anaconda3/lib/python3.9/site-packages (from scikit-image~=0.20.0->-r ../requirements.txt (line 14)) (2021.7.2)
Requirement already satisfied: PyWavelets>=1.1.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from scikit-image~=0.20.0->-r ../requirements.txt (line 14)) (1.3.0)
Requirement already satisfied: lazy_loader>=0.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from scikit-image~=0.20.0->-r ../requirements.txt (line 14)) (0.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from scikit-learn~=1.1.1->-r ../requirements.txt (line 15)) (2.2.0)
Requirement already satisfied: pyDeprecate>=0.3.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (0.3.2)
Requirement already satisfied: tensorboard>=2.9.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (2.10.1)
Requirement already satisfied: pretrainedmodels==0.7.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from segmentation-models-pytorch~=0.3.0->-r ../requirements.txt (line 17)) (0.7.4)
Requirement already satisfied: timm==0.6.12 in /home/maria/anaconda3/lib/python3.9/site-packages (from segmentation-models-pytorch~=0.3.0->-r ../requirements.txt (line 17)) (0.6.12)
Requirement already satisfied: torchvision>=0.5.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from segmentation-models-pytorch~=0.3.0->-r ../requirements.txt (line 17)) (0.15.2)
Requirement already satisfied: efficientnet-pytorch==0.7.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from segmentation-models-pytorch~=0.3.0->-r ../requirements.txt (line 17)) (0.7.1)
Requirement already satisfied: munch in /home/maria/anaconda3/lib/python3.9/site-packages (from pretrainedmodels==0.7.4->segmentation-models-pytorch~=0.3.0->-r ../requirements.txt (line 17)) (2.5.0)
Requirement already satisfied: huggingface-hub in /home/maria/anaconda3/lib/python3.9/site-packages (from timm==0.6.12->segmentation-models-pytorch~=0.3.0->-r ../requirements.txt (line 17)) (0.14.1)
Requirement already satisfied: numcodecs>=0.6.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from zarr~=2.12.0->-r ../requirements.txt (line 18)) (0.10.2)
Requirement already satisfied: fasteners in /home/maria/anaconda3/lib/python3.9/site-packages (from zarr~=2.12.0->-r ../requirements.txt (line 18)) (0.18)
Requirement already satisfied: asciitree in /home/maria/anaconda3/lib/python3.9/site-packages (from zarr~=2.12.0->-r ../requirements.txt (line 18)) (0.3.3)
Requirement already satisfied: pyts>=0.12.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from tsai~=0.3.4->-r ../requirements.txt (line 21)) (0.12.0)
Requirement already satisfied: imbalanced-learn>=0.8.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from tsai~=0.3.4->-r ../requirements.txt (line 21)) (0.10.1)
Requirement already satisfied: psutil>=5.4.8 in /home/maria/anaconda3/lib/python3.9/site-packages (from tsai~=0.3.4->-r ../requirements.txt (line 21)) (5.8.0)
Requirement already satisfied: fastai>=2.7.12 in /home/maria/anaconda3/lib/python3.9/site-packages (from tsai~=0.3.4->-r ../requirements.txt (line 21)) (2.7.12)
Requirement already satisfied: sphinx<8,>=1.6 in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx_rtd_theme->-r ../requirements.txt (line 23)) (4.4.0)
Requirement already satisfied: docutils<0.19 in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx_rtd_theme->-r ../requirements.txt (line 23)) (0.17.1)
Collecting sphinxcontrib-jquery<5,>=4
Using cached sphinxcontrib_jquery-4.1-py2.py3-none-any.whl (121 kB)
Requirement already satisfied: ipython-genutils~=0.2.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipywidgets->-r ../requirements.txt (line 25)) (0.2.0)
Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipywidgets->-r ../requirements.txt (line 25)) (3.5.2)
Requirement already satisfied: nbformat>=4.2.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipywidgets->-r ../requirements.txt (line 25)) (5.3.0)
Requirement already satisfied: traitlets>=4.3.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipywidgets->-r ../requirements.txt (line 25)) (5.1.1)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipywidgets->-r ../requirements.txt (line 25)) (1.0.0)
Requirement already satisfied: ipython>=4.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipywidgets->-r ../requirements.txt (line 25)) (8.2.0)
Requirement already satisfied: ipykernel>=4.5.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipywidgets->-r ../requirements.txt (line 25)) (6.9.1)
Collecting jupyter-server<3,>=2.4.0
Using cached jupyter_server-2.7.3-py3-none-any.whl (375 kB)
Collecting tornado>=6.2.0
Using cached tornado-6.3.3-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
Collecting notebook-shim<0.3,>=0.2
Using cached notebook_shim-0.2.3-py3-none-any.whl (13 kB)
Collecting jupyterlab-server<3,>=2.22.1
Using cached jupyterlab_server-2.24.0-py3-none-any.whl (57 kB)
Collecting jupyterlab<5,>=4.0.2
Using cached jupyterlab-4.0.5-py3-none-any.whl (9.2 MB)
Requirement already satisfied: requests in /home/maria/anaconda3/lib/python3.9/site-packages (from fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (2.27.1)
Requirement already satisfied: fastprogress>=0.2.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (1.0.3)
Requirement already satisfied: spacy<4 in /home/maria/anaconda3/lib/python3.9/site-packages (from fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (3.5.0)
Requirement already satisfied: fastdownload<2,>=0.0.5 in /home/maria/anaconda3/lib/python3.9/site-packages (from fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (0.0.7)
Requirement already satisfied: fastcore<1.6,>=1.5.29 in /home/maria/anaconda3/lib/python3.9/site-packages (from fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (1.5.29)
Requirement already satisfied: pip in /home/maria/anaconda3/lib/python3.9/site-packages (from fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (21.2.4)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /home/maria/anaconda3/lib/python3.9/site-packages (from fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (3.8.1)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/maria/anaconda3/lib/python3.9/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (4.0.1)
Requirement already satisfied: yarl<2.0,>=1.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (1.6.3)
Requirement already satisfied: attrs>=17.3.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (21.4.0)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (2.0.4)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/maria/anaconda3/lib/python3.9/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (5.2.0)
Requirement already satisfied: aiosignal>=1.1.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=0.6.0->dask~=2022.7.1->-r ../requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: debugpy<2.0,>=1.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets->-r ../requirements.txt (line 25)) (1.5.1)
Requirement already satisfied: jupyter-client<8.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets->-r ../requirements.txt (line 25)) (6.1.12)
Requirement already satisfied: nest-asyncio in /home/maria/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets->-r ../requirements.txt (line 25)) (1.5.5)
Requirement already satisfied: matplotlib-inline<0.2.0,>=0.1.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets->-r ../requirements.txt (line 25)) (0.1.2)
Requirement already satisfied: decorator in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (5.1.1)
Requirement already satisfied: pexpect>4.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (4.8.0)
Requirement already satisfied: stack-data in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.2.0)
Requirement already satisfied: pickleshare in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.7.5)
Requirement already satisfied: backcall in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.2.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (3.0.20)
Requirement already satisfied: jedi>=0.16 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.18.1)
Requirement already satisfied: pygments>=2.4.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (2.11.2)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.8.3)
Requirement already satisfied: pyzmq>=13 in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets->-r ../requirements.txt (line 25)) (22.3.0)
Requirement already satisfied: jupyter-core>=4.6.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets->-r ../requirements.txt (line 25)) (4.9.2)
Collecting traitlets>=4.3.1
Using cached traitlets-5.9.0-py3-none-any.whl (117 kB)
Collecting jupyter-client<8.0
Using cached jupyter_client-7.4.9-py3-none-any.whl (133 kB)
Requirement already satisfied: websocket-client in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.58.0)
Collecting send2trash>=1.8.2
Using cached Send2Trash-1.8.2-py3-none-any.whl (18 kB)
Collecting jupyter-server-terminals
Using cached jupyter_server_terminals-0.4.4-py3-none-any.whl (13 kB)
Requirement already satisfied: anyio>=3.1.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (3.5.0)
Collecting pyzmq>=13
Using cached pyzmq-25.1.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting overrides
Using cached overrides-7.4.0-py3-none-any.whl (17 kB)
Requirement already satisfied: prometheus-client in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.13.1)
Requirement already satisfied: terminado>=0.8.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.13.1)
Collecting jupyter-events>=0.6.0
Using cached jupyter_events-0.7.0-py3-none-any.whl (18 kB)
Collecting jupyter-core>=4.6.0
Using cached jupyter_core-5.3.1-py3-none-any.whl (93 kB)
Requirement already satisfied: nbconvert>=6.4.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (6.4.4)
Requirement already satisfied: argon2-cffi in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (21.3.0)
Requirement already satisfied: sniffio>=1.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (1.2.0)
Requirement already satisfied: idna>=2.8 in /home/maria/anaconda3/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (3.3)
Requirement already satisfied: entrypoints in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets->-r ../requirements.txt (line 25)) (0.4)
Collecting platformdirs>=2.5
Using cached platformdirs-3.10.0-py3-none-any.whl (17 kB)
Collecting jsonschema[format-nongpl]>=4.18.0
Using cached jsonschema-4.19.0-py3-none-any.whl (83 kB)
Collecting rfc3339-validator
Using cached rfc3339_validator-0.1.4-py2.py3-none-any.whl (3.5 kB)
Collecting referencing
Using cached referencing-0.30.2-py3-none-any.whl (25 kB)
Collecting python-json-logger>=2.0.4
Using cached python_json_logger-2.0.7-py3-none-any.whl (8.1 kB)
Collecting rfc3986-validator>=0.1.1
Using cached rfc3986_validator-0.1.1-py2.py3-none-any.whl (4.2 kB)
Collecting attrs>=17.3.0
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting rpds-py>=0.7.1
Downloading rpds_py-0.10.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
|████████████████████████████████| 1.2 MB 988 kB/s
Collecting jsonschema-specifications>=2023.03.6
Using cached jsonschema_specifications-2023.7.1-py3-none-any.whl (17 kB)
Collecting jsonpointer>1.13
Downloading jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)
Collecting webcolors>=1.11
Downloading webcolors-1.13-py3-none-any.whl (14 kB)
Collecting isoduration
Downloading isoduration-20.11.0-py3-none-any.whl (11 kB)
Collecting fqdn
Downloading fqdn-1.5.1-py3-none-any.whl (9.1 kB)
Collecting uri-template
Downloading uri_template-1.3.0-py3-none-any.whl (11 kB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting jupyter-lsp>=2.0.0
Using cached jupyter_lsp-2.2.0-py3-none-any.whl (65 kB)
Requirement already satisfied: importlib-metadata>=4.8.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyterlab<5,>=4.0.2->notebook~=7.0.0->-r ../requirements.txt (line 26)) (4.11.3)
Collecting async-lru>=1.0.0
Using cached async_lru-2.0.4-py3-none-any.whl (6.1 kB)
Requirement already satisfied: tomli in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyterlab<5,>=4.0.2->notebook~=7.0.0->-r ../requirements.txt (line 26)) (1.2.2)
Requirement already satisfied: zipp>=0.5 in /home/maria/anaconda3/lib/python3.9/site-packages (from importlib-metadata>=4.8.3->jupyterlab<5,>=4.0.2->notebook~=7.0.0->-r ../requirements.txt (line 26)) (3.7.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from jinja2->torch==2.0.1->-r ../requirements.txt (line 8)) (2.0.1)
Collecting requests
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Requirement already satisfied: json5>=0.9.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from jupyterlab-server<3,>=2.22.1->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.9.6)
Collecting babel>=2.10
Using cached Babel-2.12.1-py3-none-any.whl (10.1 MB)
Requirement already satisfied: jupyterlab-pygments in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.1.2)
Requirement already satisfied: testpath in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.5.0)
Requirement already satisfied: defusedxml in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.7.1)
Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.5.13)
Requirement already satisfied: bleach in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (4.1.0)
Requirement already satisfied: mistune<2,>=0.8.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.8.4)
Requirement already satisfied: pandocfilters>=1.4.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (1.5.0)
Requirement already satisfied: beautifulsoup4 in /home/maria/anaconda3/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (4.11.1)
Requirement already satisfied: fastjsonschema in /home/maria/anaconda3/lib/python3.9/site-packages (from nbformat>=4.2.0->ipywidgets->-r ../requirements.txt (line 25)) (2.15.1)
Requirement already satisfied: llvmlite<0.41,>=0.40.0dev0 in /home/maria/anaconda3/lib/python3.9/site-packages (from numba>=0.51->pyod~=1.0.5->-r ../requirements.txt (line 12)) (0.40.1)
Requirement already satisfied: locket in /home/maria/anaconda3/lib/python3.9/site-packages (from partd>=0.3.10->dask~=2022.7.1->-r ../requirements.txt (line 1)) (0.2.1)
Requirement already satisfied: ptyprocess>=0.5 in /home/maria/anaconda3/lib/python3.9/site-packages (from pexpect>4.3->ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.7.0)
Requirement already satisfied: wcwidth in /home/maria/anaconda3/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.2.5)
Requirement already satisfied: certifi in /home/maria/anaconda3/lib/python3.9/site-packages (from pyproj>=3.0.0->cartopy==0.21.0->-r ../requirements.txt (line 13)) (2023.5.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from requests->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (1.26.9)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (2.0.7)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (3.0.8)
Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (1.1.1)
Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (8.1.7)
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (3.3.0)
Requirement already satisfied: pathy>=0.10.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (0.10.1)
Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (6.3.0)
Requirement already satisfied: typer<0.8.0,>=0.3.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (0.7.0)
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (1.0.4)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (1.0.9)
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (2.4.5)
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (2.0.8)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (1.10.5)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /home/maria/anaconda3/lib/python3.9/site-packages (from spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (3.0.12)
Requirement already satisfied: sphinxcontrib-devhelp in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (1.0.2)
Requirement already satisfied: alabaster<0.8,>=0.7 in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (0.7.12)
Requirement already satisfied: sphinxcontrib-jsmath in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (1.0.1)
Requirement already satisfied: sphinxcontrib-applehelp in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (1.0.2)
Requirement already satisfied: sphinxcontrib-htmlhelp>=2.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (2.0.0)
Requirement already satisfied: imagesize in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (1.3.0)
Requirement already satisfied: snowballstemmer>=1.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (2.2.0)
Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (1.1.5)
Requirement already satisfied: sphinxcontrib-qthelp in /home/maria/anaconda3/lib/python3.9/site-packages (from sphinx<8,>=1.6->sphinx_rtd_theme->-r ../requirements.txt (line 23)) (1.0.3)
Requirement already satisfied: markdown>=2.6.8 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (3.3.4)
Requirement already satisfied: werkzeug>=1.0.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (2.0.3)
Requirement already satisfied: grpcio>=1.24.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (1.42.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (0.6.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (0.4.6)
Requirement already satisfied: absl-py>=0.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (1.2.0)
Requirement already satisfied: protobuf<3.20,>=3.9.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (3.19.1)
Requirement already satisfied: google-auth<3,>=1.6.3 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (1.33.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (1.8.1)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (4.2.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /home/maria/anaconda3/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (1.3.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/maria/anaconda3/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.9.1->pytorch_lightning~=1.7.6->-r ../requirements.txt (line 16)) (3.2.1)
Requirement already satisfied: confection<1.0.0,>=0.0.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from thinc<8.2.0,>=8.1.0->spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (0.0.4)
Requirement already satisfied: blis<0.8.0,>=0.7.8 in /home/maria/anaconda3/lib/python3.9/site-packages (from thinc<8.2.0,>=8.1.0->spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (0.7.9)
Requirement already satisfied: click<9.0.0,>=7.1.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from typer<0.8.0,>=0.3.0->spacy<4->fastai>=2.7.12->tsai~=0.3.4->-r ../requirements.txt (line 21)) (8.0.4)
Requirement already satisfied: argon2-cffi-bindings in /home/maria/anaconda3/lib/python3.9/site-packages (from argon2-cffi->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (21.2.0)
Requirement already satisfied: cffi>=1.0.1 in /home/maria/anaconda3/lib/python3.9/site-packages (from argon2-cffi-bindings->argon2-cffi->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (1.15.0)
Requirement already satisfied: pycparser in /home/maria/anaconda3/lib/python3.9/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (2.21)
Requirement already satisfied: soupsieve>1.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from beautifulsoup4->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (2.3.1)
Requirement already satisfied: webencodings in /home/maria/anaconda3/lib/python3.9/site-packages (from bleach->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (0.5.1)
Requirement already satisfied: arrow>=0.15.0 in /home/maria/anaconda3/lib/python3.9/site-packages (from isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.6.0->jupyter-server<3,>=2.4.0->notebook~=7.0.0->-r ../requirements.txt (line 26)) (1.2.2)
Requirement already satisfied: pure-eval in /home/maria/anaconda3/lib/python3.9/site-packages (from stack-data->ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.2.2)
Requirement already satisfied: asttokens in /home/maria/anaconda3/lib/python3.9/site-packages (from stack-data->ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (2.0.5)
Requirement already satisfied: executing in /home/maria/anaconda3/lib/python3.9/site-packages (from stack-data->ipython>=4.0.0->ipywidgets->-r ../requirements.txt (line 25)) (0.8.3)
Requirement already satisfied: patsy>=0.5.2 in /home/maria/anaconda3/lib/python3.9/site-packages (from statsmodels->pyod~=1.0.5->-r ../requirements.txt (line 12)) (0.5.2)
Requirement already satisfied: mpmath>=0.19 in /home/maria/anaconda3/lib/python3.9/site-packages (from sympy->torch==2.0.1->-r ../requirements.txt (line 8)) (1.2.1)
Installing collected packages: rpds-py, attrs, referencing, traitlets, platformdirs, jsonschema-specifications, tornado, pyzmq, jupyter-core, jsonschema, webcolors, uri-template, rfc3986-validator, rfc3339-validator, jupyter-client, jsonpointer, isoduration, fqdn, python-json-logger, jinja2, send2trash, overrides, jupyter-server-terminals, jupyter-events, requests, jupyter-server, babel, notebook-shim, jupyterlab-server, jupyter-lsp, async-lru, jupyterlab, notebook, sphinxcontrib-jquery, shapely, sphinx-rtd-theme, python-docs-theme
Attempting uninstall: attrs
Found existing installation: attrs 21.4.0
Uninstalling attrs-21.4.0:
Successfully uninstalled attrs-21.4.0
Attempting uninstall: traitlets
Found existing installation: traitlets 5.1.1
Uninstalling traitlets-5.1.1:
Successfully uninstalled traitlets-5.1.1
Attempting uninstall: tornado
Found existing installation: tornado 6.1
Uninstalling tornado-6.1:
Successfully uninstalled tornado-6.1
Attempting uninstall: pyzmq
Found existing installation: pyzmq 22.3.0
Uninstalling pyzmq-22.3.0:
Successfully uninstalled pyzmq-22.3.0
Attempting uninstall: jupyter-core
Found existing installation: jupyter-core 4.9.2
Uninstalling jupyter-core-4.9.2:
Successfully uninstalled jupyter-core-4.9.2
Attempting uninstall: jsonschema
Found existing installation: jsonschema 4.4.0
Uninstalling jsonschema-4.4.0:
Successfully uninstalled jsonschema-4.4.0
Attempting uninstall: jupyter-client
Found existing installation: jupyter-client 6.1.12
Uninstalling jupyter-client-6.1.12:
Successfully uninstalled jupyter-client-6.1.12
Attempting uninstall: jinja2
Found existing installation: Jinja2 2.11.3
Uninstalling Jinja2-2.11.3:
Successfully uninstalled Jinja2-2.11.3
Attempting uninstall: send2trash
Found existing installation: Send2Trash 1.8.0
Uninstalling Send2Trash-1.8.0:
Successfully uninstalled Send2Trash-1.8.0
Attempting uninstall: requests
Found existing installation: requests 2.27.1
Uninstalling requests-2.27.1:
Successfully uninstalled requests-2.27.1
Attempting uninstall: jupyter-server
Found existing installation: jupyter-server 1.13.5
Uninstalling jupyter-server-1.13.5:
Successfully uninstalled jupyter-server-1.13.5
Attempting uninstall: babel
Found existing installation: Babel 2.9.1
Uninstalling Babel-2.9.1:
Successfully uninstalled Babel-2.9.1
Attempting uninstall: jupyterlab-server
Found existing installation: jupyterlab-server 2.10.3
Uninstalling jupyterlab-server-2.10.3:
Successfully uninstalled jupyterlab-server-2.10.3
Attempting uninstall: jupyterlab
Found existing installation: jupyterlab 3.3.2
Uninstalling jupyterlab-3.3.2:
Successfully uninstalled jupyterlab-3.3.2
Attempting uninstall: notebook
Found existing installation: notebook 6.4.12
Uninstalling notebook-6.4.12:
Successfully uninstalled notebook-6.4.12
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
spyder 5.1.5 requires pyqt5<5.13, which is not installed.
spyder 5.1.5 requires pyqtwebengine<5.13, which is not installed.
anaconda-project 0.10.2 requires ruamel-yaml, which is not installed.
spyder-kernels 2.1.3 requires jupyter-client<7,>=5.3.4, but you have jupyter-client 7.4.9 which is incompatible.
nbclassic 0.3.5 requires jupyter-server~=1.8, but you have jupyter-server 2.7.3 which is incompatible.
nbclassic 0.3.5 requires notebook<7, but you have notebook 7.0.3 which is incompatible.
distributed 2022.2.1 requires dask==2022.02.1, but you have dask 2022.7.1 which is incompatible.
Successfully installed async-lru-2.0.4 attrs-23.1.0 babel-2.12.1 fqdn-1.5.1 isoduration-20.11.0 jinja2-3.1.2 jsonpointer-2.4 jsonschema-4.19.0 jsonschema-specifications-2023.7.1 jupyter-client-7.4.9 jupyter-core-5.3.1 jupyter-events-0.7.0 jupyter-lsp-2.2.0 jupyter-server-2.7.3 jupyter-server-terminals-0.4.4 jupyterlab-4.0.5 jupyterlab-server-2.24.0 notebook-7.0.3 notebook-shim-0.2.3 overrides-7.4.0 platformdirs-3.10.0 python-docs-theme-2023.7 python-json-logger-2.0.7 pyzmq-25.1.1 referencing-0.30.2 requests-2.31.0 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rpds-py-0.10.2 send2trash-1.8.2 shapely-1.8.5.post1 sphinx-rtd-theme-1.3.0 sphinxcontrib-jquery-4.1 tornado-6.3.3 traitlets-5.9.0 uri-template-1.3.0 webcolors-1.13
c) Import required libraries¶
In [5]:
# GENERAL
import os, sys
from os.path import dirname, abspath
# NUMPY
import numpy as np
# PANDAS
import pandas as pd
# XARRAY
import xarray as xr
# ITERTOOLS
import itertools
# TIME, DATETIME
import time
from datetime import datetime
# SCIPY
from scipy.interpolate import interp1d
# SKLEARN
from sklearn.preprocessing import RobustScaler
# DATASET CLASSES
from databases import *
# MODEL TEMPLATE CLASS
from backbones import *
# EVALUATION
from evaluators import *
# METRICS
from utils.misc import *
from utils.setup_config import setup
import yaml
from os.path import dirname, abspath
from pathlib import Path
import sys
# MATPLOTLIB
import matplotlib as mpl
mpl.rcParams['figure.dpi'] = 300
pd.set_option('display.max_rows', 50)
<frozen importlib._bootstrap>:228: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject /home/maria/anaconda3/lib/python3.9/site-packages/torch/cuda/__init__.py:107: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.) return torch._C._cuda_getDeviceCount() > 0
In [6]:
# Add current path into Python path
current_d = dirname(abspath("__file__"))
sys.path.insert(0, current_d)
2.1 DATABASE¶
2.1.1 DroughtED¶
The DroughtED classification database [Minixhofer, C] consists of globally available meteorological features paired with location meta-data. It can be downloaded by using this link.
- Data dimensionality:
Each entry is a daily drought level at a specific point in time in a specific US county, accompanied by 18 meteorological indicators.
- Drought levels:
DroughtED is a classification dataset with six levels of drought, depicted in the following image:
To work with the Outlier Detection algorithms, we binarize the categorization to obtain our anomalous and non-anomalous classes:
- Meteorological indicators:
The DroughtED database contains 18 meteorological indicators.
| - Minimum Wind Speed at 10 Meters (m/s) | - Specific Humidity at 2 Meters (g/kg) | - Temperature Range at 2 Meters (C) |
| - Wind Speed at 10 Meters (m/s) | - Temperature at 2 Meters (C) | - Minimum Wind Speed at 50 Meters (m/s) |
| - Maximum Temperature at 2 Meters (C) | - Wind Speed at 50 Meters (m/s) | - Earth Skin Temperature (C) |
| - Wind Speed Range at 50 Meters (m/s) | - Maximum Wind Speed at 50 Meters (m/s) | - Maximum Wind Speed at 10 Meters (m/s) |
| - Wind Speed Range at 10 Meters (m/s) | - Surface Pressure (kPa) | - Dew/Frost Point at 2 Meters (C) |
| - Minimum Temperature at 2 Meters (C) | - Wet Bulb Temperature at 2 Meters (C) | - Precipitation (mm day-1) |
- Data subsets:
This dataset contains data from 2000 up until 2020. For simplicity, in this tutorial we define the data subsets as follows:
| Split | Year Range (inclusive) |
|---|---|
| Train | 2018 |
| Validation | 2019 |
| Test | 2020 |
In [7]:
#files = {}
#for dirname, _, filenames in os.walk('databases/DroughtED/'):
# for filename in filenames:
# if 'valid' in filename:
# files['valid'] = os.path.join(dirname, filename)
#dfs = {
# k: pd.read_csv(files[k]).set_index(['fips', 'date'])
# for k in files.keys()
#}
#print(dfs['valid'])
2.1.2 The dataset file¶
For the users to be able to work with their own dataset, they have to create a dataset file NAME_database.py (DroughtED_database.py in this example) which will define a class NAME (DroughtED in this example). This class DroughtED will read, split the data and perform any preprocessing needed.
Unsupervised outlier detection algorithms available in the Python Outlier Detection (PyOD) library used as backend work with the whole set of data at the same time, so the function required in the DroughtED_database.py for this group of algorithms has to have the following structure:
Without going into detail about the whole script, in the following image you can see a snippet of how it should be:
The function returns x and labels, which have the following dimensionality:
In [8]:
# Load the validation split for demonstration purposes
config = setup(current_d + "/configs/config_DroughtED_OutlierDetection.yaml")
data_val = eval('DroughtED')(config, period = 'val').__getallitems__()
loaded 1100232 samples
loaded 1100232 samples
[86.4291349460841, 13.5708650539159]
In [9]:
# Show the returned objects from __getallitems__
print(data_val)
{'x': array([[ 0.16694717, 0.16145487, 0.17047794, ..., 0.00443387,
0.00355118, -0.00046997],
[ 0.02916026, 0.01976876, -0.09439664, ..., -0.39198976,
-0.35119544, -0.32302094]]), 'labels': array([0., 0., 0., ..., 0., 0., 0.])}
In [10]:
# Show one sample of data
print(data_val['x'].shape)
(2, 1100232)
In [11]:
# Show one sample of the ground-truth
print(data_val['labels'].shape)
(1100232,)
2.2 CHOOSING A MODEL¶
Unsupervised outlier detection algorithms are powered by the Python library PyOD [Zhao, Y.]. The user can choose up to 30 anomaly detection algorithms:
In this tutorial we will apply K-Nearest-Neighbor (KNN) [Angiulli, F.]
2.3 CONFIG FILE¶
User-defined file to specify the characteristics of the dataset and the chosen model. The following image shows the minimum required parameters that this file should contain:
For this turtorial, the architecture definition will be as follows:
The needed parameters for the architecture can be found in the PyOD API Documentation.
For the data section, we define the required arguments (name, data_dim and num_classes) but we can also make use of this configuration file to specify other parameters that will be used in the DroughtED_database.py file:
a) Load the configuration file and create folder results¶
In [12]:
YOUR_CONFIG_FILE_NAME = "config_DroughtED_OutlierDetection"
EXPERIMENT_ID = "jupyter_" + YOUR_CONFIG_FILE_NAME.split("_")[1] + "_"+ YOUR_CONFIG_FILE_NAME.split("_")[2] + "_" + str(time.time())
config_path = current_d + "/configs/" + YOUR_CONFIG_FILE_NAME + ".yaml"
config = setup(config_path)
config['experiment_id'] = EXPERIMENT_ID
#We manually set the experiment ID
config['experiment_id']= EXPERIMENT_ID
config['experiment_id']
Out[12]:
'jupyter_DroughtED_OutlierDetection_1694251312.5587056'
In [13]:
config
Out[13]:
{'name': 'AIDE',
'task': 'OutlierDetection',
'from_scratch': True,
'best_run_path': '',
'save_path': 'experiments/',
'data': {'name': 'DroughtED',
'data_dim': 1,
'num_classes': 2,
'class_bound': 1,
'root': './databases/DroughtED',
'data_file': 'data',
'input_size_train': 2,
'window_size': 1,
'features': ['PRECTOT',
'PS',
'QV2M',
'T2M',
'T2MDEW',
'T2MWET',
'T2M_MAX',
'T2M_MIN',
'T2M_RANGE',
'TS',
'WS10M',
'WS10M_MAX',
'WS10M_MIN',
'WS10M_RANGE',
'WS50M',
'WS50M_MAX',
'WS50M_MIN',
'WS50M_RANGE'],
'features_selected': [1, 3],
'train_slice': {'start': '2018-01-01', 'end': '2018-12-31'},
'val_slice': {'start': '2019-01-01', 'end': '2019-12-31'},
'test_slice': {'start': '2020-01-01', 'end': '2020-12-31'}},
'arch': {'user_defined': False,
'type': 'knn.KNN',
'args': {'contamination': 0.1,
'n_neighbors': 1,
'method': 'largest',
'radius': 1.0,
'algorithm': 'auto',
'leaf_size': 10,
'metric': 'minkowski',
'p': 2,
'metric_params': None,
'n_jobs': 1},
'input_model_dim': 1,
'output_model_dim': 1,
'step_samples_train': 5000,
'step_samples_evaluation': 2000},
'implementation': {'trainer': {'batch_size': 8},
'data_loader': {'num_workers': 16}},
'evaluation': {'visualization': {'activate': True}},
'experiment_id': 'jupyter_DroughtED_OutlierDetection_1694251312.5587056'}
b) Gather the data, splited in three sets: train, validation and test¶
In [14]:
#Create experimental folder structure if it was not already created
if not Path(config['save_path']).name == config['experiment_id'].replace('/', ''):
save_path= Path(config['save_path']) / Path(config['experiment_id'])
config['save_path']= str(save_path.resolve())
save_path.mkdir(exist_ok=True, parents=True)
else:
save_path= Path(config['save_path'])
print(f'Logging experiment data at: {save_path}')
# Datasets
classifier = PyodBackbone(config)
classifier.load_data()
Logging experiment data at: experiments/jupyter_DroughtED_OutlierDetection_1694251312.5587056
loaded 1100232 samples
loaded 1100232 samples
[73.46059740127536, 26.53940259872463]
loaded 1100232 samples
loaded 1100232 samples
[86.4291349460841, 13.5708650539159]
loaded 1103340 samples
loaded 1103340 samples
[76.66920441568328, 23.33079558431671]
c) Create the model with the parameters specified in the configuration file¶
TemplateModel1D is the python object that will be created. It does not require any modification from the user.
In [15]:
# Model
classifier.implement_model()
classifier.model
Out[15]:
<backbones.models.pyodModel.PyodModel at 0x7fe2ec3f0490>
d) Training stage¶
In [16]:
model= classifier.train()
KNN_train ROC:0.5, accuracy:0.7421, precision:0.0, recall:0.0, AUC-PR:0.2579 KNN_val ROC:0.5056, accuracy:0.8512, precision:0.4118, recall:0.0886, AUC-PR:0.1672
e) Test stage, evaluation and visualization of results¶
In [17]:
classifier.test()
KNN_test ROC:0.6595, accuracy:0.7808, precision:0.6087, recall:0.1111, AUC-PR:0.2705
In [18]:
output_data= classifier.inference()
Model 1 knn.KNN
Wrap-up:¶
In this tutorial we have seen:
- How to use your dataset with the toolbox
- How to choose your algorithm
- Required files to prepare before working with the toolbox
- How the algorithm is trained
- How the algorithm is evaluated
Reminder: To run your code you only need your database.py and the config.yaml files. After setting up them, you only need to execute the toolbox using the following instruction:
python main.py --config=./configs/config.yamlReferences¶
Angiulli, F., and Pizzuti, C., 2002. Fast outlier detection in high dimensional spaces. European Conference on Principles of Data Mining and Knowl- edge Discovery, pp. 15–27, Springer.
Minixhofer, C., Swan, M., McMeekin, C., & Andreadis, P. (2021, July). DroughtED: A dataset and methodology for drought forecasting spanning multiple climate zones. In Tackling Climate Change with Machine Learning: Workshop at ICML 2021
Zhao, Y., Nasrullah, Z. and Li, Z., 2019. PyOD: A Python Toolbox for Scalable Outlier Detection. Journal of machine learning research (JMLR), 20(96), pp.1-7.