Lab book Fluotracify
1 Technical Notes
1.1 README
1.1.1 General:
- This file corresponds to my lab book for my doctoral thesis tackling artifact correction in Fluorescence Correlation Spectroscopy (FCS) measurements using Deep Neural Networks. It also contains notes taken during the process of setting up this workflow for reproducible research.
- This file contains explanations of how things are organized, of the workflow for doing experiments, changes made to the code, and the observed behavior in the "* Data" section.
- The branching model used is described in this paper. Therefore: if you
are interested in the "* Data" section, you have to
git clonethe data branch of the repository. The main branch is clean from any results, it contains only source code and the analysis. - This project is my take on Open-notebook science. The idea was postulated in
a blog post in 2006:
… there is a URL to a laboratory notebook that is freely available and indexed on common search engines. It does not necessarily have to look like a paper notebook but it is essential that all of the information available to the researchers to make their conclusions is equally available to the rest of the world —Jean-Claude Bradley
- Proposal on how to deal with truly private data (e.g. notes from a confidential meeting with a colleague), which might otherwise be noted in a normal Lab notebook: do not include them here. Only notes relevant to the current project should be taken
1.1.2 Code block languages used in this document
# This is a sh block for shell / bash scripting. In the context of this file, # these blocks are mainly used for operations on my local computer. # In the LabBook.html rendering of this document, these blocks will have a # light green colour (#F0FBE9)
# This block can open and access tmux sessions, used for shell scripting on # remote computing clusters. # In the LabBook.html rendering of this document, these blocks will have a # distinct light green colour (#E1EED8)
# This is a python block. In the context of this file, it is seldomly used # (only for examplary scripts.) # In the LabBook.html rendering of this document, these blocks will have a # light blue colour (#E6EDF4)
# This is a jupyter-python block. The code is sent to a jupyter kernel running # on a remote high performance computing cluster. Most of my jupyter code is # executed this way. # In the LabBook.html rendering of this document, these blocks will have a # light orange colour (#FAEAE1)
;; This is a emacs-lisp block, the language used to customize Emacs, which is ;; sometimes necessary, since the reproducible workflow of this LabBook is ;; tightly integrated with Emacs and org-mode. ;; In the LabBook.html rendering of this document, these blocks will have a ;; light violet colour (#F7ECFB)
This is a literal example block. It can be used very flexibly - in the context of this document the output of most code blocks is displayed this way. In the LabBook.html rendering of this document, these blocks will have a light yellow colour (#FBFBBF)
This is a literal example block enclosed in a details block. This is useful to make the page more readable by collapsing large amounts of output. In the Labbook.html rendering of this document, the details block will have a light grey colour (#f0f0f0) and a pink color when hovering above it.
1.1.3 Experiments workflow:
- Create a new branch from
main - Print out the git log from the latest commit and the metadata
- Call the analysis scripts, follow the principles outlined in Organization of code
- All machine learning runs are saved in
data/mlruns, all other data indata/#experiment-name - Add a
** exp-<date>-<name>" section to this file under Data - Commit/push the results of this separate branch
- Merge this new branch with the remote
databranch
1.1.4 Example for experimental setup procedure
1.1.5 tools used (notes)
1.1.5.1 Emacs magit
gitflow-avh(magit-flow) to follow the flow- possibly https://github.com/magit/magit-annex for large files. Follow this: https://git-annex.branchable.com/walkthrough/
- maybe check out git-toolbelt at some point https://github.com/nvie/git-toolbelt#readme with https://nvie.com/posts/git-power-tools/
1.1.5.2 jupyter
- emacs jupyter for running and connecting to kernel on server: https://github.com/dzop/emacs-jupyter
- if I actually still would use .ipynb files, these might come handy:
- jupytext: https://github.com/mwouts/jupytext
- nbstripout: https://github.com/kynan/nbstripout
1.1.5.3 mlflow
1.1.5.4 tensorflow
1.2 Template for data entry and setup notes:
1.2.1 exp-#date-#title
1.2.1.1 git:
git log -1
1.2.1.2 System Metadata:
import os import pprint ramlist = os.popen('free -th').readlines()[-1].split()[1:] print('No of CPUs in system:', os.cpu_count()) print('No of CPUs the current process can use:', len(os.sched_getaffinity(0))) print('load average:', os.getloadavg()) print('os.uname(): ', os.uname()) print('PID of process:', os.getpid()) print('RAM total: {}, RAM used: {}, RAM free: {}'.format( ramlist[0], ramlist[1], ramlist[2])) !echo the current directory: $PWD !echo My disk usage: !df -h if _long: %conda list pprint.pprint(dict(os.environ), sort_dicts=False)
1.2.1.3 Tmux setup and scripts
rm ~/.tmux-local-socket-remote-machine REMOTE_SOCKET=$(ssh ara 'tmux ls -F "#{socket_path}"' | head -1) echo $REMOTE_SOCKET ssh ara -tfN \ -L ~/.tmux-local-socket-remote-machine:$REMOTE_SOCKET
| rm: | cannot | remove | 'home/lex.tmux-local-socket-remote-machine': | No | such | file | or | directory |
| ye53nis@ara-login01.rz.uni-jena.de's | password: | |||||||
| /tmp/tmux-67339/default | ||||||||
| > | ye53nis@ara-login01.rz.uni-jena.de's | password: |
1.2.1.4 SSH tunneling
Different applications can be run on the remote compute node. If I want to access them at the local machine, and open them with the browser, I use this tunneling script.
ssh -t -t ara -L $port:localhost:$port ssh $node -L $port:Localhost:$port
Apps I use that way:
- Jupyter lab for running Python 3-Kernels
- TensorBoard
- Mlflow ui
1.2.1.5 jupyter scripts
Starting a jupyter instance on a server where the necessary libraries are installed is easy using this script:
conda activate tf export PORT=9999 export XDG_RUNTIME_DIR='' export XDG_RUNTIME_DIR="" jupyter lab --no-browser --port=$PORT
On the compute node of the HPC, the users' environment is managed through
module files using the system Lmod. The export XDG_RUNTIME_DIR statements
are needed because of a jupyter bug which did not let it start. Right now,
ob-tmux does not support a :var header like normal org-babel does. So
the $port variable has to be set here in the template.
Now this port has to be tunnelled on our local computer (See SSH tunneling). While the tmux session above keeps running, no matter if Emacs is running or not, this following ssh tunnel needs to be active locally to connect to the notebook. If you close Emacs, it would need to be reestablished
1.2.2 Setup notes
1.2.2.1 Setting up a tmux connection from using ob-tmux in org-babel
- prerequisite: tmux versions need to be the same locally and on the server.
Let's verify that now.
- the local tmux version:
tmux -V
tmux 3.0a
- the remote tmux version:
ssh ara tmux -V
ye53nis@ara-login01.rz.uni-jena.de's password: tmux 3.0a
- the local tmux version:
- as is described in the ob-tmux readme, the following code snippet creates
a socket on the remote machine and forwards this socket to the local
machine (note that
socket_pathwas introduced in tmux version 2.2)REMOTE_SOCKET=$(ssh ara 'tmux ls -F "#{socket_path}"' | head -1) echo $REMOTE_SOCKET ssh ara -tfN \ -L ~/.tmux-local-socket-remote-machine:$REMOTE_SOCKET
ye53nis@ara-login01.rz.uni-jena.de's password: /tmp/tmux-67339/default > ye53nis@ara-login01.rz.uni-jena.de's password: - now a new tmux session with name
ob-NAMEis created when using a code block which looks like this:#+BEGIN_SRC tmux :socket ~/.tmux-local-socket-remote-machine :session NAME - Commands can be sent now to the remote tmux session, BUT note that the output is not printed yet
- there is a workaround for getting output back to our LabBook.org: A script
which allows to print the output from the tmux session in an
#+begin_example-Block below the tmux block by pressingC-c C-oorC-c C-v C-owhen the pointer is inside the tmux block.
1.2.2.2 emacs-jupyter Setup
Emacs-jupyter aims to be an API for a lot of functionalities of the
jupyter project. The documentation can be found on GitHub.
- For the whole document: connect to a running jupyter instance
M-x jupyter-server-list-kernels- set server URL, e.g.
http://localhost:8889 - set websocket URL, e.g.
http://localhost:8889
- set server URL, e.g.
- two possibilities
- kernel already exists \(\to\) list of kernels and
kernel-IDis displayed - kernel does not exist \(\to\) prompt asks if you want to start one \(\to\)
yes \(\to\) type kernel you want to start, e.g.
Python 3
- kernel already exists \(\to\) list of kernels and
- In the subtree where you want to use
jupyter-pythonblocks withorg babel- set the
:header-args:jupyter-python :session /jpy:localhost#kernel:8889-ID - customize the output folder using the following org-mode variable:
(setq org-babel-jupyter-resource-directory "./data/exp-test/plots")
./data/exp-test/plots
- set the
- For each individual block, the following customizations might be useful
- jupyter kernels can return multiple kinds of rich output (images,
html, …) or scalar data (plain text, numbers, lists, …). To force
a plain output, use
:results scalar. To show the output in the minibuffer only, use:results silent - to change the priority of different rich outputs, use
:displayheader argument, e.g.:display text/plain text/htmlprioritizes plain text over html. All supported mimetypes in default order:- text/org
- image/svg+xml, image/jpeg, image/png
- text/html
- text/markdown
- text/latex
- text/plain
- We can set jupyter to output pandas DataFrames as org tables
automatically using the source block header argument
:pandoc t - useful keybindings
M-ito open the documentation for wherever your pointer is (like pressingShift-TABin Jupyter notebooks)C-c C-ito interrupt the kernel,C-c C-rto restart the kernel
- jupyter kernels can return multiple kinds of rich output (images,
html, …) or scalar data (plain text, numbers, lists, …). To force
a plain output, use
1.2.3 Notes on archiving
1.2.3.1 Exporting the LabBook.org to html in a twbs style
- I am partial to the twitter bootstrap theme of html, since I like it's simple design, but clear structure with a nice table of contents at the side → the following org mode extension supports a seemless export to twitter bootstrap html: https://github.com/marsmining/ox-twbs
- when installed, the export can be triggered via the command
(org-twbs-export-as-html)or via the keyboard shortcut for exportC-c C-efollowed bywfor Twitter bootstrap andhfor saving the .html - Things to configure:
- in general, there are multiple export options: https://orgmode.org/manual/Export-Settings.html
- E.g. I set 2
#+OPTIONSkeywords at the begin of the file:toc:4andH:4which make sure that in my export my sidebar table of contents will show numbered headings till a depth of 4. - I configured my code blocks so that they will not be evaluated when
exporting (I would recommend this especially if you only export for
archiving) and that both the code block and the output will be exported
with the keyword:
#+PROPERTY: header-args :eval never-export :exports both - To discriminate between code blocks for different languages I gave each
of them a distinct colour using
#+HTML_HEAD_EXTRA: <style...(see above) - I had to configure a style for
table, so that thedisplay: block; overflow-x: auto;gets the table to be restricted to the width of the text and if it is larger, activates scrollingwhite-space: nowrap;makes it that there is no wrap in a column, so it might be broader, but better readable if you have scrolling anyway
- Things to do before exporting / Troubleshooting while exporting:
- when using a dark theme for you emacs, the export of the code blocks
might show some ugly dark backgrounds from the theme. If this becomes
an issue, change to a light theme for the export with
M-x (load-theme)and choosesolarized-light - only in the
databranch you set the git tags after merging. If you want to show them here, execute the corresponding function in Git TAGs - make sure your file links work properly! I recommend referencing your files relatively (e.g. [ [ f ile:./data/exp-XXXXXX-test/test.png]] without spaces). Otherwise there will be errors in your Messages buffer
- There might be errors with your code blocks
- e.g. the export function expects you to assign a default variable to your functions
- if you call a function via the
#+CALLmechanism, it wants you to include two parentheses for the function, e.g.#+CALL: test()
- check indentation of code blocks inside lists
- add a
detailsblock around large output cells. This makes them expandable. I added some#+HTML_HEAD_EXTRA: <style...inspired by alhassy. That's how thedetailsblock looks like:#+begin_details #+end_details
- If you reference a parameter with an underscore in the name, use the
org markdown tricks to style them like code (
==or~~), otherwise the part after the underscore will be rendered like a subscript:under_scorevs underscore
- when using a dark theme for you emacs, the export of the code blocks
might show some ugly dark backgrounds from the theme. If this becomes
an issue, change to a light theme for the export with
- Things to do after exporting:
- In my workflow, the exported
LabBook.htmlwith the overview of all experiments is in thedatafolder. If you move the file, you will have to fix the file links for the new location, e.g. via "Find and replace"M-%:- if you move the org file → in the org file find
[[file:./data/and replace with[[file:./→ then export withC-c C-e w h - if you export first with
C-c C-e w hand move the html file todata→ in the html file find./dataand replace with.
- if you move the org file → in the org file find
- In my workflow, the exported
1.3 Organization of git
1.3.1 remote/origin/main branch:
- contains all the source code in folder src/ which is used for experiments.
- contains the LabBook.org template
- contains setup- and metadata files such as MLproject or conda.yaml
- the log contains only lasting alterations on the folders and files mentioned above, which are e.g. used for conducting experiments or which introduce new features. Day-to-day changes in code
1.3.2 remote/origin/exp### branches:
- if an experiment is done, the code and templates will be branched out from main in an #experiment-name branch, ### meaning some meaningful descriptor.
- all data generated during the experiment (e.g. .csv files, plots, images, etc), is stored in a folder with the name data/#experiment-name, except machine learning-specific data and metadata from `mlflow` runs, which are saved under data/mlruns (this allows easily comparing machine learning runs with different experimental settings)
- The LabBook.org file is essential
- If possible, all code is executed from inside this file (meaning analysis scripts or calling the code from the scr/ directory).
- All other steps taken during an experiment are noted down, as well as conclusions or my thought process while conducting the experiment
- Provenance data, such as metadata about the environment the code was executed in, the command line output of the code, and some plots
1.3.3 remote/origin/develop branch:
- this is the branch I use for day to day work on features and exploration. All of my current activity can be followed here.
1.3.4 remote/origin/data branch:
- contains a full cronicle of the whole research process
- all #experiment-name branches are merged here. Afterwards the original branch is deleted and on the data branch there is a Git tag which shows the merge commit to make accessing single experiments easy.
- the develop branch is merged here as well.
1.3.5 Git TAGs
1.3.5.1 Stable versions:
1.3.5.2 All tags from git:
git push origin --tags git tag -n1
exp-200402-test Merge branch 'exp-200402-test' into data exp-200520-unet Merge branch 'exp-310520-unet' into data exp-200531-unet Merge branch 'heads/exp-310520-unet' into data exp-201231-clustsim exp-201231-clustsim exp-210204-unet Add exp-210204-unet LabBook part 3 exp-310520-unet move exp-310520-unet to data branch manually
1.4 Organization of code
1.4.1 scripts:
1.4.2 src/
1.4.2.1 fluotracify/
- imports/
- simulations/
- training/
- applications/
- doc/
- use Sphinx
- follow this: https://daler.github.io/sphinxdoc-test/includeme.html
- evtl export org-mode Readme to rst via https://github.com/msnoigrs/ox-rst
- possibly heavily use http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html
- for examples sphinx-galleries could be useful https://sphinx-gallery.github.io/stable/getting_started.html
- use Sphinx
1.4.2.2 nanosimpy/
- cloned from dwaithe with refactoring for Python 3-compatibility
1.5 Changes in this repository (without "* Data" in this file)
1.5.1 Changes in LabBook.org (without "* Data")
1.5.1.1 2022-02-19
- Add
#+HTML_HEAD_EXTRA: <style...fortableto enable scrolling if the table overflows
1.5.1.2 2021-12-16
- Add
detailsblocks, corresponding#+HTML_HEAD_EXTRA: <style...and documentation in Notes on archiving
1.5.1.3 2021-08-05
- Rename
masterbranch tomainbranch
1.5.1.4 2021-04-04
- Add
#+OPTIONS: H:4and#+OPTIONS: toc:4to show up to 4 levels of depth in the html (twbs) export of this LabBook in the table of contents at the side - I added Notes on archiving
1.5.1.5 2020-11-04
- update "jupyter scripts" in Template for data entry and setup notes:
for new conda environment on server (now
conda activate tf-nightly)
1.5.1.6 2020-05-31
- extend general documentation in README
- Add code block examples
- extend documentation on experiment workflow
- move setup notes from README to "Template for data entry and setup notes"
- remove emacs-lisp code for custom tmux block functions (not relevant enough)
- change named "jpt-tmux" from starting a jupyter notebook to starting
jupyter lab. Load a conda environment instead of using Lmod's
module load
1.5.1.7 2020-05-07
- extend documentation on git model
- extend documentation on jupyter setup
1.5.1.8 2020-04-22
- added parts of README which describe the experimental process
- added templates for system metadata, tmux, jupyter setup
- added organization of code
1.5.1.9 2020-03-30
- set up lab book and form git repo accoring to setup by Luka Stanisic et al
1.5.2 Changes in src/fluotracify
2 Data
2.1 exp-310520-unet
2.1.1 Connect
2.1.1.1 Tmux on Ara
| ye53nis@ara-login01.rz.uni-jena.de's | password: | |
| /tmp/tmux-67339/default | ||
| > | ye53nis@ara-login01.rz.uni-jena.de's | password: |
Test:
pwd
(tensorflow_nightly) [ye53nis@login01 drmed-git]$ pwd /beegfs/ye53nis/drmed-git
pwd
2.1.1.2 Compute node for script execution
srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
(base) [ye53nis@node151 drmed-git]$
2.1.1.3 Jupyter on Ara
- Request compute node via tmux
srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
(base) [ye53nis@node189 drmed-git]$
cd /home/ye53nis/DOKTOR
- Start Jupyter Lab
[I 10:54:50.672 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/jupyterlab [I 10:54:50.673 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tensorflow_nightly/share/jupyter/lab [I 10:54:50.678 LabApp] Serving notebooks from local directory: /home/ye53nis/DOKTOR [I 10:54:50.678 LabApp] The Jupyter Notebook is running at: [I 10:54:50.678 LabApp] http://localhost:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b [I 10:54:50.678 LabApp] or http://127.0.0.1:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b [I 10:54:50.678 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 10:54:50.724 LabApp] To access the notebook, open this file in a browser: file:///home/ye53nis/.local/share/jupyter/runtime/nbserver-365563-open.html Or copy and paste one of these URLs: http://localhost:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b or http://127.0.0.1:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
| sh-5.0$ | ye53nis@ara-login01.rz.uni-jena.de's | password: | |||||||
| ye53nis@node020's | password: | channel | 3: | open | failed: | connect | failed: | Connection | refused |
| channel | 3: | open | failed: | connect | failed: | Connection | refused | ||
| Last | login: | Sun | Jul | 5 | 09:52:47 | 2020 | from | login01.ara |
I started a Python3 kernel using jupyter-server-list-kernels. Then I added the
kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.
python3 7fe3d66b-d802-4f0f-b105-9536a917b816 a few seconds ago starting 0
Test:
No of CPUs in system: 72
No of CPUs the current process can use: 24
load average: (1.06, 1.01, 0.97)
os.uname(): posix.uname_result(sysname='Linux', nodename='node218', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
PID of process: 10254
RAM total: 199G, RAM used: 6.0G, RAM free: 181G
the current directory: /home/ye53nis/DOKTOR
My disk usage:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 3.2G 47G 7% /
devtmpfs 94G 0 94G 0% /dev
tmpfs 94G 725M 94G 1% /dev/shm
tmpfs 94G 75M 94G 1% /run
tmpfs 94G 0 94G 0% /sys/fs/cgroup
nfs02-ib:/data01 88T 61T 27T 70% /data01
nfs03-ib:/pool/work 100T 79T 22T 79% /nfsdata
nfs01-ib:/home 80T 61T 20T 76% /home
nfs01-ib:/cluster 2.0T 316G 1.7T 16% /cluster
/dev/sda3 6.0G 412M 5.6G 7% /var
/dev/sda6 169G 2.8G 166G 2% /local
/dev/sda5 2.0G 34M 2.0G 2% /tmp
beegfs_nodev 524T 453T 72T 87% /beegfs
tmpfs 19G 0 19G 0% /run/user/67339
/bin/sh: conda: command not found
{'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
'SLURM_NODELIST': 'node218',
'SLURM_JOB_NAME': 'bash',
'XDG_SESSION_ID': '8541',
'SLURMD_NODENAME': 'node218',
'SLURM_TOPOLOGY_ADDR': 'node218',
'SLURM_NTASKS_PER_NODE': '24',
'HOSTNAME': 'login01',
'SLURM_PRIO_PROCESS': '0',
'SLURM_SRUN_COMM_PORT': '45911',
'SHELL': '/bin/bash',
'TERM': 'xterm-color',
'SLURM_JOB_QOS': 'qstand',
'SLURM_PTY_WIN_ROW': '52',
'HISTSIZE': '1000',
'TMPDIR': '/tmp',
'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
'SSH_CLIENT': '10.231.190.186 48592 22',
'CONDA_SHLVL': '2',
'CONDA_PROMPT_MODIFIER': '(tensorflow_nightly) ',
'GSETTINGS_SCHEMA_DIR_CONDA_BACKUP': '',
'WINDOWID': '0',
'OLDPWD': '/beegfs/ye53nis/drmed-git',
'QTDIR': '/usr/lib64/qt-3.3',
'QTINC': '/usr/lib64/qt-3.3/include',
'SSH_TTY': '/dev/pts/57',
'QT_GRAPHICSSYSTEM_CHECKED': '1',
'SLURM_NNODES': '1',
'USER': 'ye53nis',
'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
'CONDA_EXE': '/cluster/miniconda3/bin/conda',
'SLURM_STEP_NUM_NODES': '1',
'SLURM_JOBID': '392142',
'SRUN_DEBUG': '3',
'SLURM_NTASKS': '24',
'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
'SLURM_STEP_ID': '0',
'TMUX': '/tmp/tmux-67339/default,47311,2',
'_CE_CONDA': '',
'CONDA_PREFIX_1': '/cluster/miniconda3',
'SLURM_STEP_LAUNCHER_PORT': '45911',
'SLURM_TASKS_PER_NODE': '24',
'MAIL': '/var/spool/mail/ye53nis',
'PATH': '/home/ye53nis/.conda/envs/tensorflow_nightly/bin:/home/lex/Programme/miniconda3/envs/tensorflow_env/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/lex/Programme/miniconda3/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
'GSETTINGS_SCHEMA_DIR': '/home/ye53nis/.conda/envs/tensorflow_nightly/share/glib-2.0/schemas',
'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
'SLURM_JOB_ID': '392142',
'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tensorflow_nightly',
'SLURM_JOB_USER': 'ye53nis',
'SLURM_STEPID': '0',
'PWD': '/home/ye53nis/DOKTOR',
'SLURM_SRUN_COMM_HOST': '192.168.192.5',
'LANG': 'en_US.UTF-8',
'SLURM_PTY_WIN_COL': '206',
'SLURM_UMASK': '0022',
'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
'SLURM_JOB_UID': '67339',
'LOADEDMODULES': '',
'SLURM_NODEID': '0',
'TMUX_PANE': '%2',
'SLURM_SUBMIT_DIR': '/beegfs/ye53nis/drmed-git',
'SLURM_TASK_PID': '696',
'SLURM_NPROCS': '24',
'SLURM_CPUS_ON_NODE': '24',
'SLURM_DISTRIBUTION': 'block',
'https_proxy': 'https://internet4nzm.rz.uni-jena.de:3128',
'SLURM_PROCID': '0',
'HISTCONTROL': 'ignoredups',
'_CE_M': '',
'SLURM_JOB_NODELIST': 'node218',
'SLURM_PTY_PORT': '37811',
'HOME': '/home/ye53nis',
'SHLVL': '3',
'SLURM_LOCALID': '0',
'SLURM_JOB_GID': '13280',
'SLURM_JOB_CPUS_PER_NODE': '24',
'SLURM_CLUSTER_NAME': 'hpc',
'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
'SLURM_SUBMIT_HOST': 'login01',
'SLURM_JOB_PARTITION': 's_standard',
'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
'LOGNAME': 'ye53nis',
'SLURM_STEP_NUM_TASKS': '24',
'QTLIB': '/usr/lib64/qt-3.3/lib',
'SLURM_JOB_ACCOUNT': 'iaob',
'SLURM_JOB_NUM_NODES': '1',
'MODULESHOME': '/usr/share/Modules',
'CONDA_DEFAULT_ENV': 'tensorflow_nightly',
'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
'SLURM_STEP_TASKS_PER_NODE': '24',
'PORT': '8889',
'SLURM_STEP_NODELIST': 'node218',
'DISPLAY': ':0',
'XDG_RUNTIME_DIR': '',
'XAUTHORITY': '/tmp/xauth-1000-_0',
'BASH_FUNC_module()': '() { eval `/usr/bin/modulecmd bash $*`\n}',
'_': '/home/ye53nis/.conda/envs/tensorflow_nightly/bin/jupyter',
'KERNEL_LAUNCH_TIMEOUT': '40',
'JPY_PARENT_PID': '7566',
'CLICOLOR': '1',
'PAGER': 'cat',
'GIT_PAGER': 'cat',
'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
2.1.1.4 Tensorboard tunnel, Mlflow ui tunnel
| sh-5.0$ | ye53nis@ara-login01.rz.uni-jena.de's | password: | ||||||
| ye53nis@node171's | password: | |||||||
| Last | login: | Sun | May | 31 | 15:29:02 | 2020 | from | login01.ara |
2.1.2 git exp 1
git status git log -1
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ git status
# On branch exp-310520-unet
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# data/
# experiment_params.csv
# mlruns/
# tramp.YDPCnB
nothing added to commit but untracked files present (use "git add" to track)
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ git log -1
commit 2225d6fa18cca9044960b6e86a56ec9fb4362d5d
Author: Apoplex <oligolex@vivaldi.net>
Date: Sun May 31 21:34:12 2020 +0200
Change learning rate
2.1.3 mlflow environment variables
conda activate tensorflow_nightly cd /beegfs/ye53nis/drmed-git export MLFLOW_EXPERIMENT_NAME=exp-310520-unet export MLFLOW_TRACKING_URI=file:./data/mlruns mkdir data/exp-310520-unet
(tensorflow_nightly) [ye53nis@node171 drmed-git]$
2.1.4 Learning rate schedule
learning_rate = 0.2 if epoch > 10: learning_rate = 0.02 if epoch > 20: learning_rate = 0.01 if epoch > 40: learning_rate = 0.001 if epoch > 60: learning_rate = 0.0001 if epoch > 80: learning_rate = 0.00001
2.1.5 test runs
2.1.5.1 no 1 - experiment creation failed
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P steps_per_epoch=400 -P validation_steps=200
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P
steps_per_epoch=400 -P validation_steps=200
WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
Traceback (most recent call last):
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
experiment = self._get_experiment(exp_id, view_type)
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
INFO: 'exp-310520-unet' does not exist. Creating a new experiment
WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
Traceback (most recent call last):
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
experiment = self._get_experiment(exp_id, view_type)
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
Traceback (most recent call last):
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
experiment = self._get_experiment(exp_id, view_type)
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
2020/05/31 14:00:34 INFO mlflow.projects: === Created directory /tmp/tmpfhabm3p4 for downloading remote URIs passed to arguments of type 'path' ===
2020/05/31 14:00:34 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f
luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 2 /beegfs/ye53nis/saves/firstartifact_Mar2020 400 200' in run with ID 'b37371694a004638a6cd7a94d4d2e77f' ===
2020-05-31 14:00:38.667018: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No
such file or directory
2020-05-31 14:00:38.667107: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo
ur of importlib; see the module's documentation for alternative uses
import imp
2.3.0-dev20200527
2020-05-31 14:00:42.422965: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file
or directory
2020-05-31 14:00:42.422998: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-05-31 14:00:42.423020: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist
GPUs: []
train 0 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set027.csv
train 1 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set087.csv
train 2 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set003.csv
train 3 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set056.csv
train 4 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set076.csv
train 5 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set094.csv
train 6 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set017.csv
train 7 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set074.csv
train 8 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set055.csv
train 9 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set096.csv
train 10 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set054.csv
train 11 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set093.csv
train 12 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set079.csv
train 13 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set014.csv
train 14 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set008.csv
train 15 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set031.csv
train 16 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set023.csv
train 17 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set025.csv
train 18 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set034.csv
train 19 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set009.csv
train 20 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set044.csv
train 21 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set063.csv
train 22 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set004.csv
train 23 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set072.csv
train 24 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set046.csv
train 25 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set049.csv
train 26 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set007.csv
train 27 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set100.csv
train 28 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set083.csv
train 29 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set077.csv
train 30 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set061.csv
train 31 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set081.csv
train 32 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set091.csv
train 33 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set069.csv
train 34 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set052.csv
train 35 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set028.csv
train 36 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set019.csv
train 37 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set057.csv
train 38 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set064.csv
train 39 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set075.csv
train 40 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set002.csv
train 41 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set062.csv
train 42 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set043.csv
train 43 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set042.csv
train 44 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set005.csv
train 45 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set016.csv
train 46 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set018.csv
train 47 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set041.csv
train 48 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set039.csv
train 49 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set006.csv
train 50 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set092.csv
train 51 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set060.csv
train 52 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set001.csv
train 53 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set035.csv
train 54 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set029.csv
train 55 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set051.csv
train 56 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set012.csv
train 57 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set036.csv
train 58 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set024.csv
train 59 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set053.csv
train 60 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set011.csv
train 61 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set032.csv
train 62 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set067.csv
train 63 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set058.csv
train 64 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set080.csv
train 65 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set086.csv
train 66 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set033.csv
train 67 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set085.csv
train 68 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set015.csv
train 69 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set090.csv
train 70 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set020.csv
train 71 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set030.csv
train 72 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set050.csv
train 73 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set098.csv
train 74 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set099.csv
train 75 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set070.csv
train 76 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set021.csv
train 77 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set095.csv
train 78 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set073.csv
train 79 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set078.csv
test 80 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set026.csv
test 81 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set038.csv
test 82 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set082.csv
test 83 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set047.csv
test 84 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set040.csv
test 85 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set066.csv
test 86 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set059.csv
test 87 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set013.csv
test 88 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set089.csv
test 89 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set071.csv
test 90 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set088.csv
test 91 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set037.csv
test 92 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set022.csv
test 93 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set084.csv
test 94 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set010.csv
test 95 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set097.csv
test 96 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set068.csv
test 97 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set065.csv
test 98 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set048.csv
test 99 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set045.csv
shapes of feature dataframe: (16384, 8000) and label dataframe: (16384, 8000)
shapes of feature dataframe: (16384, 2000) and label dataframe: (16384, 2000)
for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
label001_1 16384
label001_1 16384
label001_1 16384
label001_1 16384
label001_1 16384
dtype: int64
2020-05-31 14:05:15.534482: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera
tions: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-05-31 14:05:15.544530: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-05-31 14:05:15.545867: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557b10ed44b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-31 14:05:15.545896: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600
------------------------
number of test examples: 2000
input - shape: (None, 16384, 1)
output - shape: (None, 16384, 1)
2020-05-31 14:05:22.334555: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A
BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho
n 3.0, use inspect.signature() or inspect.getfullargspec()
all_param_names, _, _, all_default_values = inspect.getargspec(fn) # pylint: disable=W1505
Epoch 1/2
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from
'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
if not isinstance(values, collections.Sequence):
1/400 [..............................] - ETA: 0s - loss: 0.7153 - tp: 66972.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 14948.0000 - precision: 1.0000 - recall: 0.8175 - accuracy: 0.8175 - auc: 0.0000e+0
02020-05-31 14:05:34.823955: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager
.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-05-31 14:05:36.334154: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36
2020-05-31 14:05:36.348656: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.trace.json.gz
2020-05-31 14:05:36.375326: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36
2020-05-31 14:05:36.375452: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.memory
_profile.json.gz
2020-05-31 14:05:36.377489: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36Dumped tool data for xplane.pb to /tmp/tb/trai
n/plugins/profile/2020_05_31_14_05_36/node171.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.kernel_stats.pb
400/400 [==============================] - 651s 2s/step - loss: 0.0019 - tp: 32753052.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 14948.0000 - precision: 1.0000 - recall: 0.9995 - accuracy: 0.9995 - auc: 0
.0000e+00 - val_loss: 7.3428e-23 - val_tp: 16384000.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e
+00
Epoch 2/2
400/400 [==============================] - 645s 2s/step - loss: 2.1658e-09 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - au
c: 0.0000e+00 - val_loss: 8.7165e-23 - val_tp: 16384000.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0
000e+00
400/400 [==============================] - 119s 298ms/step - loss: 9.0078e-23 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 -
auc: 0.0000e+00
2020/05/31 14:29:15 INFO mlflow.projects: === Run (ID 'b37371694a004638a6cd7a94d4d2e77f') succeeded ===
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ Regard
mlflow ui
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow ui [2020-05-31 15:27:54 +0200] [200865] [INFO] Starting gunicorn 20.0.4 [2020-05-31 15:27:54 +0200] [200865] [INFO] Listening at: http://127.0.0.1:5000 (200865) [2020-05-31 15:27:54 +0200] [200865] [INFO] Using worker: sync [2020-05-31 15:27:54 +0200] [200871] [INFO] Booting worker with pid: 200871
2.1.5.2 no 2
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P steps_per_epoch=10 -P validation_steps=10
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P steps_per_epoch=10 -P validation_steps=10 INFO: 'exp-310520-unet' does not exist. Creating a new experiment 2020/05/31 15:45:30 INFO mlflow.projects: === Created directory /tmp/tmp2vv5kiic for downloading remote URIs passed to arguments of type 'path' === 2020/05/31 15:45:30 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 2 /beegfs/ye53nis/saves/firstartifact_Mar2020 10 10' in run with ID '7f23f6ba7a244914b3cbbebd731d50a1' === 2020-05-31 15:45:31.886019: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2020-05-31 15:45:31.886068: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo ur of importlib; see the module's documentation for alternative uses import imp 2.3.0-dev20200527 2020-05-31 15:45:35.151500: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-05-31 15:45:35.151549: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303) 2020-05-31 15:45:35.151589: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist GPUs: [] train 0 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set027.csv train 1 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set087.csv train 2 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set003.csv train 3 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set056.csv train 4 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set076.csv train 5 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set094.csv train 6 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set017.csv train 7 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set074.csv train 8 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set055.csv train 9 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set096.csv train 10 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set054.csv train 11 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set093.csv train 12 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set079.csv train 13 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set014.csv train 14 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set008.csv train 15 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set031.csv train 16 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set023.csv train 17 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set025.csv train 18 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set034.csv train 19 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set009.csv train 20 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set044.csv train 21 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set063.csv train 22 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set004.csv train 23 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set072.csv train 24 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set046.csv train 25 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set049.csv train 26 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set007.csv train 27 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set100.csv train 28 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set083.csv train 29 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set077.csv train 30 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set061.csv train 31 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set081.csv train 32 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set091.csv train 33 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set069.csv train 34 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set052.csv train 35 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set028.csv train 36 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set019.csv train 37 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set057.csv train 38 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set064.csv train 39 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set075.csv train 40 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set002.csv train 41 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set062.csv train 42 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set043.csv train 43 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set042.csv train 44 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set005.csv train 45 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set016.csv train 46 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set018.csv train 47 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set041.csv train 48 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set039.csv train 49 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set006.csv train 50 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set092.csv train 51 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set060.csv train 52 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set001.csv train 53 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set035.csv train 54 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set029.csv train 55 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set051.csv train 56 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set012.csv train 57 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set036.csv train 58 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set024.csv train 59 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set053.csv train 60 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set011.csv train 61 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set032.csv train 62 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set067.csv train 63 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set058.csv train 64 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set080.csv train 65 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set086.csv train 66 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set033.csv train 67 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set085.csv train 68 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set015.csv train 69 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set090.csv train 70 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set020.csv train 71 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set030.csv train 72 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set050.csv train 73 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set098.csv train 74 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set099.csv train 75 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set070.csv train 76 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set021.csv train 77 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set095.csv train 78 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set073.csv train 79 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set078.csv test 80 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set026.csv test 81 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set038.csv test 82 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set082.csv test 83 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set047.csv test 84 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set040.csv test 85 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set066.csv test 86 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set059.csv test 87 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set013.csv test 88 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set089.csv test 89 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set071.csv test 90 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set088.csv test 91 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set037.csv test 92 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set022.csv test 93 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set084.csv test 94 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set010.csv test 95 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set097.csv test 96 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set068.csv test 97 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set065.csv test 98 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set048.csv test 99 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set045.csv shapes of feature dataframe: (16384, 8000) and label dataframe: (16384, 8000) shapes of feature dataframe: (16384, 2000) and label dataframe: (16384, 2000) for each 20,000 timestap trace there are the following numbers of corrupted timesteps: label001_1 16384 label001_1 16384 label001_1 16384 label001_1 16384 label001_1 16384 dtype: int64 2020-05-31 15:49:25.643578: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera tions: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-05-31 15:49:25.653635: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz 2020-05-31 15:49:25.655125: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556ab2694730 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-05-31 15:49:25.655169: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version number of training examples: 6400, number of validation examples: 1600 ------------------------ number of test examples: 2000 input - shape: (None, 16384, 1) output - shape: (None, 16384, 1) 2020-05-31 15:49:32.395226: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(wrapped_dict, collections.Mapping): /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho n 3.0, use inspect.signature() or inspect.getfullargspec() all_param_names, _, _, all_default_values = inspect.getargspec(fn) # pylint: disable=W1505 Epoch 1/2 /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(values, collections.Sequence): 1/10 [==>...........................] - ETA: 0s - loss: 0.5648 - tp: 72554.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 9366.0000 - precision: 1.0000 - recall: 0.8857 - accuracy: 0.8857 - auc: 0.0000e+0020 20-05-31 15:49:44.598769: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager .profiler) is deprecated and will be removed after 2020-07-01. Instructions for updating: use `tf.profiler.experimental.stop` instead. 2020-05-31 15:49:46.123954: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46 2020-05-31 15:49:46.137599: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.trace.json.gz 2020-05-31 15:49:46.163428: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46 2020-05-31 15:49:46.163531: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.memory _profile.json.gz 2020-05-31 15:49:46.165569: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46Dumped tool data for xplane.pb to /tmp/tb/trai n/plugins/profile/2020_05_31_15_49_46/node171.xplane.pb Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.overview_page.pb Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.kernel_stats.pb 10/10 [==============================] - 23s 2s/step - loss: 0.0638 - tp: 809834.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 9366.0000 - precision: 1.0000 - recall: 0.9886 - accuracy: 0.9886 - auc: 0.0000e +00 - val_loss: 0.0000e+00 - val_tp: 819200.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e+00 Epoch 2/2 10/10 [==============================] - 18s 2s/step - loss: 6.4955e-17 - tp: 819200.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - auc: 0. 0000e+00 - val_loss: 0.0000e+00 - val_tp: 819200.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e+00 400/400 [==============================] - 120s 300ms/step - loss: 0.0000e+00 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - auc: 0.0000e+00 2020/05/31 15:52:36 INFO mlflow.projects: === Run (ID '7f23f6ba7a244914b3cbbebd731d50a1') succeeded ===
These two test runs used the wrong dataset! the firstartifact_Mar2020 dataset
is bright clusters, but the label is the uncorrupted trace, not just the
artifact information. This dataset is meant for a Variational
Autoencoder-Training.
2.1.6 experimental run
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=70 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=400 -P validation_steps=200
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=70 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 - P steps_per_epoch=400 -P validation_steps=200 2020/05/31 21:39:53 INFO mlflow.projects: === Created directory /tmp/tmpnc10f5ut for downloading remote URIs passed to arguments of type 'path' === 2020/05/31 21:39:53 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 70 /beegfs/ye53nis/saves/firstartefact_Sep2019 400 200' in run with ID '1aefda1366f04f5da5d1fc2241ad9208' === 2020-05-31 21:39:54.753597: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2020-05-31 21:39:54.753646: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo ur of importlib; see the module's documentation for alternative uses import imp 2.3.0-dev20200527 2020-05-31 21:39:57.636018: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-05-31 21:39:57.636052: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303) 2020-05-31 21:39:57.636075: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist GPUs: [] train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000) shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000) for each 20,000 timestap trace there are the following numbers of corrupted timesteps: label001_1 6286 label001_1 2568 label001_1 4495 label001_1 4414 label001_1 1105 dtype: int64 2020-05-31 21:44:15.811359: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera tions: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-05-31 21:44:15.820877: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz 2020-05-31 21:44:15.822374: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55acbe3932e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-05-31 21:44:15.822402: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version number of training examples: 6400, number of validation examples: 1600 ------------------------ number of test examples: 2000 input - shape: (None, 16384, 1) output - shape: (None, 16384, 1) 2020-05-31 21:44:19.667769: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(wrapped_dict, collections.Mapping): /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho n 3.0, use inspect.signature() or inspect.getfullargspec() all_param_names, _, _, all_default_values = inspect.getargspec(fn) # pylint: disable=W1505 Epoch 1/70 /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(values, collections.Sequence): 1/400 [..............................] - ETA: 0s - loss: 1.5974 - tp: 3783.0000 - fp: 23102.0000 - tn: 47580.0000 - fn: 7455.0000 - precision: 0.1407 - recall: 0.3366 - accuracy: 0.6270 - auc: 0.51772020- 05-31 21:44:31.349376: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager .profiler) is deprecated and will be removed after 2020-07-01. Instructions for updating: use `tf.profiler.experimental.stop` instead. 2020-05-31 21:44:32.843703: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_21_44_32 2020-05-31 21:44:32.857345: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.trace.json.gz 2020-05-31 21:44:32.882350: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_21_44_32 2020-05-31 21:44:32.882475: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.memory _profile.json.gz 2020-05-31 21:44:32.884516: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_21_44_32Dumped tool data for xplane.pb to /tmp/tb/trai n/plugins/profile/2020_05_31_21_44_32/node171.xplane.pb Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.overview_page.pb Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.kernel_stats.pb 400/400 [==============================] - 648s 2s/step - loss: 1.2332 - tp: 3788781.0000 - fp: 1219922.0000 - tn: 24049440.0000 - fn: 3709858.0000 - precision: 0.7564 - recall: 0.5053 - accuracy: 0.8496 - auc: 0.8489 - val_loss: 322.0566 - val_tp: 3646493.0000 - val_fp: 12737507.0000 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 0.2226 - val_recall: 1.0000 - val_accuracy: 0.2226 - val_auc: 0.500 0 Epoch 2/70 400/400 [==============================] - 644s 2s/step - loss: 1.0109 - tp: 5005706.0000 - fp: 1436756.0000 - tn: 23682896.0000 - fn: 2642650.0000 - precision: 0.7770 - recall: 0.6545 - accuracy: 0.8755 - auc: 0.8979 - val_loss: 606.1023 - val_tp: 3756979.0000 - val_fp: 12626343.0000 - val_tn: 645.0000 - val_fn: 33.0000 - val_precision: 0.2293 - val_recall: 1.0000 - val_accuracy: 0.2293 - val_auc: 0.5000 Epoch 3/70 400/400 [==============================] - 628s 2s/step - loss: 0.8285 - tp: 5051785.0000 - fp: 1374618.0000 - tn: 23855938.0000 - fn: 2485671.0000 - precision: 0.7861 - recall: 0.6702 - accuracy: 0.8822 - auc: 0.9026 - val_loss: 10.0331 - val_tp: 3878641.0000 - val_fp: 12500019.0000 - val_tn: 4304.0000 - val_fn: 1036.0000 - val_precision: 0.2368 - val_recall: 0.9997 - val_accuracy: 0.2370 - val_auc: 0.5000 Epoch 4/70 400/400 [==============================] - 629s 2s/step - loss: 0.6738 - tp: 5537594.0000 - fp: 1337082.0000 - tn: 23959264.0000 - fn: 1934060.0000 - precision: 0.8055 - recall: 0.7411 - accuracy: 0.9002 - auc: 0.9303 - val_loss: 1.2653 - val_tp: 3303018.0000 - val_fp: 1870945.0000 - val_tn: 10869556.0000 - val_fn: 340481.0000 - val_precision: 0.6384 - val_recall: 0.9066 - val_accuracy: 0.8650 - val_auc: 0.92 24 Epoch 5/70 400/400 [==============================] - 631s 2s/step - loss: 0.5958 - tp: 5587069.0000 - fp: 1162314.0000 - tn: 24313792.0000 - fn: 1704840.0000 - precision: 0.8278 - recall: 0.7662 - accuracy: 0.9125 - auc: 0.9409 - val_loss: 1.2564 - val_tp: 2402186.0000 - val_fp: 760086.0000 - val_tn: 11788196.0000 - val_fn: 1433532.0000 - val_precision: 0.7596 - val_recall: 0.6263 - val_accuracy: 0.8661 - val_auc: 0.84 39 Epoch 6/70 400/400 [==============================] - 632s 2s/step - loss: 0.5248 - tp: 6187578.0000 - fp: 1102970.0000 - tn: 24040936.0000 - fn: 1436530.0000 - precision: 0.8487 - recall: 0.8116 - accuracy: 0.9225 - auc: 0.9536 - val_loss: 3.8886 - val_tp: 3699587.0000 - val_fp: 5786776.0000 - val_tn: 6872632.0000 - val_fn: 25005.0000 - val_precision: 0.3900 - val_recall: 0.9933 - val_accuracy: 0.6453 - val_auc: 0.8590 Epoch 7/70 400/400 [==============================] - 629s 2s/step - loss: 0.5417 - tp: 6075103.0000 - fp: 1198710.0000 - tn: 23976388.0000 - fn: 1517805.0000 - precision: 0.8352 - recall: 0.8001 - accuracy: 0.9171 - auc: 0.9507 - val_loss: 0.6575 - val_tp: 3549691.0000 - val_fp: 1710464.0000 - val_tn: 10816997.0000 - val_fn: 306848.0000 - val_precision: 0.6748 - val_recall: 0.9204 - val_accuracy: 0.8769 - val_auc: 0.95 25 Epoch 8/70 400/400 [==============================] - 630s 2s/step - loss: 0.5261 - tp: 6041081.0000 - fp: 1000532.0000 - tn: 24306236.0000 - fn: 1420140.0000 - precision: 0.8579 - recall: 0.8097 - accuracy: 0.9261 - auc: 0.9527 - val_loss: 21.2177 - val_tp: 3715793.0000 - val_fp: 9558636.0000 - val_tn: 3103000.0000 - val_fn: 6571.0000 - val_precision: 0.2799 - val_recall: 0.9982 - val_accuracy: 0.4162 - val_auc: 0.6830 Epoch 9/70 400/400 [==============================] - 634s 2s/step - loss: 0.5777 - tp: 5925253.0000 - fp: 1206548.0000 - tn: 23980040.0000 - fn: 1656156.0000 - precision: 0.8308 - recall: 0.7816 - accuracy: 0.9126 - auc: 0.9467 - val_loss: 1.4550 - val_tp: 2525888.0000 - val_fp: 1794119.0000 - val_tn: 10974268.0000 - val_fn: 1089725.0000 - val_precision: 0.5847 - val_recall: 0.6986 - val_accuracy: 0.8240 - val_auc: 0.8 157 Epoch 10/70 400/400 [==============================] - 633s 2s/step - loss: 0.4670 - tp: 6137276.0000 - fp: 1085998.0000 - tn: 24173720.0000 - fn: 1370997.0000 - precision: 0.8497 - recall: 0.8174 - accuracy: 0.9250 - auc: 0.9596 - val_loss: 0.4228 - val_tp: 3267331.0000 - val_fp: 512452.0000 - val_tn: 12134530.0000 - val_fn: 469687.0000 - val_precision: 0.8644 - val_recall: 0.8743 - val_accuracy: 0.9401 - val_auc: 0.958 6 Epoch 11/70 400/400 [==============================] - 636s 2s/step - loss: 0.4183 - tp: 6387249.0000 - fp: 950432.0000 - tn: 24334356.0000 - fn: 1095980.0000 - precision: 0.8705 - recall: 0.8535 - accuracy: 0.9375 - a uc: 0.9662 - val_loss: 2.5835 - val_tp: 3791404.0000 - val_fp: 10983628.0000 - val_tn: 1606117.0000 - val_fn: 2851.0000 - val_precision: 0.2566 - val_recall: 0.9992 - val_accuracy: 0.3294 - val_auc: 0.9205 Epoch 12/70 400/400 [==============================] - 633s 2s/step - loss: 0.3622 - tp: 6622657.0000 - fp: 875498.0000 - tn: 24381258.0000 - fn: 888583.0000 - precision: 0.8832 - recall: 0.8817 - accuracy: 0.9462 - au c: 0.9716 - val_loss: 0.3789 - val_tp: 3055309.0000 - val_fp: 146845.0000 - val_tn: 12452879.0000 - val_fn: 728967.0000 - val_precision: 0.9541 - val_recall: 0.8074 - val_accuracy: 0.9465 - val_auc: 0.9659 Epoch 13/70 400/400 [==============================] - 631s 2s/step - loss: 0.3590 - tp: 6660677.0000 - fp: 869994.0000 - tn: 24335574.0000 - fn: 901756.0000 - precision: 0.8845 - recall: 0.8808 - accuracy: 0.9459 - au c: 0.9722 - val_loss: 0.4284 - val_tp: 2585078.0000 - val_fp: 26120.0000 - val_tn: 12713922.0000 - val_fn: 1058880.0000 - val_precision: 0.9900 - val_recall: 0.7094 - val_accuracy: 0.9338 - val_auc: 0.9595 Epoch 14/70 400/400 [==============================] - 631s 2s/step - loss: 0.3391 - tp: 6632643.0000 - fp: 838058.0000 - tn: 24461996.0000 - fn: 835290.0000 - precision: 0.8878 - recall: 0.8881 - accuracy: 0.9489 - au c: 0.9747 - val_loss: 0.3533 - val_tp: 3003363.0000 - val_fp: 127052.0000 - val_tn: 12492719.0000 - val_fn: 760866.0000 - val_precision: 0.9594 - val_recall: 0.7979 - val_accuracy: 0.9458 - val_auc: 0.9766 Epoch 15/70 400/400 [==============================] - 634s 2s/step - loss: 0.3422 - tp: 6682203.0000 - fp: 861078.0000 - tn: 24351496.0000 - fn: 873232.0000 - precision: 0.8858 - recall: 0.8844 - accuracy: 0.9471 - au c: 0.9741 - val_loss: 0.3630 - val_tp: 2919284.0000 - val_fp: 89104.0000 - val_tn: 12597566.0000 - val_fn: 778046.0000 - val_precision: 0.9704 - val_recall: 0.7896 - val_accuracy: 0.9471 - val_auc: 0.9763 Epoch 16/70 400/400 [==============================] - 632s 2s/step - loss: 0.3356 - tp: 6670288.0000 - fp: 818050.0000 - tn: 24453228.0000 - fn: 826432.0000 - precision: 0.8908 - recall: 0.8898 - accuracy: 0.9498 - au c: 0.9748 - val_loss: 0.7556 - val_tp: 1874704.0000 - val_fp: 1072.0000 - val_tn: 12589343.0000 - val_fn: 1918881.0000 - val_precision: 0.9994 - val_recall: 0.4942 - val_accuracy: 0.8828 - val_auc: 0.9213 Epoch 17/70 400/400 [==============================] - 629s 2s/step - loss: 0.3260 - tp: 6760875.0000 - fp: 795293.0000 - tn: 24413246.0000 - fn: 798585.0000 - precision: 0.8947 - recall: 0.8944 - accuracy: 0.9514 - au c: 0.9761 - val_loss: 0.4649 - val_tp: 2671759.0000 - val_fp: 36166.0000 - val_tn: 12592753.0000 - val_fn: 1083322.0000 - val_precision: 0.9866 - val_recall: 0.7115 - val_accuracy: 0.9317 - val_auc: 0.9546 Epoch 18/70 400/400 [==============================] - 631s 2s/step - loss: 0.3309 - tp: 6559233.0000 - fp: 790466.0000 - tn: 24604904.0000 - fn: 813389.0000 - precision: 0.8924 - recall: 0.8897 - accuracy: 0.9511 - au c: 0.9751 - val_loss: 0.4767 - val_tp: 2812359.0000 - val_fp: 90282.0000 - val_tn: 12389838.0000 - val_fn: 1091521.0000 - val_precision: 0.9689 - val_recall: 0.7204 - val_accuracy: 0.9279 - val_auc: 0.9586 Epoch 19/70 400/400 [==============================] - 631s 2s/step - loss: 0.3375 - tp: 6783791.0000 - fp: 866853.0000 - tn: 24263056.0000 - fn: 854305.0000 - precision: 0.8867 - recall: 0.8882 - accuracy: 0.9475 - au c: 0.9752 - val_loss: 0.3983 - val_tp: 2551077.0000 - val_fp: 35530.0000 - val_tn: 12721430.0000 - val_fn: 1075963.0000 - val_precision: 0.9863 - val_recall: 0.7033 - val_accuracy: 0.9322 - val_auc: 0.9710 Epoch 20/70 400/400 [==============================] - 632s 2s/step - loss: 0.3254 - tp: 6564303.0000 - fp: 830891.0000 - tn: 24563264.0000 - fn: 809541.0000 - precision: 0.8876 - recall: 0.8902 - accuracy: 0.9499 - au c: 0.9757 - val_loss: 0.5314 - val_tp: 2046949.0000 - val_fp: 3561.0000 - val_tn: 12714086.0000 - val_fn: 1619404.0000 - val_precision: 0.9983 - val_recall: 0.5583 - val_accuracy: 0.9009 - val_auc: 0.9545 Epoch 21/70 400/400 [==============================] - 640s 2s/step - loss: 0.3196 - tp: 6686380.0000 - fp: 775563.0000 - tn: 24496560.0000 - fn: 809502.0000 - precision: 0.8961 - recall: 0.8920 - accuracy: 0.9516 - au c: 0.9765 - val_loss: 0.4055 - val_tp: 2871943.0000 - val_fp: 218969.0000 - val_tn: 12467101.0000 - val_fn: 825987.0000 - val_precision: 0.9292 - val_recall: 0.7766 - val_accuracy: 0.9362 - val_auc: 0.9714 Epoch 22/70 400/400 [==============================] - 625s 2s/step - loss: 0.3154 - tp: 6756372.0000 - fp: 777461.0000 - tn: 24450556.0000 - fn: 783609.0000 - precision: 0.8968 - recall: 0.8961 - accuracy: 0.9524 - au c: 0.9770 - val_loss: 0.3362 - val_tp: 3143955.0000 - val_fp: 135752.0000 - val_tn: 12411791.0000 - val_fn: 692502.0000 - val_precision: 0.9586 - val_recall: 0.8195 - val_accuracy: 0.9494 - val_auc: 0.9754 Epoch 23/70 400/400 [==============================] - 633s 2s/step - loss: 0.3294 - tp: 6975026.0000 - fp: 773797.0000 - tn: 24208046.0000 - fn: 811140.0000 - precision: 0.9001 - recall: 0.8958 - accuracy: 0.9516 - au c: 0.9760 - val_loss: 0.3650 - val_tp: 2827380.0000 - val_fp: 88109.0000 - val_tn: 12626725.0000 - val_fn: 841786.0000 - val_precision: 0.9698 - val_recall: 0.7706 - val_accuracy: 0.9432 - val_auc: 0.9697 Epoch 24/70 400/400 [==============================] - 629s 2s/step - loss: 0.3184 - tp: 6701853.0000 - fp: 767247.0000 - tn: 24524388.0000 - fn: 774508.0000 - precision: 0.8973 - recall: 0.8964 - accuracy: 0.9529 - au c: 0.9767 - val_loss: 0.5394 - val_tp: 2252899.0000 - val_fp: 4050.0000 - val_tn: 12590875.0000 - val_fn: 1536176.0000 - val_precision: 0.9982 - val_recall: 0.5946 - val_accuracy: 0.9060 - val_auc: 0.9528 Epoch 25/70 400/400 [==============================] - 633s 2s/step - loss: 0.3189 - tp: 6642243.0000 - fp: 777194.0000 - tn: 24527812.0000 - fn: 820758.0000 - precision: 0.8952 - recall: 0.8900 - accuracy: 0.9512 - au c: 0.9762 - val_loss: 0.3377 - val_tp: 3281478.0000 - val_fp: 329375.0000 - val_tn: 12262368.0000 - val_fn: 510779.0000 - val_precision: 0.9088 - val_recall: 0.8653 - val_accuracy: 0.9487 - val_auc: 0.9760 Epoch 26/70 400/400 [==============================] - 636s 2s/step - loss: 0.3137 - tp: 6689776.0000 - fp: 788613.0000 - tn: 24505860.0000 - fn: 783748.0000 - precision: 0.8945 - recall: 0.8951 - accuracy: 0.9520 - au c: 0.9772 - val_loss: 0.4429 - val_tp: 2772106.0000 - val_fp: 45921.0000 - val_tn: 12444722.0000 - val_fn: 1121251.0000 - val_precision: 0.9837 - val_recall: 0.7120 - val_accuracy: 0.9288 - val_auc: 0.9625 Epoch 27/70 400/400 [==============================] - 632s 2s/step - loss: 0.3110 - tp: 6754135.0000 - fp: 761746.0000 - tn: 24463904.0000 - fn: 788220.0000 - precision: 0.8986 - recall: 0.8955 - accuracy: 0.9527 - au c: 0.9778 - val_loss: 0.3217 - val_tp: 3322329.0000 - val_fp: 341816.0000 - val_tn: 12309980.0000 - val_fn: 409875.0000 - val_precision: 0.9067 - val_recall: 0.8902 - val_accuracy: 0.9541 - val_auc: 0.9829 Epoch 28/70 400/400 [==============================] - 632s 2s/step - loss: 0.3143 - tp: 6711103.0000 - fp: 764757.0000 - tn: 24512308.0000 - fn: 779835.0000 - precision: 0.8977 - recall: 0.8959 - accuracy: 0.9529 - au c: 0.9772 - val_loss: 0.5518 - val_tp: 2137267.0000 - val_fp: 8417.0000 - val_tn: 12762333.0000 - val_fn: 1475983.0000 - val_precision: 0.9961 - val_recall: 0.5915 - val_accuracy: 0.9094 - val_auc: 0.9459 Epoch 29/70 400/400 [==============================] - 636s 2s/step - loss: 0.3172 - tp: 6808597.0000 - fp: 762146.0000 - tn: 24410300.0000 - fn: 786963.0000 - precision: 0.8993 - recall: 0.8964 - accuracy: 0.9527 - au c: 0.9773 - val_loss: 0.3880 - val_tp: 3284429.0000 - val_fp: 149353.0000 - val_tn: 12279814.0000 - val_fn: 670404.0000 - val_precision: 0.9565 - val_recall: 0.8305 - val_accuracy: 0.9500 - val_auc: 0.9824 Epoch 30/70 400/400 [==============================] - 636s 2s/step - loss: 0.3119 - tp: 6732407.0000 - fp: 734825.0000 - tn: 24527008.0000 - fn: 773770.0000 - precision: 0.9016 - recall: 0.8969 - accuracy: 0.9540 - au c: 0.9770 - val_loss: 0.3304 - val_tp: 3197972.0000 - val_fp: 378897.0000 - val_tn: 12377137.0000 - val_fn: 429994.0000 - val_precision: 0.8941 - val_recall: 0.8815 - val_accuracy: 0.9506 - val_auc: 0.9788 Epoch 31/70 400/400 [==============================] - 639s 2s/step - loss: 0.3172 - tp: 6584008.0000 - fp: 749012.0000 - tn: 24635250.0000 - fn: 799720.0000 - precision: 0.8979 - recall: 0.8917 - accuracy: 0.9527 - au c: 0.9763 - val_loss: 0.4527 - val_tp: 3349009.0000 - val_fp: 338939.0000 - val_tn: 12288817.0000 - val_fn: 407235.0000 - val_precision: 0.9081 - val_recall: 0.8916 - val_accuracy: 0.9545 - val_auc: 0.9822 Epoch 32/70 400/400 [==============================] - 632s 2s/step - loss: 0.3146 - tp: 6690763.0000 - fp: 766270.0000 - tn: 24531178.0000 - fn: 779790.0000 - precision: 0.8972 - recall: 0.8956 - accuracy: 0.9528 - au c: 0.9763 - val_loss: 0.3255 - val_tp: 2945281.0000 - val_fp: 103765.0000 - val_tn: 12623298.0000 - val_fn: 711656.0000 - val_precision: 0.9660 - val_recall: 0.8054 - val_accuracy: 0.9502 - val_auc: 0.9761 Epoch 33/70 400/400 [==============================] - 632s 2s/step - loss: 0.2975 - tp: 6744548.0000 - fp: 747413.0000 - tn: 24547794.0000 - fn: 728248.0000 - precision: 0.9002 - recall: 0.9025 - accuracy: 0.9550 - au c: 0.9787 - val_loss: 0.3461 - val_tp: 2865337.0000 - val_fp: 80568.0000 - val_tn: 12713439.0000 - val_fn: 724656.0000 - val_precision: 0.9727 - val_recall: 0.7981 - val_accuracy: 0.9509 - val_auc: 0.9780 Epoch 34/70 400/400 [==============================] - 635s 2s/step - loss: 0.3060 - tp: 6610739.0000 - fp: 717355.0000 - tn: 24665928.0000 - fn: 773982.0000 - precision: 0.9021 - recall: 0.8952 - accuracy: 0.9545 - au c: 0.9768 - val_loss: 0.3895 - val_tp: 2875845.0000 - val_fp: 64187.0000 - val_tn: 12593693.0000 - val_fn: 850275.0000 - val_precision: 0.9782 - val_recall: 0.7718 - val_accuracy: 0.9442 - val_auc: 0.9762 Epoch 35/70 400/400 [==============================] - 632s 2s/step - loss: 0.3131 - tp: 6658644.0000 - fp: 787948.0000 - tn: 24555564.0000 - fn: 765835.0000 - precision: 0.8942 - recall: 0.8968 - accuracy: 0.9526 - au c: 0.9770 - val_loss: 0.3298 - val_tp: 3291381.0000 - val_fp: 128257.0000 - val_tn: 12266428.0000 - val_fn: 697934.0000 - val_precision: 0.9625 - val_recall: 0.8250 - val_accuracy: 0.9496 - val_auc: 0.9787 Epoch 36/70 400/400 [==============================] - 631s 2s/step - loss: 0.3076 - tp: 6714273.0000 - fp: 757539.0000 - tn: 24536108.0000 - fn: 760070.0000 - precision: 0.8986 - recall: 0.8983 - accuracy: 0.9537 - au c: 0.9776 - val_loss: 0.3586 - val_tp: 2828629.0000 - val_fp: 70731.0000 - val_tn: 12691571.0000 - val_fn: 793069.0000 - val_precision: 0.9756 - val_recall: 0.7810 - val_accuracy: 0.9473 - val_auc: 0.9764 Epoch 37/70 400/400 [==============================] - 630s 2s/step - loss: 0.3019 - tp: 6673190.0000 - fp: 759238.0000 - tn: 24567992.0000 - fn: 767563.0000 - precision: 0.8978 - recall: 0.8968 - accuracy: 0.9534 - au c: 0.9783 - val_loss: 0.3614 - val_tp: 3049928.0000 - val_fp: 93276.0000 - val_tn: 12445161.0000 - val_fn: 795635.0000 - val_precision: 0.9703 - val_recall: 0.7931 - val_accuracy: 0.9457 - val_auc: 0.9755 Epoch 38/70 400/400 [==============================] - 631s 2s/step - loss: 0.3022 - tp: 6902334.0000 - fp: 772344.0000 - tn: 24353052.0000 - fn: 740273.0000 - precision: 0.8994 - recall: 0.9031 - accuracy: 0.9538 - au c: 0.9785 - val_loss: 0.3220 - val_tp: 2932889.0000 - val_fp: 104394.0000 - val_tn: 12669356.0000 - val_fn: 677361.0000 - val_precision: 0.9656 - val_recall: 0.8124 - val_accuracy: 0.9523 - val_auc: 0.9783 Epoch 39/70 400/400 [==============================] - 632s 2s/step - loss: 0.2978 - tp: 6800883.0000 - fp: 734721.0000 - tn: 24491858.0000 - fn: 740536.0000 - precision: 0.9025 - recall: 0.9018 - accuracy: 0.9550 - au c: 0.9785 - val_loss: 0.3295 - val_tp: 3082233.0000 - val_fp: 128611.0000 - val_tn: 12542874.0000 - val_fn: 630282.0000 - val_precision: 0.9599 - val_recall: 0.8302 - val_accuracy: 0.9537 - val_auc: 0.9799 Epoch 40/70 400/400 [==============================] - 634s 2s/step - loss: 0.3083 - tp: 6769252.0000 - fp: 762049.0000 - tn: 24472648.0000 - fn: 764043.0000 - precision: 0.8988 - recall: 0.8986 - accuracy: 0.9534 - au c: 0.9771 - val_loss: 0.3593 - val_tp: 2826188.0000 - val_fp: 59581.0000 - val_tn: 12671875.0000 - val_fn: 826356.0000 - val_precision: 0.9794 - val_recall: 0.7738 - val_accuracy: 0.9459 - val_auc: 0.9756 Epoch 41/70 400/400 [==============================] - 630s 2s/step - loss: 0.2953 - tp: 6718090.0000 - fp: 747344.0000 - tn: 24561188.0000 - fn: 741388.0000 - precision: 0.8999 - recall: 0.9006 - accuracy: 0.9546 - au c: 0.9793 - val_loss: 0.3628 - val_tp: 2996938.0000 - val_fp: 97647.0000 - val_tn: 12518986.0000 - val_fn: 770429.0000 - val_precision: 0.9684 - val_recall: 0.7955 - val_accuracy: 0.9470 - val_auc: 0.9774 Epoch 42/70 400/400 [==============================] - 629s 2s/step - loss: 0.3039 - tp: 6654758.0000 - fp: 731543.0000 - tn: 24611838.0000 - fn: 769861.0000 - precision: 0.9010 - recall: 0.8963 - accuracy: 0.9542 - au c: 0.9775 - val_loss: 0.3282 - val_tp: 2923757.0000 - val_fp: 81924.0000 - val_tn: 12615035.0000 - val_fn: 763284.0000 - val_precision: 0.9727 - val_recall: 0.7930 - val_accuracy: 0.9484 - val_auc: 0.9759 Epoch 43/70 400/400 [==============================] - 636s 2s/step - loss: 0.2937 - tp: 6435979.0000 - fp: 733398.0000 - tn: 24870358.0000 - fn: 728270.0000 - precision: 0.8977 - recall: 0.8983 - accuracy: 0.9554 - au c: 0.9781 - val_loss: 0.3116 - val_tp: 3021625.0000 - val_fp: 126888.0000 - val_tn: 12599400.0000 - val_fn: 636087.0000 - val_precision: 0.9597 - val_recall: 0.8261 - val_accuracy: 0.9534 - val_auc: 0.9791 Epoch 44/70 400/400 [==============================] - 638s 2s/step - loss: 0.3207 - tp: 6810749.0000 - fp: 779963.0000 - tn: 24383824.0000 - fn: 793469.0000 - precision: 0.8972 - recall: 0.8957 - accuracy: 0.9520 - au c: 0.9763 - val_loss: 0.3609 - val_tp: 3046190.0000 - val_fp: 96429.0000 - val_tn: 12529182.0000 - val_fn: 712199.0000 - val_precision: 0.9693 - val_recall: 0.8105 - val_accuracy: 0.9506 - val_auc: 0.9784 Epoch 45/70 400/400 [==============================] - 631s 2s/step - loss: 0.3025 - tp: 7086768.0000 - fp: 750575.0000 - tn: 24182492.0000 - fn: 748166.0000 - precision: 0.9042 - recall: 0.9045 - accuracy: 0.9543 - au c: 0.9790 - val_loss: 0.3397 - val_tp: 2823520.0000 - val_fp: 64000.0000 - val_tn: 12722924.0000 - val_fn: 773556.0000 - val_precision: 0.9778 - val_recall: 0.7849 - val_accuracy: 0.9489 - val_auc: 0.9768 Epoch 46/70 400/400 [==============================] - 639s 2s/step - loss: 0.3029 - tp: 6575419.0000 - fp: 752434.0000 - tn: 24679216.0000 - fn: 760935.0000 - precision: 0.8973 - recall: 0.8963 - accuracy: 0.9538 - au c: 0.9777 - val_loss: 0.3449 - val_tp: 3214822.0000 - val_fp: 77887.0000 - val_tn: 12327207.0000 - val_fn: 764084.0000 - val_precision: 0.9763 - val_recall: 0.8080 - val_accuracy: 0.9486 - val_auc: 0.9800 Epoch 47/70 400/400 [==============================] - 638s 2s/step - loss: 0.2988 - tp: 7041341.0000 - fp: 779154.0000 - tn: 24212120.0000 - fn: 735391.0000 - precision: 0.9004 - recall: 0.9054 - accuracy: 0.9538 - au c: 0.9794 - val_loss: 0.3507 - val_tp: 2992658.0000 - val_fp: 83764.0000 - val_tn: 12550469.0000 - val_fn: 757109.0000 - val_precision: 0.9728 - val_recall: 0.7981 - val_accuracy: 0.9487 - val_auc: 0.9795 Epoch 48/70 400/400 [==============================] - 633s 2s/step - loss: 0.3015 - tp: 6740728.0000 - fp: 737863.0000 - tn: 24521192.0000 - fn: 768220.0000 - precision: 0.9013 - recall: 0.8977 - accuracy: 0.9540 - au c: 0.9781 - val_loss: 0.3456 - val_tp: 3135590.0000 - val_fp: 126895.0000 - val_tn: 12407028.0000 - val_fn: 714487.0000 - val_precision: 0.9611 - val_recall: 0.8144 - val_accuracy: 0.9486 - val_auc: 0.9755 Epoch 49/70 400/400 [==============================] - 630s 2s/step - loss: 0.2965 - tp: 6929622.0000 - fp: 773455.0000 - tn: 24334720.0000 - fn: 730210.0000 - precision: 0.8996 - recall: 0.9047 - accuracy: 0.9541 - au c: 0.9793 - val_loss: 0.3253 - val_tp: 2923894.0000 - val_fp: 87045.0000 - val_tn: 12668721.0000 - val_fn: 704340.0000 - val_precision: 0.9711 - val_recall: 0.8059 - val_accuracy: 0.9517 - val_auc: 0.9786 Epoch 50/70 400/400 [==============================] - 631s 2s/step - loss: 0.3053 - tp: 6698368.0000 - fp: 772097.0000 - tn: 24544316.0000 - fn: 753212.0000 - precision: 0.8966 - recall: 0.8989 - accuracy: 0.9535 - au c: 0.9778 - val_loss: 0.3687 - val_tp: 2958570.0000 - val_fp: 52523.0000 - val_tn: 12513477.0000 - val_fn: 859430.0000 - val_precision: 0.9826 - val_recall: 0.7749 - val_accuracy: 0.9443 - val_auc: 0.9771 Epoch 51/70 400/400 [==============================] - 635s 2s/step - loss: 0.2971 - tp: 6664028.0000 - fp: 737094.0000 - tn: 24626012.0000 - fn: 740867.0000 - precision: 0.9004 - recall: 0.8999 - accuracy: 0.9549 - au c: 0.9786 - val_loss: 0.3459 - val_tp: 3029844.0000 - val_fp: 116426.0000 - val_tn: 12539360.0000 - val_fn: 698370.0000 - val_precision: 0.9630 - val_recall: 0.8127 - val_accuracy: 0.9503 - val_auc: 0.9793 Epoch 52/70 400/400 [==============================] - 633s 2s/step - loss: 0.3026 - tp: 6753347.0000 - fp: 678789.0000 - tn: 24527348.0000 - fn: 808513.0000 - precision: 0.9087 - recall: 0.8931 - accuracy: 0.9546 - au c: 0.9785 - val_loss: 0.3330 - val_tp: 2851126.0000 - val_fp: 109294.0000 - val_tn: 12699895.0000 - val_fn: 723685.0000 - val_precision: 0.9631 - val_recall: 0.7976 - val_accuracy: 0.9492 - val_auc: 0.9762 Epoch 53/70 400/400 [==============================] - 636s 2s/step - loss: 0.2980 - tp: 6803599.0000 - fp: 685524.0000 - tn: 24488690.0000 - fn: 790195.0000 - precision: 0.9085 - recall: 0.8959 - accuracy: 0.9550 - au c: 0.9786 - val_loss: 0.3384 - val_tp: 2978082.0000 - val_fp: 84198.0000 - val_tn: 12570606.0000 - val_fn: 751114.0000 - val_precision: 0.9725 - val_recall: 0.7986 - val_accuracy: 0.9490 - val_auc: 0.9782 Epoch 54/70 400/400 [==============================] - 636s 2s/step - loss: 0.3008 - tp: 6617352.0000 - fp: 660115.0000 - tn: 24672528.0000 - fn: 818008.0000 - precision: 0.9093 - recall: 0.8900 - accuracy: 0.9549 - au c: 0.9776 - val_loss: 0.3487 - val_tp: 3069049.0000 - val_fp: 100663.0000 - val_tn: 12450296.0000 - val_fn: 763992.0000 - val_precision: 0.9682 - val_recall: 0.8007 - val_accuracy: 0.9472 - val_auc: 0.9767 Epoch 55/70 400/400 [==============================] - 634s 2s/step - loss: 0.3041 - tp: 6727196.0000 - fp: 696140.0000 - tn: 24540392.0000 - fn: 804270.0000 - precision: 0.9062 - recall: 0.8932 - accuracy: 0.9542 - au c: 0.9781 - val_loss: 0.3201 - val_tp: 2941382.0000 - val_fp: 99527.0000 - val_tn: 12647545.0000 - val_fn: 695546.0000 - val_precision: 0.9673 - val_recall: 0.8088 - val_accuracy: 0.9515 - val_auc: 0.9784 Epoch 56/70 400/400 [==============================] - 636s 2s/step - loss: 0.2960 - tp: 6826521.0000 - fp: 708283.0000 - tn: 24472022.0000 - fn: 761183.0000 - precision: 0.9060 - recall: 0.8997 - accuracy: 0.9552 - au c: 0.9793 - val_loss: 0.3209 - val_tp: 2942017.0000 - val_fp: 95854.0000 - val_tn: 12658328.0000 - val_fn: 687801.0000 - val_precision: 0.9684 - val_recall: 0.8105 - val_accuracy: 0.9522 - val_auc: 0.9787 Epoch 57/70 400/400 [==============================] - 638s 2s/step - loss: 0.2978 - tp: 6648554.0000 - fp: 686959.0000 - tn: 24654000.0000 - fn: 778489.0000 - precision: 0.9064 - recall: 0.8952 - accuracy: 0.9553 - au c: 0.9782 - val_loss: 0.3418 - val_tp: 3047748.0000 - val_fp: 90564.0000 - val_tn: 12482669.0000 - val_fn: 763019.0000 - val_precision: 0.9711 - val_recall: 0.7998 - val_accuracy: 0.9479 - val_auc: 0.9784 Epoch 58/70 400/400 [==============================] - 637s 2s/step - loss: 0.3043 - tp: 6628231.0000 - fp: 693416.0000 - tn: 24639726.0000 - fn: 806627.0000 - precision: 0.9053 - recall: 0.8915 - accuracy: 0.9542 - au c: 0.9777 - val_loss: 0.3313 - val_tp: 3159765.0000 - val_fp: 106822.0000 - val_tn: 12398325.0000 - val_fn: 719088.0000 - val_precision: 0.9673 - val_recall: 0.8146 - val_accuracy: 0.9496 - val_auc: 0.9780 Epoch 59/70 400/400 [==============================] - 637s 2s/step - loss: 0.3006 - tp: 6652559.0000 - fp: 718848.0000 - tn: 24625476.0000 - fn: 771096.0000 - precision: 0.9025 - recall: 0.8961 - accuracy: 0.9545 - au c: 0.9786 - val_loss: 0.3533 - val_tp: 3040082.0000 - val_fp: 82199.0000 - val_tn: 12530147.0000 - val_fn: 731572.0000 - val_precision: 0.9737 - val_recall: 0.8060 - val_accuracy: 0.9503 - val_auc: 0.9795 Epoch 60/70 400/400 [==============================] - 634s 2s/step - loss: 0.3137 - tp: 6891311.0000 - fp: 715443.0000 - tn: 24333106.0000 - fn: 828145.0000 - precision: 0.9059 - recall: 0.8927 - accuracy: 0.9529 - au c: 0.9774 - val_loss: 0.3250 - val_tp: 2930627.0000 - val_fp: 99442.0000 - val_tn: 12670646.0000 - val_fn: 683285.0000 - val_precision: 0.9672 - val_recall: 0.8109 - val_accuracy: 0.9522 - val_auc: 0.9785 Epoch 61/70 400/400 [==============================] - 641s 2s/step - loss: 0.3028 - tp: 6738706.0000 - fp: 737970.0000 - tn: 24529770.0000 - fn: 761545.0000 - precision: 0.9013 - recall: 0.8985 - accuracy: 0.9542 - au c: 0.9783 - val_loss: 0.3347 - val_tp: 3211606.0000 - val_fp: 100010.0000 - val_tn: 12350882.0000 - val_fn: 721502.0000 - val_precision: 0.9698 - val_recall: 0.8166 - val_accuracy: 0.9499 - val_auc: 0.9791 Epoch 62/70 400/400 [==============================] - 635s 2s/step - loss: 0.3047 - tp: 6492872.0000 - fp: 710243.0000 - tn: 24764572.0000 - fn: 800309.0000 - precision: 0.9014 - recall: 0.8903 - accuracy: 0.9539 - au c: 0.9771 - val_loss: 0.3868 - val_tp: 3110246.0000 - val_fp: 71777.0000 - val_tn: 12369449.0000 - val_fn: 832528.0000 - val_precision: 0.9774 - val_recall: 0.7888 - val_accuracy: 0.9448 - val_auc: 0.9764 Epoch 63/70 400/400 [==============================] - 633s 2s/step - loss: 0.3033 - tp: 6979600.0000 - fp: 720858.0000 - tn: 24300348.0000 - fn: 767184.0000 - precision: 0.9064 - recall: 0.9010 - accuracy: 0.9546 - au c: 0.9788 - val_loss: 0.3340 - val_tp: 3093435.0000 - val_fp: 85835.0000 - val_tn: 12442850.0000 - val_fn: 761880.0000 - val_precision: 0.9730 - val_recall: 0.8024 - val_accuracy: 0.9483 - val_auc: 0.9784 Epoch 64/70 400/400 [==============================] - 637s 2s/step - loss: 0.2992 - tp: 6684955.0000 - fp: 717935.0000 - tn: 24597880.0000 - fn: 767241.0000 - precision: 0.9030 - recall: 0.8970 - accuracy: 0.9547 - au c: 0.9787 - val_loss: 0.3498 - val_tp: 2864609.0000 - val_fp: 82878.0000 - val_tn: 12711063.0000 - val_fn: 725450.0000 - val_precision: 0.9719 - val_recall: 0.7979 - val_accuracy: 0.9507 - val_auc: 0.9786 Epoch 65/70 400/400 [==============================] - 637s 2s/step - loss: 0.2992 - tp: 6881058.0000 - fp: 713083.0000 - tn: 24387492.0000 - fn: 786363.0000 - precision: 0.9061 - recall: 0.8974 - accuracy: 0.9542 - au c: 0.9792 - val_loss: 0.3161 - val_tp: 3129890.0000 - val_fp: 123769.0000 - val_tn: 12482708.0000 - val_fn: 647633.0000 - val_precision: 0.9620 - val_recall: 0.8286 - val_accuracy: 0.9529 - val_auc: 0.9799 Epoch 66/70 400/400 [==============================] - 638s 2s/step - loss: 0.2998 - tp: 6513297.0000 - fp: 696712.0000 - tn: 24803000.0000 - fn: 754991.0000 - precision: 0.9034 - recall: 0.8961 - accuracy: 0.9557 - au c: 0.9779 - val_loss: 0.3222 - val_tp: 3039732.0000 - val_fp: 95061.0000 - val_tn: 12541210.0000 - val_fn: 707997.0000 - val_precision: 0.9697 - val_recall: 0.8111 - val_accuracy: 0.9510 - val_auc: 0.9783 Epoch 67/70 400/400 [==============================] - 635s 2s/step - loss: 0.3011 - tp: 6869971.0000 - fp: 709956.0000 - tn: 24396884.0000 - fn: 791177.0000 - precision: 0.9063 - recall: 0.8967 - accuracy: 0.9542 - au c: 0.9787 - val_loss: 0.3435 - val_tp: 3137473.0000 - val_fp: 93385.0000 - val_tn: 12425599.0000 - val_fn: 727543.0000 - val_precision: 0.9711 - val_recall: 0.8118 - val_accuracy: 0.9499 - val_auc: 0.9791 Epoch 68/70 400/400 [==============================] - 632s 2s/step - loss: 0.3034 - tp: 6640807.0000 - fp: 757137.0000 - tn: 24604356.0000 - fn: 765698.0000 - precision: 0.8977 - recall: 0.8966 - accuracy: 0.9535 - au c: 0.9781 - val_loss: 0.3824 - val_tp: 3081203.0000 - val_fp: 98009.0000 - val_tn: 12448722.0000 - val_fn: 756066.0000 - val_precision: 0.9692 - val_recall: 0.8030 - val_accuracy: 0.9479 - val_auc: 0.9776 Epoch 69/70 400/400 [==============================] - 636s 2s/step - loss: 0.2988 - tp: 6698533.0000 - fp: 734236.0000 - tn: 24588786.0000 - fn: 746437.0000 - precision: 0.9012 - recall: 0.8997 - accuracy: 0.9548 - au c: 0.9788 - val_loss: 0.3604 - val_tp: 2967778.0000 - val_fp: 93005.0000 - val_tn: 12540155.0000 - val_fn: 783062.0000 - val_precision: 0.9696 - val_recall: 0.7912 - val_accuracy: 0.9465 - val_auc: 0.9780 Epoch 70/70 400/400 [==============================] - 638s 2s/step - loss: 0.2996 - tp: 6725869.0000 - fp: 711970.0000 - tn: 24565164.0000 - fn: 765019.0000 - precision: 0.9043 - recall: 0.8979 - accuracy: 0.9549 - au c: 0.9786 - val_loss: 0.3505 - val_tp: 3050870.0000 - val_fp: 101786.0000 - val_tn: 12476995.0000 - val_fn: 754349.0000 - val_precision: 0.9677 - val_recall: 0.8018 - val_accuracy: 0.9477 - val_auc: 0.9771 400/400 [==============================] - 121s 301ms/step - loss: 0.3684 - tp: 6266358.0000 - fp: 308665.0000 - tn: 24781578.0000 - fn: 1411378.0000 - precision: 0.9531 - recall: 0.8162 - accuracy: 0.9475 - auc: 0.9778 2020/06/01 10:07:50 INFO mlflow.projects: === Run (ID '1aefda1366f04f5da5d1fc2241ad9208') succeeded === (tensorflow_nightly) [ye53nis@node171 drmed-git]$
mlflow ui
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow ui [2020-06-01 10:38:40 +0200] [298787] [INFO] Starting gunicorn 20.0.4 [2020-06-01 10:38:40 +0200] [298787] [INFO] Listening at: http://127.0.0.1:5000 (298787) [2020-06-01 10:38:40 +0200] [298787] [INFO] Using worker: sync [2020-06-01 10:38:40 +0200] [298793] [INFO] Booting worker with pid: 298793
2.1.7 read out logs
2.1.7.1 read out mlflow logs using CLI
conda activate tensorflow_env cd Programme/drmed-git export MLFLOW_EXPERIMENT_NAME=exp-devtest export MLFLOW_TRACKING_URI=file:./data/mlruns
mlflow experiments list
mlflow runs list --experiment-id 1
mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3 mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3 -a model mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3 -a model_summary.txt mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3 -a tensorboard_logs/train/
mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3 mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3 -a model mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3 -a model_summary.txt mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3 -a tensorboard_logs
mlflow runs describe --run-id 47b870b8fdcb4445956635c6758caff3
tensorboard --logdir=data/mlruns/1/47b870b8fdcb4445956635c6758caff3/artifacts/tensorboard_logs
mlflow ui --backend-store-uri file:///home/lex/Programme/drmed-git/data/mlruns
2.1.7.2 Reading out mlflow logs with Python API
I started local machine:
import mlflow import pprint %cd /home/lex/Programme/drmed-git/
/home/lex/Programme/drmed-git
uri = 'file:///home/lex/Programme/drmed-git/data/mlruns' mlflow.set_tracking_uri(uri) runs = mlflow.search_runs(experiment_ids='0') print('run_id of first in list: ', runs.iloc[0].run_id) print('no of runs in list: ', len(runs)) print() runs
run_id of first in list: 1aefda1366f04f5da5d1fc2241ad9208 no of runs in list: 1
| run\id | experiment\id | status | artifact\uri | start\time | end\time | metrics.val\recall | metrics.tn | metrics.auc | metrics.val\loss | … | tags.mlflow.log-model.history | tags.mlflow.source.type | tags.mlflow.user | tags.mlflow.project.entryPoint | tags.mlflow.source.git.repoURL | tags.mlflow.source.name | tags.mlflow.source.git.commit | tags.mlflow.project.backend | tags.mlflow.gitRepoURL | tags.mlflow.project.env | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1aefda1366f04f5da5d1fc2241ad9208 | 0 | FINISHED | ./data/mlruns/0/1aefda1366f04f5da5d1fc224… | 2020-05-31 19:39:51.905000+00:00 | 2020-06-01 08:07:51.011000+00:00 | 0.801759 | 24565164.0 | 0.978632 | 0.350498 | … | [{"run\id": "1aefda1366f04f5da5d1fc2241ad9208"… | PROJECT | ye53nis | main | https://github.com/aseltmann/fluotracify | file:///beegfs/ye53nis/drmed-git | 2225d6fa18cca9044960b6e86a56ec9fb4362d5d | local | https://github.com/aseltmann/fluotracify | conda |
1 rows × 52 columns
client = mlflow.tracking.MlflowClient(tracking_uri=uri)
run_idx = 0 # mlflow.entities.Experiment print('client.get_experiment()\n', exp.to_proto()) # mlflow.entities.Metric # metent = client.get_metric_history(run_id=runs.iloc[run_idx].run_id, key='loss') # for i in metent: # print(i) print('- - - mlflow.entities.Run - - -') run = client.get_run(runs.iloc[run_idx].run_id) print('run.info.to_proto\n', run.info.to_proto()) pprint.pprint(run.data.metrics, sort_dicts=False) model_path = client.download_artifacts( run_id=runs.iloc[run_idx].run_id, path='model') tensorboard_path = client.download_artifacts( run_id=runs.iloc[run_idx].run_id, path='tensorboard_logs') print('\nmodel_path\n', model_path) print('\nMLmodel file') %cat $model_path/MLmodel print('\nconda.yaml') %cat $model_path/conda.yaml print('\nkeras_module.txt') %cat $model_path/data/keras_module.txt
client.get_experiment()
experiment_id: "0"
name: "exp-310520-unet"
artifact_location: "file:./data/mlruns/0"
lifecycle_stage: "active"
- - - mlflow.entities.Run - - -
run.info.to_proto
run_uuid: "1aefda1366f04f5da5d1fc2241ad9208"
experiment_id: "0"
user_id: "ye53nis"
status: FINISHED
start_time: 1590953991905
end_time: 1590998871011
artifact_uri: "file:./data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts"
lifecycle_stage: "active"
run_id: "1aefda1366f04f5da5d1fc2241ad9208"
{'learning rate': 1e-05,
'val_loss': 0.3504984378814697,
'precision': 0.9042773246765137,
'fp': 711970.0,
'loss': 0.2996121346950531,
'val_tp': 3050870.0,
'recall': 0.8978734016418457,
'accuracy': 0.9549258947372437,
'val_precision': 0.9677141904830933,
'val_recall': 0.8017593622207642,
'lr': 1e-05,
'fn': 765019.0,
'auc': 0.9786317348480225,
'val_fp': 101786.0,
'val_accuracy': 0.947745680809021,
'val_tn': 12476995.0,
'val_auc': 0.977130651473999,
'val_fn': 754349.0,
'tn': 24565164.0,
'tp': 6725869.0}
model_path
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model
MLmodel file
artifact_path: model
flavors:
keras:
data: data
keras_module: tensorflow.keras
keras_version: 2.2.4-tf
python_function:
data: data
env: conda.yaml
loader_module: mlflow.keras
python_version: 3.8.3
run_id: 1aefda1366f04f5da5d1fc2241ad9208
utc_time_created: '2020-06-01 08:05:46.358978'
conda.yaml
channels:
- defaults
dependencies:
- python=3.8.3
- pip
- pip:
- mlflow
- tensorflow==2.3.0-dev20200527
name: mlflow-env
keras_module.txt
tensorflow.keras
summary_path
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model_summary.txt
tensorboard_path
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs
summary_path = client.download_artifacts( run_id=runs.iloc[run_idx].run_id, path='model_summary.txt') print('summary_path\n', summary_path, '\n') %cat $summary_path
summary_path
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model_summary.txt
Model: "functional_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 16384, 1)] 0
__________________________________________________________________________________________________
encode0 (Sequential) (None, 16384, 64) 13120 input_1[0][0]
__________________________________________________________________________________________________
mp_encode0 (MaxPooling1D) (None, 8192, 64) 0 encode0[0][0]
__________________________________________________________________________________________________
encode1 (Sequential) (None, 8192, 128) 75008 mp_encode0[0][0]
__________________________________________________________________________________________________
mp_encode1 (MaxPooling1D) (None, 4096, 128) 0 encode1[0][0]
__________________________________________________________________________________________________
encode2 (Sequential) (None, 4096, 256) 297472 mp_encode1[0][0]
__________________________________________________________________________________________________
mp_encode2 (MaxPooling1D) (None, 2048, 256) 0 encode2[0][0]
__________________________________________________________________________________________________
encode3 (Sequential) (None, 2048, 512) 1184768 mp_encode2[0][0]
__________________________________________________________________________________________________
mp_encode3 (MaxPooling1D) (None, 1024, 512) 0 encode3[0][0]
__________________________________________________________________________________________________
encode4 (Sequential) (None, 1024, 512) 1577984 mp_encode3[0][0]
__________________________________________________________________________________________________
mp_encode4 (MaxPooling1D) (None, 512, 512) 0 encode4[0][0]
__________________________________________________________________________________________________
encode5 (Sequential) (None, 512, 512) 1577984 mp_encode4[0][0]
__________________________________________________________________________________________________
mp_encode5 (MaxPooling1D) (None, 256, 512) 0 encode5[0][0]
__________________________________________________________________________________________________
encode6 (Sequential) (None, 256, 512) 1577984 mp_encode5[0][0]
__________________________________________________________________________________________________
mp_encode6 (MaxPooling1D) (None, 128, 512) 0 encode6[0][0]
__________________________________________________________________________________________________
encode7 (Sequential) (None, 128, 512) 1577984 mp_encode6[0][0]
__________________________________________________________________________________________________
mp_encode7 (MaxPooling1D) (None, 64, 512) 0 encode7[0][0]
__________________________________________________________________________________________________
encode8 (Sequential) (None, 64, 512) 1577984 mp_encode7[0][0]
__________________________________________________________________________________________________
mp_encode8 (MaxPooling1D) (None, 32, 512) 0 encode8[0][0]
__________________________________________________________________________________________________
two_conv_center (Sequential) (None, 32, 1024) 4728832 mp_encode8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder8 (Sequen (None, 64, 512) 1051136 two_conv_center[0][0]
__________________________________________________________________________________________________
decoder8 (Concatenate) (None, 64, 1024) 0 encode8[0][0]
conv_transpose_decoder8[0][0]
__________________________________________________________________________________________________
two_conv_decoder8 (Sequential) (None, 64, 512) 2364416 decoder8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder7 (Sequen (None, 128, 512) 526848 two_conv_decoder8[0][0]
__________________________________________________________________________________________________
decoder7 (Concatenate) (None, 128, 1024) 0 encode7[0][0]
conv_transpose_decoder7[0][0]
__________________________________________________________________________________________________
two_conv_decoder7 (Sequential) (None, 128, 512) 2364416 decoder7[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder6 (Sequen (None, 256, 512) 526848 two_conv_decoder7[0][0]
__________________________________________________________________________________________________
decoder6 (Concatenate) (None, 256, 1024) 0 encode6[0][0]
conv_transpose_decoder6[0][0]
__________________________________________________________________________________________________
two_conv_decoder6 (Sequential) (None, 256, 512) 2364416 decoder6[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder5 (Sequen (None, 512, 512) 526848 two_conv_decoder6[0][0]
__________________________________________________________________________________________________
decoder5 (Concatenate) (None, 512, 1024) 0 encode5[0][0]
conv_transpose_decoder5[0][0]
__________________________________________________________________________________________________
two_conv_decoder5 (Sequential) (None, 512, 512) 2364416 decoder5[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder4 (Sequen (None, 1024, 512) 526848 two_conv_decoder5[0][0]
__________________________________________________________________________________________________
decoder4 (Concatenate) (None, 1024, 1024) 0 encode4[0][0]
conv_transpose_decoder4[0][0]
__________________________________________________________________________________________________
two_conv_decoder4 (Sequential) (None, 1024, 512) 2364416 decoder4[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder3 (Sequen (None, 2048, 512) 526848 two_conv_decoder4[0][0]
__________________________________________________________________________________________________
decoder3 (Concatenate) (None, 2048, 1024) 0 encode3[0][0]
conv_transpose_decoder3[0][0]
__________________________________________________________________________________________________
two_conv_decoder3 (Sequential) (None, 2048, 512) 2364416 decoder3[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder2 (Sequen (None, 4096, 256) 263424 two_conv_decoder3[0][0]
__________________________________________________________________________________________________
decoder2 (Concatenate) (None, 4096, 512) 0 encode2[0][0]
conv_transpose_decoder2[0][0]
__________________________________________________________________________________________________
two_conv_decoder2 (Sequential) (None, 4096, 256) 592384 decoder2[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder1 (Sequen (None, 8192, 128) 66176 two_conv_decoder2[0][0]
__________________________________________________________________________________________________
decoder1 (Concatenate) (None, 8192, 256) 0 encode1[0][0]
conv_transpose_decoder1[0][0]
__________________________________________________________________________________________________
two_conv_decoder1 (Sequential) (None, 8192, 128) 148736 decoder1[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder0 (Sequen (None, 16384, 64) 16704 two_conv_decoder1[0][0]
__________________________________________________________________________________________________
decoder0 (Concatenate) (None, 16384, 128) 0 encode0[0][0]
conv_transpose_decoder0[0][0]
__________________________________________________________________________________________________
two_conv_decoder0 (Sequential) (None, 16384, 64) 37504 decoder0[0][0]
__________________________________________________________________________________________________
conv1d_38 (Conv1D) (None, 16384, 1) 65 two_conv_decoder0[0][0]
==================================================================================================
Total params: 33,185,985
Trainable params: 33,146,689
Non-trainable params: 39,296
__________________________________________________________________________________________________
print('\ntensorboard_path\n', tensorboard_path) !ls $tensorboard_path # https://stackoverflow.com/questions/41074688/how-do-you-read-tensorboard-files-programmatically from tensorboard.backend.event_processing import event_accumulator path = str(tensorboard_path) + '/train' path2 = str() print(path) ea = event_accumulator.EventAccumulator(path=path, size_guidance={ # see below regarding this argument event_accumulator.COMPRESSED_HISTOGRAMS: 500, event_accumulator.IMAGES: 4, event_accumulator.AUDIO: 4, event_accumulator.SCALARS: 0, event_accumulator.HISTOGRAMS: 1, }) ea.Reload() # loads events from file
tensorboard_path /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs image metrics train validation /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs/train <tensorboard.backend.event_processing.event_accumulator.EventAccumulator at 0x7fdb70770c70>
ea.Tags()
{'images': ['encode0/conv1d/kernel_0/image/0',
'encode0/conv1d/bias_0/image/0',
'encode0/batch_normalization/gamma_0/image/0',
'encode0/batch_normalization/beta_0/image/0',
'encode0/batch_normalization/moving_mean_0/image/0',
'encode0/batch_normalization/moving_variance_0/image/0',
'encode0/conv1d_1/kernel_0/image/0',
'encode0/conv1d_1/kernel_0/image/1',
'encode0/conv1d_1/kernel_0/image/2',
'encode0/conv1d_1/bias_0/image/0',
'encode0/batch_normalization_1/gamma_0/image/0',
'encode0/batch_normalization_1/beta_0/image/0',
'encode0/batch_normalization_1/moving_mean_0/image/0',
'encode0/batch_normalization_1/moving_variance_0/image/0',
'encode1/conv1d_2/kernel_0/image/0',
'encode1/conv1d_2/kernel_0/image/1',
'encode1/conv1d_2/kernel_0/image/2',
'encode1/conv1d_2/bias_0/image/0',
'encode1/batch_normalization_2/gamma_0/image/0',
'encode1/batch_normalization_2/beta_0/image/0',
'encode1/batch_normalization_2/moving_mean_0/image/0',
'encode1/batch_normalization_2/moving_variance_0/image/0',
'encode1/conv1d_3/kernel_0/image/0',
'encode1/conv1d_3/kernel_0/image/1',
'encode1/conv1d_3/kernel_0/image/2',
'encode1/conv1d_3/bias_0/image/0',
'encode1/batch_normalization_3/gamma_0/image/0',
'encode1/batch_normalization_3/beta_0/image/0',
'encode1/batch_normalization_3/moving_mean_0/image/0',
'encode1/batch_normalization_3/moving_variance_0/image/0',
'encode2/conv1d_4/kernel_0/image/0',
'encode2/conv1d_4/kernel_0/image/1',
'encode2/conv1d_4/kernel_0/image/2',
'encode2/conv1d_4/bias_0/image/0',
'encode2/batch_normalization_4/gamma_0/image/0',
'encode2/batch_normalization_4/beta_0/image/0',
'encode2/batch_normalization_4/moving_mean_0/image/0',
'encode2/batch_normalization_4/moving_variance_0/image/0',
'encode2/conv1d_5/kernel_0/image/0',
'encode2/conv1d_5/kernel_0/image/1',
'encode2/conv1d_5/kernel_0/image/2',
'encode2/conv1d_5/bias_0/image/0',
'encode2/batch_normalization_5/gamma_0/image/0',
'encode2/batch_normalization_5/beta_0/image/0',
'encode2/batch_normalization_5/moving_mean_0/image/0',
'encode2/batch_normalization_5/moving_variance_0/image/0',
'encode3/conv1d_6/kernel_0/image/0',
'encode3/conv1d_6/kernel_0/image/1',
'encode3/conv1d_6/kernel_0/image/2',
'encode3/conv1d_6/bias_0/image/0',
'encode3/batch_normalization_6/gamma_0/image/0',
'encode3/batch_normalization_6/beta_0/image/0',
'encode3/batch_normalization_6/moving_mean_0/image/0',
'encode3/batch_normalization_6/moving_variance_0/image/0',
'encode3/conv1d_7/kernel_0/image/0',
'encode3/conv1d_7/kernel_0/image/1',
'encode3/conv1d_7/kernel_0/image/2',
'encode3/conv1d_7/bias_0/image/0',
'encode3/batch_normalization_7/gamma_0/image/0',
'encode3/batch_normalization_7/beta_0/image/0',
'encode3/batch_normalization_7/moving_mean_0/image/0',
'encode3/batch_normalization_7/moving_variance_0/image/0',
'encode4/conv1d_8/kernel_0/image/0',
'encode4/conv1d_8/kernel_0/image/1',
'encode4/conv1d_8/kernel_0/image/2',
'encode4/conv1d_8/bias_0/image/0',
'encode4/batch_normalization_8/gamma_0/image/0',
'encode4/batch_normalization_8/beta_0/image/0',
'encode4/batch_normalization_8/moving_mean_0/image/0',
'encode4/batch_normalization_8/moving_variance_0/image/0',
'encode4/conv1d_9/kernel_0/image/0',
'encode4/conv1d_9/kernel_0/image/1',
'encode4/conv1d_9/kernel_0/image/2',
'encode4/conv1d_9/bias_0/image/0',
'encode4/batch_normalization_9/gamma_0/image/0',
'encode4/batch_normalization_9/beta_0/image/0',
'encode4/batch_normalization_9/moving_mean_0/image/0',
'encode4/batch_normalization_9/moving_variance_0/image/0',
'encode5/conv1d_10/kernel_0/image/0',
'encode5/conv1d_10/kernel_0/image/1',
'encode5/conv1d_10/kernel_0/image/2',
'encode5/conv1d_10/bias_0/image/0',
'encode5/batch_normalization_10/gamma_0/image/0',
'encode5/batch_normalization_10/beta_0/image/0',
'encode5/batch_normalization_10/moving_mean_0/image/0',
'encode5/batch_normalization_10/moving_variance_0/image/0',
'encode5/conv1d_11/kernel_0/image/0',
'encode5/conv1d_11/kernel_0/image/1',
'encode5/conv1d_11/kernel_0/image/2',
'encode5/conv1d_11/bias_0/image/0',
'encode5/batch_normalization_11/gamma_0/image/0',
'encode5/batch_normalization_11/beta_0/image/0',
'encode5/batch_normalization_11/moving_mean_0/image/0',
'encode5/batch_normalization_11/moving_variance_0/image/0',
'encode6/conv1d_12/kernel_0/image/0',
'encode6/conv1d_12/kernel_0/image/1',
'encode6/conv1d_12/kernel_0/image/2',
'encode6/conv1d_12/bias_0/image/0',
'encode6/batch_normalization_12/gamma_0/image/0',
'encode6/batch_normalization_12/beta_0/image/0',
'encode6/batch_normalization_12/moving_mean_0/image/0',
'encode6/batch_normalization_12/moving_variance_0/image/0',
'encode6/conv1d_13/kernel_0/image/0',
'encode6/conv1d_13/kernel_0/image/1',
'encode6/conv1d_13/kernel_0/image/2',
'encode6/conv1d_13/bias_0/image/0',
'encode6/batch_normalization_13/gamma_0/image/0',
'encode6/batch_normalization_13/beta_0/image/0',
'encode6/batch_normalization_13/moving_mean_0/image/0',
'encode6/batch_normalization_13/moving_variance_0/image/0',
'encode7/conv1d_14/kernel_0/image/0',
'encode7/conv1d_14/kernel_0/image/1',
'encode7/conv1d_14/kernel_0/image/2',
'encode7/conv1d_14/bias_0/image/0',
'encode7/batch_normalization_14/gamma_0/image/0',
'encode7/batch_normalization_14/beta_0/image/0',
'encode7/batch_normalization_14/moving_mean_0/image/0',
'encode7/batch_normalization_14/moving_variance_0/image/0',
'encode7/conv1d_15/kernel_0/image/0',
'encode7/conv1d_15/kernel_0/image/1',
'encode7/conv1d_15/kernel_0/image/2',
'encode7/conv1d_15/bias_0/image/0',
'encode7/batch_normalization_15/gamma_0/image/0',
'encode7/batch_normalization_15/beta_0/image/0',
'encode7/batch_normalization_15/moving_mean_0/image/0',
'encode7/batch_normalization_15/moving_variance_0/image/0',
'encode8/conv1d_16/kernel_0/image/0',
'encode8/conv1d_16/kernel_0/image/1',
'encode8/conv1d_16/kernel_0/image/2',
'encode8/conv1d_16/bias_0/image/0',
'encode8/batch_normalization_16/gamma_0/image/0',
'encode8/batch_normalization_16/beta_0/image/0',
'encode8/batch_normalization_16/moving_mean_0/image/0',
'encode8/batch_normalization_16/moving_variance_0/image/0',
'encode8/conv1d_17/kernel_0/image/0',
'encode8/conv1d_17/kernel_0/image/1',
'encode8/conv1d_17/kernel_0/image/2',
'encode8/conv1d_17/bias_0/image/0',
'encode8/batch_normalization_17/gamma_0/image/0',
'encode8/batch_normalization_17/beta_0/image/0',
'encode8/batch_normalization_17/moving_mean_0/image/0',
'encode8/batch_normalization_17/moving_variance_0/image/0',
'two_conv_center/conv1d_18/kernel_0/image/0',
'two_conv_center/conv1d_18/kernel_0/image/1',
'two_conv_center/conv1d_18/kernel_0/image/2',
'two_conv_center/conv1d_18/bias_0/image/0',
'two_conv_center/batch_normalization_18/gamma_0/image/0',
'two_conv_center/batch_normalization_18/beta_0/image/0',
'two_conv_center/batch_normalization_18/moving_mean_0/image/0',
'two_conv_center/batch_normalization_18/moving_variance_0/image/0',
'two_conv_center/conv1d_19/kernel_0/image/0',
'two_conv_center/conv1d_19/kernel_0/image/1',
'two_conv_center/conv1d_19/kernel_0/image/2',
'two_conv_center/conv1d_19/bias_0/image/0',
'two_conv_center/batch_normalization_19/gamma_0/image/0',
'two_conv_center/batch_normalization_19/beta_0/image/0',
'two_conv_center/batch_normalization_19/moving_mean_0/image/0',
'two_conv_center/batch_normalization_19/moving_variance_0/image/0',
'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/0',
'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/1',
'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/2',
'conv_transpose_decoder8/conv1d_transpose/bias_0/image/0',
'conv_transpose_decoder8/batch_normalization_20/gamma_0/image/0',
'conv_transpose_decoder8/batch_normalization_20/beta_0/image/0',
'conv_transpose_decoder8/batch_normalization_20/moving_mean_0/image/0',
'conv_transpose_decoder8/batch_normalization_20/moving_variance_0/image/0',
'two_conv_decoder8/conv1d_20/kernel_0/image/0',
'two_conv_decoder8/conv1d_20/kernel_0/image/1',
'two_conv_decoder8/conv1d_20/kernel_0/image/2',
'two_conv_decoder8/conv1d_20/bias_0/image/0',
'two_conv_decoder8/batch_normalization_21/gamma_0/image/0',
'two_conv_decoder8/batch_normalization_21/beta_0/image/0',
'two_conv_decoder8/batch_normalization_21/moving_mean_0/image/0',
'two_conv_decoder8/batch_normalization_21/moving_variance_0/image/0',
'two_conv_decoder8/conv1d_21/kernel_0/image/0',
'two_conv_decoder8/conv1d_21/kernel_0/image/1',
'two_conv_decoder8/conv1d_21/kernel_0/image/2',
'two_conv_decoder8/conv1d_21/bias_0/image/0',
'two_conv_decoder8/batch_normalization_22/gamma_0/image/0',
'two_conv_decoder8/batch_normalization_22/beta_0/image/0',
'two_conv_decoder8/batch_normalization_22/moving_mean_0/image/0',
'two_conv_decoder8/batch_normalization_22/moving_variance_0/image/0',
'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/0',
'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/1',
'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/2',
'conv_transpose_decoder7/conv1d_transpose_1/bias_0/image/0',
'conv_transpose_decoder7/batch_normalization_23/gamma_0/image/0',
'conv_transpose_decoder7/batch_normalization_23/beta_0/image/0',
'conv_transpose_decoder7/batch_normalization_23/moving_mean_0/image/0',
'conv_transpose_decoder7/batch_normalization_23/moving_variance_0/image/0',
'two_conv_decoder7/conv1d_22/kernel_0/image/0',
'two_conv_decoder7/conv1d_22/kernel_0/image/1',
'two_conv_decoder7/conv1d_22/kernel_0/image/2',
'two_conv_decoder7/conv1d_22/bias_0/image/0',
'two_conv_decoder7/batch_normalization_24/gamma_0/image/0',
'two_conv_decoder7/batch_normalization_24/beta_0/image/0',
'two_conv_decoder7/batch_normalization_24/moving_mean_0/image/0',
'two_conv_decoder7/batch_normalization_24/moving_variance_0/image/0',
'two_conv_decoder7/conv1d_23/kernel_0/image/0',
'two_conv_decoder7/conv1d_23/kernel_0/image/1',
'two_conv_decoder7/conv1d_23/kernel_0/image/2',
'two_conv_decoder7/conv1d_23/bias_0/image/0',
'two_conv_decoder7/batch_normalization_25/gamma_0/image/0',
'two_conv_decoder7/batch_normalization_25/beta_0/image/0',
'two_conv_decoder7/batch_normalization_25/moving_mean_0/image/0',
'two_conv_decoder7/batch_normalization_25/moving_variance_0/image/0',
'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/0',
'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/1',
'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/2',
'conv_transpose_decoder6/conv1d_transpose_2/bias_0/image/0',
'conv_transpose_decoder6/batch_normalization_26/gamma_0/image/0',
'conv_transpose_decoder6/batch_normalization_26/beta_0/image/0',
'conv_transpose_decoder6/batch_normalization_26/moving_mean_0/image/0',
'conv_transpose_decoder6/batch_normalization_26/moving_variance_0/image/0',
'two_conv_decoder6/conv1d_24/kernel_0/image/0',
'two_conv_decoder6/conv1d_24/kernel_0/image/1',
'two_conv_decoder6/conv1d_24/kernel_0/image/2',
'two_conv_decoder6/conv1d_24/bias_0/image/0',
'two_conv_decoder6/batch_normalization_27/gamma_0/image/0',
'two_conv_decoder6/batch_normalization_27/beta_0/image/0',
'two_conv_decoder6/batch_normalization_27/moving_mean_0/image/0',
'two_conv_decoder6/batch_normalization_27/moving_variance_0/image/0',
'two_conv_decoder6/conv1d_25/kernel_0/image/0',
'two_conv_decoder6/conv1d_25/kernel_0/image/1',
'two_conv_decoder6/conv1d_25/kernel_0/image/2',
'two_conv_decoder6/conv1d_25/bias_0/image/0',
'two_conv_decoder6/batch_normalization_28/gamma_0/image/0',
'two_conv_decoder6/batch_normalization_28/beta_0/image/0',
'two_conv_decoder6/batch_normalization_28/moving_mean_0/image/0',
'two_conv_decoder6/batch_normalization_28/moving_variance_0/image/0',
'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/0',
'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/1',
'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/2',
'conv_transpose_decoder5/conv1d_transpose_3/bias_0/image/0',
'conv_transpose_decoder5/batch_normalization_29/gamma_0/image/0',
'conv_transpose_decoder5/batch_normalization_29/beta_0/image/0',
'conv_transpose_decoder5/batch_normalization_29/moving_mean_0/image/0',
'conv_transpose_decoder5/batch_normalization_29/moving_variance_0/image/0',
'two_conv_decoder5/conv1d_26/kernel_0/image/0',
'two_conv_decoder5/conv1d_26/kernel_0/image/1',
'two_conv_decoder5/conv1d_26/kernel_0/image/2',
'two_conv_decoder5/conv1d_26/bias_0/image/0',
'two_conv_decoder5/batch_normalization_30/gamma_0/image/0',
'two_conv_decoder5/batch_normalization_30/beta_0/image/0',
'two_conv_decoder5/batch_normalization_30/moving_mean_0/image/0',
'two_conv_decoder5/batch_normalization_30/moving_variance_0/image/0',
'two_conv_decoder5/conv1d_27/kernel_0/image/0',
'two_conv_decoder5/conv1d_27/kernel_0/image/1',
'two_conv_decoder5/conv1d_27/kernel_0/image/2',
'two_conv_decoder5/conv1d_27/bias_0/image/0',
'two_conv_decoder5/batch_normalization_31/gamma_0/image/0',
'two_conv_decoder5/batch_normalization_31/beta_0/image/0',
'two_conv_decoder5/batch_normalization_31/moving_mean_0/image/0',
'two_conv_decoder5/batch_normalization_31/moving_variance_0/image/0',
'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/0',
'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/1',
'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/2',
'conv_transpose_decoder4/conv1d_transpose_4/bias_0/image/0',
'conv_transpose_decoder4/batch_normalization_32/gamma_0/image/0',
'conv_transpose_decoder4/batch_normalization_32/beta_0/image/0',
'conv_transpose_decoder4/batch_normalization_32/moving_mean_0/image/0',
'conv_transpose_decoder4/batch_normalization_32/moving_variance_0/image/0',
'two_conv_decoder4/conv1d_28/kernel_0/image/0',
'two_conv_decoder4/conv1d_28/kernel_0/image/1',
'two_conv_decoder4/conv1d_28/kernel_0/image/2',
'two_conv_decoder4/conv1d_28/bias_0/image/0',
'two_conv_decoder4/batch_normalization_33/gamma_0/image/0',
'two_conv_decoder4/batch_normalization_33/beta_0/image/0',
'two_conv_decoder4/batch_normalization_33/moving_mean_0/image/0',
'two_conv_decoder4/batch_normalization_33/moving_variance_0/image/0',
'two_conv_decoder4/conv1d_29/kernel_0/image/0',
'two_conv_decoder4/conv1d_29/kernel_0/image/1',
'two_conv_decoder4/conv1d_29/kernel_0/image/2',
'two_conv_decoder4/conv1d_29/bias_0/image/0',
'two_conv_decoder4/batch_normalization_34/gamma_0/image/0',
'two_conv_decoder4/batch_normalization_34/beta_0/image/0',
'two_conv_decoder4/batch_normalization_34/moving_mean_0/image/0',
'two_conv_decoder4/batch_normalization_34/moving_variance_0/image/0',
'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/0',
'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/1',
'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/2',
'conv_transpose_decoder3/conv1d_transpose_5/bias_0/image/0',
'conv_transpose_decoder3/batch_normalization_35/gamma_0/image/0',
'conv_transpose_decoder3/batch_normalization_35/beta_0/image/0',
'conv_transpose_decoder3/batch_normalization_35/moving_mean_0/image/0',
'conv_transpose_decoder3/batch_normalization_35/moving_variance_0/image/0',
'two_conv_decoder3/conv1d_30/kernel_0/image/0',
'two_conv_decoder3/conv1d_30/kernel_0/image/1',
'two_conv_decoder3/conv1d_30/kernel_0/image/2',
'two_conv_decoder3/conv1d_30/bias_0/image/0',
'two_conv_decoder3/batch_normalization_36/gamma_0/image/0',
'two_conv_decoder3/batch_normalization_36/beta_0/image/0',
'two_conv_decoder3/batch_normalization_36/moving_mean_0/image/0',
'two_conv_decoder3/batch_normalization_36/moving_variance_0/image/0',
'two_conv_decoder3/conv1d_31/kernel_0/image/0',
'two_conv_decoder3/conv1d_31/kernel_0/image/1',
'two_conv_decoder3/conv1d_31/kernel_0/image/2',
'two_conv_decoder3/conv1d_31/bias_0/image/0',
'two_conv_decoder3/batch_normalization_37/gamma_0/image/0',
'two_conv_decoder3/batch_normalization_37/beta_0/image/0',
'two_conv_decoder3/batch_normalization_37/moving_mean_0/image/0',
'two_conv_decoder3/batch_normalization_37/moving_variance_0/image/0',
'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/0',
'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/1',
'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/2',
'conv_transpose_decoder2/conv1d_transpose_6/bias_0/image/0',
'conv_transpose_decoder2/batch_normalization_38/gamma_0/image/0',
'conv_transpose_decoder2/batch_normalization_38/beta_0/image/0',
'conv_transpose_decoder2/batch_normalization_38/moving_mean_0/image/0',
'conv_transpose_decoder2/batch_normalization_38/moving_variance_0/image/0',
'two_conv_decoder2/conv1d_32/kernel_0/image/0',
'two_conv_decoder2/conv1d_32/kernel_0/image/1',
'two_conv_decoder2/conv1d_32/kernel_0/image/2',
'two_conv_decoder2/conv1d_32/bias_0/image/0',
'two_conv_decoder2/batch_normalization_39/gamma_0/image/0',
'two_conv_decoder2/batch_normalization_39/beta_0/image/0',
'two_conv_decoder2/batch_normalization_39/moving_mean_0/image/0',
'two_conv_decoder2/batch_normalization_39/moving_variance_0/image/0',
'two_conv_decoder2/conv1d_33/kernel_0/image/0',
'two_conv_decoder2/conv1d_33/kernel_0/image/1',
'two_conv_decoder2/conv1d_33/kernel_0/image/2',
'two_conv_decoder2/conv1d_33/bias_0/image/0',
'two_conv_decoder2/batch_normalization_40/gamma_0/image/0',
'two_conv_decoder2/batch_normalization_40/beta_0/image/0',
'two_conv_decoder2/batch_normalization_40/moving_mean_0/image/0',
'two_conv_decoder2/batch_normalization_40/moving_variance_0/image/0',
'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/0',
'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/1',
'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/2',
'conv_transpose_decoder1/conv1d_transpose_7/bias_0/image/0',
'conv_transpose_decoder1/batch_normalization_41/gamma_0/image/0',
'conv_transpose_decoder1/batch_normalization_41/beta_0/image/0',
'conv_transpose_decoder1/batch_normalization_41/moving_mean_0/image/0',
'conv_transpose_decoder1/batch_normalization_41/moving_variance_0/image/0',
'two_conv_decoder1/conv1d_34/kernel_0/image/0',
'two_conv_decoder1/conv1d_34/kernel_0/image/1',
'two_conv_decoder1/conv1d_34/kernel_0/image/2',
'two_conv_decoder1/conv1d_34/bias_0/image/0',
'two_conv_decoder1/batch_normalization_42/gamma_0/image/0',
'two_conv_decoder1/batch_normalization_42/beta_0/image/0',
'two_conv_decoder1/batch_normalization_42/moving_mean_0/image/0',
'two_conv_decoder1/batch_normalization_42/moving_variance_0/image/0',
'two_conv_decoder1/conv1d_35/kernel_0/image/0',
'two_conv_decoder1/conv1d_35/kernel_0/image/1',
'two_conv_decoder1/conv1d_35/kernel_0/image/2',
'two_conv_decoder1/conv1d_35/bias_0/image/0',
'two_conv_decoder1/batch_normalization_43/gamma_0/image/0',
'two_conv_decoder1/batch_normalization_43/beta_0/image/0',
'two_conv_decoder1/batch_normalization_43/moving_mean_0/image/0',
'two_conv_decoder1/batch_normalization_43/moving_variance_0/image/0',
'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/0',
'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/1',
'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/2',
'conv_transpose_decoder0/conv1d_transpose_8/bias_0/image/0',
'conv_transpose_decoder0/batch_normalization_44/gamma_0/image/0',
'conv_transpose_decoder0/batch_normalization_44/beta_0/image/0',
'conv_transpose_decoder0/batch_normalization_44/moving_mean_0/image/0',
'conv_transpose_decoder0/batch_normalization_44/moving_variance_0/image/0',
'two_conv_decoder0/conv1d_36/kernel_0/image/0',
'two_conv_decoder0/conv1d_36/kernel_0/image/1',
'two_conv_decoder0/conv1d_36/kernel_0/image/2',
'two_conv_decoder0/conv1d_36/bias_0/image/0',
'two_conv_decoder0/batch_normalization_45/gamma_0/image/0',
'two_conv_decoder0/batch_normalization_45/beta_0/image/0',
'two_conv_decoder0/batch_normalization_45/moving_mean_0/image/0',
'two_conv_decoder0/batch_normalization_45/moving_variance_0/image/0',
'two_conv_decoder0/conv1d_37/kernel_0/image/0',
'two_conv_decoder0/conv1d_37/kernel_0/image/1',
'two_conv_decoder0/conv1d_37/kernel_0/image/2',
'two_conv_decoder0/conv1d_37/bias_0/image/0',
'two_conv_decoder0/batch_normalization_46/gamma_0/image/0',
'two_conv_decoder0/batch_normalization_46/beta_0/image/0',
'two_conv_decoder0/batch_normalization_46/moving_mean_0/image/0',
'two_conv_decoder0/batch_normalization_46/moving_variance_0/image/0',
'conv1d_38/kernel_0/image/0'],
'audio': [],
'histograms': ['encode0/conv1d/kernel_0',
'encode0/conv1d/bias_0',
'encode0/batch_normalization/gamma_0',
'encode0/batch_normalization/beta_0',
'encode0/batch_normalization/moving_mean_0',
'encode0/batch_normalization/moving_variance_0',
'encode0/conv1d_1/kernel_0',
'encode0/conv1d_1/bias_0',
'encode0/batch_normalization_1/gamma_0',
'encode0/batch_normalization_1/beta_0',
'encode0/batch_normalization_1/moving_mean_0',
'encode0/batch_normalization_1/moving_variance_0',
'encode1/conv1d_2/kernel_0',
'encode1/conv1d_2/bias_0',
'encode1/batch_normalization_2/gamma_0',
'encode1/batch_normalization_2/beta_0',
'encode1/batch_normalization_2/moving_mean_0',
'encode1/batch_normalization_2/moving_variance_0',
'encode1/conv1d_3/kernel_0',
'encode1/conv1d_3/bias_0',
'encode1/batch_normalization_3/gamma_0',
'encode1/batch_normalization_3/beta_0',
'encode1/batch_normalization_3/moving_mean_0',
'encode1/batch_normalization_3/moving_variance_0',
'encode2/conv1d_4/kernel_0',
'encode2/conv1d_4/bias_0',
'encode2/batch_normalization_4/gamma_0',
'encode2/batch_normalization_4/beta_0',
'encode2/batch_normalization_4/moving_mean_0',
'encode2/batch_normalization_4/moving_variance_0',
'encode2/conv1d_5/kernel_0',
'encode2/conv1d_5/bias_0',
'encode2/batch_normalization_5/gamma_0',
'encode2/batch_normalization_5/beta_0',
'encode2/batch_normalization_5/moving_mean_0',
'encode2/batch_normalization_5/moving_variance_0',
'encode3/conv1d_6/kernel_0',
'encode3/conv1d_6/bias_0',
'encode3/batch_normalization_6/gamma_0',
'encode3/batch_normalization_6/beta_0',
'encode3/batch_normalization_6/moving_mean_0',
'encode3/batch_normalization_6/moving_variance_0',
'encode3/conv1d_7/kernel_0',
'encode3/conv1d_7/bias_0',
'encode3/batch_normalization_7/gamma_0',
'encode3/batch_normalization_7/beta_0',
'encode3/batch_normalization_7/moving_mean_0',
'encode3/batch_normalization_7/moving_variance_0',
'encode4/conv1d_8/kernel_0',
'encode4/conv1d_8/bias_0',
'encode4/batch_normalization_8/gamma_0',
'encode4/batch_normalization_8/beta_0',
'encode4/batch_normalization_8/moving_mean_0',
'encode4/batch_normalization_8/moving_variance_0',
'encode4/conv1d_9/kernel_0',
'encode4/conv1d_9/bias_0',
'encode4/batch_normalization_9/gamma_0',
'encode4/batch_normalization_9/beta_0',
'encode4/batch_normalization_9/moving_mean_0',
'encode4/batch_normalization_9/moving_variance_0',
'encode5/conv1d_10/kernel_0',
'encode5/conv1d_10/bias_0',
'encode5/batch_normalization_10/gamma_0',
'encode5/batch_normalization_10/beta_0',
'encode5/batch_normalization_10/moving_mean_0',
'encode5/batch_normalization_10/moving_variance_0',
'encode5/conv1d_11/kernel_0',
'encode5/conv1d_11/bias_0',
'encode5/batch_normalization_11/gamma_0',
'encode5/batch_normalization_11/beta_0',
'encode5/batch_normalization_11/moving_mean_0',
'encode5/batch_normalization_11/moving_variance_0',
'encode6/conv1d_12/kernel_0',
'encode6/conv1d_12/bias_0',
'encode6/batch_normalization_12/gamma_0',
'encode6/batch_normalization_12/beta_0',
'encode6/batch_normalization_12/moving_mean_0',
'encode6/batch_normalization_12/moving_variance_0',
'encode6/conv1d_13/kernel_0',
'encode6/conv1d_13/bias_0',
'encode6/batch_normalization_13/gamma_0',
'encode6/batch_normalization_13/beta_0',
'encode6/batch_normalization_13/moving_mean_0',
'encode6/batch_normalization_13/moving_variance_0',
'encode7/conv1d_14/kernel_0',
'encode7/conv1d_14/bias_0',
'encode7/batch_normalization_14/gamma_0',
'encode7/batch_normalization_14/beta_0',
'encode7/batch_normalization_14/moving_mean_0',
'encode7/batch_normalization_14/moving_variance_0',
'encode7/conv1d_15/kernel_0',
'encode7/conv1d_15/bias_0',
'encode7/batch_normalization_15/gamma_0',
'encode7/batch_normalization_15/beta_0',
'encode7/batch_normalization_15/moving_mean_0',
'encode7/batch_normalization_15/moving_variance_0',
'encode8/conv1d_16/kernel_0',
'encode8/conv1d_16/bias_0',
'encode8/batch_normalization_16/gamma_0',
'encode8/batch_normalization_16/beta_0',
'encode8/batch_normalization_16/moving_mean_0',
'encode8/batch_normalization_16/moving_variance_0',
'encode8/conv1d_17/kernel_0',
'encode8/conv1d_17/bias_0',
'encode8/batch_normalization_17/gamma_0',
'encode8/batch_normalization_17/beta_0',
'encode8/batch_normalization_17/moving_mean_0',
'encode8/batch_normalization_17/moving_variance_0',
'two_conv_center/conv1d_18/kernel_0',
'two_conv_center/conv1d_18/bias_0',
'two_conv_center/batch_normalization_18/gamma_0',
'two_conv_center/batch_normalization_18/beta_0',
'two_conv_center/batch_normalization_18/moving_mean_0',
'two_conv_center/batch_normalization_18/moving_variance_0',
'two_conv_center/conv1d_19/kernel_0',
'two_conv_center/conv1d_19/bias_0',
'two_conv_center/batch_normalization_19/gamma_0',
'two_conv_center/batch_normalization_19/beta_0',
'two_conv_center/batch_normalization_19/moving_mean_0',
'two_conv_center/batch_normalization_19/moving_variance_0',
'conv_transpose_decoder8/conv1d_transpose/kernel_0',
'conv_transpose_decoder8/conv1d_transpose/bias_0',
'conv_transpose_decoder8/batch_normalization_20/gamma_0',
'conv_transpose_decoder8/batch_normalization_20/beta_0',
'conv_transpose_decoder8/batch_normalization_20/moving_mean_0',
'conv_transpose_decoder8/batch_normalization_20/moving_variance_0',
'two_conv_decoder8/conv1d_20/kernel_0',
'two_conv_decoder8/conv1d_20/bias_0',
'two_conv_decoder8/batch_normalization_21/gamma_0',
'two_conv_decoder8/batch_normalization_21/beta_0',
'two_conv_decoder8/batch_normalization_21/moving_mean_0',
'two_conv_decoder8/batch_normalization_21/moving_variance_0',
'two_conv_decoder8/conv1d_21/kernel_0',
'two_conv_decoder8/conv1d_21/bias_0',
'two_conv_decoder8/batch_normalization_22/gamma_0',
'two_conv_decoder8/batch_normalization_22/beta_0',
'two_conv_decoder8/batch_normalization_22/moving_mean_0',
'two_conv_decoder8/batch_normalization_22/moving_variance_0',
'conv_transpose_decoder7/conv1d_transpose_1/kernel_0',
'conv_transpose_decoder7/conv1d_transpose_1/bias_0',
'conv_transpose_decoder7/batch_normalization_23/gamma_0',
'conv_transpose_decoder7/batch_normalization_23/beta_0',
'conv_transpose_decoder7/batch_normalization_23/moving_mean_0',
'conv_transpose_decoder7/batch_normalization_23/moving_variance_0',
'two_conv_decoder7/conv1d_22/kernel_0',
'two_conv_decoder7/conv1d_22/bias_0',
'two_conv_decoder7/batch_normalization_24/gamma_0',
'two_conv_decoder7/batch_normalization_24/beta_0',
'two_conv_decoder7/batch_normalization_24/moving_mean_0',
'two_conv_decoder7/batch_normalization_24/moving_variance_0',
'two_conv_decoder7/conv1d_23/kernel_0',
'two_conv_decoder7/conv1d_23/bias_0',
'two_conv_decoder7/batch_normalization_25/gamma_0',
'two_conv_decoder7/batch_normalization_25/beta_0',
'two_conv_decoder7/batch_normalization_25/moving_mean_0',
'two_conv_decoder7/batch_normalization_25/moving_variance_0',
'conv_transpose_decoder6/conv1d_transpose_2/kernel_0',
'conv_transpose_decoder6/conv1d_transpose_2/bias_0',
'conv_transpose_decoder6/batch_normalization_26/gamma_0',
'conv_transpose_decoder6/batch_normalization_26/beta_0',
'conv_transpose_decoder6/batch_normalization_26/moving_mean_0',
'conv_transpose_decoder6/batch_normalization_26/moving_variance_0',
'two_conv_decoder6/conv1d_24/kernel_0',
'two_conv_decoder6/conv1d_24/bias_0',
'two_conv_decoder6/batch_normalization_27/gamma_0',
'two_conv_decoder6/batch_normalization_27/beta_0',
'two_conv_decoder6/batch_normalization_27/moving_mean_0',
'two_conv_decoder6/batch_normalization_27/moving_variance_0',
'two_conv_decoder6/conv1d_25/kernel_0',
'two_conv_decoder6/conv1d_25/bias_0',
'two_conv_decoder6/batch_normalization_28/gamma_0',
'two_conv_decoder6/batch_normalization_28/beta_0',
'two_conv_decoder6/batch_normalization_28/moving_mean_0',
'two_conv_decoder6/batch_normalization_28/moving_variance_0',
'conv_transpose_decoder5/conv1d_transpose_3/kernel_0',
'conv_transpose_decoder5/conv1d_transpose_3/bias_0',
'conv_transpose_decoder5/batch_normalization_29/gamma_0',
'conv_transpose_decoder5/batch_normalization_29/beta_0',
'conv_transpose_decoder5/batch_normalization_29/moving_mean_0',
'conv_transpose_decoder5/batch_normalization_29/moving_variance_0',
'two_conv_decoder5/conv1d_26/kernel_0',
'two_conv_decoder5/conv1d_26/bias_0',
'two_conv_decoder5/batch_normalization_30/gamma_0',
'two_conv_decoder5/batch_normalization_30/beta_0',
'two_conv_decoder5/batch_normalization_30/moving_mean_0',
'two_conv_decoder5/batch_normalization_30/moving_variance_0',
'two_conv_decoder5/conv1d_27/kernel_0',
'two_conv_decoder5/conv1d_27/bias_0',
'two_conv_decoder5/batch_normalization_31/gamma_0',
'two_conv_decoder5/batch_normalization_31/beta_0',
'two_conv_decoder5/batch_normalization_31/moving_mean_0',
'two_conv_decoder5/batch_normalization_31/moving_variance_0',
'conv_transpose_decoder4/conv1d_transpose_4/kernel_0',
'conv_transpose_decoder4/conv1d_transpose_4/bias_0',
'conv_transpose_decoder4/batch_normalization_32/gamma_0',
'conv_transpose_decoder4/batch_normalization_32/beta_0',
'conv_transpose_decoder4/batch_normalization_32/moving_mean_0',
'conv_transpose_decoder4/batch_normalization_32/moving_variance_0',
'two_conv_decoder4/conv1d_28/kernel_0',
'two_conv_decoder4/conv1d_28/bias_0',
'two_conv_decoder4/batch_normalization_33/gamma_0',
'two_conv_decoder4/batch_normalization_33/beta_0',
'two_conv_decoder4/batch_normalization_33/moving_mean_0',
'two_conv_decoder4/batch_normalization_33/moving_variance_0',
'two_conv_decoder4/conv1d_29/kernel_0',
'two_conv_decoder4/conv1d_29/bias_0',
'two_conv_decoder4/batch_normalization_34/gamma_0',
'two_conv_decoder4/batch_normalization_34/beta_0',
'two_conv_decoder4/batch_normalization_34/moving_mean_0',
'two_conv_decoder4/batch_normalization_34/moving_variance_0',
'conv_transpose_decoder3/conv1d_transpose_5/kernel_0',
'conv_transpose_decoder3/conv1d_transpose_5/bias_0',
'conv_transpose_decoder3/batch_normalization_35/gamma_0',
'conv_transpose_decoder3/batch_normalization_35/beta_0',
'conv_transpose_decoder3/batch_normalization_35/moving_mean_0',
'conv_transpose_decoder3/batch_normalization_35/moving_variance_0',
'two_conv_decoder3/conv1d_30/kernel_0',
'two_conv_decoder3/conv1d_30/bias_0',
'two_conv_decoder3/batch_normalization_36/gamma_0',
'two_conv_decoder3/batch_normalization_36/beta_0',
'two_conv_decoder3/batch_normalization_36/moving_mean_0',
'two_conv_decoder3/batch_normalization_36/moving_variance_0',
'two_conv_decoder3/conv1d_31/kernel_0',
'two_conv_decoder3/conv1d_31/bias_0',
'two_conv_decoder3/batch_normalization_37/gamma_0',
'two_conv_decoder3/batch_normalization_37/beta_0',
'two_conv_decoder3/batch_normalization_37/moving_mean_0',
'two_conv_decoder3/batch_normalization_37/moving_variance_0',
'conv_transpose_decoder2/conv1d_transpose_6/kernel_0',
'conv_transpose_decoder2/conv1d_transpose_6/bias_0',
'conv_transpose_decoder2/batch_normalization_38/gamma_0',
'conv_transpose_decoder2/batch_normalization_38/beta_0',
'conv_transpose_decoder2/batch_normalization_38/moving_mean_0',
'conv_transpose_decoder2/batch_normalization_38/moving_variance_0',
'two_conv_decoder2/conv1d_32/kernel_0',
'two_conv_decoder2/conv1d_32/bias_0',
'two_conv_decoder2/batch_normalization_39/gamma_0',
'two_conv_decoder2/batch_normalization_39/beta_0',
'two_conv_decoder2/batch_normalization_39/moving_mean_0',
'two_conv_decoder2/batch_normalization_39/moving_variance_0',
'two_conv_decoder2/conv1d_33/kernel_0',
'two_conv_decoder2/conv1d_33/bias_0',
'two_conv_decoder2/batch_normalization_40/gamma_0',
'two_conv_decoder2/batch_normalization_40/beta_0',
'two_conv_decoder2/batch_normalization_40/moving_mean_0',
'two_conv_decoder2/batch_normalization_40/moving_variance_0',
'conv_transpose_decoder1/conv1d_transpose_7/kernel_0',
'conv_transpose_decoder1/conv1d_transpose_7/bias_0',
'conv_transpose_decoder1/batch_normalization_41/gamma_0',
'conv_transpose_decoder1/batch_normalization_41/beta_0',
'conv_transpose_decoder1/batch_normalization_41/moving_mean_0',
'conv_transpose_decoder1/batch_normalization_41/moving_variance_0',
'two_conv_decoder1/conv1d_34/kernel_0',
'two_conv_decoder1/conv1d_34/bias_0',
'two_conv_decoder1/batch_normalization_42/gamma_0',
'two_conv_decoder1/batch_normalization_42/beta_0',
'two_conv_decoder1/batch_normalization_42/moving_mean_0',
'two_conv_decoder1/batch_normalization_42/moving_variance_0',
'two_conv_decoder1/conv1d_35/kernel_0',
'two_conv_decoder1/conv1d_35/bias_0',
'two_conv_decoder1/batch_normalization_43/gamma_0',
'two_conv_decoder1/batch_normalization_43/beta_0',
'two_conv_decoder1/batch_normalization_43/moving_mean_0',
'two_conv_decoder1/batch_normalization_43/moving_variance_0',
'conv_transpose_decoder0/conv1d_transpose_8/kernel_0',
'conv_transpose_decoder0/conv1d_transpose_8/bias_0',
'conv_transpose_decoder0/batch_normalization_44/gamma_0',
'conv_transpose_decoder0/batch_normalization_44/beta_0',
'conv_transpose_decoder0/batch_normalization_44/moving_mean_0',
'conv_transpose_decoder0/batch_normalization_44/moving_variance_0',
'two_conv_decoder0/conv1d_36/kernel_0',
'two_conv_decoder0/conv1d_36/bias_0',
'two_conv_decoder0/batch_normalization_45/gamma_0',
'two_conv_decoder0/batch_normalization_45/beta_0',
'two_conv_decoder0/batch_normalization_45/moving_mean_0',
'two_conv_decoder0/batch_normalization_45/moving_variance_0',
'two_conv_decoder0/conv1d_37/kernel_0',
'two_conv_decoder0/conv1d_37/bias_0',
'two_conv_decoder0/batch_normalization_46/gamma_0',
'two_conv_decoder0/batch_normalization_46/beta_0',
'two_conv_decoder0/batch_normalization_46/moving_mean_0',
'two_conv_decoder0/batch_normalization_46/moving_variance_0',
'conv1d_38/kernel_0',
'conv1d_38/bias_0'],
'scalars': ['epoch_loss',
'epoch_tp',
'epoch_fp',
'epoch_tn',
'epoch_fn',
'epoch_precision',
'epoch_recall',
'epoch_accuracy',
'epoch_auc'],
'distributions': ['encode0/conv1d/kernel_0',
'encode0/conv1d/bias_0',
'encode0/batch_normalization/gamma_0',
'encode0/batch_normalization/beta_0',
'encode0/batch_normalization/moving_mean_0',
'encode0/batch_normalization/moving_variance_0',
'encode0/conv1d_1/kernel_0',
'encode0/conv1d_1/bias_0',
'encode0/batch_normalization_1/gamma_0',
'encode0/batch_normalization_1/beta_0',
'encode0/batch_normalization_1/moving_mean_0',
'encode0/batch_normalization_1/moving_variance_0',
'encode1/conv1d_2/kernel_0',
'encode1/conv1d_2/bias_0',
'encode1/batch_normalization_2/gamma_0',
'encode1/batch_normalization_2/beta_0',
'encode1/batch_normalization_2/moving_mean_0',
'encode1/batch_normalization_2/moving_variance_0',
'encode1/conv1d_3/kernel_0',
'encode1/conv1d_3/bias_0',
'encode1/batch_normalization_3/gamma_0',
'encode1/batch_normalization_3/beta_0',
'encode1/batch_normalization_3/moving_mean_0',
'encode1/batch_normalization_3/moving_variance_0',
'encode2/conv1d_4/kernel_0',
'encode2/conv1d_4/bias_0',
'encode2/batch_normalization_4/gamma_0',
'encode2/batch_normalization_4/beta_0',
'encode2/batch_normalization_4/moving_mean_0',
'encode2/batch_normalization_4/moving_variance_0',
'encode2/conv1d_5/kernel_0',
'encode2/conv1d_5/bias_0',
'encode2/batch_normalization_5/gamma_0',
'encode2/batch_normalization_5/beta_0',
'encode2/batch_normalization_5/moving_mean_0',
'encode2/batch_normalization_5/moving_variance_0',
'encode3/conv1d_6/kernel_0',
'encode3/conv1d_6/bias_0',
'encode3/batch_normalization_6/gamma_0',
'encode3/batch_normalization_6/beta_0',
'encode3/batch_normalization_6/moving_mean_0',
'encode3/batch_normalization_6/moving_variance_0',
'encode3/conv1d_7/kernel_0',
'encode3/conv1d_7/bias_0',
'encode3/batch_normalization_7/gamma_0',
'encode3/batch_normalization_7/beta_0',
'encode3/batch_normalization_7/moving_mean_0',
'encode3/batch_normalization_7/moving_variance_0',
'encode4/conv1d_8/kernel_0',
'encode4/conv1d_8/bias_0',
'encode4/batch_normalization_8/gamma_0',
'encode4/batch_normalization_8/beta_0',
'encode4/batch_normalization_8/moving_mean_0',
'encode4/batch_normalization_8/moving_variance_0',
'encode4/conv1d_9/kernel_0',
'encode4/conv1d_9/bias_0',
'encode4/batch_normalization_9/gamma_0',
'encode4/batch_normalization_9/beta_0',
'encode4/batch_normalization_9/moving_mean_0',
'encode4/batch_normalization_9/moving_variance_0',
'encode5/conv1d_10/kernel_0',
'encode5/conv1d_10/bias_0',
'encode5/batch_normalization_10/gamma_0',
'encode5/batch_normalization_10/beta_0',
'encode5/batch_normalization_10/moving_mean_0',
'encode5/batch_normalization_10/moving_variance_0',
'encode5/conv1d_11/kernel_0',
'encode5/conv1d_11/bias_0',
'encode5/batch_normalization_11/gamma_0',
'encode5/batch_normalization_11/beta_0',
'encode5/batch_normalization_11/moving_mean_0',
'encode5/batch_normalization_11/moving_variance_0',
'encode6/conv1d_12/kernel_0',
'encode6/conv1d_12/bias_0',
'encode6/batch_normalization_12/gamma_0',
'encode6/batch_normalization_12/beta_0',
'encode6/batch_normalization_12/moving_mean_0',
'encode6/batch_normalization_12/moving_variance_0',
'encode6/conv1d_13/kernel_0',
'encode6/conv1d_13/bias_0',
'encode6/batch_normalization_13/gamma_0',
'encode6/batch_normalization_13/beta_0',
'encode6/batch_normalization_13/moving_mean_0',
'encode6/batch_normalization_13/moving_variance_0',
'encode7/conv1d_14/kernel_0',
'encode7/conv1d_14/bias_0',
'encode7/batch_normalization_14/gamma_0',
'encode7/batch_normalization_14/beta_0',
'encode7/batch_normalization_14/moving_mean_0',
'encode7/batch_normalization_14/moving_variance_0',
'encode7/conv1d_15/kernel_0',
'encode7/conv1d_15/bias_0',
'encode7/batch_normalization_15/gamma_0',
'encode7/batch_normalization_15/beta_0',
'encode7/batch_normalization_15/moving_mean_0',
'encode7/batch_normalization_15/moving_variance_0',
'encode8/conv1d_16/kernel_0',
'encode8/conv1d_16/bias_0',
'encode8/batch_normalization_16/gamma_0',
'encode8/batch_normalization_16/beta_0',
'encode8/batch_normalization_16/moving_mean_0',
'encode8/batch_normalization_16/moving_variance_0',
'encode8/conv1d_17/kernel_0',
'encode8/conv1d_17/bias_0',
'encode8/batch_normalization_17/gamma_0',
'encode8/batch_normalization_17/beta_0',
'encode8/batch_normalization_17/moving_mean_0',
'encode8/batch_normalization_17/moving_variance_0',
'two_conv_center/conv1d_18/kernel_0',
'two_conv_center/conv1d_18/bias_0',
'two_conv_center/batch_normalization_18/gamma_0',
'two_conv_center/batch_normalization_18/beta_0',
'two_conv_center/batch_normalization_18/moving_mean_0',
'two_conv_center/batch_normalization_18/moving_variance_0',
'two_conv_center/conv1d_19/kernel_0',
'two_conv_center/conv1d_19/bias_0',
'two_conv_center/batch_normalization_19/gamma_0',
'two_conv_center/batch_normalization_19/beta_0',
'two_conv_center/batch_normalization_19/moving_mean_0',
'two_conv_center/batch_normalization_19/moving_variance_0',
'conv_transpose_decoder8/conv1d_transpose/kernel_0',
'conv_transpose_decoder8/conv1d_transpose/bias_0',
'conv_transpose_decoder8/batch_normalization_20/gamma_0',
'conv_transpose_decoder8/batch_normalization_20/beta_0',
'conv_transpose_decoder8/batch_normalization_20/moving_mean_0',
'conv_transpose_decoder8/batch_normalization_20/moving_variance_0',
'two_conv_decoder8/conv1d_20/kernel_0',
'two_conv_decoder8/conv1d_20/bias_0',
'two_conv_decoder8/batch_normalization_21/gamma_0',
'two_conv_decoder8/batch_normalization_21/beta_0',
'two_conv_decoder8/batch_normalization_21/moving_mean_0',
'two_conv_decoder8/batch_normalization_21/moving_variance_0',
'two_conv_decoder8/conv1d_21/kernel_0',
'two_conv_decoder8/conv1d_21/bias_0',
'two_conv_decoder8/batch_normalization_22/gamma_0',
'two_conv_decoder8/batch_normalization_22/beta_0',
'two_conv_decoder8/batch_normalization_22/moving_mean_0',
'two_conv_decoder8/batch_normalization_22/moving_variance_0',
'conv_transpose_decoder7/conv1d_transpose_1/kernel_0',
'conv_transpose_decoder7/conv1d_transpose_1/bias_0',
'conv_transpose_decoder7/batch_normalization_23/gamma_0',
'conv_transpose_decoder7/batch_normalization_23/beta_0',
'conv_transpose_decoder7/batch_normalization_23/moving_mean_0',
'conv_transpose_decoder7/batch_normalization_23/moving_variance_0',
'two_conv_decoder7/conv1d_22/kernel_0',
'two_conv_decoder7/conv1d_22/bias_0',
'two_conv_decoder7/batch_normalization_24/gamma_0',
'two_conv_decoder7/batch_normalization_24/beta_0',
'two_conv_decoder7/batch_normalization_24/moving_mean_0',
'two_conv_decoder7/batch_normalization_24/moving_variance_0',
'two_conv_decoder7/conv1d_23/kernel_0',
'two_conv_decoder7/conv1d_23/bias_0',
'two_conv_decoder7/batch_normalization_25/gamma_0',
'two_conv_decoder7/batch_normalization_25/beta_0',
'two_conv_decoder7/batch_normalization_25/moving_mean_0',
'two_conv_decoder7/batch_normalization_25/moving_variance_0',
'conv_transpose_decoder6/conv1d_transpose_2/kernel_0',
'conv_transpose_decoder6/conv1d_transpose_2/bias_0',
'conv_transpose_decoder6/batch_normalization_26/gamma_0',
'conv_transpose_decoder6/batch_normalization_26/beta_0',
'conv_transpose_decoder6/batch_normalization_26/moving_mean_0',
'conv_transpose_decoder6/batch_normalization_26/moving_variance_0',
'two_conv_decoder6/conv1d_24/kernel_0',
'two_conv_decoder6/conv1d_24/bias_0',
'two_conv_decoder6/batch_normalization_27/gamma_0',
'two_conv_decoder6/batch_normalization_27/beta_0',
'two_conv_decoder6/batch_normalization_27/moving_mean_0',
'two_conv_decoder6/batch_normalization_27/moving_variance_0',
'two_conv_decoder6/conv1d_25/kernel_0',
'two_conv_decoder6/conv1d_25/bias_0',
'two_conv_decoder6/batch_normalization_28/gamma_0',
'two_conv_decoder6/batch_normalization_28/beta_0',
'two_conv_decoder6/batch_normalization_28/moving_mean_0',
'two_conv_decoder6/batch_normalization_28/moving_variance_0',
'conv_transpose_decoder5/conv1d_transpose_3/kernel_0',
'conv_transpose_decoder5/conv1d_transpose_3/bias_0',
'conv_transpose_decoder5/batch_normalization_29/gamma_0',
'conv_transpose_decoder5/batch_normalization_29/beta_0',
'conv_transpose_decoder5/batch_normalization_29/moving_mean_0',
'conv_transpose_decoder5/batch_normalization_29/moving_variance_0',
'two_conv_decoder5/conv1d_26/kernel_0',
'two_conv_decoder5/conv1d_26/bias_0',
'two_conv_decoder5/batch_normalization_30/gamma_0',
'two_conv_decoder5/batch_normalization_30/beta_0',
'two_conv_decoder5/batch_normalization_30/moving_mean_0',
'two_conv_decoder5/batch_normalization_30/moving_variance_0',
'two_conv_decoder5/conv1d_27/kernel_0',
'two_conv_decoder5/conv1d_27/bias_0',
'two_conv_decoder5/batch_normalization_31/gamma_0',
'two_conv_decoder5/batch_normalization_31/beta_0',
'two_conv_decoder5/batch_normalization_31/moving_mean_0',
'two_conv_decoder5/batch_normalization_31/moving_variance_0',
'conv_transpose_decoder4/conv1d_transpose_4/kernel_0',
'conv_transpose_decoder4/conv1d_transpose_4/bias_0',
'conv_transpose_decoder4/batch_normalization_32/gamma_0',
'conv_transpose_decoder4/batch_normalization_32/beta_0',
'conv_transpose_decoder4/batch_normalization_32/moving_mean_0',
'conv_transpose_decoder4/batch_normalization_32/moving_variance_0',
'two_conv_decoder4/conv1d_28/kernel_0',
'two_conv_decoder4/conv1d_28/bias_0',
'two_conv_decoder4/batch_normalization_33/gamma_0',
'two_conv_decoder4/batch_normalization_33/beta_0',
'two_conv_decoder4/batch_normalization_33/moving_mean_0',
'two_conv_decoder4/batch_normalization_33/moving_variance_0',
'two_conv_decoder4/conv1d_29/kernel_0',
'two_conv_decoder4/conv1d_29/bias_0',
'two_conv_decoder4/batch_normalization_34/gamma_0',
'two_conv_decoder4/batch_normalization_34/beta_0',
'two_conv_decoder4/batch_normalization_34/moving_mean_0',
'two_conv_decoder4/batch_normalization_34/moving_variance_0',
'conv_transpose_decoder3/conv1d_transpose_5/kernel_0',
'conv_transpose_decoder3/conv1d_transpose_5/bias_0',
'conv_transpose_decoder3/batch_normalization_35/gamma_0',
'conv_transpose_decoder3/batch_normalization_35/beta_0',
'conv_transpose_decoder3/batch_normalization_35/moving_mean_0',
'conv_transpose_decoder3/batch_normalization_35/moving_variance_0',
'two_conv_decoder3/conv1d_30/kernel_0',
'two_conv_decoder3/conv1d_30/bias_0',
'two_conv_decoder3/batch_normalization_36/gamma_0',
'two_conv_decoder3/batch_normalization_36/beta_0',
'two_conv_decoder3/batch_normalization_36/moving_mean_0',
'two_conv_decoder3/batch_normalization_36/moving_variance_0',
'two_conv_decoder3/conv1d_31/kernel_0',
'two_conv_decoder3/conv1d_31/bias_0',
'two_conv_decoder3/batch_normalization_37/gamma_0',
'two_conv_decoder3/batch_normalization_37/beta_0',
'two_conv_decoder3/batch_normalization_37/moving_mean_0',
'two_conv_decoder3/batch_normalization_37/moving_variance_0',
'conv_transpose_decoder2/conv1d_transpose_6/kernel_0',
'conv_transpose_decoder2/conv1d_transpose_6/bias_0',
'conv_transpose_decoder2/batch_normalization_38/gamma_0',
'conv_transpose_decoder2/batch_normalization_38/beta_0',
'conv_transpose_decoder2/batch_normalization_38/moving_mean_0',
'conv_transpose_decoder2/batch_normalization_38/moving_variance_0',
'two_conv_decoder2/conv1d_32/kernel_0',
'two_conv_decoder2/conv1d_32/bias_0',
'two_conv_decoder2/batch_normalization_39/gamma_0',
'two_conv_decoder2/batch_normalization_39/beta_0',
'two_conv_decoder2/batch_normalization_39/moving_mean_0',
'two_conv_decoder2/batch_normalization_39/moving_variance_0',
'two_conv_decoder2/conv1d_33/kernel_0',
'two_conv_decoder2/conv1d_33/bias_0',
'two_conv_decoder2/batch_normalization_40/gamma_0',
'two_conv_decoder2/batch_normalization_40/beta_0',
'two_conv_decoder2/batch_normalization_40/moving_mean_0',
'two_conv_decoder2/batch_normalization_40/moving_variance_0',
'conv_transpose_decoder1/conv1d_transpose_7/kernel_0',
'conv_transpose_decoder1/conv1d_transpose_7/bias_0',
'conv_transpose_decoder1/batch_normalization_41/gamma_0',
'conv_transpose_decoder1/batch_normalization_41/beta_0',
'conv_transpose_decoder1/batch_normalization_41/moving_mean_0',
'conv_transpose_decoder1/batch_normalization_41/moving_variance_0',
'two_conv_decoder1/conv1d_34/kernel_0',
'two_conv_decoder1/conv1d_34/bias_0',
'two_conv_decoder1/batch_normalization_42/gamma_0',
'two_conv_decoder1/batch_normalization_42/beta_0',
'two_conv_decoder1/batch_normalization_42/moving_mean_0',
'two_conv_decoder1/batch_normalization_42/moving_variance_0',
'two_conv_decoder1/conv1d_35/kernel_0',
'two_conv_decoder1/conv1d_35/bias_0',
'two_conv_decoder1/batch_normalization_43/gamma_0',
'two_conv_decoder1/batch_normalization_43/beta_0',
'two_conv_decoder1/batch_normalization_43/moving_mean_0',
'two_conv_decoder1/batch_normalization_43/moving_variance_0',
'conv_transpose_decoder0/conv1d_transpose_8/kernel_0',
'conv_transpose_decoder0/conv1d_transpose_8/bias_0',
'conv_transpose_decoder0/batch_normalization_44/gamma_0',
'conv_transpose_decoder0/batch_normalization_44/beta_0',
'conv_transpose_decoder0/batch_normalization_44/moving_mean_0',
'conv_transpose_decoder0/batch_normalization_44/moving_variance_0',
'two_conv_decoder0/conv1d_36/kernel_0',
'two_conv_decoder0/conv1d_36/bias_0',
'two_conv_decoder0/batch_normalization_45/gamma_0',
'two_conv_decoder0/batch_normalization_45/beta_0',
'two_conv_decoder0/batch_normalization_45/moving_mean_0',
'two_conv_decoder0/batch_normalization_45/moving_variance_0',
'two_conv_decoder0/conv1d_37/kernel_0',
'two_conv_decoder0/conv1d_37/bias_0',
'two_conv_decoder0/batch_normalization_46/gamma_0',
'two_conv_decoder0/batch_normalization_46/beta_0',
'two_conv_decoder0/batch_normalization_46/moving_mean_0',
'two_conv_decoder0/batch_normalization_46/moving_variance_0',
'conv1d_38/kernel_0',
'conv1d_38/bias_0'],
'tensors': ['batch_2'],
'graph': False,
'meta_graph': False,
'run_metadata': []}
ea.Scalars('epoch_loss')
2.1.8 Load model from mlflow logs and predict separately loaded data
First, check if the model file has to be downloaded via git lfs
cd /home/lex/Programme/drmed-git/ git lfs ls-files git lfs checkout
(base) [lex@Topialex drmed-git]$ git lfs ls-files 6526c5abca - data/mlruns/1/47b870b8fdcb4445956635c6758caff3/artifacts/model/data/model.h5 (base) [lex@Topialex drmed-git]$ git lfs checkout Skipped checkout for "data/mlruns/1/47b870b8fdcb4445956635c6758caff3/artifacts/model/data/model.h5", content not local. Use fetch to download. Checking out LFS objects: 100% (1/1), 399 MB | 0 B/s, done.
git lfs pull
fetch: Fetching reference refs/heads/develop Username for 'https://github.com': aseltmann B/s Password for 'https://aseltmann@github.com': (base) [lex@Topialex drmed-git]$ 1), 399 MB | 1.9 MB/s
Note: after these downloads, the GitHub-Version of git lfs didn't work for me anymore without a payment for download/upload. Initially I thought of git lfs as a distributed way of storing files without the need of a central storage. GitHub does not seem to support that, which is sad - I will move away from it in the future. See here.
import mlflow.keras import mlflow.tensorflow import sys import numpy as np import tensorflow as tf project_path = '/home/lex/Programme/drmed-git' fluotracify_path = '{}/src/'.format(project_path) sys.path.append(fluotracify_path) from fluotracify.simulations import import_simulation_from_csv as isfc from fluotracify.training import build_model as bm, preprocess_data as ppd %cd /home/lex/Programme/drmed-git print(tf.__version__)
/home/lex/Programme/drmed-git 2.3.0-dev20200519
mlflow.set_tracking_uri('file://{}/data/mlruns'.format(project_path)) client = mlflow.tracking.MlflowClient(tracking_uri=mlflow.get_tracking_uri()) model_path = client.download_artifacts( run_id='1aefda1366f04f5da5d1fc2241ad9208', path='model') print(model_path)
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model
bm.binary_ce_dice_loss()
<function fluotracify.training.build_model.binary_ce_dice_loss.<locals>.binary_ce_dice(y_true, y_pred)>
# mlflow.keras module model_keras = mlflow.keras.load_model(model_uri=model_path, custom_objects={'binary_ce_dice': bm.binary_ce_dice_loss()}) model_keras
<tensorflow.python.keras.engine.functional.Functional at 0x7f3afa29d0a0>
data, _, nsamples, experiment_params = isfc.import_from_csv( path='/home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/', header=12, frac_train=1, col_per_example=2, dropindex=None, dropcolumns='Unnamed: 200')
train 0 /home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/traces_brightclust_rand_Sep2019_set003.csv train 1 /home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/traces_brightclust_rand_Sep2019_set002.csv train 2 /home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/traces_brightclust_rand_Sep2019_set001.csv
input - shape: (None, 16384, 1)
output - shape: (None, 16384, 1)
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 16384, 1)] 0
__________________________________________________________________________________________________
encode0 (Sequential) (None, 16384, 64) 13120 input_1[0][0]
__________________________________________________________________________________________________
mp_encode0 (MaxPooling1D) (None, 8192, 64) 0 encode0[0][0]
__________________________________________________________________________________________________
encode1 (Sequential) (None, 8192, 128) 75008 mp_encode0[0][0]
__________________________________________________________________________________________________
mp_encode1 (MaxPooling1D) (None, 4096, 128) 0 encode1[0][0]
__________________________________________________________________________________________________
encode2 (Sequential) (None, 4096, 256) 297472 mp_encode1[0][0]
__________________________________________________________________________________________________
mp_encode2 (MaxPooling1D) (None, 2048, 256) 0 encode2[0][0]
__________________________________________________________________________________________________
encode3 (Sequential) (None, 2048, 512) 1184768 mp_encode2[0][0]
__________________________________________________________________________________________________
mp_encode3 (MaxPooling1D) (None, 1024, 512) 0 encode3[0][0]
__________________________________________________________________________________________________
encode4 (Sequential) (None, 1024, 512) 1577984 mp_encode3[0][0]
__________________________________________________________________________________________________
mp_encode4 (MaxPooling1D) (None, 512, 512) 0 encode4[0][0]
__________________________________________________________________________________________________
encode5 (Sequential) (None, 512, 512) 1577984 mp_encode4[0][0]
__________________________________________________________________________________________________
mp_encode5 (MaxPooling1D) (None, 256, 512) 0 encode5[0][0]
__________________________________________________________________________________________________
encode6 (Sequential) (None, 256, 512) 1577984 mp_encode5[0][0]
__________________________________________________________________________________________________
mp_encode6 (MaxPooling1D) (None, 128, 512) 0 encode6[0][0]
__________________________________________________________________________________________________
encode7 (Sequential) (None, 128, 512) 1577984 mp_encode6[0][0]
__________________________________________________________________________________________________
mp_encode7 (MaxPooling1D) (None, 64, 512) 0 encode7[0][0]
__________________________________________________________________________________________________
encode8 (Sequential) (None, 64, 512) 1577984 mp_encode7[0][0]
__________________________________________________________________________________________________
mp_encode8 (MaxPooling1D) (None, 32, 512) 0 encode8[0][0]
__________________________________________________________________________________________________
two_conv_center (Sequential) (None, 32, 1024) 4728832 mp_encode8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder8 (Sequen (None, 64, 512) 1051136 two_conv_center[0][0]
__________________________________________________________________________________________________
decoder8 (Concatenate) (None, 64, 1024) 0 encode8[0][0]
conv_transpose_decoder8[0][0]
__________________________________________________________________________________________________
two_conv_decoder8 (Sequential) (None, 64, 512) 2364416 decoder8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder7 (Sequen (None, 128, 512) 526848 two_conv_decoder8[0][0]
__________________________________________________________________________________________________
decoder7 (Concatenate) (None, 128, 1024) 0 encode7[0][0]
conv_transpose_decoder7[0][0]
__________________________________________________________________________________________________
two_conv_decoder7 (Sequential) (None, 128, 512) 2364416 decoder7[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder6 (Sequen (None, 256, 512) 526848 two_conv_decoder7[0][0]
__________________________________________________________________________________________________
decoder6 (Concatenate) (None, 256, 1024) 0 encode6[0][0]
conv_transpose_decoder6[0][0]
__________________________________________________________________________________________________
two_conv_decoder6 (Sequential) (None, 256, 512) 2364416 decoder6[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder5 (Sequen (None, 512, 512) 526848 two_conv_decoder6[0][0]
__________________________________________________________________________________________________
decoder5 (Concatenate) (None, 512, 1024) 0 encode5[0][0]
conv_transpose_decoder5[0][0]
__________________________________________________________________________________________________
two_conv_decoder5 (Sequential) (None, 512, 512) 2364416 decoder5[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder4 (Sequen (None, 1024, 512) 526848 two_conv_decoder5[0][0]
__________________________________________________________________________________________________
decoder4 (Concatenate) (None, 1024, 1024) 0 encode4[0][0]
conv_transpose_decoder4[0][0]
__________________________________________________________________________________________________
two_conv_decoder4 (Sequential) (None, 1024, 512) 2364416 decoder4[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder3 (Sequen (None, 2048, 512) 526848 two_conv_decoder4[0][0]
__________________________________________________________________________________________________
decoder3 (Concatenate) (None, 2048, 1024) 0 encode3[0][0]
conv_transpose_decoder3[0][0]
__________________________________________________________________________________________________
two_conv_decoder3 (Sequential) (None, 2048, 512) 2364416 decoder3[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder2 (Sequen (None, 4096, 256) 263424 two_conv_decoder3[0][0]
__________________________________________________________________________________________________
decoder2 (Concatenate) (None, 4096, 512) 0 encode2[0][0]
conv_transpose_decoder2[0][0]
__________________________________________________________________________________________________
two_conv_decoder2 (Sequential) (None, 4096, 256) 592384 decoder2[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder1 (Sequen (None, 8192, 128) 66176 two_conv_decoder2[0][0]
__________________________________________________________________________________________________
decoder1 (Concatenate) (None, 8192, 256) 0 encode1[0][0]
conv_transpose_decoder1[0][0]
__________________________________________________________________________________________________
two_conv_decoder1 (Sequential) (None, 8192, 128) 148736 decoder1[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder0 (Sequen (None, 16384, 64) 16704 two_conv_decoder1[0][0]
__________________________________________________________________________________________________
decoder0 (Concatenate) (None, 16384, 128) 0 encode0[0][0]
conv_transpose_decoder0[0][0]
__________________________________________________________________________________________________
two_conv_decoder0 (Sequential) (None, 16384, 64) 37504 decoder0[0][0]
__________________________________________________________________________________________________
conv1d_38 (Conv1D) (None, 16384, 1) 65 two_conv_decoder0[0][0]
==================================================================================================
Total params: 33,185,985
Trainable params: 33,146,689
Non-trainable params: 39,296
__________________________________________________________________________________________________
None
prediction = model_keras.predict(np.array(data.iloc[:16384, 0]).reshape(1, -1, 1))
I noticed one problem: directly outputting the prediction array for large
predictions (e.g. 16384 time steps), will freeze emacs. Especially with my setup
with conversion to org tables. It is possible to output it in the REPL, or print
it using print(prediction), because there the output gets truncated, e.g. like
this:
array([[[1.],
[1.],
[1.],
...,
[1.],
[1.],
[1.]]], dtype=float32)
While here it tries to print everything (I guess). Maybe I should test the
:pandoc t argument instead of the OrgFormatter class
Basically, no matter if I choosnpe the :results header argument, it always gets
printed with :display org like this.
log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S") tensorboard_callback = tf.keras.callbacks.TensorBoard( log_dir=log_dir, histogram_freq=5, write_images=True, update_freq='batch') epochs = 50 history = model.fit(x=dataset_train, epochs=epochs, steps_per_epoch=400, validation_data=dataset_val, validation_steps=tf.math.ceil(num_val_examples / batch_size), callbacks=[tensorboard_callback])
WARNING: Logging before flag parsing goes to stderr. W0413 03:23:05.926374 47620222508736 summary_ops_v2.py:1132] Model failed to serialize as JSON. Ignoring... Layers with arguments in `__init__` must override `get_config`. Train for 400 steps, validate for 534.0 steps Epoch 1/50 8/400 [..............................] - ETA: 1:05:18 - loss: 1.4586 - mean_io_u: 0.4039 - precision: 0.2347 - recall: 0.1408
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last)
prediction
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 2043 | 2044 | 2045 | 2046 | 2047 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
To make the output work the desired way (like in the REPL or print() and
without freezing emacs for large tables), use :display plain
prediction
array([[[1.],
[1.],
[1.],
...,
[1.],
[1.],
[1.]]], dtype=float32)
2.1.9 git exp 2
git status git log -1
(tensorflow_nightly) [ye53nis@node151 drmed-git]$ git status
# On branch exp-310520-unet
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
# ...
nothing added to commit but untracked files present (use "git add" to track)
(tensorflow_nightly) [ye53nis@node151 drmed-git]$ git log -1
commit 09c1d2a1bc083695026b46c74ea175c9168bb5f2
Author: Apoplex <oligolex@vivaldi.net>
Date: Fri Jun 12 14:17:14 2020 +0200
New learning rate schedule
current learning rate schedule:
learning_rate = 0.1 if epoch > 10: learning_rate = 0.01 if epoch > 20: learning_rate = 0.001 if epoch > 30: learning_rate = 0.0001 if epoch > 40: learning_rate = 0.00001 if epoch > 50: learning_rate = 0.000001 if epoch > 60: learning_rate = 0.00000001 if epoch > 70: learning_rate = 0.0000000001 if epoch > 80: learning_rate = 0.000000000001 if epoch > 90: learning_rate = 0.00000000000001
2.1.10 experimental run 2 - full dataset
This time new learning rate
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=100 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=1280 -P validation_steps=320
(tensorflow_nightly) [ye53nis@node151 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=100 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=1280 -P val idation_steps=320 2020/06/12 14:20:34 INFO mlflow.projects: === Created directory /tmp/tmp2s79px_z for downloading remote URIs passed to arguments of type 'path' === 2020/06/12 14:20:34 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py / beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 100 /beegfs/ye53nis/saves/firstartefact_Sep2019 1280 320' in run with ID '037e1d9e4ad74784974f4aaac11138cc' === 2020-06-12 14:20:53.286502: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2020-06-12 14:20:53.286593: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul e's documentation for alternative uses import imp 2.3.0-dev20200527 2020-06-12 14:21:33.782337: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-06-12 14:21:33.782422: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303) 2020-06-12 14:21:33.782480: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node151): /proc/driver/nvidia/version does not exist GPUs: [] train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000) shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000) for each 20,000 timestap trace there are the following numbers of corrupted timesteps: label001_1 6286 label001_1 2568 label001_1 4495 label001_1 4414 label001_1 1105 dtype: int64 2020-06-12 14:26:49.734444: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-06-12 14:26:49.744132: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz 2020-06-12 14:26:49.745558: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55721c34af40 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-06-12 14:26:49.745586: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version number of training examples: 6400, number of validation examples: 1600 ------------------------ number of test examples: 2000 input - shape: (None, 16384, 1) output - shape: (None, 16384, 1) 2020-06-12 14:26:53.653730: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(wrapped_dict, collections.Mapping): /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec() all_param_names, _, _, all_default_values = inspect.getargspec(fn) # pylint: disable=W1505 Epoch 1/100 /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(values, collections.Sequence): 1/1280 [..............................] - ETA: 0s - loss: 1.5430 - tp: 5048.0000 - fp: 18093.0000 - tn: 50755.0000 - fn: 8024.0000 - precision: 0.2181 - recall: 0.3862 - accuracy: 0.6812 - auc: 0.58482020-06-12 14:27:06.190184: I ten sorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w ill be removed after 2020-07-01. Instructions for updating: use `tf.profiler.experimental.stop` instead. 2020-06-12 14:27:07.670394: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_12_14_27_07 2020-06-12 14:27:07.684980: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.trace.json.gz 2020-06-12 14:27:07.711539: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_12_14_27_07 2020-06-12 14:27:07.711638: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.memory_profile.json.gz 2020-06-12 14:27:07.713732: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_06_12_14_27_07Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_06_12_1 4_27_07/node151.xplane.pb Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.overview_page.pb Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.kernel_stats.pb 1280/1280 [==============================] - 2053s 2s/step - loss: 0.9576 - tp: 14931613.0000 - fp: 6406557.0000 - tn: 74369864.0000 - fn: 9149518.0000 - precision: 0.6998 - recall: 0.6201 - accuracy: 0.8516 - auc: 0.8805 - val_loss: 1. 1981 - val_tp: 3451021.0000 - val_fp: 465330.0000 - val_tn: 19870236.0000 - val_fn: 2427812.0000 - val_precision: 0.8812 - val_recall: 0.5870 - val_accuracy: 0.8896 - val_auc: 0.8828 Epoch 2/100 679/1280 [==============>...............] - ETA: 14:47 - loss: 0.6900 - tp: 9867994.0000 - fp: 2756931.0000 - tn: 40089676.0000 - fn: 2909066.0000 - precision: 0.7816 - recall: 0.7723 - accuracy: 0.8981 - auc: 0.9307 1280/1280 [==============================] - 1998s 2s/step - loss: 0.6748 - tp: 18707488.0000 - fp: 4980630.0000 - tn: 75750952.0000 - fn: 5418544.0000 - precision: 0.7897 - recall: 0.7754 - accuracy: 0.9008 - auc: 0.9335 - val_loss: 1. 8923 - val_tp: 1511411.0000 - val_fp: 71511.0000 - val_tn: 20387136.0000 - val_fn: 4244342.0000 - val_precision: 0.9548 - val_recall: 0.2626 - val_accuracy: 0.8354 - val_auc: 0.8004 Epoch 3/100 1280/1280 [==============================] - 1974s 2s/step - loss: 0.6086 - tp: 19388796.0000 - fp: 4672424.0000 - tn: 76028144.0000 - fn: 4768219.0000 - precision: 0.8058 - recall: 0.8026 - accuracy: 0.9100 - auc: 0.9440 - val_loss: 4. 4772 - val_tp: 497608.0000 - val_fp: 47.0000 - val_tn: 20133974.0000 - val_fn: 5582775.0000 - val_precision: 0.9999 - val_recall: 0.0818 - val_accuracy: 0.7870 - val_auc: 0.6095 Epoch 4/100 1280/1280 [==============================] - 2036s 2s/step - loss: 0.5105 - tp: 20225824.0000 - fp: 3816004.0000 - tn: 76833904.0000 - fn: 3981872.0000 - precision: 0.8413 - recall: 0.8355 - accuracy: 0.9256 - auc: 0.9568 - val_loss: 1. 5316 - val_tp: 1395128.0000 - val_fp: 5102.0000 - val_tn: 20164212.0000 - val_fn: 4649960.0000 - val_precision: 0.9964 - val_recall: 0.2308 - val_accuracy: 0.8224 - val_auc: 0.7861 Epoch 5/100 1280/1280 [==============================] - 1961s 2s/step - loss: 0.4310 - tp: 20871492.0000 - fp: 3257518.0000 - tn: 77358976.0000 - fn: 3369640.0000 - precision: 0.8650 - recall: 0.8610 - accuracy: 0.9368 - auc: 0.9655 - val_loss: 0. 6155 - val_tp: 4878880.0000 - val_fp: 1771345.0000 - val_tn: 18827236.0000 - val_fn: 736940.0000 - val_precision: 0.7336 - val_recall: 0.8688 - val_accuracy: 0.9043 - val_auc: 0.9524 Epoch 6/100 1280/1280 [==============================] - 1934s 2s/step - loss: 0.3911 - tp: 20849656.0000 - fp: 2983615.0000 - tn: 77906056.0000 - fn: 3118252.0000 - precision: 0.8748 - recall: 0.8699 - accuracy: 0.9418 - auc: 0.9695 - val_loss: 13 .7751 - val_tp: 5829853.0000 - val_fp: 12729657.0000 - val_tn: 7490981.0000 - val_fn: 163909.0000 - val_precision: 0.3141 - val_recall: 0.9727 - val_accuracy: 0.5081 - val_auc: 0.7568 Epoch 7/100 1280/1280 [==============================] - 1954s 2s/step - loss: 0.3763 - tp: 20945158.0000 - fp: 2869675.0000 - tn: 78062216.0000 - fn: 2980532.0000 - precision: 0.8795 - recall: 0.8754 - accuracy: 0.9442 - auc: 0.9712 - val_loss: 1. 4221 - val_tp: 2763303.0000 - val_fp: 537658.0000 - val_tn: 19870444.0000 - val_fn: 3042995.0000 - val_precision: 0.8371 - val_recall: 0.4759 - val_accuracy: 0.8634 - val_auc: 0.8209 Epoch 8/100 1280/1280 [==============================] - 1959s 2s/step - loss: 0.3565 - tp: 21224852.0000 - fp: 2648995.0000 - tn: 78077912.0000 - fn: 2905816.0000 - precision: 0.8890 - recall: 0.8796 - accuracy: 0.9470 - auc: 0.9734 - val_loss: 4. 1428 - val_tp: 671439.0000 - val_fp: 5.0000 - val_tn: 19926294.0000 - val_fn: 5616654.0000 - val_precision: 1.0000 - val_recall: 0.1068 - val_accuracy: 0.7857 - val_auc: 0.6147 Epoch 9/100 1280/1280 [==============================] - 1959s 2s/step - loss: 0.3532 - tp: 21099540.0000 - fp: 2612022.0000 - tn: 78277824.0000 - fn: 2868197.0000 - precision: 0.8898 - recall: 0.8803 - accuracy: 0.9477 - auc: 0.9736 - val_loss: 0. 4181 - val_tp: 5276605.0000 - val_fp: 1364866.0000 - val_tn: 19070268.0000 - val_fn: 502655.0000 - val_precision: 0.7945 - val_recall: 0.9130 - val_accuracy: 0.9288 - val_auc: 0.9733 Epoch 10/100 1280/1280 [==============================] - 1964s 2s/step - loss: 0.3303 - tp: 21854000.0000 - fp: 2601074.0000 - tn: 77778032.0000 - fn: 2624482.0000 - precision: 0.8936 - recall: 0.8928 - accuracy: 0.9502 - auc: 0.9758 - val_loss: 0. 8186 - val_tp: 4852372.0000 - val_fp: 1259191.0000 - val_tn: 19046032.0000 - val_fn: 1056801.0000 - val_precision: 0.7940 - val_recall: 0.8212 - val_accuracy: 0.9117 - val_auc: 0.9279 Epoch 11/100 1280/1280 [==============================] - 1954s 2s/step - loss: 0.3367 - tp: 21247744.0000 - fp: 2633337.0000 - tn: 78318176.0000 - fn: 2658349.0000 - precision: 0.8897 - recall: 0.8888 - accuracy: 0.9495 - auc: 0.9748 - val_loss: 14 .1746 - val_tp: 5315183.0000 - val_fp: 9292374.0000 - val_tn: 10878559.0000 - val_fn: 728284.0000 - val_precision: 0.3639 - val_recall: 0.8795 - val_accuracy: 0.6177 - val_auc: 0.7350 Epoch 12/100 1280/1280 [==============================] - 1962s 2s/step - loss: 0.3097 - tp: 21383088.0000 - fp: 2165215.0000 - tn: 78747360.0000 - fn: 2561919.0000 - precision: 0.9081 - recall: 0.8930 - accuracy: 0.9549 - auc: 0.9777 - val_loss: 0. 3274 - val_tp: 4925695.0000 - val_fp: 165902.0000 - val_tn: 20004090.0000 - val_fn: 1118702.0000 - val_precision: 0.9674 - val_recall: 0.8149 - val_accuracy: 0.9510 - val_auc: 0.9759 Epoch 13/100 1280/1280 [==============================] - 1969s 2s/step - loss: 0.2912 - tp: 21857506.0000 - fp: 2213488.0000 - tn: 78446488.0000 - fn: 2340152.0000 - precision: 0.9080 - recall: 0.9033 - accuracy: 0.9566 - auc: 0.9793 - val_loss: 0. 4428 - val_tp: 4338892.0000 - val_fp: 35099.0000 - val_tn: 20239610.0000 - val_fn: 1600799.0000 - val_precision: 0.9920 - val_recall: 0.7305 - val_accuracy: 0.9376 - val_auc: 0.9713 Epoch 14/100 1280/1280 [==============================] - 1957s 2s/step - loss: 0.2883 - tp: 21882504.0000 - fp: 2239269.0000 - tn: 78431648.0000 - fn: 2304140.0000 - precision: 0.9072 - recall: 0.9047 - accuracy: 0.9567 - auc: 0.9796 - val_loss: 0. 2957 - val_tp: 5289461.0000 - val_fp: 449961.0000 - val_tn: 19835942.0000 - val_fn: 639032.0000 - val_precision: 0.9216 - val_recall: 0.8922 - val_accuracy: 0.9585 - val_auc: 0.9820 Epoch 15/100 1280/1280 [==============================] - 2044s 2s/step - loss: 0.2825 - tp: 21866824.0000 - fp: 2177254.0000 - tn: 78541248.0000 - fn: 2272316.0000 - precision: 0.9094 - recall: 0.9059 - accuracy: 0.9576 - auc: 0.9802 - val_loss: 0. 3209 - val_tp: 5233737.0000 - val_fp: 520949.0000 - val_tn: 19784424.0000 - val_fn: 675290.0000 - val_precision: 0.9095 - val_recall: 0.8857 - val_accuracy: 0.9544 - val_auc: 0.9800 Epoch 16/100 1280/1280 [==============================] - 1957s 2s/step - loss: 0.2769 - tp: 21749612.0000 - fp: 2170557.0000 - tn: 78722872.0000 - fn: 2214598.0000 - precision: 0.9093 - recall: 0.9076 - accuracy: 0.9582 - auc: 0.9802 - val_loss: 0. 3247 - val_tp: 5660332.0000 - val_fp: 828589.0000 - val_tn: 19303372.0000 - val_fn: 422111.0000 - val_precision: 0.8723 - val_recall: 0.9306 - val_accuracy: 0.9523 - val_auc: 0.9859 Epoch 17/100 1280/1280 [==============================] - 1949s 2s/step - loss: 0.2774 - tp: 21816756.0000 - fp: 2164372.0000 - tn: 78635128.0000 - fn: 2241237.0000 - precision: 0.9097 - recall: 0.9068 - accuracy: 0.9580 - auc: 0.9806 - val_loss: 0. 3148 - val_tp: 4953313.0000 - val_fp: 118813.0000 - val_tn: 20032204.0000 - val_fn: 1110073.0000 - val_precision: 0.9766 - val_recall: 0.8169 - val_accuracy: 0.9531 - val_auc: 0.9788 Epoch 18/100 1280/1280 [==============================] - 1944s 2s/step - loss: 0.2725 - tp: 21678578.0000 - fp: 2114671.0000 - tn: 78865952.0000 - fn: 2198422.0000 - precision: 0.9111 - recall: 0.9079 - accuracy: 0.9589 - auc: 0.9807 - val_loss: 0. 3363 - val_tp: 4425577.0000 - val_fp: 55980.0000 - val_tn: 20363536.0000 - val_fn: 1369309.0000 - val_precision: 0.9875 - val_recall: 0.7637 - val_accuracy: 0.9456 - val_auc: 0.9739 Epoch 19/100 1280/1280 [==============================] - 1947s 2s/step - loss: 0.2757 - tp: 22040606.0000 - fp: 2177952.0000 - tn: 78420008.0000 - fn: 2218997.0000 - precision: 0.9101 - recall: 0.9085 - accuracy: 0.9581 - auc: 0.9805 - val_loss: 0. 3061 - val_tp: 5376668.0000 - val_fp: 295874.0000 - val_tn: 19715332.0000 - val_fn: 826533.0000 - val_precision: 0.9478 - val_recall: 0.8668 - val_accuracy: 0.9572 - val_auc: 0.9796 Epoch 20/100 1280/1280 [==============================] - 1953s 2s/step - loss: 0.2758 - tp: 21804466.0000 - fp: 2129789.0000 - tn: 78704432.0000 - fn: 2218904.0000 - precision: 0.9110 - recall: 0.9076 - accuracy: 0.9585 - auc: 0.9805 - val_loss: 0. 3864 - val_tp: 4404185.0000 - val_fp: 40930.0000 - val_tn: 20226700.0000 - val_fn: 1542578.0000 - val_precision: 0.9908 - val_recall: 0.7406 - val_accuracy: 0.9396 - val_auc: 0.9670 Epoch 21/100 1280/1280 [==============================] - 1956s 2s/step - loss: 0.2715 - tp: 21833378.0000 - fp: 2128388.0000 - tn: 78697688.0000 - fn: 2198099.0000 - precision: 0.9112 - recall: 0.9085 - accuracy: 0.9587 - auc: 0.9812 - val_loss: 0. 3215 - val_tp: 4933923.0000 - val_fp: 100047.0000 - val_tn: 20044032.0000 - val_fn: 1136392.0000 - val_precision: 0.9801 - val_recall: 0.8128 - val_accuracy: 0.9528 - val_auc: 0.9780 Epoch 22/100 1280/1280 [==============================] - 1957s 2s/step - loss: 0.2685 - tp: 22131982.0000 - fp: 2123964.0000 - tn: 78455072.0000 - fn: 2146571.0000 - precision: 0.9124 - recall: 0.9116 - accuracy: 0.9593 - auc: 0.9811 - val_loss: 0. 2843 - val_tp: 5065657.0000 - val_fp: 196579.0000 - val_tn: 20093300.0000 - val_fn: 858874.0000 - val_precision: 0.9626 - val_recall: 0.8550 - val_accuracy: 0.9597 - val_auc: 0.9806 Epoch 23/100 1280/1280 [==============================] - 1974s 2s/step - loss: 0.2667 - tp: 21803614.0000 - fp: 2090465.0000 - tn: 78828520.0000 - fn: 2134989.0000 - precision: 0.9125 - recall: 0.9108 - accuracy: 0.9597 - auc: 0.9811 - val_loss: 0. 3048 - val_tp: 5024525.0000 - val_fp: 147570.0000 - val_tn: 20040236.0000 - val_fn: 1002069.0000 - val_precision: 0.9715 - val_recall: 0.8337 - val_accuracy: 0.9561 - val_auc: 0.9796 Epoch 24/100 1280/1280 [==============================] - 1969s 2s/step - loss: 0.2636 - tp: 21975042.0000 - fp: 2062614.0000 - tn: 78687080.0000 - fn: 2132792.0000 - precision: 0.9142 - recall: 0.9115 - accuracy: 0.9600 - auc: 0.9817 - val_loss: 0. 2889 - val_tp: 5170852.0000 - val_fp: 206375.0000 - val_tn: 19954888.0000 - val_fn: 882289.0000 - val_precision: 0.9616 - val_recall: 0.8542 - val_accuracy: 0.9585 - val_auc: 0.9811 Epoch 25/100 1280/1280 [==============================] - 1966s 2s/step - loss: 0.2654 - tp: 21864694.0000 - fp: 2102611.0000 - tn: 78755144.0000 - fn: 2135188.0000 - precision: 0.9123 - recall: 0.9110 - accuracy: 0.9596 - auc: 0.9815 - val_loss: 0. 2742 - val_tp: 5232587.0000 - val_fp: 220712.0000 - val_tn: 19933466.0000 - val_fn: 827639.0000 - val_precision: 0.9595 - val_recall: 0.8634 - val_accuracy: 0.9600 - val_auc: 0.9812 Epoch 26/100 1280/1280 [==============================] - 1968s 2s/step - loss: 0.2665 - tp: 21870048.0000 - fp: 2112301.0000 - tn: 78741728.0000 - fn: 2133559.0000 - precision: 0.9119 - recall: 0.9111 - accuracy: 0.9595 - auc: 0.9816 - val_loss: 0. 3169 - val_tp: 4986570.0000 - val_fp: 130988.0000 - val_tn: 20053542.0000 - val_fn: 1043297.0000 - val_precision: 0.9744 - val_recall: 0.8270 - val_accuracy: 0.9552 - val_auc: 0.9802 Epoch 27/100 1280/1280 [==============================] - 1952s 2s/step - loss: 0.2678 - tp: 21893164.0000 - fp: 2085167.0000 - tn: 78709264.0000 - fn: 2170034.0000 - precision: 0.9130 - recall: 0.9098 - accuracy: 0.9594 - auc: 0.9813 - val_loss: 0. 2900 - val_tp: 5054986.0000 - val_fp: 200466.0000 - val_tn: 20084900.0000 - val_fn: 874040.0000 - val_precision: 0.9619 - val_recall: 0.8526 - val_accuracy: 0.9590 - val_auc: 0.9804 Epoch 28/100 1280/1280 [==============================] - 1958s 2s/step - loss: 0.2650 - tp: 22049098.0000 - fp: 2106978.0000 - tn: 78571512.0000 - fn: 2130046.0000 - precision: 0.9128 - recall: 0.9119 - accuracy: 0.9596 - auc: 0.9817 - val_loss: 0. 2835 - val_tp: 4962208.0000 - val_fp: 180333.0000 - val_tn: 20181980.0000 - val_fn: 889878.0000 - val_precision: 0.9649 - val_recall: 0.8479 - val_accuracy: 0.9592 - val_auc: 0.9790 Epoch 29/100 1280/1280 [==============================] - 1966s 2s/step - loss: 0.2624 - tp: 21985904.0000 - fp: 2088019.0000 - tn: 78675856.0000 - fn: 2107816.0000 - precision: 0.9133 - recall: 0.9125 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0. 2769 - val_tp: 5178520.0000 - val_fp: 252398.0000 - val_tn: 20002776.0000 - val_fn: 780712.0000 - val_precision: 0.9535 - val_recall: 0.8690 - val_accuracy: 0.9606 - val_auc: 0.9797 Epoch 30/100 1280/1280 [==============================] - 1938s 2s/step - loss: 0.2616 - tp: 21893676.0000 - fp: 2096355.0000 - tn: 78765992.0000 - fn: 2101589.0000 - precision: 0.9126 - recall: 0.9124 - accuracy: 0.9600 - auc: 0.9820 - val_loss: 0. 2931 - val_tp: 5091699.0000 - val_fp: 180837.0000 - val_tn: 20020972.0000 - val_fn: 920891.0000 - val_precision: 0.9657 - val_recall: 0.8468 - val_accuracy: 0.9580 - val_auc: 0.9803 Epoch 31/100 1280/1280 [==============================] - 1932s 2s/step - loss: 0.2629 - tp: 21900190.0000 - fp: 2102823.0000 - tn: 78738360.0000 - fn: 2116254.0000 - precision: 0.9124 - recall: 0.9119 - accuracy: 0.9598 - auc: 0.9818 - val_loss: 0. 2833 - val_tp: 5068565.0000 - val_fp: 190985.0000 - val_tn: 20068752.0000 - val_fn: 886105.0000 - val_precision: 0.9637 - val_recall: 0.8512 - val_accuracy: 0.9589 - val_auc: 0.9797 Epoch 32/100 1280/1280 [==============================] - 1936s 2s/step - loss: 0.2656 - tp: 21855648.0000 - fp: 2070417.0000 - tn: 78781648.0000 - fn: 2149897.0000 - precision: 0.9135 - recall: 0.9104 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0. 3086 - val_tp: 5081221.0000 - val_fp: 145540.0000 - val_tn: 20010884.0000 - val_fn: 976755.0000 - val_precision: 0.9722 - val_recall: 0.8388 - val_accuracy: 0.9572 - val_auc: 0.9796 Epoch 33/100 1280/1280 [==============================] - 1896s 1s/step - loss: 0.2601 - tp: 21847706.0000 - fp: 2017783.0000 - tn: 78866344.0000 - fn: 2125796.0000 - precision: 0.9155 - recall: 0.9113 - accuracy: 0.9605 - auc: 0.9820 - val_loss: 0. 3225 - val_tp: 5136058.0000 - val_fp: 129931.0000 - val_tn: 19900672.0000 - val_fn: 1047735.0000 - val_precision: 0.9753 - val_recall: 0.8306 - val_accuracy: 0.9551 - val_auc: 0.9803 Epoch 34/100 1280/1280 [==============================] - 1883s 1s/step - loss: 0.2647 - tp: 21866856.0000 - fp: 2063199.0000 - tn: 78792768.0000 - fn: 2134767.0000 - precision: 0.9138 - recall: 0.9111 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0. 2877 - val_tp: 5081384.0000 - val_fp: 183837.0000 - val_tn: 20092414.0000 - val_fn: 856762.0000 - val_precision: 0.9651 - val_recall: 0.8557 - val_accuracy: 0.9603 - val_auc: 0.9819 Epoch 35/100 1280/1280 [==============================] - 1909s 1s/step - loss: 0.2644 - tp: 21793352.0000 - fp: 2043704.0000 - tn: 78858192.0000 - fn: 2162348.0000 - precision: 0.9143 - recall: 0.9097 - accuracy: 0.9599 - auc: 0.9813 - val_loss: 0. 2804 - val_tp: 5229124.0000 - val_fp: 199996.0000 - val_tn: 19913378.0000 - val_fn: 871912.0000 - val_precision: 0.9632 - val_recall: 0.8571 - val_accuracy: 0.9591 - val_auc: 0.9811 Epoch 36/100 1280/1280 [==============================] - 1926s 2s/step - loss: 0.2624 - tp: 21694132.0000 - fp: 2078119.0000 - tn: 78982728.0000 - fn: 2102522.0000 - precision: 0.9126 - recall: 0.9116 - accuracy: 0.9601 - auc: 0.9816 - val_loss: 0. 2822 - val_tp: 5091321.0000 - val_fp: 163063.0000 - val_tn: 20040276.0000 - val_fn: 919745.0000 - val_precision: 0.9690 - val_recall: 0.8470 - val_accuracy: 0.9587 - val_auc: 0.9807 Epoch 37/100 1280/1280 [==============================] - 1904s 1s/step - loss: 0.2629 - tp: 21760348.0000 - fp: 2034859.0000 - tn: 78921984.0000 - fn: 2140485.0000 - precision: 0.9145 - recall: 0.9104 - accuracy: 0.9602 - auc: 0.9816 - val_loss: 0. 3176 - val_tp: 5181435.0000 - val_fp: 158434.0000 - val_tn: 19879032.0000 - val_fn: 995505.0000 - val_precision: 0.9703 - val_recall: 0.8388 - val_accuracy: 0.9560 - val_auc: 0.9790 Epoch 38/100 1280/1280 [==============================] - 1894s 1s/step - loss: 0.2660 - tp: 21873788.0000 - fp: 2074315.0000 - tn: 78773216.0000 - fn: 2136309.0000 - precision: 0.9134 - recall: 0.9110 - accuracy: 0.9598 - auc: 0.9814 - val_loss: 0. 2881 - val_tp: 4927277.0000 - val_fp: 173887.0000 - val_tn: 20220536.0000 - val_fn: 892705.0000 - val_precision: 0.9659 - val_recall: 0.8466 - val_accuracy: 0.9593 - val_auc: 0.9789 Epoch 39/100 1280/1280 [==============================] - 1899s 1s/step - loss: 0.2622 - tp: 21838440.0000 - fp: 2067724.0000 - tn: 78836336.0000 - fn: 2115209.0000 - precision: 0.9135 - recall: 0.9117 - accuracy: 0.9601 - auc: 0.9818 - val_loss: 0. 2978 - val_tp: 4918815.0000 - val_fp: 143772.0000 - val_tn: 20190378.0000 - val_fn: 961431.0000 - val_precision: 0.9716 - val_recall: 0.8365 - val_accuracy: 0.9578 - val_auc: 0.9807 Epoch 40/100 1280/1280 [==============================] - 1888s 1s/step - loss: 0.2648 - tp: 21872584.0000 - fp: 2050789.0000 - tn: 78773392.0000 - fn: 2160846.0000 - precision: 0.9143 - recall: 0.9101 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0. 2909 - val_tp: 5274649.0000 - val_fp: 191573.0000 - val_tn: 19822700.0000 - val_fn: 925468.0000 - val_precision: 0.9650 - val_recall: 0.8507 - val_accuracy: 0.9574 - val_auc: 0.9799 Epoch 41/100 1280/1280 [==============================] - 1910s 1s/step - loss: 0.2648 - tp: 21696836.0000 - fp: 2093251.0000 - tn: 78944688.0000 - fn: 2122855.0000 - precision: 0.9120 - recall: 0.9109 - accuracy: 0.9598 - auc: 0.9814 - val_loss: 0. 2811 - val_tp: 5266549.0000 - val_fp: 194645.0000 - val_tn: 19876088.0000 - val_fn: 877120.0000 - val_precision: 0.9644 - val_recall: 0.8572 - val_accuracy: 0.9591 - val_auc: 0.9805 Epoch 42/100 1280/1280 [==============================] - 1912s 1s/step - loss: 0.2653 - tp: 21816640.0000 - fp: 2044165.0000 - tn: 78832952.0000 - fn: 2163912.0000 - precision: 0.9143 - recall: 0.9098 - accuracy: 0.9599 - auc: 0.9813 - val_loss: 0. 2983 - val_tp: 4907063.0000 - val_fp: 162204.0000 - val_tn: 20208222.0000 - val_fn: 936914.0000 - val_precision: 0.9680 - val_recall: 0.8397 - val_accuracy: 0.9581 - val_auc: 0.9799 Epoch 43/100 1280/1280 [==============================] - 1898s 1s/step - loss: 0.2646 - tp: 22222472.0000 - fp: 2064761.0000 - tn: 78412904.0000 - fn: 2157526.0000 - precision: 0.9150 - recall: 0.9115 - accuracy: 0.9597 - auc: 0.9817 - val_loss: 0. 2900 - val_tp: 5266505.0000 - val_fp: 189111.0000 - val_tn: 19834046.0000 - val_fn: 924743.0000 - val_precision: 0.9653 - val_recall: 0.8506 - val_accuracy: 0.9575 - val_auc: 0.9807 Epoch 44/100 1280/1280 [==============================] - 1937s 2s/step - loss: 0.2625 - tp: 21883992.0000 - fp: 2040394.0000 - tn: 78792808.0000 - fn: 2140429.0000 - precision: 0.9147 - recall: 0.9109 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0. 3128 - val_tp: 4872292.0000 - val_fp: 140248.0000 - val_tn: 20261372.0000 - val_fn: 940479.0000 - val_precision: 0.9720 - val_recall: 0.8382 - val_accuracy: 0.9588 - val_auc: 0.9805 Epoch 45/100 1280/1280 [==============================] - 1931s 2s/step - loss: 0.2641 - tp: 21804156.0000 - fp: 2047906.0000 - tn: 78877472.0000 - fn: 2128092.0000 - precision: 0.9141 - recall: 0.9111 - accuracy: 0.9602 - auc: 0.9814 - val_loss: 0. 2983 - val_tp: 5262537.0000 - val_fp: 177583.0000 - val_tn: 19827930.0000 - val_fn: 946346.0000 - val_precision: 0.9674 - val_recall: 0.8476 - val_accuracy: 0.9571 - val_auc: 0.9792 Epoch 46/100 1280/1280 [==============================] - 1941s 2s/step - loss: 0.2640 - tp: 21923374.0000 - fp: 2080069.0000 - tn: 78717264.0000 - fn: 2136918.0000 - precision: 0.9133 - recall: 0.9112 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0. 2822 - val_tp: 5087710.0000 - val_fp: 182249.0000 - val_tn: 20063030.0000 - val_fn: 881411.0000 - val_precision: 0.9654 - val_recall: 0.8523 - val_accuracy: 0.9594 - val_auc: 0.9803 Epoch 47/100 1280/1280 [==============================] - 1944s 2s/step - loss: 0.2620 - tp: 21961636.0000 - fp: 2068701.0000 - tn: 78701112.0000 - fn: 2126225.0000 - precision: 0.9139 - recall: 0.9117 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0. 2982 - val_tp: 5201830.0000 - val_fp: 170225.0000 - val_tn: 19876590.0000 - val_fn: 965741.0000 - val_precision: 0.9683 - val_recall: 0.8434 - val_accuracy: 0.9567 - val_auc: 0.9799 Epoch 48/100 1280/1280 [==============================] - 1948s 2s/step - loss: 0.2631 - tp: 21971392.0000 - fp: 2056324.0000 - tn: 78695536.0000 - fn: 2134300.0000 - precision: 0.9144 - recall: 0.9115 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0. 2906 - val_tp: 5083102.0000 - val_fp: 170364.0000 - val_tn: 20046178.0000 - val_fn: 914747.0000 - val_precision: 0.9676 - val_recall: 0.8475 - val_accuracy: 0.9586 - val_auc: 0.9792 Epoch 49/100 1280/1280 [==============================] - 1937s 2s/step - loss: 0.2629 - tp: 21911328.0000 - fp: 2071634.0000 - tn: 78755200.0000 - fn: 2119426.0000 - precision: 0.9136 - recall: 0.9118 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0. 2839 - val_tp: 5050322.0000 - val_fp: 183661.0000 - val_tn: 20111232.0000 - val_fn: 869193.0000 - val_precision: 0.9649 - val_recall: 0.8532 - val_accuracy: 0.9598 - val_auc: 0.9806 Epoch 50/100 1280/1280 [==============================] - 1964s 2s/step - loss: 0.2634 - tp: 21943356.0000 - fp: 2042601.0000 - tn: 78728936.0000 - fn: 2142696.0000 - precision: 0.9148 - recall: 0.9110 - accuracy: 0.9601 - auc: 0.9816 - val_loss: 0. 2906 - val_tp: 5176414.0000 - val_fp: 195927.0000 - val_tn: 19950280.0000 - val_fn: 891781.0000 - val_precision: 0.9635 - val_recall: 0.8530 - val_accuracy: 0.9585 - val_auc: 0.9800 Epoch 51/100 1280/1280 [==============================] - 1949s 2s/step - loss: 0.2619 - tp: 21905776.0000 - fp: 2054187.0000 - tn: 78790400.0000 - fn: 2107240.0000 - precision: 0.9143 - recall: 0.9122 - accuracy: 0.9603 - auc: 0.9818 - val_loss: 0. 2845 - val_tp: 5128164.0000 - val_fp: 168438.0000 - val_tn: 19986128.0000 - val_fn: 931667.0000 - val_precision: 0.9682 - val_recall: 0.8463 - val_accuracy: 0.9580 - val_auc: 0.9810 Epoch 52/100 1280/1280 [==============================] - 1959s 2s/step - loss: 0.2638 - tp: 21870924.0000 - fp: 2087711.0000 - tn: 78782424.0000 - fn: 2116607.0000 - precision: 0.9129 - recall: 0.9118 - accuracy: 0.9599 - auc: 0.9817 - val_loss: 0. 3062 - val_tp: 5273162.0000 - val_fp: 169306.0000 - val_tn: 19832368.0000 - val_fn: 939566.0000 - val_precision: 0.9689 - val_recall: 0.8488 - val_accuracy: 0.9577 - val_auc: 0.9815 Epoch 53/100 1280/1280 [==============================] - 1958s 2s/step - loss: 0.2648 - tp: 21947556.0000 - fp: 2065779.0000 - tn: 78702432.0000 - fn: 2141837.0000 - precision: 0.9140 - recall: 0.9111 - accuracy: 0.9599 - auc: 0.9816 - val_loss: 0. 2945 - val_tp: 5204859.0000 - val_fp: 153095.0000 - val_tn: 19890892.0000 - val_fn: 965553.0000 - val_precision: 0.9714 - val_recall: 0.8435 - val_accuracy: 0.9573 - val_auc: 0.9800 Epoch 54/100 1280/1280 [==============================] - 1937s 2s/step - loss: 0.2635 - tp: 21905812.0000 - fp: 2049643.0000 - tn: 78761584.0000 - fn: 2140551.0000 - precision: 0.9144 - recall: 0.9110 - accuracy: 0.9600 - auc: 0.9816 - val_loss: 0. 3064 - val_tp: 5062076.0000 - val_fp: 149992.0000 - val_tn: 20054216.0000 - val_fn: 948115.0000 - val_precision: 0.9712 - val_recall: 0.8422 - val_accuracy: 0.9581 - val_auc: 0.9815 Epoch 55/100 1280/1280 [==============================] - 1958s 2s/step - loss: 0.2653 - tp: 22046804.0000 - fp: 2122362.0000 - tn: 78569416.0000 - fn: 2119021.0000 - precision: 0.9122 - recall: 0.9123 - accuracy: 0.9596 - auc: 0.9818 - val_loss: 0. 2913 - val_tp: 5030327.0000 - val_fp: 162722.0000 - val_tn: 20130282.0000 - val_fn: 891077.0000 - val_precision: 0.9687 - val_recall: 0.8495 - val_accuracy: 0.9598 - val_auc: 0.9812 Epoch 56/100 1280/1280 [==============================] - 1946s 2s/step - loss: 0.2644 - tp: 21869356.0000 - fp: 2060246.0000 - tn: 78783952.0000 - fn: 2144064.0000 - precision: 0.9139 - recall: 0.9107 - accuracy: 0.9599 - auc: 0.9814 - val_loss: 0. 2816 - val_tp: 5105422.0000 - val_fp: 167571.0000 - val_tn: 20081292.0000 - val_fn: 860105.0000 - val_precision: 0.9682 - val_recall: 0.8558 - val_accuracy: 0.9608 - val_auc: 0.9805 Epoch 57/100 1280/1280 [==============================] - 1921s 2s/step - loss: 0.2642 - tp: 22001298.0000 - fp: 2070341.0000 - tn: 78641616.0000 - fn: 2144355.0000 - precision: 0.9140 - recall: 0.9112 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0. 3304 - val_tp: 4955272.0000 - val_fp: 133108.0000 - val_tn: 20129192.0000 - val_fn: 996829.0000 - val_precision: 0.9738 - val_recall: 0.8325 - val_accuracy: 0.9569 - val_auc: 0.9802 Epoch 58/100 1280/1280 [==============================] - 1919s 1s/step - loss: 0.2644 - tp: 21756178.0000 - fp: 2097986.0000 - tn: 78861704.0000 - fn: 2141728.0000 - precision: 0.9120 - recall: 0.9104 - accuracy: 0.9596 - auc: 0.9815 - val_loss: 0. 2800 - val_tp: 5173984.0000 - val_fp: 172737.0000 - val_tn: 19995672.0000 - val_fn: 872006.0000 - val_precision: 0.9677 - val_recall: 0.8558 - val_accuracy: 0.9601 - val_auc: 0.9811 Epoch 59/100 1280/1280 [==============================] - 1937s 2s/step - loss: 0.2639 - tp: 21965856.0000 - fp: 2104414.0000 - tn: 78670312.0000 - fn: 2117059.0000 - precision: 0.9126 - recall: 0.9121 - accuracy: 0.9597 - auc: 0.9817 - val_loss: 0. 2969 - val_tp: 5288252.0000 - val_fp: 187540.0000 - val_tn: 19831520.0000 - val_fn: 907092.0000 - val_precision: 0.9658 - val_recall: 0.8536 - val_accuracy: 0.9582 - val_auc: 0.9792 Epoch 60/100 1280/1280 [==============================] - 1942s 2s/step - loss: 0.2652 - tp: 21972316.0000 - fp: 2103459.0000 - tn: 78666280.0000 - fn: 2115554.0000 - precision: 0.9126 - recall: 0.9122 - accuracy: 0.9598 - auc: 0.9817 - val_loss: 0. 2818 - val_tp: 5166245.0000 - val_fp: 190846.0000 - val_tn: 19971494.0000 - val_fn: 885813.0000 - val_precision: 0.9644 - val_recall: 0.8536 - val_accuracy: 0.9589 - val_auc: 0.9810 Epoch 61/100 1280/1280 [==============================] - 1925s 2s/step - loss: 0.2651 - tp: 21878652.0000 - fp: 2066501.0000 - tn: 78782176.0000 - fn: 2130193.0000 - precision: 0.9137 - recall: 0.9113 - accuracy: 0.9600 - auc: 0.9814 - val_loss: 0. 2793 - val_tp: 5085499.0000 - val_fp: 194718.0000 - val_tn: 20060124.0000 - val_fn: 874059.0000 - val_precision: 0.9631 - val_recall: 0.8533 - val_accuracy: 0.9592 - val_auc: 0.9805 Epoch 62/100 1280/1280 [==============================] - 1922s 2s/step - loss: 0.2645 - tp: 21871498.0000 - fp: 2095977.0000 - tn: 78758960.0000 - fn: 2131121.0000 - precision: 0.9125 - recall: 0.9112 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0. 3215 - val_tp: 5111082.0000 - val_fp: 129336.0000 - val_tn: 19952116.0000 - val_fn: 1021864.0000 - val_precision: 0.9753 - val_recall: 0.8334 - val_accuracy: 0.9561 - val_auc: 0.9808 Epoch 63/100 1280/1280 [==============================] - 1938s 2s/step - loss: 0.2653 - tp: 21667542.0000 - fp: 2040165.0000 - tn: 78980168.0000 - fn: 2169717.0000 - precision: 0.9139 - recall: 0.9090 - accuracy: 0.9599 - auc: 0.9811 - val_loss: 0. 2935 - val_tp: 5105651.0000 - val_fp: 157582.0000 - val_tn: 19997984.0000 - val_fn: 953187.0000 - val_precision: 0.9701 - val_recall: 0.8427 - val_accuracy: 0.9576 - val_auc: 0.9795 Epoch 64/100 1280/1280 [==============================] - 1922s 2s/step - loss: 0.2621 - tp: 21566260.0000 - fp: 2041653.0000 - tn: 79126792.0000 - fn: 2122825.0000 - precision: 0.9135 - recall: 0.9104 - accuracy: 0.9603 - auc: 0.9816 - val_loss: 0. 2868 - val_tp: 5295960.0000 - val_fp: 181316.0000 - val_tn: 19817780.0000 - val_fn: 919349.0000 - val_precision: 0.9669 - val_recall: 0.8521 - val_accuracy: 0.9580 - val_auc: 0.9802 Epoch 65/100 1280/1280 [==============================] - 1930s 2s/step - loss: 0.2618 - tp: 21803380.0000 - fp: 2046884.0000 - tn: 78875232.0000 - fn: 2132114.0000 - precision: 0.9142 - recall: 0.9109 - accuracy: 0.9601 - auc: 0.9818 - val_loss: 0. 2655 - val_tp: 4998172.0000 - val_fp: 220002.0000 - val_tn: 20208396.0000 - val_fn: 787827.0000 - val_precision: 0.9578 - val_recall: 0.8638 - val_accuracy: 0.9616 - val_auc: 0.9810 Epoch 66/100 1280/1280 [==============================] - 1940s 2s/step - loss: 0.2621 - tp: 21715956.0000 - fp: 2060856.0000 - tn: 78976016.0000 - fn: 2104835.0000 - precision: 0.9133 - recall: 0.9116 - accuracy: 0.9603 - auc: 0.9817 - val_loss: 0. 2934 - val_tp: 4858098.0000 - val_fp: 162111.0000 - val_tn: 20270832.0000 - val_fn: 923350.0000 - val_precision: 0.9677 - val_recall: 0.8403 - val_accuracy: 0.9586 - val_auc: 0.9797 Epoch 67/100 1280/1280 [==============================] - 1936s 2s/step - loss: 0.2641 - tp: 22101508.0000 - fp: 2096796.0000 - tn: 78514056.0000 - fn: 2145167.0000 - precision: 0.9133 - recall: 0.9115 - accuracy: 0.9595 - auc: 0.9819 - val_loss: 0. 2935 - val_tp: 5311098.0000 - val_fp: 189758.0000 - val_tn: 19779840.0000 - val_fn: 933704.0000 - val_precision: 0.9655 - val_recall: 0.8505 - val_accuracy: 0.9571 - val_auc: 0.9804 Epoch 68/100 1280/1280 [==============================] - 1929s 2s/step - loss: 0.2639 - tp: 21973892.0000 - fp: 2058296.0000 - tn: 78690816.0000 - fn: 2134585.0000 - precision: 0.9144 - recall: 0.9115 - accuracy: 0.9600 - auc: 0.9816 - val_loss: 0. 3021 - val_tp: 5393303.0000 - val_fp: 200762.0000 - val_tn: 19679066.0000 - val_fn: 941275.0000 - val_precision: 0.9641 - val_recall: 0.8514 - val_accuracy: 0.9564 - val_auc: 0.9795 Epoch 69/100 1280/1280 [==============================] - 1928s 2s/step - loss: 0.2632 - tp: 21797492.0000 - fp: 2054212.0000 - tn: 78874840.0000 - fn: 2131042.0000 - precision: 0.9139 - recall: 0.9109 - accuracy: 0.9601 - auc: 0.9816 - val_loss: 0. 2970 - val_tp: 5102175.0000 - val_fp: 191344.0000 - val_tn: 19995848.0000 - val_fn: 925037.0000 - val_precision: 0.9639 - val_recall: 0.8465 - val_accuracy: 0.9574 - val_auc: 0.9797 Epoch 70/100 1280/1280 [==============================] - 1936s 2s/step - loss: 0.2629 - tp: 21925510.0000 - fp: 2063262.0000 - tn: 78733408.0000 - fn: 2135445.0000 - precision: 0.9140 - recall: 0.9112 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0. 2835 - val_tp: 5162856.0000 - val_fp: 228043.0000 - val_tn: 19969488.0000 - val_fn: 854014.0000 - val_precision: 0.9577 - val_recall: 0.8581 - val_accuracy: 0.9587 - val_auc: 0.9802 Epoch 71/100 1280/1280 [==============================] - 1930s 2s/step - loss: 0.2606 - tp: 21986884.0000 - fp: 2118880.0000 - tn: 78684536.0000 - fn: 2067241.0000 - precision: 0.9121 - recall: 0.9141 - accuracy: 0.9601 - auc: 0.9822 - val_loss: 0. 2809 - val_tp: 5176839.0000 - val_fp: 235191.0000 - val_tn: 19977128.0000 - val_fn: 825244.0000 - val_precision: 0.9565 - val_recall: 0.8625 - val_accuracy: 0.9595 - val_auc: 0.9812 Epoch 72/100 1280/1280 [==============================] - 1952s 2s/step - loss: 0.2618 - tp: 22038310.0000 - fp: 2056830.0000 - tn: 78655040.0000 - fn: 2107382.0000 - precision: 0.9146 - recall: 0.9127 - accuracy: 0.9603 - auc: 0.9820 - val_loss: 0. 2895 - val_tp: 5367415.0000 - val_fp: 210844.0000 - val_tn: 19760872.0000 - val_fn: 875261.0000 - val_precision: 0.9622 - val_recall: 0.8598 - val_accuracy: 0.9586 - val_auc: 0.9801 Epoch 73/100 1280/1280 [==============================] - 1931s 2s/step - loss: 0.2642 - tp: 22040244.0000 - fp: 2079148.0000 - tn: 78597176.0000 - fn: 2140976.0000 - precision: 0.9138 - recall: 0.9115 - accuracy: 0.9598 - auc: 0.9817 - val_loss: 0. 3093 - val_tp: 5018548.0000 - val_fp: 149682.0000 - val_tn: 20096504.0000 - val_fn: 949668.0000 - val_precision: 0.9710 - val_recall: 0.8409 - val_accuracy: 0.9581 - val_auc: 0.9804 Epoch 74/100 1280/1280 [==============================] - 1891s 1s/step - loss: 0.2648 - tp: 21823536.0000 - fp: 2026553.0000 - tn: 78842960.0000 - fn: 2164550.0000 - precision: 0.9150 - recall: 0.9098 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0. 2941 - val_tp: 5138005.0000 - val_fp: 183811.0000 - val_tn: 19970134.0000 - val_fn: 922446.0000 - val_precision: 0.9655 - val_recall: 0.8478 - val_accuracy: 0.9578 - val_auc: 0.9795 Epoch 75/100 1280/1280 [==============================] - 1896s 1s/step - loss: 0.2599 - tp: 21797956.0000 - fp: 2051529.0000 - tn: 78902288.0000 - fn: 2105822.0000 - precision: 0.9140 - recall: 0.9119 - accuracy: 0.9604 - auc: 0.9820 - val_loss: 0. 2866 - val_tp: 5237317.0000 - val_fp: 179478.0000 - val_tn: 19883768.0000 - val_fn: 913840.0000 - val_precision: 0.9669 - val_recall: 0.8514 - val_accuracy: 0.9583 - val_auc: 0.9806 Epoch 76/100 1280/1280 [==============================] - 1920s 2s/step - loss: 0.2644 - tp: 21883500.0000 - fp: 2043967.0000 - tn: 78765848.0000 - fn: 2164260.0000 - precision: 0.9146 - recall: 0.9100 - accuracy: 0.9599 - auc: 0.9815 - val_loss: 0. 2969 - val_tp: 4856771.0000 - val_fp: 158534.0000 - val_tn: 20318780.0000 - val_fn: 880306.0000 - val_precision: 0.9684 - val_recall: 0.8466 - val_accuracy: 0.9604 - val_auc: 0.9810 Epoch 77/100 1280/1280 [==============================] - 1950s 2s/step - loss: 0.2649 - tp: 21977528.0000 - fp: 2098101.0000 - tn: 78653504.0000 - fn: 2128494.0000 - precision: 0.9129 - recall: 0.9117 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0. 2794 - val_tp: 5248088.0000 - val_fp: 201804.0000 - val_tn: 19923706.0000 - val_fn: 840800.0000 - val_precision: 0.9630 - val_recall: 0.8619 - val_accuracy: 0.9602 - val_auc: 0.9806 Epoch 78/100 1280/1280 [==============================] - 1951s 2s/step - loss: 0.2630 - tp: 21619006.0000 - fp: 2054910.0000 - tn: 79065992.0000 - fn: 2117649.0000 - precision: 0.9132 - recall: 0.9108 - accuracy: 0.9602 - auc: 0.9815 - val_loss: 0. 2924 - val_tp: 5121101.0000 - val_fp: 170363.0000 - val_tn: 20022854.0000 - val_fn: 900073.0000 - val_precision: 0.9678 - val_recall: 0.8505 - val_accuracy: 0.9592 - val_auc: 0.9811 Epoch 79/100 1280/1280 [==============================] - 1956s 2s/step - loss: 0.2646 - tp: 21744384.0000 - fp: 2060624.0000 - tn: 78899240.0000 - fn: 2153398.0000 - precision: 0.9134 - recall: 0.9099 - accuracy: 0.9598 - auc: 0.9814 - val_loss: 0. 2982 - val_tp: 4993843.0000 - val_fp: 185702.0000 - val_tn: 20120108.0000 - val_fn: 914746.0000 - val_precision: 0.9641 - val_recall: 0.8452 - val_accuracy: 0.9580 - val_auc: 0.9797 Epoch 80/100 1280/1280 [==============================] - 1953s 2s/step - loss: 0.2669 - tp: 22055420.0000 - fp: 2135651.0000 - tn: 78526352.0000 - fn: 2140199.0000 - precision: 0.9117 - recall: 0.9115 - accuracy: 0.9592 - auc: 0.9815 - val_loss: 0. 2787 - val_tp: 4984180.0000 - val_fp: 197102.0000 - val_tn: 20178072.0000 - val_fn: 855046.0000 - val_precision: 0.9620 - val_recall: 0.8536 - val_accuracy: 0.9599 - val_auc: 0.9800 Epoch 81/100 1280/1280 [==============================] - 1956s 2s/step - loss: 0.2622 - tp: 21705808.0000 - fp: 2077811.0000 - tn: 78966440.0000 - fn: 2107549.0000 - precision: 0.9126 - recall: 0.9115 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0. 2877 - val_tp: 4988818.0000 - val_fp: 163659.0000 - val_tn: 20154334.0000 - val_fn: 907589.0000 - val_precision: 0.9682 - val_recall: 0.8461 - val_accuracy: 0.9591 - val_auc: 0.9802 Epoch 82/100 1280/1280 [==============================] - 1945s 2s/step - loss: 0.2660 - tp: 22038244.0000 - fp: 2055404.0000 - tn: 78596896.0000 - fn: 2167084.0000 - precision: 0.9147 - recall: 0.9105 - accuracy: 0.9597 - auc: 0.9813 - val_loss: 0. 2860 - val_tp: 4915941.0000 - val_fp: 173380.0000 - val_tn: 20253348.0000 - val_fn: 871720.0000 - val_precision: 0.9659 - val_recall: 0.8494 - val_accuracy: 0.9601 - val_auc: 0.9808 Epoch 83/100 1280/1280 [==============================] - 1960s 2s/step - loss: 0.2620 - tp: 21828352.0000 - fp: 2096385.0000 - tn: 78818296.0000 - fn: 2114614.0000 - precision: 0.9124 - recall: 0.9117 - accuracy: 0.9598 - auc: 0.9819 - val_loss: 0. 2882 - val_tp: 5181476.0000 - val_fp: 172365.0000 - val_tn: 19932176.0000 - val_fn: 928376.0000 - val_precision: 0.9678 - val_recall: 0.8481 - val_accuracy: 0.9580 - val_auc: 0.9790 Epoch 84/100 1280/1280 [==============================] - 1956s 2s/step - loss: 0.2650 - tp: 21972974.0000 - fp: 2043197.0000 - tn: 78688096.0000 - fn: 2153339.0000 - precision: 0.9149 - recall: 0.9107 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0. 2884 - val_tp: 5125310.0000 - val_fp: 199354.0000 - val_tn: 20006488.0000 - val_fn: 883247.0000 - val_precision: 0.9626 - val_recall: 0.8530 - val_accuracy: 0.9587 - val_auc: 0.9792 Epoch 85/100 1280/1280 [==============================] - 1932s 2s/step - loss: 0.2665 - tp: 21841062.0000 - fp: 2087044.0000 - tn: 78779464.0000 - fn: 2150003.0000 - precision: 0.9128 - recall: 0.9104 - accuracy: 0.9596 - auc: 0.9813 - val_loss: 0. 2990 - val_tp: 4966995.0000 - val_fp: 155834.0000 - val_tn: 20153514.0000 - val_fn: 938068.0000 - val_precision: 0.9696 - val_recall: 0.8411 - val_accuracy: 0.9583 - val_auc: 0.9797 Epoch 86/100 1280/1280 [==============================] - 1950s 2s/step - loss: 0.2613 - tp: 21989296.0000 - fp: 2028370.0000 - tn: 78716320.0000 - fn: 2123700.0000 - precision: 0.9155 - recall: 0.9119 - accuracy: 0.9604 - auc: 0.9819 - val_loss: 0. 3127 - val_tp: 5106508.0000 - val_fp: 142899.0000 - val_tn: 19982462.0000 - val_fn: 982530.0000 - val_precision: 0.9728 - val_recall: 0.8386 - val_accuracy: 0.9571 - val_auc: 0.9803 Epoch 87/100 1280/1280 [==============================] - 1949s 2s/step - loss: 0.2652 - tp: 21800292.0000 - fp: 2065815.0000 - tn: 78849960.0000 - fn: 2141540.0000 - precision: 0.9134 - recall: 0.9106 - accuracy: 0.9599 - auc: 0.9813 - val_loss: 0. 2862 - val_tp: 5040894.0000 - val_fp: 182758.0000 - val_tn: 20103008.0000 - val_fn: 887735.0000 - val_precision: 0.9650 - val_recall: 0.8503 - val_accuracy: 0.9592 - val_auc: 0.9799 Epoch 88/100 1280/1280 [==============================] - 1920s 2s/step - loss: 0.2629 - tp: 21758816.0000 - fp: 2035708.0000 - tn: 78916872.0000 - fn: 2146157.0000 - precision: 0.9144 - recall: 0.9102 - accuracy: 0.9601 - auc: 0.9815 - val_loss: 0. 2883 - val_tp: 5067502.0000 - val_fp: 156913.0000 - val_tn: 20054418.0000 - val_fn: 935565.0000 - val_precision: 0.9700 - val_recall: 0.8442 - val_accuracy: 0.9583 - val_auc: 0.9804 Epoch 89/100 1280/1280 [==============================] - 1902s 1s/step - loss: 0.2661 - tp: 21907896.0000 - fp: 2077818.0000 - tn: 78708656.0000 - fn: 2163245.0000 - precision: 0.9134 - recall: 0.9101 - accuracy: 0.9596 - auc: 0.9814 - val_loss: 0. 3095 - val_tp: 5120703.0000 - val_fp: 137657.0000 - val_tn: 19965744.0000 - val_fn: 990302.0000 - val_precision: 0.9738 - val_recall: 0.8379 - val_accuracy: 0.9570 - val_auc: 0.9806 Epoch 90/100 1280/1280 [==============================] - 1920s 1s/step - loss: 0.2652 - tp: 21984496.0000 - fp: 2114012.0000 - tn: 78621976.0000 - fn: 2137152.0000 - precision: 0.9123 - recall: 0.9114 - accuracy: 0.9595 - auc: 0.9815 - val_loss: 0. 2829 - val_tp: 5208131.0000 - val_fp: 193268.0000 - val_tn: 19946136.0000 - val_fn: 866871.0000 - val_precision: 0.9642 - val_recall: 0.8573 - val_accuracy: 0.9596 - val_auc: 0.9809 Epoch 91/100 1280/1280 [==============================] - 1917s 1s/step - loss: 0.2631 - tp: 21814944.0000 - fp: 2037460.0000 - tn: 78857032.0000 - fn: 2148231.0000 - precision: 0.9146 - recall: 0.9104 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0. 2921 - val_tp: 4988873.0000 - val_fp: 154974.0000 - val_tn: 20127408.0000 - val_fn: 943141.0000 - val_precision: 0.9699 - val_recall: 0.8410 - val_accuracy: 0.9581 - val_auc: 0.9797 Epoch 92/100 1280/1280 [==============================] - 1920s 1s/step - loss: 0.2641 - tp: 21926682.0000 - fp: 2066839.0000 - tn: 78722320.0000 - fn: 2141742.0000 - precision: 0.9139 - recall: 0.9110 - accuracy: 0.9599 - auc: 0.9816 - val_loss: 0. 3016 - val_tp: 5074157.0000 - val_fp: 161845.0000 - val_tn: 20024836.0000 - val_fn: 953570.0000 - val_precision: 0.9691 - val_recall: 0.8418 - val_accuracy: 0.9575 - val_auc: 0.9794 Epoch 93/100 1280/1280 [==============================] - 1915s 1s/step - loss: 0.2639 - tp: 21858124.0000 - fp: 2086217.0000 - tn: 78785032.0000 - fn: 2128289.0000 - precision: 0.9129 - recall: 0.9113 - accuracy: 0.9598 - auc: 0.9817 - val_loss: 0. 3349 - val_tp: 5110189.0000 - val_fp: 138626.0000 - val_tn: 19910190.0000 - val_fn: 1055394.0000 - val_precision: 0.9736 - val_recall: 0.8288 - val_accuracy: 0.9545 - val_auc: 0.9781 Epoch 94/100 1280/1280 [==============================] - 1932s 2s/step - loss: 0.2661 - tp: 21972080.0000 - fp: 2067147.0000 - tn: 78645520.0000 - fn: 2172891.0000 - precision: 0.9140 - recall: 0.9100 - accuracy: 0.9596 - auc: 0.9814 - val_loss: 0. 3010 - val_tp: 5379887.0000 - val_fp: 190356.0000 - val_tn: 19720614.0000 - val_fn: 923543.0000 - val_precision: 0.9658 - val_recall: 0.8535 - val_accuracy: 0.9575 - val_auc: 0.9800 Epoch 95/100 1280/1280 [==============================] - 1925s 2s/step - loss: 0.2641 - tp: 21773760.0000 - fp: 2069301.0000 - tn: 78892664.0000 - fn: 2121866.0000 - precision: 0.9132 - recall: 0.9112 - accuracy: 0.9600 - auc: 0.9814 - val_loss: 0. 2834 - val_tp: 5148394.0000 - val_fp: 196890.0000 - val_tn: 20004664.0000 - val_fn: 864450.0000 - val_precision: 0.9632 - val_recall: 0.8562 - val_accuracy: 0.9595 - val_auc: 0.9803 Epoch 96/100 1280/1280 [==============================] - 1924s 2s/step - loss: 0.2620 - tp: 21883636.0000 - fp: 2083221.0000 - tn: 78785256.0000 - fn: 2105458.0000 - precision: 0.9131 - recall: 0.9122 - accuracy: 0.9601 - auc: 0.9819 - val_loss: 0. 3238 - val_tp: 5063114.0000 - val_fp: 150917.0000 - val_tn: 20012972.0000 - val_fn: 987390.0000 - val_precision: 0.9711 - val_recall: 0.8368 - val_accuracy: 0.9566 - val_auc: 0.9792 Epoch 97/100 1280/1280 [==============================] - 1926s 2s/step - loss: 0.2629 - tp: 21759104.0000 - fp: 2051085.0000 - tn: 78912920.0000 - fn: 2134447.0000 - precision: 0.9139 - recall: 0.9107 - accuracy: 0.9601 - auc: 0.9815 - val_loss: 0. 2802 - val_tp: 5228317.0000 - val_fp: 190075.0000 - val_tn: 19926608.0000 - val_fn: 869396.0000 - val_precision: 0.9649 - val_recall: 0.8574 - val_accuracy: 0.9596 - val_auc: 0.9809 Epoch 98/100 1280/1280 [==============================] - 1929s 2s/step - loss: 0.2630 - tp: 21930218.0000 - fp: 2067388.0000 - tn: 78741992.0000 - fn: 2117977.0000 - precision: 0.9139 - recall: 0.9119 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0. 3097 - val_tp: 4899275.0000 - val_fp: 135668.0000 - val_tn: 20163436.0000 - val_fn: 1016018.0000 - val_precision: 0.9731 - val_recall: 0.8282 - val_accuracy: 0.9561 - val_auc: 0.9789 Epoch 99/100 1280/1280 [==============================] - 1957s 2s/step - loss: 0.2648 - tp: 21993828.0000 - fp: 2052826.0000 - tn: 78663616.0000 - fn: 2147339.0000 - precision: 0.9146 - recall: 0.9111 - accuracy: 0.9599 - auc: 0.9816 - val_loss: 0. 2889 - val_tp: 5106607.0000 - val_fp: 161956.0000 - val_tn: 20025448.0000 - val_fn: 920399.0000 - val_precision: 0.9693 - val_recall: 0.8473 - val_accuracy: 0.9587 - val_auc: 0.9814 Epoch 100/100 1280/1280 [==============================] - 1941s 2s/step - loss: 0.2598 - tp: 21924942.0000 - fp: 2029354.0000 - tn: 78777592.0000 - fn: 2125714.0000 - precision: 0.9153 - recall: 0.9116 - accuracy: 0.9604 - auc: 0.9819 - val_loss: 0. 3001 - val_tp: 5185959.0000 - val_fp: 162584.0000 - val_tn: 19884838.0000 - val_fn: 981025.0000 - val_precision: 0.9696 - val_recall: 0.8409 - val_accuracy: 0.9564 - val_auc: 0.9800 400/400 [==============================] - 121s 302ms/step - loss: 0.3130 - tp: 6560497.0000 - fp: 267592.0000 - tn: 24822684.0000 - fn: 1117239.0000 - precision: 0.9608 - recall: 0.8545 - accuracy: 0.9577 - auc: 0.9801 2020/06/14 20:28:16 INFO mlflow.projects: === Run (ID '037e1d9e4ad74784974f4aaac11138cc') succeeded ===
2.1.11 read out logs experiment 2
2.1.11.1 read out mlflow logs using CLI
conda activate tensorflow_env cd Programme/drmed-git export MLFLOW_EXPERIMENT_NAME=exp-310520-unet export MLFLOW_TRACKING_URI=file:./data/mlruns
mlflow experiments list
Experiment Id Name Artifact Location
--------------- --------------- --------------------
0 exp-310520-unet file:./data/mlruns/0
1 exp-devtest file:./data/mlruns/1
mlflow runs list --experiment-id 0
Date Name ID ------------------------ ------ -------------------------------- 2020-06-12 14:20:30 CEST 037e1d9e4ad74784974f4aaac11138cc 2020-05-31 21:39:51 CEST 1aefda1366f04f5da5d1fc2241ad9208
export EXP2=037e1d9e4ad74784974f4aaac11138cc
I have accidentally run the model with the wrong experiment ID. I'll check if I
can change the experiment. I will do it later with
mlflow.tracking.MlflowClient().rename_experiment()
mlflow artifacts list -r $EXP2 mlflow artifacts list -r $EXP2 -a model mlflow artifacts list -r $EXP2 -a model_summary.txt mlflow artifacts list -r $EXP2 -a tensorboard_logs/train/
(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2
[{
"path": "model",
"is_dir": true
}, {
"path": "model_summary.txt",
"is_dir": false,
"file_size": "10895"
}, {
"path": "tensorboard_logs",
"is_dir": true
}]
(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a model
[{
"path": "model/MLmodel",
"is_dir": false,
"file_size": "317"
}, {
"path": "model/conda.yaml",
"is_dir": false,
"file_size": "125"
}, {
"path": "model/data",
"is_dir": true
}]
(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a model_summary.txt
[]
(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a tensorboard_logs/train/
[{
"path": "tensorboard_logs/train/events.out.tfevents.1591964813.node151.309868.9202.v2",
"is_dir": false,
"file_size": "20270485"
}, {
"path": "tensorboard_logs/train/events.out.tfevents.1591964827.node151.profile-empty",
"is_dir": false,
"file_size": "40"
}, {
"path": "tensorboard_logs/train/plugins",
"is_dir": true
}]
mlflow artifacts download -r $EXP2 mlflow artifacts download -r $EXP2 -a model mlflow artifacts download -r $EXP2 -a model_summary.txt mlflow artifacts download -r $EXP2 -a tensorboard_logs
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts /home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/model /home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/model_summary.txt /home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/tensorboard_logs
mlflow runs describe --run-id $EXP2
{
"info": {
"artifact_uri": "file:./data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts",
"end_time": 1592159296429,
"experiment_id": "0",
"lifecycle_stage": "active",
"run_id": "037e1d9e4ad74784974f4aaac11138cc",
"run_uuid": "037e1d9e4ad74784974f4aaac11138cc",
"start_time": 1591964430233,
"status": "FINISHED",
"user_id": "ye53nis"
},
"data": {
"metrics": {
"learning rate": 1e-14,
"val_loss": 0.3001052141189575,
"precision": 0.9152822494506836,
"fp": 2029354.0,
"loss": 0.2597578763961792,
"val_tp": 5185959.0,
"recall": 0.9116151332855225,
"accuracy": 0.9603750705718994,
"val_precision": 0.9696021676063538,
"val_recall": 0.8409230709075928,
"lr": 1e-14,
"fn": 2125714.0,
"auc": 0.9819398522377014,
"val_fp": 162584.0,
"val_accuracy": 0.9563749432563782,
"val_tn": 19884838.0,
"val_auc": 0.9799740314483643,
"val_fn": 981025.0,
"tn": 78777592.0,
"tp": 21924942.0
},
"params": {
"opt_beta_1": "0.9",
"validation_steps": "320",
"opt_learning_rate": "0.001",
"fluotracify_path": "/beegfs/ye53nis/drmed-git/src/",
"opt_amsgrad": "False",
"frac_val": "0.2",
"batch_size": "5",
"epochs": "100",
"opt_decay": "0.0",
"steps_per_epoch": "1280",
"length_delimiter": "16384",
"opt_name": "Adam",
"opt_beta_2": "0.999",
"csv_path": "/beegfs/ye53nis/saves/firstartefact_Sep2019",
"learning_rate": "None",
"opt_epsilon": "1e-07"
},
"tags": {
"mlflow.source.git.repoURL": "https://github.com/aseltmann/fluotracify",
"mlflow.source.type": "PROJECT",
"mlflow.source.name": "file:///beegfs/ye53nis/drmed-git",
"mlflow.user": "ye53nis",
"mlflow.source.git.commit": "09c1d2a1bc083695026b46c74ea175c9168bb5f2",
"mlflow.gitRepoURL": "https://github.com/aseltmann/fluotracify",
"mlflow.project.backend": "local",
"mlflow.project.env": "conda",
"mlflow.project.entryPoint": "main",
"mlflow.log-model.history": "[{
\"run_id\": \"037e1d9e4ad74784974f4aaac11138cc\",
\"artifact_path\": \"model\",
\"utc_time_created\": \"2020-06-14 18:26:02.386982\",
\"flavors\": {
\"keras\": {
\"keras_module\": \"tensorflow.keras\",
\"keras_version\": \"2.2.4-tf\",
\"data\": \"data\"
},
\"python_function\": {
\"loader_module\": \"mlflow.keras\",
\"python_version\": \"3.8.3\",
\"data\": \"data\",
\"env\": \"conda.yaml\"
}
}
}]"
}
}
}
tensorboard --logdir=data/mlruns/0/$EXP2/artifacts/tensorboard_logs
mlflow ui --backend-store-uri file:///home/lex/Programme/drmed-git/data/mlruns
2.1.11.2 plots via mlflow ui, comparing run 1 and 2
AUC:
Loss:
Precision / Recall:
2.1.11.3 plots via tensorboard ui
Prediction 03
Prediction 20
Prediction 24
Prediction 32
Prediction 99
Distribution of Conv1D-Kernel (final layer)
Histograms of Conv1D-Kernel (final layer)
2.1.11.4 for next runs
- more steps = good: better recall
- strong fluctuation of validation metrics in beginning and already convergence after ~30 Epochs for the second run → maybe use learning rate a bit more gently and start the lr schedule with lower rates
- judging from the tensorboard histograms, the model architecture can clearly be made simpler
- doing some background reading: in this SO thread it is argued that while
using Adam, an extensive learning rate schedule, as I used, should not
really be necessary.
- Adam (adaptive moment estimation) updates any parameter with an individual learning rate
- every learning rate can vary from 0 (no update) to the given learning rate as an upper limit
- others argue, that it definitely helps (or that you have to start with a very low lr, which I can confirm from own experience) and that it especially could help reducing the loss in the late steps of training
- finding optimal lr:
- start with very very low lr, than increase till loss stops decreasing, and look at where the slope of the loss curve and pick the learning rate that is associated with the fastest decrease
- effect of batch size
- see effect of batch size
- from pracitcal point: larger batch size → computational speedups from parallelism of GPUs. In theory, using a batch equal to the entire dataset guarantees convergence to global optima of objective function - hower on cost of slower, empirical convergence
- well known: too large of a batch size → poor generalization (hoffer,
hubara, soudry argue here that this is not inherently true).
- why poor gen: competing gradients of different training examples → sequential optimization is easier than simultaneous optimization in complex, high dimensional parameter spaces
- smaller batch size pro: faster convergence to "good" solutions
- smaller batch size con: model not guaranteed to converge to global optima, will bounce around, staying outside some \(\epsilon\) - ball of the optima, where \(\epsilon\) depends on the ratio of the batch size to the data size
- if no computational constraints: start at small batch size, steadily grow batch size through training.
- non-convex models: "sweet spot" between batch size of 1 (bad, "noisy", nn prone to overfitting) and entire training dataset.
- on other hand: "noise" in small batch size might be good → "tug-and-pull" dynamic which might prevent nn from overfitting
- what is "large number of epochs" → number of epochs such that any further training provides little to no boost in test accuracy → difficult to determine → best guess
- example MNIST: bs=64 gets test acc of 98%, bs=1024 gets test acc of 95% → increasing learning rate can compensate for larger batch sizes (for bs=1024 from 0.01 to 0.1 → problem solved) → could be more difficult in more complex datasets
- example MNIST: starting with large batch size does not "get the model stuck" in bad local optimums, better test accuracy can be achieved anytime by switching to lower batch size or higher learning rate
- investigation of how stuff is updated with large batch size:
- larger batch sizes make larger gradient steps than smaller batch sizes for the same number of samples seen
- for same average euclidean norm distance from initial weights of the model, larger batch sizes have larger variance in the distance → if smaller batch sizes are more "noisy", this would be the other way around
- conclusion:
- large batch size → model makes very large gradient updates and very small gradient updates → size depends on which particular samples are drawn.
- Small batch size → model makes updates of about the same size → weakly dependend on which particular samples are drawn.
- better solutions can be far away from inital weights → if loss is averaged over the batch then large batch sizes simply do not allow the model to travel far enough to reach the better solutions for the same number of training epochs → you take fewer steps → increasing lr makes steps larger → and then they can even move further after seeing the same number of samples
- Adam vs SGD
- Adam claims: insensitivity to weight initialization and initial learning rate choice
- finding: adam finds solutions with much larger weights, which might explain why it has lower test accuracy and is not generalizing as well → this is why weight decay is recommended with Adam
- in SGD the weights are initialized to approx the magnitude you want them to be and most of the learning is shuffling the weights along the hyper-sphere of the initial radius
- in Adam, the model ignores the initialization
- hoffer et al
- propose that in SGD, initial learning phase can be described using high-dimensional "random walk on a random potential" process, with an "ultra-slow" logarithmic increase in the distance of the weights from their initialization (observed empirically)
- remedies
- Use SGD with momentum, gradient clipping, and a decreasing learning rate schedule
- adapt learning rate with batch size (e.g. square root scaling)
- compute batch-norm statistics over several partitions ("ghost batch-norm")
- use sufficient number of high learning rate training iterations
- the "generalization gap" problem is not related to the batch size but rather the amount of updates
- Smith et al, Google Brain
- instead of learning rate decay, increase batch size with training → works with Adam, Nesterov momentum, SGD with momentum, SGD → reaches equivalent test accuracies after same no of epochs, but with fewer parameter updates → greater parallelism (good for GPU), shorter training time
- using dropout layers with Unet
- Bartolome et al did it in DeepCell for automatic nuclei detection
- encoding block:
- maxpool
- 2 times conv+ELU+batchnorm
- decoding block:
- upsample
- dropout (0.3)
- 2 times conv+ELU
- last block
- 1x1 conv
- sigmoid
- encoding block:
- from this reddit
- Srivastava/Hinton dropout paper: additional gain in performacne obtained by adding dropout in conv layers is worth noting (3% to 2.55%). Dropout in lower layers help, because it provides noisy inputs for the higher fully connected layers which prevents them from overfitting → they use 0.7 prob for conv dropout and 0.5 for FC
- Bartolome et al did it in DeepCell for automatic nuclei detection
2.1.12 use model from run 2 to correct data
2.1.12.1 simulated data from the test set
2.1.12.2 experimental data from Pablo (structured experiment)
- biological metadata
- Who took the data: Pablo Carravilla in November 2019 in Oxford
- a bit of 400 "dirty" and 400 "clean" curves
- AF488: small dye, homogeneous signal
- "clean"
- in folders
GroupMeas_5" andGroupMeas_6 - Hs-PEX5-eGFP - PEX5 from Homo sapiens, labelled with eGFP
- typical filename:
20 nM AF488147_T1754s_1.ptu
- in folders
- "dirty"
- in folders
GroupMeas_1toGroupMeas_4_4 - Tp-PEX5-eGFP - PEX5 from Trypanosoma brucei, labelled with eGFP
- typical filename:
DiO LUV 10uM in 20 nM AF48816_T197s_1.ptu - Dio LUV: big vesicles, spikes
- in folders
- TODO microscope metadata - paste from pickled pandas dfs
- anecdotal, semi-structured plotting of correlation curves
- first: "clean" data of Hs-PEX5-eGFP
- from plotting 10 traces with the respective binning windows using
correct_correlation_by_unet_prediction:- prediction didn't find anything - awesome! So no values were removed and correlations are "pure"
- for correlation at \(1ms\) binning window:
- transit time ~\(0.25...0.5ms\)
- diffusion coefficient ~\(22.5...45 \mu m^2 / s\)
- for correlation at \(100\mu s\) binning window:
- transit time ~\(0.4...0.53ms\)
- diffusion coefficient ~\(21...29 \mu m^2 / s\)
- This is probably the most accurate!
- for correlation at \(100\mu s\) binning window:
- for correlation at \(10\mu s\) and at 1us binning window:
- transit time ~\(0.09ms\)
- diffusion coefficient ~\(120...133\mu m^2 / s\)
- fit didn't look too good - probably something is not working correctly here with multipletau. Looks like the start of the correlation curve is flattening, but multipletau fits an increase
- for correlation at \(10\mu s\) and at 1us binning window:
- second: "dirty" data of Tb-PEX5-eGFP
- from plotting 10 traces with the respective binning windows using
correct_correlation_by_unet_prediction:- for correlation at \(1ms\) binning wnidow:
- transit time
- without correction: ~\(6...20ms\)
- with correction: ~\(2...5ms\)
- diffusion coefficients:
- without correction: ~\(0.6...1.9\mu m^2 / s\)
- with correction: ~\(2...6.3 \mu m^2 / s\)
- transit time
- for correlation at \(1ms\) binning wnidow:
- for correlation at \(100 \mu s\) binninw window:
- transit time
- without correction: ~\(43...149ms\)
- with correction: ~\(1.6...42ms\)
- diffusion coefficients:
- without correction: ~\(0.08...0.26\mu m^2/s\)
- with correction: ~\(0.3...7\mu m^2 / s\)
- transit time
- for correlation at \(100 \mu s\) binninw window:
- for correlation at \(10\mu s\) binning window:
- fitting looks HORRIBLE for the traces corrected by prediction, while looking "okayish" for traces without correction - even though the values are bad.
- transit time
- without correction: ~\(153...1329ms\)
- with correction: ~\(0.2...346ms\)
- diffusion coefficients:
- without correction: ~\(0.008...0.07 \mu m^2 / s\)
- with correction: ~\(0.03...52\mu m^2 / s\)
- for correlation at \(10\mu s\) binning window:
- for correlation at \(1\mu s\) binning window (only 3 traces, because ):
- fits have to be better!
- for correlation at \(1\mu s\) binning window (only 3 traces, because ):
2.1.13 git exp 3
git status git log -1
(base) [ye53nis@node161 drmed-git]$ git status
# On branch exp-310520-unet
# Untracked files:
# ...
nothing added to commit but untracked files present (use "git add" to track)
(base) [ye53nis@node161 drmed-git]$ git log -1
commit 193d9e3f8d126828b253121e613ddaf5363d7c3d
Author: Apoplex <oligolex@vivaldi.net>
Date: Wed Jun 17 12:09:44 2020 +0200
Move experiment to correct mlflow id
2.1.14 experimental run 3 - bs=3
conda activate tensorflow_nightly cd /beegfs/ye53nis/drmed-git export MLFLOW_EXPERIMENT_NAME=exp-310520-unet export MLFLOW_TRACKING_URI=file:./data/mlruns
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=3 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=2100 -P validation_steps=320
2020/06/20 00:43:51 INFO mlflow.projects: === Created directory /tmp/tmpxwtlwi2b for downloading remote URIs passed to arguments of type 'path' === 2020/06/20 00:43:51 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py / beegfs/ye53nis/drmed-git/src 3 0.2 16384 None 40 /beegfs/ye53nis/saves/firstartefact_Sep2019 2100 320' in run with ID '1e98b3ed2e1d421da9592058bd5587a8' === 2020-06-20 00:44:08.580506: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2020-06-20 00:44:08.580645: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul e's documentation for alternative uses import imp 2.3.0-dev20200527 2020-06-20 00:44:36.776071: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-06-20 00:44:36.776132: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303) 2020-06-20 00:44:36.776174: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node161): /proc/driver/nvidia/version does not exist GPUs: [] train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000) shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000) for each 20,000 timestap trace there are the following numbers of corrupted timesteps: label001_1 6286 label001_1 2568 label001_1 4495 label001_1 4414 label001_1 1105 dtype: int64 2020-06-20 00:49:48.824441: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-06-20 00:49:48.834205: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz 2020-06-20 00:49:48.835707: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5643aca77c10 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-06-20 00:49:48.835736: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version number of training examples: 6400, number of validation examples: 1600 ------------------------ number of test examples: 2000 input - shape: (None, 16384, 1) output - shape: (None, 16384, 1) 2020-06-20 00:49:53.167773: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(wrapped_dict, collections.Mapping): /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec() all_param_names, _, _, all_default_values = inspect.getargspec(fn) # pylint: disable=W1505 Epoch 1/40 /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(values, collections.Sequence): 1/2100 [..............................] - ETA: 0s - loss: 1.4336 - tp: 11096.0000 - fp: 13747.0000 - tn: 16434.0000 - fn: 7875.0000 - precision: 0.4466 - recall: 0.5849 - accuracy: 0.5601 - auc: 0.59502020-06-20 00:50:06.341154: I te nsorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w ill be removed after 2020-07-01. Instructions for updating: use `tf.profiler.experimental.stop` instead. 2020-06-20 00:50:07.661012: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_20_00_50_07 2020-06-20 00:50:07.677958: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.trace.json.gz 2020-06-20 00:50:07.711562: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_20_00_50_07 2020-06-20 00:50:07.711709: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.memory_profile.json.gz 2020-06-20 00:50:07.713980: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_06_20_00_50_07Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_06_20_0 0_50_07/node161.xplane.pb Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.overview_page.pb Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.kernel_stats.pb 2100/2100 [==============================] - 2619s 1s/step - loss: 1.1651 - tp: 14036256.0000 - fp: 5361485.0000 - tn: 74321616.0000 - fn: 9499881.0000 - precision: 0.7236 - recall: 0.5964 - accuracy: 0.8560 - auc: 0.8712 - val_loss: 6. 7480 - val_tp: 3658812.0000 - val_fp: 7608396.0000 - val_tn: 4305440.0000 - val_fn: 155992.0000 - val_precision: 0.3247 - val_recall: 0.9591 - val_accuracy: 0.5064 - val_auc: 0.7199 Epoch 2/40 2100/2100 [==============================] - 2616s 1s/step - loss: 0.7522 - tp: 16978660.0000 - fp: 4731235.0000 - tn: 74935072.0000 - fn: 6574314.0000 - precision: 0.7821 - recall: 0.7209 - accuracy: 0.8905 - auc: 0.9195 - val_loss: 2. 3699 - val_tp: 1583199.0000 - val_fp: 169213.0000 - val_tn: 11937019.0000 - val_fn: 2039209.0000 - val_precision: 0.9034 - val_recall: 0.4371 - val_accuracy: 0.8596 - val_auc: 0.7654 Epoch 3/40 2100/2100 [==============================] - 2615s 1s/step - loss: 0.5253 - tp: 19558076.0000 - fp: 3583985.0000 - tn: 75779848.0000 - fn: 4297304.0000 - precision: 0.8451 - recall: 0.8199 - accuracy: 0.9236 - auc: 0.9546 - val_loss: 20 .3139 - val_tp: 3551544.0000 - val_fp: 6920318.0000 - val_tn: 5181055.0000 - val_fn: 75723.0000 - val_precision: 0.3392 - val_recall: 0.9791 - val_accuracy: 0.5552 - val_auc: 0.7820 Epoch 4/40 2100/2100 [==============================] - 2593s 1s/step - loss: 0.4607 - tp: 19722116.0000 - fp: 3070380.0000 - tn: 76557328.0000 - fn: 3869410.0000 - precision: 0.8653 - recall: 0.8360 - accuracy: 0.9328 - auc: 0.9621 - val_loss: 0. 6783 - val_tp: 2809999.0000 - val_fp: 1055013.0000 - val_tn: 11273173.0000 - val_fn: 590455.0000 - val_precision: 0.7270 - val_recall: 0.8264 - val_accuracy: 0.8954 - val_auc: 0.9256 Epoch 5/40 2100/2100 [==============================] - 2579s 1s/step - loss: 0.4347 - tp: 20035524.0000 - fp: 2906267.0000 - tn: 76615392.0000 - fn: 3662016.0000 - precision: 0.8733 - recall: 0.8455 - accuracy: 0.9364 - auc: 0.9644 - val_loss: 0. 9250 - val_tp: 2881407.0000 - val_fp: 1446289.0000 - val_tn: 10683025.0000 - val_fn: 717919.0000 - val_precision: 0.6658 - val_recall: 0.8005 - val_accuracy: 0.8624 - val_auc: 0.9110 Epoch 6/40 2100/2100 [==============================] - 2587s 1s/step - loss: 0.3938 - tp: 20159010.0000 - fp: 2640041.0000 - tn: 77171896.0000 - fn: 3248240.0000 - precision: 0.8842 - recall: 0.8612 - accuracy: 0.9430 - auc: 0.9692 - val_loss: 0. 9069 - val_tp: 3350060.0000 - val_fp: 2334741.0000 - val_tn: 9777228.0000 - val_fn: 266611.0000 - val_precision: 0.5893 - val_recall: 0.9263 - val_accuracy: 0.8346 - val_auc: 0.9424 Epoch 7/40 2100/2100 [==============================] - 2596s 1s/step - loss: 0.3898 - tp: 20604918.0000 - fp: 2623827.0000 - tn: 76744272.0000 - fn: 3246144.0000 - precision: 0.8870 - recall: 0.8639 - accuracy: 0.9431 - auc: 0.9700 - val_loss: 1. 1869 - val_tp: 2154011.0000 - val_fp: 161830.0000 - val_tn: 11907682.0000 - val_fn: 1505117.0000 - val_precision: 0.9301 - val_recall: 0.5887 - val_accuracy: 0.8940 - val_auc: 0.8640 Epoch 8/40 2100/2100 [==============================] - 2585s 1s/step - loss: 0.3615 - tp: 20559308.0000 - fp: 2451944.0000 - tn: 77195760.0000 - fn: 3012324.0000 - precision: 0.8934 - recall: 0.8722 - accuracy: 0.9471 - auc: 0.9725 - val_loss: 2. 2166 - val_tp: 457245.0000 - val_fp: 0.0000e+00 - val_tn: 12111846.0000 - val_fn: 3159549.0000 - val_precision: 1.0000 - val_recall: 0.1264 - val_accuracy: 0.7991 - val_auc: 0.7373 Epoch 9/40 2100/2100 [==============================] - 2575s 1s/step - loss: 0.3614 - tp: 20474248.0000 - fp: 2479592.0000 - tn: 77192832.0000 - fn: 3072574.0000 - precision: 0.8920 - recall: 0.8695 - accuracy: 0.9462 - auc: 0.9720 - val_loss: 1. 6778 - val_tp: 3153657.0000 - val_fp: 3799634.0000 - val_tn: 8459831.0000 - val_fn: 315518.0000 - val_precision: 0.4535 - val_recall: 0.9091 - val_accuracy: 0.7384 - val_auc: 0.8806 Epoch 10/40 2100/2100 [==============================] - 2574s 1s/step - loss: 0.3525 - tp: 20915356.0000 - fp: 2417250.0000 - tn: 76968064.0000 - fn: 2918540.0000 - precision: 0.8964 - recall: 0.8775 - accuracy: 0.9483 - auc: 0.9734 - val_loss: 0. 7001 - val_tp: 3011358.0000 - val_fp: 1423566.0000 - val_tn: 10810936.0000 - val_fn: 482780.0000 - val_precision: 0.6790 - val_recall: 0.8618 - val_accuracy: 0.8788 - val_auc: 0.9191 Epoch 11/40 2100/2100 [==============================] - 2583s 1s/step - loss: 0.3536 - tp: 20693556.0000 - fp: 2409057.0000 - tn: 77110336.0000 - fn: 3006188.0000 - precision: 0.8957 - recall: 0.8732 - accuracy: 0.9475 - auc: 0.9731 - val_loss: 0. 8146 - val_tp: 3341267.0000 - val_fp: 1466467.0000 - val_tn: 10619280.0000 - val_fn: 301626.0000 - val_precision: 0.6950 - val_recall: 0.9172 - val_accuracy: 0.8876 - val_auc: 0.9452 Epoch 12/40 2100/2100 [==============================] - 2559s 1s/step - loss: 0.3165 - tp: 21370760.0000 - fp: 2314648.0000 - tn: 77018928.0000 - fn: 2514801.0000 - precision: 0.9023 - recall: 0.8947 - accuracy: 0.9532 - auc: 0.9771 - val_loss: 0. 3977 - val_tp: 2929496.0000 - val_fp: 337784.0000 - val_tn: 11799466.0000 - val_fn: 661894.0000 - val_precision: 0.8966 - val_recall: 0.8157 - val_accuracy: 0.9364 - val_auc: 0.9624 Epoch 13/40 2100/2100 [==============================] - 2555s 1s/step - loss: 0.2959 - tp: 21260244.0000 - fp: 2183093.0000 - tn: 77404592.0000 - fn: 2371187.0000 - precision: 0.9069 - recall: 0.8997 - accuracy: 0.9559 - auc: 0.9785 - val_loss: 0. 4174 - val_tp: 2796648.0000 - val_fp: 277583.0000 - val_tn: 11853440.0000 - val_fn: 800969.0000 - val_precision: 0.9097 - val_recall: 0.7774 - val_accuracy: 0.9314 - val_auc: 0.9595 Epoch 14/40 2100/2100 [==============================] - 2567s 1s/step - loss: 0.2983 - tp: 21268994.0000 - fp: 2193443.0000 - tn: 77331256.0000 - fn: 2425591.0000 - precision: 0.9065 - recall: 0.8976 - accuracy: 0.9552 - auc: 0.9784 - val_loss: 0. 3715 - val_tp: 3222237.0000 - val_fp: 299190.0000 - val_tn: 11646772.0000 - val_fn: 560441.0000 - val_precision: 0.9150 - val_recall: 0.8518 - val_accuracy: 0.9453 - val_auc: 0.9688 Epoch 15/40 2100/2100 [==============================] - 2562s 1s/step - loss: 0.2919 - tp: 21573232.0000 - fp: 2138857.0000 - tn: 77160424.0000 - fn: 2346711.0000 - precision: 0.9098 - recall: 0.9019 - accuracy: 0.9565 - auc: 0.9791 - val_loss: 0. 4316 - val_tp: 3028795.0000 - val_fp: 656530.0000 - val_tn: 11568012.0000 - val_fn: 475303.0000 - val_precision: 0.8219 - val_recall: 0.8644 - val_accuracy: 0.9280 - val_auc: 0.9580 Epoch 16/40 2100/2100 [==============================] - 2544s 1s/step - loss: 0.2915 - tp: 21637290.0000 - fp: 2191368.0000 - tn: 77039000.0000 - fn: 2351546.0000 - precision: 0.9080 - recall: 0.9020 - accuracy: 0.9560 - auc: 0.9792 - val_loss: 0. 4676 - val_tp: 2601966.0000 - val_fp: 397290.0000 - val_tn: 12006863.0000 - val_fn: 722521.0000 - val_precision: 0.8675 - val_recall: 0.7827 - val_accuracy: 0.9288 - val_auc: 0.9455 Epoch 17/40 2100/2100 [==============================] - 2550s 1s/step - loss: 0.2894 - tp: 21593576.0000 - fp: 2107461.0000 - tn: 77151016.0000 - fn: 2367164.0000 - precision: 0.9111 - recall: 0.9012 - accuracy: 0.9566 - auc: 0.9796 - val_loss: 0. 5073 - val_tp: 2723777.0000 - val_fp: 267464.0000 - val_tn: 11851181.0000 - val_fn: 886218.0000 - val_precision: 0.9106 - val_recall: 0.7545 - val_accuracy: 0.9267 - val_auc: 0.9421 Epoch 18/40 2100/2100 [==============================] - 2565s 1s/step - loss: 0.2915 - tp: 21220710.0000 - fp: 2089500.0000 - tn: 77529384.0000 - fn: 2379606.0000 - precision: 0.9104 - recall: 0.8992 - accuracy: 0.9567 - auc: 0.9792 - val_loss: 0. 4440 - val_tp: 3014233.0000 - val_fp: 194294.0000 - val_tn: 11773351.0000 - val_fn: 746762.0000 - val_precision: 0.9394 - val_recall: 0.8014 - val_accuracy: 0.9402 - val_auc: 0.9537 Epoch 19/40 2100/2100 [==============================] - 2555s 1s/step - loss: 0.2859 - tp: 21497494.0000 - fp: 2128539.0000 - tn: 77269760.0000 - fn: 2323349.0000 - precision: 0.9099 - recall: 0.9025 - accuracy: 0.9569 - auc: 0.9798 - val_loss: 0. 4337 - val_tp: 2842516.0000 - val_fp: 264846.0000 - val_tn: 11878934.0000 - val_fn: 742344.0000 - val_precision: 0.9148 - val_recall: 0.7929 - val_accuracy: 0.9360 - val_auc: 0.9550 Epoch 20/40 2100/2100 [==============================] - 2618s 1s/step - loss: 0.2902 - tp: 21548910.0000 - fp: 2102526.0000 - tn: 77221952.0000 - fn: 2345775.0000 - precision: 0.9111 - recall: 0.9018 - accuracy: 0.9569 - auc: 0.9791 - val_loss: 0. 5074 - val_tp: 2805182.0000 - val_fp: 105571.0000 - val_tn: 11861966.0000 - val_fn: 955921.0000 - val_precision: 0.9637 - val_recall: 0.7458 - val_accuracy: 0.9325 - val_auc: 0.9465 Epoch 21/40 2100/2100 [==============================] - 2633s 1s/step - loss: 0.2838 - tp: 21355176.0000 - fp: 2103115.0000 - tn: 77459040.0000 - fn: 2301871.0000 - precision: 0.9103 - recall: 0.9027 - accuracy: 0.9573 - auc: 0.9797 - val_loss: 0. 5292 - val_tp: 2876840.0000 - val_fp: 499247.0000 - val_tn: 11594560.0000 - val_fn: 757993.0000 - val_precision: 0.8521 - val_recall: 0.7915 - val_accuracy: 0.9201 - val_auc: 0.9410 Epoch 22/40 2100/2100 [==============================] - 2623s 1s/step - loss: 0.2770 - tp: 21105648.0000 - fp: 2105094.0000 - tn: 77782320.0000 - fn: 2226166.0000 - precision: 0.9093 - recall: 0.9046 - accuracy: 0.9580 - auc: 0.9804 - val_loss: 0. 4563 - val_tp: 2796040.0000 - val_fp: 290458.0000 - val_tn: 11870456.0000 - val_fn: 771686.0000 - val_precision: 0.9059 - val_recall: 0.7837 - val_accuracy: 0.9325 - val_auc: 0.9511 Epoch 23/40 2100/2100 [==============================] - 2604s 1s/step - loss: 0.2779 - tp: 21230712.0000 - fp: 2051525.0000 - tn: 77688872.0000 - fn: 2248134.0000 - precision: 0.9119 - recall: 0.9042 - accuracy: 0.9583 - auc: 0.9800 - val_loss: 0. 4772 - val_tp: 2839169.0000 - val_fp: 344193.0000 - val_tn: 11784262.0000 - val_fn: 761016.0000 - val_precision: 0.8919 - val_recall: 0.7886 - val_accuracy: 0.9297 - val_auc: 0.9486 Epoch 24/40 2100/2100 [==============================] - 2607s 1s/step - loss: 0.2831 - tp: 21599932.0000 - fp: 2114749.0000 - tn: 77206248.0000 - fn: 2298204.0000 - precision: 0.9108 - recall: 0.9038 - accuracy: 0.9572 - auc: 0.9799 - val_loss: 0. 4515 - val_tp: 3000318.0000 - val_fp: 428123.0000 - val_tn: 11603006.0000 - val_fn: 697193.0000 - val_precision: 0.8751 - val_recall: 0.8114 - val_accuracy: 0.9285 - val_auc: 0.9539 Epoch 25/40 2100/2100 [==============================] - 2608s 1s/step - loss: 0.2802 - tp: 21684768.0000 - fp: 2090639.0000 - tn: 77166856.0000 - fn: 2276865.0000 - precision: 0.9121 - recall: 0.9050 - accuracy: 0.9577 - auc: 0.9803 - val_loss: 0. 4666 - val_tp: 2881923.0000 - val_fp: 449549.0000 - val_tn: 11719217.0000 - val_fn: 677951.0000 - val_precision: 0.8651 - val_recall: 0.8096 - val_accuracy: 0.9283 - val_auc: 0.9492 Epoch 26/40 2100/2100 [==============================] - 2601s 1s/step - loss: 0.2817 - tp: 21424128.0000 - fp: 2071632.0000 - tn: 77450872.0000 - fn: 2272584.0000 - precision: 0.9118 - recall: 0.9041 - accuracy: 0.9579 - auc: 0.9798 - val_loss: 0. 4771 - val_tp: 2777236.0000 - val_fp: 523426.0000 - val_tn: 11747034.0000 - val_fn: 680944.0000 - val_precision: 0.8414 - val_recall: 0.8031 - val_accuracy: 0.9234 - val_auc: 0.9473 Epoch 27/40 2100/2100 [==============================] - 2557s 1s/step - loss: 0.2761 - tp: 21149424.0000 - fp: 2050887.0000 - tn: 77759664.0000 - fn: 2259208.0000 - precision: 0.9116 - recall: 0.9035 - accuracy: 0.9582 - auc: 0.9802 - val_loss: 0. 4809 - val_tp: 2766963.0000 - val_fp: 356057.0000 - val_tn: 11830343.0000 - val_fn: 775277.0000 - val_precision: 0.8860 - val_recall: 0.7811 - val_accuracy: 0.9281 - val_auc: 0.9473 Epoch 28/40 2100/2100 [==============================] - 2572s 1s/step - loss: 0.2787 - tp: 21614004.0000 - fp: 2105764.0000 - tn: 77265320.0000 - fn: 2234175.0000 - precision: 0.9112 - recall: 0.9063 - accuracy: 0.9580 - auc: 0.9804 - val_loss: 0. 4297 - val_tp: 2776882.0000 - val_fp: 304747.0000 - val_tn: 11905345.0000 - val_fn: 741666.0000 - val_precision: 0.9011 - val_recall: 0.7892 - val_accuracy: 0.9335 - val_auc: 0.9558 Epoch 29/40 2100/2100 [==============================] - 2592s 1s/step - loss: 0.2759 - tp: 21330060.0000 - fp: 2032687.0000 - tn: 77600192.0000 - fn: 2256365.0000 - precision: 0.9130 - recall: 0.9043 - accuracy: 0.9584 - auc: 0.9805 - val_loss: 0. 4579 - val_tp: 2999477.0000 - val_fp: 494545.0000 - val_tn: 11578967.0000 - val_fn: 655651.0000 - val_precision: 0.8585 - val_recall: 0.8206 - val_accuracy: 0.9269 - val_auc: 0.9534 Epoch 30/40 2100/2100 [==============================] - 2581s 1s/step - loss: 0.2724 - tp: 21130112.0000 - fp: 2033978.0000 - tn: 77848336.0000 - fn: 2206652.0000 - precision: 0.9122 - recall: 0.9054 - accuracy: 0.9589 - auc: 0.9804 - val_loss: 0. 4905 - val_tp: 2843808.0000 - val_fp: 315711.0000 - val_tn: 11777024.0000 - val_fn: 792097.0000 - val_precision: 0.9001 - val_recall: 0.7821 - val_accuracy: 0.9296 - val_auc: 0.9485 Epoch 31/40 2100/2100 [==============================] - 2600s 1s/step - loss: 0.2785 - tp: 21125680.0000 - fp: 2037613.0000 - tn: 77795080.0000 - fn: 2260802.0000 - precision: 0.9120 - recall: 0.9033 - accuracy: 0.9584 - auc: 0.9796 - val_loss: 0. 4386 - val_tp: 2841391.0000 - val_fp: 466262.0000 - val_tn: 11790714.0000 - val_fn: 630273.0000 - val_precision: 0.8590 - val_recall: 0.8185 - val_accuracy: 0.9303 - val_auc: 0.9549 Epoch 32/40 2100/2100 [==============================] - 2569s 1s/step - loss: 0.2780 - tp: 21701982.0000 - fp: 2120334.0000 - tn: 77174816.0000 - fn: 2221998.0000 - precision: 0.9110 - recall: 0.9071 - accuracy: 0.9579 - auc: 0.9802 - val_loss: 0. 4458 - val_tp: 2760528.0000 - val_fp: 256875.0000 - val_tn: 11974282.0000 - val_fn: 736955.0000 - val_precision: 0.9149 - val_recall: 0.7893 - val_accuracy: 0.9368 - val_auc: 0.9520 Epoch 33/40 2100/2100 [==============================] - 2558s 1s/step - loss: 0.2798 - tp: 21431360.0000 - fp: 2104855.0000 - tn: 77436496.0000 - fn: 2246416.0000 - precision: 0.9106 - recall: 0.9051 - accuracy: 0.9578 - auc: 0.9799 - val_loss: 0. 4787 - val_tp: 2902400.0000 - val_fp: 434973.0000 - val_tn: 11637810.0000 - val_fn: 753457.0000 - val_precision: 0.8697 - val_recall: 0.7939 - val_accuracy: 0.9244 - val_auc: 0.9494 Epoch 34/40 2100/2100 [==============================] - 2540s 1s/step - loss: 0.2773 - tp: 21209216.0000 - fp: 2046492.0000 - tn: 77703984.0000 - fn: 2259570.0000 - precision: 0.9120 - recall: 0.9037 - accuracy: 0.9583 - auc: 0.9800 - val_loss: 0. 4282 - val_tp: 2892899.0000 - val_fp: 378247.0000 - val_tn: 11801593.0000 - val_fn: 655901.0000 - val_precision: 0.8844 - val_recall: 0.8152 - val_accuracy: 0.9343 - val_auc: 0.9564 Epoch 35/40 2100/2100 [==============================] - 2548s 1s/step - loss: 0.2784 - tp: 21515622.0000 - fp: 2084525.0000 - tn: 77370184.0000 - fn: 2248908.0000 - precision: 0.9117 - recall: 0.9054 - accuracy: 0.9580 - auc: 0.9804 - val_loss: 0. 4553 - val_tp: 3086603.0000 - val_fp: 467495.0000 - val_tn: 11473592.0000 - val_fn: 700950.0000 - val_precision: 0.8685 - val_recall: 0.8149 - val_accuracy: 0.9257 - val_auc: 0.9544 Epoch 36/40 2100/2100 [==============================] - 2541s 1s/step - loss: 0.2708 - tp: 20892576.0000 - fp: 2005985.0000 - tn: 78121936.0000 - fn: 2198694.0000 - precision: 0.9124 - recall: 0.9048 - accuracy: 0.9593 - auc: 0.9805 - val_loss: 0. 4393 - val_tp: 2703510.0000 - val_fp: 394771.0000 - val_tn: 11969160.0000 - val_fn: 661199.0000 - val_precision: 0.8726 - val_recall: 0.8035 - val_accuracy: 0.9329 - val_auc: 0.9538 Epoch 37/40 2100/2100 [==============================] - 2552s 1s/step - loss: 0.2788 - tp: 21802620.0000 - fp: 2101735.0000 - tn: 77067912.0000 - fn: 2246918.0000 - precision: 0.9121 - recall: 0.9066 - accuracy: 0.9579 - auc: 0.9805 - val_loss: 0. 4474 - val_tp: 2806677.0000 - val_fp: 219556.0000 - val_tn: 11887825.0000 - val_fn: 814582.0000 - val_precision: 0.9274 - val_recall: 0.7751 - val_accuracy: 0.9343 - val_auc: 0.9537 Epoch 38/40 2100/2100 [==============================] - 2545s 1s/step - loss: 0.2769 - tp: 21191620.0000 - fp: 2046041.0000 - tn: 77717744.0000 - fn: 2263741.0000 - precision: 0.9120 - recall: 0.9035 - accuracy: 0.9582 - auc: 0.9801 - val_loss: 0. 4414 - val_tp: 2927919.0000 - val_fp: 286995.0000 - val_tn: 11783958.0000 - val_fn: 729768.0000 - val_precision: 0.9107 - val_recall: 0.8005 - val_accuracy: 0.9354 - val_auc: 0.9544 Epoch 39/40 2100/2100 [==============================] - 2545s 1s/step - loss: 0.2785 - tp: 21171298.0000 - fp: 2047544.0000 - tn: 77738952.0000 - fn: 2261357.0000 - precision: 0.9118 - recall: 0.9035 - accuracy: 0.9583 - auc: 0.9798 - val_loss: 0. 4576 - val_tp: 2779792.0000 - val_fp: 346642.0000 - val_tn: 11869174.0000 - val_fn: 733032.0000 - val_precision: 0.8891 - val_recall: 0.7913 - val_accuracy: 0.9314 - val_auc: 0.9517 Epoch 40/40 2100/2100 [==============================] - 2538s 1s/step - loss: 0.2811 - tp: 21804190.0000 - fp: 2110823.0000 - tn: 77014800.0000 - fn: 2289448.0000 - precision: 0.9117 - recall: 0.9050 - accuracy: 0.9574 - auc: 0.9803 - val_loss: 0. 4445 - val_tp: 2933937.0000 - val_fp: 327429.0000 - val_tn: 11730665.0000 - val_fn: 736609.0000 - val_precision: 0.8996 - val_recall: 0.7993 - val_accuracy: 0.9324 - val_auc: 0.9543 667/667 [==============================] - 163s 244ms/step - loss: 0.5248 - tp: 6207113.0000 - fp: 1521051.0000 - tn: 23569214.0000 - fn: 1470623.0000 - precision: 0.8032 - recall: 0.8085 - accuracy: 0.9087 - auc: 0.9439 2020/06/21 05:32:13 INFO mlflow.projects: === Run (ID '1e98b3ed2e1d421da9592058bd5587a8') succeeded ===
2.1.15 git exp 4
git status git log -1
(tensorflow_nightly) [ye53nis@node161 drmed-git]$ git status
git log -1
# On branch exp-310520-unet
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
# ...
nothing added to commit but untracked files present (use "git add" to track)
(tensorflow_nightly) [ye53nis@node161 drmed-git]$ git log -1
commit 8adaca82d8e092997828338bc2b730568c3ff74b
Author: Alex Seltmann <seltmann@posteo.de>
Date: Mon Jun 22 00:45:48 2020 +0200
exp-310520-unet run 3 bs=3
2.1.16 experimental run 4 - bs=7
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=7 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=910 -P validation_steps=225
2020/06/22 01:46:18 INFO mlflow.projects: === Created directory /tmp/tmpbqom5l09 for downloading remote URIs passed to arguments of type 'path' === 2020/06/22 01:46:18 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py / beegfs/ye53nis/drmed-git/src 7 0.2 16384 None 40 /beegfs/ye53nis/saves/firstartefact_Sep2019 910 225' in run with ID '306234c75c9c48058cbd694579eff31b' === 2020-06-22 01:46:38.397450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2020-06-22 01:46:38.397497: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul e's documentation for alternative uses import imp 2.3.0-dev20200527 2020-06-22 01:47:08.147584: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-06-22 01:47:08.147650: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303) 2020-06-22 01:47:08.147696: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node161): /proc/driver/nvidia/version does not exist GPUs: [] train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000) shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000) for each 20,000 timestap trace there are the following numbers of corrupted timesteps: label001_1 6286 label001_1 2568 label001_1 4495 label001_1 4414 label001_1 1105 dtype: int64 2020-06-22 01:52:37.438938: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-06-22 01:52:37.449993: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz 2020-06-22 01:52:37.451959: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5592ec827380 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-06-22 01:52:37.451991: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version number of training examples: 6400, number of validation examples: 1600 ------------------------ number of test examples: 2000 input - shape: (None, 16384, 1) output - shape: (None, 16384, 1) 2020-06-22 01:52:41.793790: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(wrapped_dict, collections.Mapping): /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec() all_param_names, _, _, all_default_values = inspect.getargspec(fn) # pylint: disable=W1505 Epoch 1/40 /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(values, collections.Sequence): 1/910 [..............................] - ETA: 0s - loss: 1.5123 - tp: 3489.0000 - fp: 12945.0000 - tn: 73461.0000 - fn: 24793.0000 - precision: 0.2123 - recall: 0.1234 - accuracy: 0.6710 - auc: 0.49312020-06-22 01:52:55.410673: I tens orflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w ill be removed after 2020-07-01. Instructions for updating: use `tf.profiler.experimental.stop` instead. 2020-06-22 01:52:57.883904: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_22_01_52_57 2020-06-22 01:52:57.900920: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.trace.json.gz 2020-06-22 01:52:57.938257: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_22_01_52_57 2020-06-22 01:52:57.938413: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.memory_profile.json.gz 2020-06-22 01:52:57.941154: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_06_22_01_52_57Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_06_22_0 1_52_57/node161.xplane.pb Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.overview_page.pb Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.kernel_stats.pb 910/910 [==============================] - 2234s 2s/step - loss: 1.1284 - tp: 16644571.0000 - fp: 5060115.0000 - tn: 75211888.0000 - fn: 7449514.0000 - precision: 0.7669 - recall: 0.6908 - accuracy: 0.8801 - auc: 0.9092 - val_loss: 44.1 496 - val_tp: 5991771.0000 - val_fp: 19810554.0000 - val_tn: 2354.0000 - val_fn: 117.0000 - val_precision: 0.2322 - val_recall: 1.0000 - val_accuracy: 0.2323 - val_auc: 0.5235 Epoch 2/40 910/910 [==============================] - 2228s 2s/step - loss: 0.6506 - tp: 18676100.0000 - fp: 3725706.0000 - tn: 76768256.0000 - fn: 5196001.0000 - precision: 0.8337 - recall: 0.7823 - accuracy: 0.9145 - auc: 0.9421 - val_loss: 4.02 94 - val_tp: 5567715.0000 - val_fp: 7766004.0000 - val_tn: 12285671.0000 - val_fn: 185410.0000 - val_precision: 0.4176 - val_recall: 0.9678 - val_accuracy: 0.6919 - val_auc: 0.8550 Epoch 3/40 910/910 [==============================] - 2209s 2s/step - loss: 0.4869 - tp: 19773668.0000 - fp: 3339711.0000 - tn: 77156064.0000 - fn: 4096586.0000 - precision: 0.8555 - recall: 0.8284 - accuracy: 0.9287 - auc: 0.9571 - val_loss: 15.0 428 - val_tp: 5964847.0000 - val_fp: 16182339.0000 - val_tn: 3635866.0000 - val_fn: 21748.0000 - val_precision: 0.2693 - val_recall: 0.9964 - val_accuracy: 0.3721 - val_auc: 0.6686 Epoch 4/40 910/910 [==============================] - 2196s 2s/step - loss: 0.4160 - tp: 20489738.0000 - fp: 2947256.0000 - tn: 77507616.0000 - fn: 3421442.0000 - precision: 0.8742 - recall: 0.8569 - accuracy: 0.9390 - auc: 0.9659 - val_loss: 1.94 92 - val_tp: 5844006.0000 - val_fp: 12298817.0000 - val_tn: 7631163.0000 - val_fn: 30814.0000 - val_precision: 0.3221 - val_recall: 0.9948 - val_accuracy: 0.5222 - val_auc: 0.9445 Epoch 5/40 910/910 [==============================] - 2197s 2s/step - loss: 0.3808 - tp: 20651254.0000 - fp: 2753622.0000 - tn: 77775072.0000 - fn: 3186176.0000 - precision: 0.8823 - recall: 0.8663 - accuracy: 0.9431 - auc: 0.9697 - val_loss: 0.48 60 - val_tp: 5068080.0000 - val_fp: 972440.0000 - val_tn: 19094874.0000 - val_fn: 669407.0000 - val_precision: 0.8390 - val_recall: 0.8833 - val_accuracy: 0.9364 - val_auc: 0.9722 Epoch 6/40 910/910 [==============================] - 2186s 2s/step - loss: 0.3788 - tp: 20770170.0000 - fp: 2708911.0000 - tn: 77787896.0000 - fn: 3099100.0000 - precision: 0.8846 - recall: 0.8702 - accuracy: 0.9444 - auc: 0.9703 - val_loss: 0.86 28 - val_tp: 5389513.0000 - val_fp: 2524440.0000 - val_tn: 17394656.0000 - val_fn: 496188.0000 - val_precision: 0.6810 - val_recall: 0.9157 - val_accuracy: 0.8829 - val_auc: 0.9492 Epoch 7/40 910/910 [==============================] - 2187s 2s/step - loss: 0.3621 - tp: 21038052.0000 - fp: 2650316.0000 - tn: 77692112.0000 - fn: 2985530.0000 - precision: 0.8881 - recall: 0.8757 - accuracy: 0.9460 - auc: 0.9724 - val_loss: 0.90 81 - val_tp: 5562172.0000 - val_fp: 3465595.0000 - val_tn: 16430461.0000 - val_fn: 346572.0000 - val_precision: 0.6161 - val_recall: 0.9413 - val_accuracy: 0.8523 - val_auc: 0.9356 Epoch 8/40 910/910 [==============================] - 2193s 2s/step - loss: 0.3553 - tp: 20964808.0000 - fp: 2536411.0000 - tn: 77880080.0000 - fn: 2984777.0000 - precision: 0.8921 - recall: 0.8754 - accuracy: 0.9471 - auc: 0.9726 - val_loss: 1.75 52 - val_tp: 5773672.0000 - val_fp: 6575655.0000 - val_tn: 13365159.0000 - val_fn: 90314.0000 - val_precision: 0.4675 - val_recall: 0.9846 - val_accuracy: 0.7417 - val_auc: 0.9457 Epoch 9/40 910/910 [==============================] - 2191s 2s/step - loss: 0.3477 - tp: 20855856.0000 - fp: 2530233.0000 - tn: 78070912.0000 - fn: 2908993.0000 - precision: 0.8918 - recall: 0.8776 - accuracy: 0.9479 - auc: 0.9732 - val_loss: 5.37 18 - val_tp: 5639818.0000 - val_fp: 10545935.0000 - val_tn: 9573186.0000 - val_fn: 45861.0000 - val_precision: 0.3484 - val_recall: 0.9919 - val_accuracy: 0.5895 - val_auc: 0.8627 Epoch 10/40 910/910 [==============================] - 2198s 2s/step - loss: 0.3453 - tp: 20967864.0000 - fp: 2537372.0000 - tn: 77932464.0000 - fn: 2928327.0000 - precision: 0.8921 - recall: 0.8775 - accuracy: 0.9476 - auc: 0.9739 - val_loss: 6.18 03 - val_tp: 5843714.0000 - val_fp: 6972956.0000 - val_tn: 12840040.0000 - val_fn: 148090.0000 - val_precision: 0.4559 - val_recall: 0.9753 - val_accuracy: 0.7240 - val_auc: 0.8662 Epoch 11/40 910/910 [==============================] - 2199s 2s/step - loss: 0.3386 - tp: 21061526.0000 - fp: 2493371.0000 - tn: 77979232.0000 - fn: 2831932.0000 - precision: 0.8941 - recall: 0.8815 - accuracy: 0.9490 - auc: 0.9744 - val_loss: 1.93 77 - val_tp: 5141006.0000 - val_fp: 5024287.0000 - val_tn: 14889905.0000 - val_fn: 749602.0000 - val_precision: 0.5057 - val_recall: 0.8727 - val_accuracy: 0.7762 - val_auc: 0.8851 Epoch 12/40 910/910 [==============================] - 2189s 2s/step - loss: 0.3032 - tp: 21443394.0000 - fp: 2244059.0000 - tn: 78197192.0000 - fn: 2481465.0000 - precision: 0.9053 - recall: 0.8963 - accuracy: 0.9547 - auc: 0.9782 - val_loss: 0.41 02 - val_tp: 4447303.0000 - val_fp: 81983.0000 - val_tn: 19938508.0000 - val_fn: 1337008.0000 - val_precision: 0.9819 - val_recall: 0.7689 - val_accuracy: 0.9450 - val_auc: 0.9689 Epoch 13/40 910/910 [==============================] - 2190s 2s/step - loss: 0.2934 - tp: 21384318.0000 - fp: 2224470.0000 - tn: 78382704.0000 - fn: 2374534.0000 - precision: 0.9058 - recall: 0.9001 - accuracy: 0.9559 - auc: 0.9790 - val_loss: 0.39 33 - val_tp: 4709935.0000 - val_fp: 185454.0000 - val_tn: 19874024.0000 - val_fn: 1035376.0000 - val_precision: 0.9621 - val_recall: 0.8198 - val_accuracy: 0.9527 - val_auc: 0.9764 Epoch 14/40 910/910 [==============================] - 2191s 2s/step - loss: 0.2891 - tp: 21659198.0000 - fp: 2221361.0000 - tn: 78137264.0000 - fn: 2348280.0000 - precision: 0.9070 - recall: 0.9022 - accuracy: 0.9562 - auc: 0.9795 - val_loss: 0.34 77 - val_tp: 4962272.0000 - val_fp: 238349.0000 - val_tn: 19698052.0000 - val_fn: 906131.0000 - val_precision: 0.9542 - val_recall: 0.8456 - val_accuracy: 0.9556 - val_auc: 0.9784 Epoch 15/40 910/910 [==============================] - 2190s 2s/step - loss: 0.2855 - tp: 21305984.0000 - fp: 2168209.0000 - tn: 78570984.0000 - fn: 2320894.0000 - precision: 0.9076 - recall: 0.9018 - accuracy: 0.9570 - auc: 0.9794 - val_loss: 0.42 92 - val_tp: 5448927.0000 - val_fp: 691788.0000 - val_tn: 19190140.0000 - val_fn: 473948.0000 - val_precision: 0.8873 - val_recall: 0.9200 - val_accuracy: 0.9548 - val_auc: 0.9843 Epoch 16/40 910/910 [==============================] - 2206s 2s/step - loss: 0.2860 - tp: 21768336.0000 - fp: 2197317.0000 - tn: 78071472.0000 - fn: 2328913.0000 - precision: 0.9083 - recall: 0.9034 - accuracy: 0.9566 - auc: 0.9798 - val_loss: 0.38 76 - val_tp: 5178300.0000 - val_fp: 782337.0000 - val_tn: 19316320.0000 - val_fn: 527843.0000 - val_precision: 0.8687 - val_recall: 0.9075 - val_accuracy: 0.9492 - val_auc: 0.9825 Epoch 17/40 910/910 [==============================] - 2195s 2s/step - loss: 0.2846 - tp: 21434488.0000 - fp: 2169029.0000 - tn: 78438552.0000 - fn: 2324017.0000 - precision: 0.9081 - recall: 0.9022 - accuracy: 0.9570 - auc: 0.9797 - val_loss: 0.30 14 - val_tp: 5075497.0000 - val_fp: 245322.0000 - val_tn: 19617398.0000 - val_fn: 866585.0000 - val_precision: 0.9539 - val_recall: 0.8542 - val_accuracy: 0.9569 - val_auc: 0.9784 Epoch 18/40 910/910 [==============================] - 2194s 2s/step - loss: 0.2779 - tp: 21615928.0000 - fp: 2131081.0000 - tn: 78362064.0000 - fn: 2257103.0000 - precision: 0.9103 - recall: 0.9055 - accuracy: 0.9580 - auc: 0.9802 - val_loss: 0.35 91 - val_tp: 5315886.0000 - val_fp: 630662.0000 - val_tn: 19377184.0000 - val_fn: 481064.0000 - val_precision: 0.8939 - val_recall: 0.9170 - val_accuracy: 0.9569 - val_auc: 0.9853 Epoch 19/40 910/910 [==============================] - 2199s 2s/step - loss: 0.2810 - tp: 21823838.0000 - fp: 2154952.0000 - tn: 78086368.0000 - fn: 2300819.0000 - precision: 0.9101 - recall: 0.9046 - accuracy: 0.9573 - auc: 0.9804 - val_loss: 0.29 36 - val_tp: 5247304.0000 - val_fp: 327869.0000 - val_tn: 19475176.0000 - val_fn: 754455.0000 - val_precision: 0.9412 - val_recall: 0.8743 - val_accuracy: 0.9581 - val_auc: 0.9805 Epoch 20/40 910/910 [==============================] - 2192s 2s/step - loss: 0.2764 - tp: 21738176.0000 - fp: 2123769.0000 - tn: 78235688.0000 - fn: 2268399.0000 - precision: 0.9110 - recall: 0.9055 - accuracy: 0.9579 - auc: 0.9807 - val_loss: 0.31 80 - val_tp: 5172404.0000 - val_fp: 301019.0000 - val_tn: 19568382.0000 - val_fn: 762993.0000 - val_precision: 0.9450 - val_recall: 0.8715 - val_accuracy: 0.9588 - val_auc: 0.9816 Epoch 21/40 910/910 [==============================] - 2198s 2s/step - loss: 0.2753 - tp: 21565306.0000 - fp: 2109464.0000 - tn: 78441872.0000 - fn: 2249478.0000 - precision: 0.9109 - recall: 0.9055 - accuracy: 0.9582 - auc: 0.9805 - val_loss: 0.32 99 - val_tp: 4898108.0000 - val_fp: 133276.0000 - val_tn: 19742242.0000 - val_fn: 1031169.0000 - val_precision: 0.9735 - val_recall: 0.8261 - val_accuracy: 0.9549 - val_auc: 0.9758 Epoch 22/40 910/910 [==============================] - 2194s 2s/step - loss: 0.2705 - tp: 21872064.0000 - fp: 2096978.0000 - tn: 78186784.0000 - fn: 2210292.0000 - precision: 0.9125 - recall: 0.9082 - accuracy: 0.9587 - auc: 0.9812 - val_loss: 0.32 48 - val_tp: 5379907.0000 - val_fp: 361207.0000 - val_tn: 19421612.0000 - val_fn: 642071.0000 - val_precision: 0.9371 - val_recall: 0.8934 - val_accuracy: 0.9611 - val_auc: 0.9833 Epoch 23/40 910/910 [==============================] - 2192s 2s/step - loss: 0.2693 - tp: 21757518.0000 - fp: 2089232.0000 - tn: 78320768.0000 - fn: 2198508.0000 - precision: 0.9124 - recall: 0.9082 - accuracy: 0.9589 - auc: 0.9810 - val_loss: 0.29 60 - val_tp: 5305212.0000 - val_fp: 364327.0000 - val_tn: 19508656.0000 - val_fn: 626605.0000 - val_precision: 0.9357 - val_recall: 0.8944 - val_accuracy: 0.9616 - val_auc: 0.9831 Epoch 24/40 910/910 [==============================] - 2182s 2s/step - loss: 0.2644 - tp: 21582250.0000 - fp: 2055042.0000 - tn: 78581192.0000 - fn: 2147551.0000 - precision: 0.9131 - recall: 0.9095 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0.31 56 - val_tp: 5259415.0000 - val_fp: 362064.0000 - val_tn: 19527056.0000 - val_fn: 656259.0000 - val_precision: 0.9356 - val_recall: 0.8891 - val_accuracy: 0.9605 - val_auc: 0.9826 Epoch 25/40 910/910 [==============================] - 2182s 2s/step - loss: 0.2646 - tp: 21752094.0000 - fp: 2060925.0000 - tn: 78391832.0000 - fn: 2161242.0000 - precision: 0.9135 - recall: 0.9096 - accuracy: 0.9595 - auc: 0.9816 - val_loss: 0.30 17 - val_tp: 5312792.0000 - val_fp: 398301.0000 - val_tn: 19491284.0000 - val_fn: 602422.0000 - val_precision: 0.9303 - val_recall: 0.8982 - val_accuracy: 0.9612 - val_auc: 0.9832 Epoch 26/40 910/910 [==============================] - 2111s 2s/step - loss: 0.2703 - tp: 21731022.0000 - fp: 2103277.0000 - tn: 78346144.0000 - fn: 2185602.0000 - precision: 0.9118 - recall: 0.9086 - accuracy: 0.9589 - auc: 0.9811 - val_loss: 0.31 48 - val_tp: 5491565.0000 - val_fp: 401219.0000 - val_tn: 19289576.0000 - val_fn: 622446.0000 - val_precision: 0.9319 - val_recall: 0.8982 - val_accuracy: 0.9603 - val_auc: 0.9844 Epoch 27/40 910/910 [==============================] - 2066s 2s/step - loss: 0.2638 - tp: 21622916.0000 - fp: 2036434.0000 - tn: 78564656.0000 - fn: 2142006.0000 - precision: 0.9139 - recall: 0.9099 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0.28 12 - val_tp: 5310123.0000 - val_fp: 347345.0000 - val_tn: 19514636.0000 - val_fn: 632692.0000 - val_precision: 0.9386 - val_recall: 0.8935 - val_accuracy: 0.9620 - val_auc: 0.9826 Epoch 28/40 910/910 [==============================] - 2084s 2s/step - loss: 0.2644 - tp: 21746928.0000 - fp: 2054824.0000 - tn: 78423840.0000 - fn: 2140490.0000 - precision: 0.9137 - recall: 0.9104 - accuracy: 0.9598 - auc: 0.9815 - val_loss: 0.29 02 - val_tp: 5105988.0000 - val_fp: 301224.0000 - val_tn: 19705344.0000 - val_fn: 692246.0000 - val_precision: 0.9443 - val_recall: 0.8806 - val_accuracy: 0.9615 - val_auc: 0.9816 Epoch 29/40 910/910 [==============================] - 2091s 2s/step - loss: 0.2657 - tp: 21903456.0000 - fp: 2055350.0000 - tn: 78244200.0000 - fn: 2163118.0000 - precision: 0.9142 - recall: 0.9101 - accuracy: 0.9596 - auc: 0.9816 - val_loss: 0.29 28 - val_tp: 5111563.0000 - val_fp: 355341.0000 - val_tn: 19707988.0000 - val_fn: 629909.0000 - val_precision: 0.9350 - val_recall: 0.8903 - val_accuracy: 0.9618 - val_auc: 0.9825 Epoch 30/40 910/910 [==============================] - 2097s 2s/step - loss: 0.2663 - tp: 22007504.0000 - fp: 2072930.0000 - tn: 78112200.0000 - fn: 2173470.0000 - precision: 0.9139 - recall: 0.9101 - accuracy: 0.9593 - auc: 0.9816 - val_loss: 0.30 39 - val_tp: 5361046.0000 - val_fp: 369167.0000 - val_tn: 19426364.0000 - val_fn: 648220.0000 - val_precision: 0.9356 - val_recall: 0.8921 - val_accuracy: 0.9606 - val_auc: 0.9831 Epoch 31/40 910/910 [==============================] - 2092s 2s/step - loss: 0.2670 - tp: 21883968.0000 - fp: 2114417.0000 - tn: 78182464.0000 - fn: 2185257.0000 - precision: 0.9119 - recall: 0.9092 - accuracy: 0.9588 - auc: 0.9816 - val_loss: 0.29 74 - val_tp: 5307833.0000 - val_fp: 308903.0000 - val_tn: 19507264.0000 - val_fn: 680801.0000 - val_precision: 0.9450 - val_recall: 0.8863 - val_accuracy: 0.9616 - val_auc: 0.9830 Epoch 32/40 910/910 [==============================] - 2168s 2s/step - loss: 0.2678 - tp: 21697564.0000 - fp: 2002839.0000 - tn: 78462496.0000 - fn: 2203202.0000 - precision: 0.9155 - recall: 0.9078 - accuracy: 0.9597 - auc: 0.9811 - val_loss: 0.29 81 - val_tp: 4991380.0000 - val_fp: 318033.0000 - val_tn: 19848366.0000 - val_fn: 647024.0000 - val_precision: 0.9401 - val_recall: 0.8852 - val_accuracy: 0.9626 - val_auc: 0.9819 Epoch 33/40 910/910 [==============================] - 2143s 2s/step - loss: 0.2641 - tp: 21608556.0000 - fp: 2029292.0000 - tn: 78569336.0000 - fn: 2158886.0000 - precision: 0.9142 - recall: 0.9092 - accuracy: 0.9599 - auc: 0.9815 - val_loss: 0.30 89 - val_tp: 5377773.0000 - val_fp: 383774.0000 - val_tn: 19419054.0000 - val_fn: 624197.0000 - val_precision: 0.9334 - val_recall: 0.8960 - val_accuracy: 0.9609 - val_auc: 0.9835 Epoch 34/40 910/910 [==============================] - 2148s 2s/step - loss: 0.2636 - tp: 21989192.0000 - fp: 2079253.0000 - tn: 78164408.0000 - fn: 2133219.0000 - precision: 0.9136 - recall: 0.9116 - accuracy: 0.9596 - auc: 0.9818 - val_loss: 0.28 72 - val_tp: 5242507.0000 - val_fp: 383622.0000 - val_tn: 19573328.0000 - val_fn: 605341.0000 - val_precision: 0.9318 - val_recall: 0.8965 - val_accuracy: 0.9617 - val_auc: 0.9836 Epoch 35/40 910/910 [==============================] - 2156s 2s/step - loss: 0.2642 - tp: 21771006.0000 - fp: 2059886.0000 - tn: 78393632.0000 - fn: 2141526.0000 - precision: 0.9136 - recall: 0.9104 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0.30 80 - val_tp: 5192984.0000 - val_fp: 352885.0000 - val_tn: 19633610.0000 - val_fn: 625321.0000 - val_precision: 0.9364 - val_recall: 0.8925 - val_accuracy: 0.9621 - val_auc: 0.9832 Epoch 36/40 910/910 [==============================] - 2114s 2s/step - loss: 0.2641 - tp: 21936120.0000 - fp: 2076747.0000 - tn: 78200080.0000 - fn: 2153085.0000 - precision: 0.9135 - recall: 0.9106 - accuracy: 0.9595 - auc: 0.9818 - val_loss: 0.29 62 - val_tp: 5151439.0000 - val_fp: 372261.0000 - val_tn: 19674524.0000 - val_fn: 606575.0000 - val_precision: 0.9326 - val_recall: 0.8947 - val_accuracy: 0.9621 - val_auc: 0.9835 Epoch 37/40 910/910 [==============================] - 2083s 2s/step - loss: 0.2591 - tp: 21745408.0000 - fp: 2014129.0000 - tn: 78506376.0000 - fn: 2100146.0000 - precision: 0.9152 - recall: 0.9119 - accuracy: 0.9606 - auc: 0.9821 - val_loss: 0.28 98 - val_tp: 5332435.0000 - val_fp: 349131.0000 - val_tn: 19493176.0000 - val_fn: 630050.0000 - val_precision: 0.9386 - val_recall: 0.8943 - val_accuracy: 0.9621 - val_auc: 0.9841 Epoch 38/40 910/910 [==============================] - 2067s 2s/step - loss: 0.2677 - tp: 22040308.0000 - fp: 2034232.0000 - tn: 78099864.0000 - fn: 2191664.0000 - precision: 0.9155 - recall: 0.9096 - accuracy: 0.9595 - auc: 0.9813 - val_loss: 0.30 58 - val_tp: 5293916.0000 - val_fp: 364251.0000 - val_tn: 19503016.0000 - val_fn: 643618.0000 - val_precision: 0.9356 - val_recall: 0.8916 - val_accuracy: 0.9609 - val_auc: 0.9818 Epoch 39/40 910/910 [==============================] - 2088s 2s/step - loss: 0.2665 - tp: 21782964.0000 - fp: 2071501.0000 - tn: 78364600.0000 - fn: 2146992.0000 - precision: 0.9132 - recall: 0.9103 - accuracy: 0.9596 - auc: 0.9812 - val_loss: 0.28 94 - val_tp: 5190450.0000 - val_fp: 358739.0000 - val_tn: 19624484.0000 - val_fn: 631125.0000 - val_precision: 0.9354 - val_recall: 0.8916 - val_accuracy: 0.9616 - val_auc: 0.9823 Epoch 40/40 910/910 [==============================] - 2093s 2s/step - loss: 0.2610 - tp: 21703476.0000 - fp: 2048057.0000 - tn: 78489664.0000 - fn: 2124881.0000 - precision: 0.9138 - recall: 0.9108 - accuracy: 0.9600 - auc: 0.9819 - val_loss: 0.28 97 - val_tp: 5014376.0000 - val_fp: 339650.0000 - val_tn: 19831600.0000 - val_fn: 619172.0000 - val_precision: 0.9366 - val_recall: 0.8901 - val_accuracy: 0.9628 - val_auc: 0.9828 286/286 [==============================] - 134s 469ms/step - loss: 0.3156 - tp: 6887079.0000 - fp: 534559.0000 - tn: 24555692.0000 - fn: 790657.0000 - precision: 0.9280 - recall: 0.8970 - accuracy: 0.9596 - auc: 0.9821 2020/06/23 01:58:39 INFO mlflow.projects: === Run (ID '306234c75c9c48058cbd694579eff31b') succeeded ===
2.1.17 git exp 5
git status git log -1
(base) [ye53nis@node011 drmed-git]$ git status
git log -1
# On branch exp-310520-unet
# ...
no changes added to commit (use "git add" and/or "git commit -a")
(base) [ye53nis@node011 drmed-git]$ git log -1
commit 8811f54920c7089b8a27d7f39a50acede5be64c9
Author: Apoplex <oligolex@vivaldi.net>
Date: Fri Jul 3 00:51:59 2020 +0200
Incorporate unet prediction in plotting function
2.1.18 experimental run 5 - full dataset, length=2**13=8192
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=5 -P length_delimiter=8192 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=1280 -P validation_steps=320
(tensorflow_nightly) [ye53nis@node011 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=5 -P length_delimiter=8192 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact _Sep2019 -P steps_per_epoch=1280 -P validation_steps=320 2020/07/03 13:45:30 INFO mlflow.projects: === Created directory /tmp/tmpqf8ewz0u for downloading remote URIs passed to arguments of type 'path' === 2020/07/03 13:45:30 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py / beegfs/ye53nis/drmed-git/src 5 0.2 8192 None 40 /beegfs/ye53nis/saves/firstartefact_Sep2019 1280 320' in run with ID 'd9b44dc2e3d44ea1a71129808b642af6' === 2020-07-03 13:45:57.151501: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2020-07-03 13:45:57.151600: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul e's documentation for alternative uses import imp 2.3.0-dev20200527 2020-07-03 13:46:34.363762: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-07-03 13:46:34.363823: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303) 2020-07-03 13:46:34.363865: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node011): /proc/driver/nvidia/version does not exist GPUs: [] train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000) shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000) for each 20,000 timestap trace there are the following numbers of corrupted timesteps: label001_1 6286 label001_1 2568 label001_1 4495 label001_1 4414 label001_1 1105 dtype: int64 2020-07-03 13:52:37.767508: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-07-03 13:52:37.782937: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2194930000 Hz 2020-07-03 13:52:37.784247: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e467051fe0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-07-03 13:52:37.784290: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version number of training examples: 6400, number of validation examples: 1600 ------------------------ number of test examples: 2000 input - shape: (None, 8192, 1) output - shape: (None, 8192, 1) 2020-07-03 13:52:43.215627: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(wrapped_dict, collections.Mapping): /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec() all_param_names, _, _, all_default_values = inspect.getargspec(fn) # pylint: disable=W1505 Epoch 1/40 /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working if not isinstance(values, collections.Sequence): 1/1280 [..............................] - ETA: 0s - loss: 1.2862 - tp: 16168.0000 - fp: 12373.0000 - tn: 7437.0000 - fn: 4982.0000 - precision: 0.5665 - recall: 0.7644 - accuracy: 0.5763 - auc: 0.63882020-07-03 13:52:58.138783: I ten sorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started. WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w ill be removed after 2020-07-01. Instructions for updating: use `tf.profiler.experimental.stop` instead. 2020-07-03 13:52:59.481566: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_07_03_13_52_59 2020-07-03 13:52:59.501564: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.trace.json.gz 2020-07-03 13:52:59.538369: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_07_03_13_52_59 2020-07-03 13:52:59.538526: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.memory_profile.json.gz 2020-07-03 13:52:59.541206: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_07_03_13_52_59Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_07_03_1 3_52_59/node011.xplane.pb Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.overview_page.pb Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.kernel_stats.pb 1280/1280 [==============================] - 1780s 1s/step - loss: 1.1858 - tp: 7339984.0000 - fp: 2874113.0000 - tn: 37719448.0000 - fn: 4495237.0000 - precision: 0.7186 - recall: 0.6202 - accuracy: 0.8594 - auc: 0.8832 - val_loss: 2.1 472 - val_tp: 2536431.0000 - val_fp: 7204421.0000 - val_tn: 3012796.0000 - val_fn: 353552.0000 - val_precision: 0.2604 - val_recall: 0.8777 - val_accuracy: 0.4234 - val_auc: 0.7624 Epoch 2/40 1280/1280 [==============================] - 1779s 1s/step - loss: 1.0617 - tp: 7423177.0000 - fp: 2931677.0000 - tn: 37621800.0000 - fn: 4452150.0000 - precision: 0.7169 - recall: 0.6251 - accuracy: 0.8592 - auc: 0.8778 - val_loss: 3.9 428 - val_tp: 2825884.0000 - val_fp: 7982935.0000 - val_tn: 2238761.0000 - val_fn: 59620.0000 - val_precision: 0.2614 - val_recall: 0.9793 - val_accuracy: 0.3864 - val_auc: 0.8231 Epoch 3/40 1280/1280 [==============================] - 1777s 1s/step - loss: 0.7857 - tp: 8301792.0000 - fp: 2630361.0000 - tn: 37976972.0000 - fn: 3519672.0000 - precision: 0.7594 - recall: 0.7023 - accuracy: 0.8827 - auc: 0.9097 - val_loss: 66. 2140 - val_tp: 3034899.0000 - val_fp: 9956908.0000 - val_tn: 115041.0000 - val_fn: 352.0000 - val_precision: 0.2336 - val_recall: 0.9999 - val_accuracy: 0.2403 - val_auc: 0.5115 Epoch 4/40 1280/1280 [==============================] - 1760s 1s/step - loss: 0.6022 - tp: 9369523.0000 - fp: 2098170.0000 - tn: 38477736.0000 - fn: 2483339.0000 - precision: 0.8170 - recall: 0.7905 - accuracy: 0.9126 - auc: 0.9419 - val_loss: 1.4 866 - val_tp: 2661698.0000 - val_fp: 4816937.0000 - val_tn: 5366742.0000 - val_fn: 261823.0000 - val_precision: 0.3559 - val_recall: 0.9104 - val_accuracy: 0.6125 - val_auc: 0.8933 Epoch 5/40 1280/1280 [==============================] - 1771s 1s/step - loss: 0.5592 - tp: 9674903.0000 - fp: 1962418.0000 - tn: 38513608.0000 - fn: 2277853.0000 - precision: 0.8314 - recall: 0.8094 - accuracy: 0.9191 - auc: 0.9488 - val_loss: 1.0 996 - val_tp: 565756.0000 - val_fp: 6305.0000 - val_tn: 10145255.0000 - val_fn: 2389884.0000 - val_precision: 0.9890 - val_recall: 0.1914 - val_accuracy: 0.8172 - val_auc: 0.9230 Epoch 6/40 1280/1280 [==============================] - 1774s 1s/step - loss: 0.5210 - tp: 9808780.0000 - fp: 1924251.0000 - tn: 38545780.0000 - fn: 2149975.0000 - precision: 0.8360 - recall: 0.8202 - accuracy: 0.9223 - auc: 0.9528 - val_loss: 1.9 950 - val_tp: 2773239.0000 - val_fp: 6335919.0000 - val_tn: 3933010.0000 - val_fn: 65032.0000 - val_precision: 0.3044 - val_recall: 0.9771 - val_accuracy: 0.5116 - val_auc: 0.9026 Epoch 7/40 1280/1280 [==============================] - 1767s 1s/step - loss: 0.4799 - tp: 9927548.0000 - fp: 1743956.0000 - tn: 38779680.0000 - fn: 1977563.0000 - precision: 0.8506 - recall: 0.8339 - accuracy: 0.9290 - auc: 0.9587 - val_loss: 0.4 051 - val_tp: 2537115.0000 - val_fp: 260830.0000 - val_tn: 9808854.0000 - val_fn: 500401.0000 - val_precision: 0.9068 - val_recall: 0.8353 - val_accuracy: 0.9419 - val_auc: 0.9654 Epoch 8/40 1280/1280 [==============================] - 1771s 1s/step - loss: 0.4655 - tp: 9972310.0000 - fp: 1610651.0000 - tn: 38928064.0000 - fn: 1917779.0000 - precision: 0.8609 - recall: 0.8387 - accuracy: 0.9327 - auc: 0.9611 - val_loss: 1.6 014 - val_tp: 1146828.0000 - val_fp: 2196.0000 - val_tn: 10286639.0000 - val_fn: 1671537.0000 - val_precision: 0.9981 - val_recall: 0.4069 - val_accuracy: 0.8723 - val_auc: 0.7663 Epoch 9/40 1280/1280 [==============================] - 1770s 1s/step - loss: 0.4410 - tp: 9975626.0000 - fp: 1547724.0000 - tn: 39046088.0000 - fn: 1859375.0000 - precision: 0.8657 - recall: 0.8429 - accuracy: 0.9350 - auc: 0.9628 - val_loss: 0.6 413 - val_tp: 2584588.0000 - val_fp: 554056.0000 - val_tn: 9603651.0000 - val_fn: 364905.0000 - val_precision: 0.8235 - val_recall: 0.8763 - val_accuracy: 0.9299 - val_auc: 0.9693 Epoch 10/40 1280/1280 [==============================] - 1761s 1s/step - loss: 0.4217 - tp: 10018262.0000 - fp: 1432905.0000 - tn: 39172740.0000 - fn: 1804910.0000 - precision: 0.8749 - recall: 0.8473 - accuracy: 0.9382 - auc: 0.9649 - val_loss: 0. 6441 - val_tp: 1928738.0000 - val_fp: 27586.0000 - val_tn: 10052344.0000 - val_fn: 1098532.0000 - val_precision: 0.9859 - val_recall: 0.6371 - val_accuracy: 0.9141 - val_auc: 0.9337 Epoch 11/40 1280/1280 [==============================] - 1767s 1s/step - loss: 0.4095 - tp: 10121675.0000 - fp: 1466373.0000 - tn: 39131704.0000 - fn: 1709053.0000 - precision: 0.8735 - recall: 0.8555 - accuracy: 0.9394 - auc: 0.9669 - val_loss: 1. 0952 - val_tp: 2779103.0000 - val_fp: 2929574.0000 - val_tn: 7332225.0000 - val_fn: 66298.0000 - val_precision: 0.4868 - val_recall: 0.9767 - val_accuracy: 0.7714 - val_auc: 0.9444 Epoch 12/40 1280/1280 [==============================] - 1766s 1s/step - loss: 0.3508 - tp: 10558538.0000 - fp: 1308014.0000 - tn: 39148768.0000 - fn: 1413463.0000 - precision: 0.8898 - recall: 0.8819 - accuracy: 0.9481 - auc: 0.9727 - val_loss: 0. 4481 - val_tp: 2197978.0000 - val_fp: 77726.0000 - val_tn: 10018559.0000 - val_fn: 812937.0000 - val_precision: 0.9658 - val_recall: 0.7300 - val_accuracy: 0.9320 - val_auc: 0.9684 Epoch 13/40 1280/1280 [==============================] - 1768s 1s/step - loss: 0.3459 - tp: 10634920.0000 - fp: 1277214.0000 - tn: 39126304.0000 - fn: 1390329.0000 - precision: 0.8928 - recall: 0.8844 - accuracy: 0.9491 - auc: 0.9732 - val_loss: 0. 4022 - val_tp: 2391272.0000 - val_fp: 184993.0000 - val_tn: 9955529.0000 - val_fn: 575406.0000 - val_precision: 0.9282 - val_recall: 0.8060 - val_accuracy: 0.9420 - val_auc: 0.9744 Epoch 14/40 1280/1280 [==============================] - 1771s 1s/step - loss: 0.3308 - tp: 10495827.0000 - fp: 1232895.0000 - tn: 39364996.0000 - fn: 1335085.0000 - precision: 0.8949 - recall: 0.8872 - accuracy: 0.9510 - auc: 0.9745 - val_loss: 0. 3876 - val_tp: 2364208.0000 - val_fp: 159464.0000 - val_tn: 9940852.0000 - val_fn: 642676.0000 - val_precision: 0.9368 - val_recall: 0.7863 - val_accuracy: 0.9388 - val_auc: 0.9704 Epoch 15/40 1280/1280 [==============================] - 1778s 1s/step - loss: 0.3349 - tp: 10513378.0000 - fp: 1247874.0000 - tn: 39301756.0000 - fn: 1365808.0000 - precision: 0.8939 - recall: 0.8850 - accuracy: 0.9501 - auc: 0.9742 - val_loss: 0. 4307 - val_tp: 2108827.0000 - val_fp: 89619.0000 - val_tn: 10175358.0000 - val_fn: 733396.0000 - val_precision: 0.9592 - val_recall: 0.7420 - val_accuracy: 0.9372 - val_auc: 0.9676 Epoch 16/40 1280/1280 [==============================] - 1774s 1s/step - loss: 0.3307 - tp: 10595335.0000 - fp: 1236928.0000 - tn: 39255112.0000 - fn: 1341378.0000 - precision: 0.8955 - recall: 0.8876 - accuracy: 0.9508 - auc: 0.9744 - val_loss: 0. 4616 - val_tp: 2311103.0000 - val_fp: 115583.0000 - val_tn: 10000757.0000 - val_fn: 679757.0000 - val_precision: 0.9524 - val_recall: 0.7727 - val_accuracy: 0.9393 - val_auc: 0.9704 Epoch 17/40 1280/1280 [==============================] - 1767s 1s/step - loss: 0.3289 - tp: 10535633.0000 - fp: 1223924.0000 - tn: 39330720.0000 - fn: 1338507.0000 - precision: 0.8959 - recall: 0.8873 - accuracy: 0.9511 - auc: 0.9748 - val_loss: 0. 4047 - val_tp: 2225304.0000 - val_fp: 97248.0000 - val_tn: 10022325.0000 - val_fn: 762323.0000 - val_precision: 0.9581 - val_recall: 0.7448 - val_accuracy: 0.9344 - val_auc: 0.9659 Epoch 18/40 1280/1280 [==============================] - 1772s 1s/step - loss: 0.3248 - tp: 10613594.0000 - fp: 1201191.0000 - tn: 39285904.0000 - fn: 1328110.0000 - precision: 0.8983 - recall: 0.8888 - accuracy: 0.9518 - auc: 0.9753 - val_loss: 0. 5139 - val_tp: 2072016.0000 - val_fp: 36763.0000 - val_tn: 10080347.0000 - val_fn: 918074.0000 - val_precision: 0.9826 - val_recall: 0.6930 - val_accuracy: 0.9272 - val_auc: 0.9530 Epoch 19/40 1280/1280 [==============================] - 1774s 1s/step - loss: 0.3211 - tp: 10667086.0000 - fp: 1209967.0000 - tn: 39238632.0000 - fn: 1313139.0000 - precision: 0.8981 - recall: 0.8904 - accuracy: 0.9519 - auc: 0.9753 - val_loss: 0. 3749 - val_tp: 2493058.0000 - val_fp: 176033.0000 - val_tn: 9926603.0000 - val_fn: 511506.0000 - val_precision: 0.9340 - val_recall: 0.8298 - val_accuracy: 0.9475 - val_auc: 0.9723 Epoch 20/40 1280/1280 [==============================] - 1771s 1s/step - loss: 0.3176 - tp: 10604079.0000 - fp: 1175083.0000 - tn: 39349048.0000 - fn: 1300586.0000 - precision: 0.9002 - recall: 0.8907 - accuracy: 0.9528 - auc: 0.9758 - val_loss: 0. 4352 - val_tp: 2225431.0000 - val_fp: 169949.0000 - val_tn: 9987117.0000 - val_fn: 724703.0000 - val_precision: 0.9291 - val_recall: 0.7543 - val_accuracy: 0.9317 - val_auc: 0.9660 Epoch 21/40 1280/1280 [==============================] - 1777s 1s/step - loss: 0.3183 - tp: 10753725.0000 - fp: 1207924.0000 - tn: 39171120.0000 - fn: 1296058.0000 - precision: 0.8990 - recall: 0.8924 - accuracy: 0.9522 - auc: 0.9760 - val_loss: 0. 5143 - val_tp: 1923402.0000 - val_fp: 8402.0000 - val_tn: 10080301.0000 - val_fn: 1095095.0000 - val_precision: 0.9957 - val_recall: 0.6372 - val_accuracy: 0.9158 - val_auc: 0.9559 Epoch 22/40 1280/1280 [==============================] - 1772s 1s/step - loss: 0.3155 - tp: 10593298.0000 - fp: 1171798.0000 - tn: 39369164.0000 - fn: 1294568.0000 - precision: 0.9004 - recall: 0.8911 - accuracy: 0.9530 - auc: 0.9766 - val_loss: 0. 3698 - val_tp: 2362795.0000 - val_fp: 98123.0000 - val_tn: 10020849.0000 - val_fn: 625433.0000 - val_precision: 0.9601 - val_recall: 0.7907 - val_accuracy: 0.9448 - val_auc: 0.9722 Epoch 23/40 1280/1280 [==============================] - 1770s 1s/step - loss: 0.3081 - tp: 10733583.0000 - fp: 1175054.0000 - tn: 39255684.0000 - fn: 1264464.0000 - precision: 0.9013 - recall: 0.8946 - accuracy: 0.9535 - auc: 0.9772 - val_loss: 0. 4154 - val_tp: 2216921.0000 - val_fp: 66152.0000 - val_tn: 10061258.0000 - val_fn: 762869.0000 - val_precision: 0.9710 - val_recall: 0.7440 - val_accuracy: 0.9368 - val_auc: 0.9674 Epoch 24/40 1280/1280 [==============================] - 1773s 1s/step - loss: 0.3108 - tp: 10737070.0000 - fp: 1185865.0000 - tn: 39247804.0000 - fn: 1258035.0000 - precision: 0.9005 - recall: 0.8951 - accuracy: 0.9534 - auc: 0.9769 - val_loss: 0. 4095 - val_tp: 2189914.0000 - val_fp: 65716.0000 - val_tn: 10122627.0000 - val_fn: 728943.0000 - val_precision: 0.9709 - val_recall: 0.7503 - val_accuracy: 0.9394 - val_auc: 0.9691 Epoch 25/40 1280/1280 [==============================] - 1778s 1s/step - loss: 0.3092 - tp: 10574009.0000 - fp: 1168082.0000 - tn: 39411864.0000 - fn: 1274842.0000 - precision: 0.9005 - recall: 0.8924 - accuracy: 0.9534 - auc: 0.9769 - val_loss: 0. 3740 - val_tp: 2271238.0000 - val_fp: 99057.0000 - val_tn: 10104250.0000 - val_fn: 632655.0000 - val_precision: 0.9582 - val_recall: 0.7821 - val_accuracy: 0.9442 - val_auc: 0.9706 Epoch 26/40 1280/1280 [==============================] - 1772s 1s/step - loss: 0.3127 - tp: 10661534.0000 - fp: 1178414.0000 - tn: 39303136.0000 - fn: 1285691.0000 - precision: 0.9005 - recall: 0.8924 - accuracy: 0.9530 - auc: 0.9766 - val_loss: 0. 4100 - val_tp: 2160141.0000 - val_fp: 93000.0000 - val_tn: 10111722.0000 - val_fn: 742337.0000 - val_precision: 0.9587 - val_recall: 0.7442 - val_accuracy: 0.9363 - val_auc: 0.9656 Epoch 27/40 1280/1280 [==============================] - 1772s 1s/step - loss: 0.3054 - tp: 10663139.0000 - fp: 1174175.0000 - tn: 39348628.0000 - fn: 1242863.0000 - precision: 0.9008 - recall: 0.8956 - accuracy: 0.9539 - auc: 0.9770 - val_loss: 0. 3826 - val_tp: 2295025.0000 - val_fp: 98538.0000 - val_tn: 10045922.0000 - val_fn: 667715.0000 - val_precision: 0.9588 - val_recall: 0.7746 - val_accuracy: 0.9415 - val_auc: 0.9695 Epoch 28/40 1280/1280 [==============================] - 1766s 1s/step - loss: 0.3101 - tp: 10638913.0000 - fp: 1161985.0000 - tn: 39375784.0000 - fn: 1252135.0000 - precision: 0.9015 - recall: 0.8947 - accuracy: 0.9540 - auc: 0.9766 - val_loss: 0. 3907 - val_tp: 2166669.0000 - val_fp: 37875.0000 - val_tn: 10143155.0000 - val_fn: 759501.0000 - val_precision: 0.9828 - val_recall: 0.7404 - val_accuracy: 0.9392 - val_auc: 0.9686 Epoch 29/40 1280/1280 [==============================] - 1773s 1s/step - loss: 0.3063 - tp: 10554201.0000 - fp: 1162046.0000 - tn: 39456136.0000 - fn: 1256420.0000 - precision: 0.9008 - recall: 0.8936 - accuracy: 0.9539 - auc: 0.9767 - val_loss: 0. 3597 - val_tp: 2286152.0000 - val_fp: 91290.0000 - val_tn: 10085049.0000 - val_fn: 644709.0000 - val_precision: 0.9616 - val_recall: 0.7800 - val_accuracy: 0.9438 - val_auc: 0.9718 Epoch 30/40 1280/1280 [==============================] - 1777s 1s/step - loss: 0.3054 - tp: 10552880.0000 - fp: 1163086.0000 - tn: 39473676.0000 - fn: 1239134.0000 - precision: 0.9007 - recall: 0.8949 - accuracy: 0.9542 - auc: 0.9771 - val_loss: 0. 4182 - val_tp: 2195277.0000 - val_fp: 32841.0000 - val_tn: 10111337.0000 - val_fn: 767745.0000 - val_precision: 0.9853 - val_recall: 0.7409 - val_accuracy: 0.9389 - val_auc: 0.9704 Epoch 31/40 1280/1280 [==============================] - 1779s 1s/step - loss: 0.3083 - tp: 10553649.0000 - fp: 1149510.0000 - tn: 39466480.0000 - fn: 1259216.0000 - precision: 0.9018 - recall: 0.8934 - accuracy: 0.9541 - auc: 0.9765 - val_loss: 0. 4366 - val_tp: 2167915.0000 - val_fp: 58039.0000 - val_tn: 10066034.0000 - val_fn: 815212.0000 - val_precision: 0.9739 - val_recall: 0.7267 - val_accuracy: 0.9334 - val_auc: 0.9673 Epoch 32/40 1280/1280 [==============================] - 1768s 1s/step - loss: 0.3090 - tp: 10850631.0000 - fp: 1193472.0000 - tn: 39117152.0000 - fn: 1267532.0000 - precision: 0.9009 - recall: 0.8954 - accuracy: 0.9531 - auc: 0.9774 - val_loss: 0. 3760 - val_tp: 2399785.0000 - val_fp: 89881.0000 - val_tn: 9935425.0000 - val_fn: 682109.0000 - val_precision: 0.9639 - val_recall: 0.7787 - val_accuracy: 0.9411 - val_auc: 0.9706 Epoch 33/40 1280/1280 [==============================] - 1770s 1s/step - loss: 0.3075 - tp: 10591815.0000 - fp: 1173635.0000 - tn: 39401344.0000 - fn: 1261969.0000 - precision: 0.9002 - recall: 0.8935 - accuracy: 0.9535 - auc: 0.9770 - val_loss: 0. 3749 - val_tp: 2266851.0000 - val_fp: 113739.0000 - val_tn: 10069260.0000 - val_fn: 657350.0000 - val_precision: 0.9522 - val_recall: 0.7752 - val_accuracy: 0.9412 - val_auc: 0.9685 Epoch 34/40 1280/1280 [==============================] - 1774s 1s/step - loss: 0.3046 - tp: 10617309.0000 - fp: 1139098.0000 - tn: 39400540.0000 - fn: 1271864.0000 - precision: 0.9031 - recall: 0.8930 - accuracy: 0.9540 - auc: 0.9772 - val_loss: 0. 4175 - val_tp: 2241531.0000 - val_fp: 87208.0000 - val_tn: 10047153.0000 - val_fn: 731308.0000 - val_precision: 0.9626 - val_recall: 0.7540 - val_accuracy: 0.9376 - val_auc: 0.9662 Epoch 35/40 1280/1280 [==============================] - 1773s 1s/step - loss: 0.3112 - tp: 10604940.0000 - fp: 1177338.0000 - tn: 39382680.0000 - fn: 1263836.0000 - precision: 0.9001 - recall: 0.8935 - accuracy: 0.9534 - auc: 0.9763 - val_loss: 0. 3769 - val_tp: 2350085.0000 - val_fp: 99126.0000 - val_tn: 10001938.0000 - val_fn: 656051.0000 - val_precision: 0.9595 - val_recall: 0.7818 - val_accuracy: 0.9424 - val_auc: 0.9697 Epoch 36/40 1280/1280 [==============================] - 1782s 1s/step - loss: 0.3079 - tp: 10576699.0000 - fp: 1181888.0000 - tn: 39421312.0000 - fn: 1248879.0000 - precision: 0.8995 - recall: 0.8944 - accuracy: 0.9536 - auc: 0.9767 - val_loss: 0. 3834 - val_tp: 2322480.0000 - val_fp: 71956.0000 - val_tn: 10012406.0000 - val_fn: 700358.0000 - val_precision: 0.9699 - val_recall: 0.7683 - val_accuracy: 0.9411 - val_auc: 0.9705 Epoch 37/40 1280/1280 [==============================] - 1775s 1s/step - loss: 0.3129 - tp: 10722707.0000 - fp: 1163322.0000 - tn: 39255528.0000 - fn: 1287232.0000 - precision: 0.9021 - recall: 0.8928 - accuracy: 0.9533 - auc: 0.9765 - val_loss: 0. 4019 - val_tp: 2314261.0000 - val_fp: 62408.0000 - val_tn: 9985730.0000 - val_fn: 744801.0000 - val_precision: 0.9737 - val_recall: 0.7565 - val_accuracy: 0.9384 - val_auc: 0.9702 Epoch 38/40 1280/1280 [==============================] - 1783s 1s/step - loss: 0.3083 - tp: 10594205.0000 - fp: 1165411.0000 - tn: 39412032.0000 - fn: 1257165.0000 - precision: 0.9009 - recall: 0.8939 - accuracy: 0.9538 - auc: 0.9767 - val_loss: 0. 3835 - val_tp: 2223637.0000 - val_fp: 52165.0000 - val_tn: 10117258.0000 - val_fn: 714140.0000 - val_precision: 0.9771 - val_recall: 0.7569 - val_accuracy: 0.9415 - val_auc: 0.9715 Epoch 39/40 1280/1280 [==============================] - 1776s 1s/step - loss: 0.3073 - tp: 10690271.0000 - fp: 1156059.0000 - tn: 39323200.0000 - fn: 1259274.0000 - precision: 0.9024 - recall: 0.8946 - accuracy: 0.9539 - auc: 0.9769 - val_loss: 0. 3965 - val_tp: 2204023.0000 - val_fp: 67630.0000 - val_tn: 10139880.0000 - val_fn: 695667.0000 - val_precision: 0.9702 - val_recall: 0.7601 - val_accuracy: 0.9418 - val_auc: 0.9679 Epoch 40/40 1280/1280 [==============================] - 1771s 1s/step - loss: 0.3049 - tp: 10682842.0000 - fp: 1183309.0000 - tn: 39323412.0000 - fn: 1239227.0000 - precision: 0.9003 - recall: 0.8961 - accuracy: 0.9538 - auc: 0.9773 - val_loss: 0. 3758 - val_tp: 2309020.0000 - val_fp: 116124.0000 - val_tn: 10004803.0000 - val_fn: 677253.0000 - val_precision: 0.9521 - val_recall: 0.7732 - val_accuracy: 0.9395 - val_auc: 0.9702 400/400 [==============================] - 118s 296ms/step - loss: 0.4356 - tp: 2969562.0000 - fp: 289346.0000 - tn: 12303178.0000 - fn: 821914.0000 - precision: 0.9112 - recall: 0.7832 - accuracy: 0.9322 - auc: 0.9648 2020/07/04 09:37:42 INFO mlflow.projects: === Run (ID 'd9b44dc2e3d44ea1a71129808b642af6') succeeded ===