Commit 764539fb authored by Manuela Kuhn's avatar Manuela Kuhn

Merge branch 'release-3.1.0'

parents 5a8d779f 645547c9
[bumpversion]
current_version = 3.0.2
current_version = 3.1.0
tag = False
commit = False
files = freeze_setup.py src/APIs/hidra/_version.py src/shared/_version.py
......@@ -12,6 +12,6 @@ replace = HiDRA develop
[bumpversion:file:hidra.spec]
search = %changelog
replace = %changelog
{now:%a %b %d %Y} Manuela Kuhn <manuela.kuhn@desy.de> - {new_version}-1
Bump version
* {now:%a %b %d %Y} Manuela Kuhn <manuela.kuhn@desy.de> - {new_version}-1
- Bump version
[bumpversion]
current_version = 3.0.2
current_version = 3.1.0
tag = False
commit = False
......
FROM ubuntu
RUN apt update && apt install -y \
git \
python \
python-pip
RUN pip install --upgrade pip && pip install \
pyzmq
RUN cd /opt && git clone https://stash.desy.de/scm/hidra/hidra.git
RUN cd /opt/hidra && git checkout develop
RUN cp -r /opt/hidra/src/APIs/hidra /usr/local/lib/python2.7/dist-packages/
FROM ubuntu
RUN apt update && apt install -y \
git \
python \
python-pip \
vim
RUN pip install --upgrade pip && pip install \
pyzmq \
setproctitle \
six
RUN cd /opt && \
git clone https://stash.desy.de/scm/hidra/hidra.git && \
cd hidra && \
git checkout develop && \
mkdir /var/log/hidra
ENTRYPOINT ["python", "/opt/hidra/src/receiver/datareceiver.py", "--verbose", "--config_file", "/opt/hidra/conf/datareceiver_docker.conf"]
FROM ubuntu
RUN apt update && apt install -y \
git \
python \
python-pip \
vim \
iputils-ping
RUN pip install --upgrade pip && pip install \
pyzmq \
setproctitle \
six \
inotifyx
RUN cd /opt && \
git clone https://stash.desy.de/scm/hidra/hidra.git && \
cd hidra && \
git checkout develop && \
mkdir /var/log/hidra
ENTRYPOINT ["python", "/opt/hidra/src/sender/datamanager.py", "--verbose", "--config_file", "/opt/hidra/conf/datamanager_docker.conf"]
Dependencies:
Python moduls:
- pyzmq, version 14.5.0 or newer
- setproctitle
- six
- inotifyx (for inotifyx event detector: file transfer on Linux)
- watchdog (for watchdog event detector: file transfer on Windows)
- requests (for HTTP event detector, used for Eiger)
for further desciptions see:
https://confluence.desy.de/display/hidra/HiDRA
HiDRA
======
__HiDRA__ is a generic tool set for high performance data multiplexing with different qualities of service
and is based on __Python__ and __ZeroMQ__. It can be used to directly store the data in the storage
system but also to send it to some kind of online monitoring or analysis framework. Together
with [OnDA] (https://github.com/ondateam/onda), data can be analyzed with a delay of seconds resulting in
an increase of the quality of the generated scientific data by 20 %. The modular architecture of
the tool (divided into event detectors, data fetchers and receivers) makes it easily extendible and
even gives the possibility to adapt the software to specific detectors directly (for example, Eiger
and Lambda detector).
__HiDRA__ (High Data Rate Access) is a generic tool set for high performance data multiplexing with different qualities of service and based on __Python__ and __ZeroMQ__.
On the one hand, it was developed to support 20 or more of the next generation detectors generating data up to 10GB/sec and images with kHz frequencies.
On the other hand, its purpose is to decouple persistent storage and selective image collection to support next generations of experiment setups, where the experiment conditions have to be monitored/analyzed in close to real time to prevent wasting the precious sample. <br />
This open source and facility independent software can be used to store the data directly in the storage system but also to send it to some kind of online monitoring or analysis framework.
Together with the tool [OnDA] (https://stash.desy.de/projects/ONDA), data can be analyzed with a delay of seconds resulting in test to an increase of the quality of the generated scientific data by 20 %.
The modular architecture of the tool makes it easily extendible and even gives the possibility to adapt the software to specific detectors directly (e.g. Eiger detector).
Contributors
============
......@@ -22,53 +21,4 @@ The project is licensed under __AGPL v3__.
Getting Started
===============
The file [INSTALL.md](INSTALL.md) describes how to compile HiDRA code and build various packages.
How to contribute
=================
**HiDRA** uses the linux kernel model where git is not only source repository,
but also the way to track contributions and copyrights.
Each submitted patch must have a "Signed-off-by" line. Patches without
this line will not be accepted.
The sign-off is a simple line at the end of the explanation for the
patch, which certifies that you wrote it or otherwise have the right to
pass it on as an open-source patch. The rules are pretty simple: if you
can certify the below:
```
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
```
then you just add a line saying ( git commit -s )
Signed-off-by: Random J Developer <random@developer.example.org>
using your real name (sorry, no pseudonyms or anonymous contributions.)
Further information can be found [in the documentation space] (https://confluence.desy.de/display/hidra/HiDRA)
#/bin/sh
#!/bin/sh
usage() { echo "Usage: $0 [-r <major|minor|patch>] [-d]" 1>&2; exit 1; }
......
......@@ -4,6 +4,7 @@
# Path where the logfile will be created
log_path = /opt/hidra/logs
#log_path = /var/log/hidra
# Filename used for logging
log_name = datamanager.log
......@@ -15,11 +16,9 @@ log_size = 10485760
# Name with which the service should be running
procname = hidra
# IP of the interface to bind to for external communication
# IP/DNS name of the interface to bind to for external communication
ext_ip = 0.0.0.0
#ext_ip = zitpcx19282
#ext_ip = 131.169.185.121 ; #zitpcx19282
#ext_ip = 131.169.251.55 ; #asap3-p00
#ext_ip = asap3-p00
#########################################
#### SignalHandler Configuration ####
......@@ -29,8 +28,9 @@ ext_ip = 0.0.0.0
com_port = 50000
# List of hosts allowed to connect
whitelist = ["localhost", "zitpcx19282", "zitpcx22614", "zitpcx22614w", "zitpcx22614.fritz.box", "lsdma-lab04", "haspp11eval01", "it-hpc-cxi03", "max-cfel015"]
#whitelist = None
#whitelist = []
#whitelist = ["localhost", "zitpcx19282", "zitpcx22614", "zitpcx22614w", "lsdma-lab04"]
whitelist = None
# ZMQ port to get new requests
request_port = 50001
......@@ -45,7 +45,7 @@ control_pub_port = 50005
control_sub_port = 50006
#########################################
#### EventDetector Configuration ####
#### eventdetector Configuration ####
#########################################
# Type of event detector to use (options are:
......@@ -54,13 +54,18 @@ control_sub_port = 50006
# see https://bugs.launchpad.net/inotifyx/+bug/1006053
event_detector_type = inotifyx_events
#event_detector_type = watchdog_events
#event_detector_type = zmq_events
#event_detector_type = http_events
#event_detector_type = hidra_events
#event_detector_type = zmq_events
# Subdirectories to be monitored and to store data to
# (needed if eventDetector is inotifyx_events or watchdog_events
# or if dataFetcher is file_fetcher)
# Subdirectories to be monitored and to store data to. These directory have to
# exist when HiDRA is started and should not be removed during the run.
# (needed if event_detector is inotifyx_events or watchdog_events
# or if data_fetcher is file_fetcher)
fix_subdirs = ["commissioning", "current", "local"]
#fix_subdirs = ["commissioning/raw", "commissioning/scratch_bl",
# "current/raw", "current/scratch_bl",
# "local"]
# Directory to be monitor for changes
# Inside this directory only the subdirectories "commissioning", "current"
......@@ -68,75 +73,89 @@ fix_subdirs = ["commissioning", "current", "local"]
# (needed if event detector is inotifyx_events or watchdog_events)
monitored_dir = /opt/hidra/data/source
#monitored_dir = /rd
#monitored_dir = /ramdisk
# Event type of files (options are: IN_CLOSE_WRITE, IN_MOVED_TO, ...) and
# the formats to be monitored, files in an other format will be be neglected
# (needed if event detector is inotifyx_events or watchdog_events)
monitored_events = {"IN_CLOSE_WRITE": [".tif", ".cbf", ".nxs"],
"IN_MOVED_TO": [".log"]}
# (needed if event_detector is inotifyx_events or watchdog_events)
monitored_events = {"IN_CLOSE_WRITE": [""]}
#monitored_events = {"IN_CLOSE_WRITE": [".tif", ".cbf", ".nxs"],
# "IN_MOVED_TO": [".log"]}
#monitored events = {"IN_MOVED_TO": [".tif", ".cbf", ".nxs"],
# "IN_CLOSE_WRITE": [".log"]}
#monitored_events = {"NO_EVENT_MONITORED": [".cbf"]}
# Number of events stored to look for doubles
# (needed if eventDetector is inotifyx_events or http_events)
# (needed if event_detector is inotifyx_events or http_events)
history_size = 0
# Flag describing if a clean up thread which regularly checks
# if some files were missed should be activated
# (needed if eventDetector is inotifyx_events)
# (needed if event_detector is inotifyx_events)
use_cleanup = False
# Intervall time (in seconds) used for clean up resp. checking of events
# (only needed if event_detector_type is inotifyx_events or watchdog_events)
# Interval time (in seconds) used for clean up resp. checking of events
# (only needed if event_detector_type is inotifyx_events together with
# use_cleanup enabled or if event_detector is watchdog_events)
#action_time = 150
action_time = 10
# Time (in seconds) since last modification after which a file will be seen as closed
# (needed if eventDetector is inotifyx_events (for clean up) or watchdog_events)
# Time (in seconds) since last modification after which a file
# will be seen as closed
# (needed if event_detector is inotifyx_events (for clean up) or
# watchdog_events)
time_till_closed = 2
# ZMQ port to get incoming data from
# (needed if event_detector_type is hidra_events)
ext_data_port = 50101
# ZMQ port to get events from
# (needed if event_detector_type is zmq_events)
event_det_port = 50003
# IP of the Eiger detector
# IP/DNS name of the detector
# (needed if event_detector_type is http_events)
#haspp11e1m.desy.de
#eiger_ip = 192.168.138.52
eiger_ip = lsdma-lab04
#lsdma-lab04.desy.de
#eiger_ip = 131.169.55.170
# API version of the Eiger detector
det_ip = asap3-mon
# API version of the detector
# (needed if event_detector_type is http_events)
eiger_api_version = 1.5.0
det_api_version = 1.5.0
#########################################
#### DataFetcher Configuration ####
#### datafetcher Configuration ####
#########################################
# Module with methods specifying how to get the data
# (options are "file_fetcher", "zmq_fetcher", "http_fetcher")
data_fetcher_type = file_fetcher
#data_fetcher_type = zmq_fetcher
#data_fetcher_type = http_fetcher
#data_fetcher_type = hidra_fetcher
#data_fetcher_type = zmq_fetcher
# If "zmq_fetcher" is specified as data_fetcher_type it needs a port to listen to
# (needed if event_detector_type is zmq_events)
# (needed if event_detector_type is zmq_events or if on windows and
# event_detector_type is http_events)
data_fetcher_port = 50010
# Test signal port to notify the data sender about problems
# (needed if event_detector_type is hidra_events)
status_check_resp_port = 50011
# Confirmation socket to send a confirmation for each data message sent
# (needed if event_detector_type is hidra_events)
confirmation_resp_port = 50012
# Number of parallel data streams
number_of_streams = 1
number_of_streams = 8
# Enable ZMQ pipe into storage system
# (uses the fixed_stream_host and fixed_stream_port)
use_data_stream = False
# Fixed host to send the data to with highest priority
fixed_stream_host = zitpcx19282
#fixed_stream_host = zitpcx22614w
# Fixed Port to send the data to with highest priority
fixed_stream_port = 50100
# Fixed host and port to send the data to with highest priority
# (only needed if use_data_stream is enabled)
data_stream_targets = [["asap3-p00", 50100]]
#data_stream_targets = [["zitpcx19282", 50100]]
#data_stream_targets = [["zitpcx22614w", 50100]]
# Chunk size of file-parts getting send via zmq
#1024*1024*10
......@@ -148,14 +167,37 @@ chunksize = 10485760
# (needed if running on Windows)
router_port = 50004
# Target to move the files into
# Target directory to move the files into (needed if store_data is enabled)
local_target = /opt/hidra/data/target
#local_target = /gpfs
# Flag describing if the data should be stored in localTarget
# Flag describing if the data should be stored in local_target
# (needed if data_fetcher_type is file_fetcher or http_fetcher)
store_data = False
# Flag describing if the files should be removed from the source
# options are:
# True - data stays on the source
# False - data is removed from the source after processing it
# deferred_error_handling - only supported if use_data_stream is enabled;
# datamanager gets a feedback if the data writing was
# successful
# with_confirmation - only supported if use_data_stream is enabled and
# data_fetcher_type is file_fetcher or http_fetcher;
# data is removed from the source only after the target
# sent a verification
# (needed if data_fetcher_type is file_fetcher or http_fetcher)
#remove_data = True
remove_data = False
#remove_data = deferred_error_handling
#remove_data = with_confirmation
# Test signal port to check if the data receiver is having problems
status_check_port = 50050
# ZMQ port to communicate with cleaner process
# (needed if running on Windows)
cleaner_port = 50051
# Confirmation socket to get a confirmation for each data message sent
confirmation_port = 50052
#########################################
#### Logging Configuration ####
#########################################
# Path where the logfile will be created
log_path = /var/log/hidra
# Filename used for logging
log_name = datamanager.log
# File size before rollover in B (linux only)
# 10 MB
log_size = 10485760
# Name with which the service should be running
procname = hidra
# IP/DNS name of the interface to bind to for external communication
ext_ip = 0.0.0.0
#########################################
#### SignalHandler Configuration ####
#########################################
# Port number to receive signals from
com_port = 50000
# List of hosts allowed to connect
#whitelist = []
whitelist = None
# ZMQ port to get new requests
request_port = 50001
# ZMQ port to forward requests
# (needed if running on Windows)
request_fw_port = 50002
# ZMQ port to disribute control signals
# (needed if running on Windows)
control_pub_port = 50005
control_sub_port = 50006
#########################################
#### eventdetector Configuration ####
#########################################
# Type of event detector to use (options are:
# inotifyx_events, watchdog_events, zmq_events, http_events)
# Inotifyx is not python3 compatible,
# see https://bugs.launchpad.net/inotifyx/+bug/1006053
event_detector_type = inotifyx_events
# Subdirectories to be monitored and to store data to. These directory have to
# exist when HiDRA is started and should not be removed during the run.
# (needed if event_detector is inotifyx_events or watchdog_events
# or if data_fetcher is file_fetcher)
fix_subdirs = ["commissioning", "current", "local"]
#fix_subdirs = ["commissioning/raw", "commissioning/scratch_bl",
# "current/raw", "current/scratch_bl",
# "local"]
# Directory to be monitor for changes
# Inside this directory only the subdirectories "commissioning", "current"
# and "local" are monitored
# (needed if event detector is inotifyx_events or watchdog_events)
monitored_dir = /ramdisk
# Event type of files (options are: IN_CLOSE_WRITE, IN_MOVED_TO, ...) and
# the formats to be monitored, files in an other format will be be neglected
# (needed if event_detector is inotifyx_events or watchdog_events)
monitored_events = {"IN_CLOSE_WRITE": [""]}
#monitored_events = {"IN_CLOSE_WRITE": [".tif", ".cbf", ".nxs"],
# "IN_MOVED_TO": [".log"]}
#monitored events = {"IN_MOVED_TO": [".tif", ".cbf", ".nxs"],
# "IN_CLOSE_WRITE": [".log"]}
#monitored_events = {"NO_EVENT_MONITORED": [".cbf"]}
# Number of events stored to look for doubles
# (needed if event_detector is inotifyx_events or http_events)
history_size = 0
# Flag describing if a clean up thread which regularly checks
# if some files were missed should be activated
# (needed if event_detector is inotifyx_events)
use_cleanup = False
# Interval time (in seconds) used for clean up resp. checking of events
# (only needed if event_detector_type is inotifyx_events together with
# use_cleanup enabled or if event_detector is watchdog_events)
#action_time = 150
action_time = 10
# Time (in seconds) since last modification after which a file will be seen as closed
# (needed if event_detector is inotifyx_events (for clean up) or watchdog_events)
time_till_closed = 2
# ZMQ port to get events from
# (needed if event_detector_type is zmq_events)
event_det_port = 50003
# IP/DNS name of the detector
# (needed if event_detector_type is http_events)
det_ip = lsdma-lab04
# API version of the detector
# (needed if event_detector_type is http_events)
det_api_version = 1.5.0
#########################################
#### datafetcher Configuration ####
#########################################
# Module with methods specifying how to get the data
# (options are "file_fetcher", "zmq_fetcher", "http_fetcher")
data_fetcher_type = file_fetcher
# If "zmq_fetcher" is specified as data_fetcher_type it needs a port to listen to
# (needed if event_detector_type is zmq_events)
data_fetcher_port = 50010
# Number of parallel data streams
number_of_streams = 8
# Enable ZMQ pipe into storage system
use_data_stream = True
# Fixed host and port to send the data to with highest priority
# (only needed if use_data_stream is enabled)
#data_stream_targets = [["asap3-p00", 50100]]
data_stream_targets = [["zitpcx19282", 50100]]
# Chunk size of file-parts getting send via zmq
#1024*1024*10
#chunksize = 102400
chunksize = 10485760
#1024*1024*1024
#chunksize = 1073741824
# ZMQ-router port which coordinates the load-balancing to the worker-processes
# (needed if running on Windows)
router_port = 50004
# Flag describing if the data should be stored in local_target
# (needed if data_fetcher_type is file_fetcher or http_fetcher)
store_data = False
# Flag describing if the files should be removed from the source
# options are:
# True - data stays on the source
# False - data is removed from the source after processing it
# with_confirmation - only supported if use_data_stream is enabled;
# data is removed from the source only after the target
# sent a verification
# (needed if data_fetcher_type is file_fetcher or http_fetcher)
remove_data = True
#########################################
#### Logging Configuration ####
#########################################
# Path where the logfile will be created
log_path = /var/log/hidra
# Filename used for logging
log_name = datamanager.log
# File size before rollover in B (linux only)
# 10 MB
log_size = 10485760
# Name with which the service should be running
procname = hidra
# IP/DNS name of the interface to bind to for external communication
ext_ip = 0.0.0.0
#########################################
#### SignalHandler Configuration ####
#########################################
# Port number to receive signals from
com_port = 50000
# List of hosts allowed to connect
whitelist = ["localhost"]
# ZMQ port to get new requests
request_port = 50001
#########################################
#### EventDetector Configuration ####
#########################################
# Type of event detector to use (options are:
# inotifyx_events, watchdog_events, zmq_events, http_events)
# Inotifyx is not python3 compatible,
# see https://bugs.launchpad.net/inotifyx/+bug/1006053
event_detector_type = inotifyx_events
# Subdirectories to be monitored and to store data to. These directory have to
# exist when HiDRA is started and should not be removed during the run.
# (needed if eventDetector is inotifyx_events or watchdog_events
# or if dataFetcher is file_fetcher)
fix_subdirs = ["commissioning/raw", "commissioning/scratch_bl",
"current/raw", "current/scratch_bl",
"local"]
# Directory to be monitor for changes
# Inside this directory only the subdirectories "commissioning", "current"
# and "local" are monitored
# (needed if event detector is inotifyx_events or watchdog_events)
monitored_dir = /ramdisk
# Event type of files (options are: IN_CLOSE_WRITE, IN_MOVED_TO, ...) and
# the formats to be monitored, files in an other format will be be neglected
# (needed if event detector is inotifyx_events or watchdog_events)
monitored_events = {"IN_MOVED_TO": [".tif", ".cbf", ".img", ".edf"],
"IN_CLOSE_WRITE": [".log"]}
# Number of events stored to look for doubles
# (needed if eventDetector is inotifyx_events or http_events)
history_size = 0
# Flag describing if a clean up thread which regularly checks
# if some files were missed should be activated
# (needed if eventDetector is inotifyx_events)
use_cleanup = False
#########################################
#### DataFetcher Configuration ####
#########################################
# Module with methods specifying how to get the data
# (options are "file_fetcher", "zmq_fetcher", "http_fetcher")
data_fetcher_type = file_fetcher
# Number of parallel data streams
number_of_streams = 4
# Enable ZMQ pipe into storage system
# (uses data_stream_targets)
use_data_stream = True
# Fixed host and port to send the data to with highest priority
data_stream_targets = [["asap3-p00", 50100]]
# Chunk size of file-parts getting send via zmq
#1024*1024*10
chunksize = 10485760
# Flag describing if the data should be stored in localTarget
# (needed if data_fetcher_type is file_fetcher or http_fetcher)
store_data = False
# Flag describing if the files should be removed from the source
# options are:
# True - data stays on the source
# False - data is removed from the source after processing it
# (needed if data_fetcher_type is file_fetcher or http_fetcher)
remove_data = True
......@@ -15,11 +15,8 @@ log_size = 10485760
# Name with which the service should be running
procname = hidra
# IP of the interface to bind to for external communication
# IP/DNS name of the interface to bind to for external communication
ext_ip = 0.0.0.0
#ext_ip = zitpcx19282
#ext_ip = 131.169.185.121 ; #zitpcx19282
#ext_ip = 131.169.251.55 ; #asap3-p00
#########################################
#### SignalHandler Configuration ####
......@@ -29,8 +26,7 @@ ext_ip = 0.0.0.0
com_port = 50000
# List of hosts allowed to connect
whitelist = ["localhost", "zitpcx19282", "zitpcx22614", "zitpcx22614w", "zitpcx22614.fritz.box", "lsdma-lab04", "haspp11eval01", "it-hpc-cxi03", "max-cfel015"]
#whitelist = None
whitelist = None
# ZMQ port to get new requests
request_port = 50001
......@@ -45,115 +41,144 @@ control_pub_port = 50005
control_sub_port = 50006
#########################################
#### EventDetector Configuration ####
#### eventdetector Configuration ####
#########################################
# Type of event detector to use (options are:
# inotifyx_events, watchdog_events, zmq_events, http_events)
# Inotifyx is not python3 compatible,
# see https://bugs.launchpad.net/inotifyx/+bug/1006053
#event_detector_type = inotifyx_events
event_detector_type = watchdog_events
#event_detector_type = zmq_events
#event_detector_type = http_events
# Subdirectories to be monitored and to store data to
# (needed if eventDetector is inotifyx_events or watchdog_events
# or if dataFetcher is file_fetcher)
fix_subdirs = ["commissioning", "current", "local"]
# Subdirectories to be monitored and to store data to. These
# directory have to exist when HiDRA is started and should not
# be removed during the run.
# (needed if event_detector is inotifyx_events or
# watchdog_events
# or if data_fetcher is file_fetcher)
fix_subdirs = ["commissioning/raw", "commissioning/scratch_bl",
"current/raw", "current/scratch_bl",
"local"]
# Directory to be monitor for changes
# Inside this directory only the subdirectories "commissioning", "current"
# and "local" are monitored
# Inside this directory only the subdirectories "commissioning",
# "current" and "local" are monitored
# (needed if event detector is inotifyx_events or watchdog_events)
monitored_dir = D:\hidra\data\source