Skip to content
Snippets Groups Projects
Commit f380f476 authored by Boris Baldassari's avatar Boris Baldassari
Browse files

Merge branch 'bba_aice_aura_demonstrator' into 'main'

More minor fixes to article.

See merge request !2
parents b26dbb97 e0b76dda
No related branches found
No related tags found
No related merge requests found
......@@ -157,7 +157,7 @@ It is also very important to interpret visually and discuss the outcomes of the
The new repository has a sound and clean structure, with passing tests, a complete documentation to exploit and run the various steps, and has everything needed for further developments. All scripts are stored under the src/ directory and are copied to the docker images during the build, thus always relying on a single source of tested truth.
Furthermore, the automatic building of containers for multiple execution targets (Airflow, Docker, Kubernetes) can easily be reproduced . As a result the new, improved structure will be reused and is set to become the reference implementation for the next developments.
Furthermore, the automatic building of containers for multiple execution targets (Airflow, Docker, Kubernetes) can easily be reproduced. As a result the new, improved structure will be reused and is set to become the reference implementation for the next developments.
### Portability and deployment
......@@ -168,7 +168,7 @@ We also installed a fresh instance of AI4EU Experiments on our dedicated hardwar
### Better performances
The major performance gain was achieved by setting up dedicated containers to run atomic tasks (e.g. data preparation, visulisation imports) in parallel. Most computers, both in the lab and for high-end execution platforms, have multiple threads and enough memory to manage several containers simultaneously, and we need to take advantage of the full computing power we have. Another major gain was obviously to run the process on a more powerful system, with enough memory, CPUs and disk throughput.
The major performance gain was achieved by setting up dedicated containers to run atomic tasks (e.g. data preparation, visualisation imports) in parallel. Most computers, both in the lab and for high-end execution platforms, have multiple threads and enough memory to manage several containers simultaneously, and we need to take advantage of the full computing power we have. Another major gain was obviously to run the process on a more powerful system, with enough memory, CPUs and disk throughput.
All considered we were able to scale down the full execution time on the TUH dataset from 17 hours on the lab's laptop to roughly 4 hours in our cluster.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment