Commit f380f476 authored by Boris Baldassari's avatar Boris Baldassari
Browse files

Merge branch 'bba_aice_aura_demonstrator' into 'main'

More minor fixes to article.

See merge request eclipsefdn/it/websites/aice.eclipse.org!2
parents b26dbb97 e0b76dda
...@@ -157,7 +157,7 @@ It is also very important to interpret visually and discuss the outcomes of the ...@@ -157,7 +157,7 @@ It is also very important to interpret visually and discuss the outcomes of the
The new repository has a sound and clean structure, with passing tests, a complete documentation to exploit and run the various steps, and has everything needed for further developments. All scripts are stored under the src/ directory and are copied to the docker images during the build, thus always relying on a single source of tested truth. The new repository has a sound and clean structure, with passing tests, a complete documentation to exploit and run the various steps, and has everything needed for further developments. All scripts are stored under the src/ directory and are copied to the docker images during the build, thus always relying on a single source of tested truth.
Furthermore, the automatic building of containers for multiple execution targets (Airflow, Docker, Kubernetes) can easily be reproduced . As a result the new, improved structure will be reused and is set to become the reference implementation for the next developments. Furthermore, the automatic building of containers for multiple execution targets (Airflow, Docker, Kubernetes) can easily be reproduced. As a result the new, improved structure will be reused and is set to become the reference implementation for the next developments.
### Portability and deployment ### Portability and deployment
...@@ -168,7 +168,7 @@ We also installed a fresh instance of AI4EU Experiments on our dedicated hardwar ...@@ -168,7 +168,7 @@ We also installed a fresh instance of AI4EU Experiments on our dedicated hardwar
### Better performances ### Better performances
The major performance gain was achieved by setting up dedicated containers to run atomic tasks (e.g. data preparation, visulisation imports) in parallel. Most computers, both in the lab and for high-end execution platforms, have multiple threads and enough memory to manage several containers simultaneously, and we need to take advantage of the full computing power we have. Another major gain was obviously to run the process on a more powerful system, with enough memory, CPUs and disk throughput. The major performance gain was achieved by setting up dedicated containers to run atomic tasks (e.g. data preparation, visualisation imports) in parallel. Most computers, both in the lab and for high-end execution platforms, have multiple threads and enough memory to manage several containers simultaneously, and we need to take advantage of the full computing power we have. Another major gain was obviously to run the process on a more powerful system, with enough memory, CPUs and disk throughput.
All considered we were able to scale down the full execution time on the TUH dataset from 17 hours on the lab's laptop to roughly 4 hours in our cluster. All considered we were able to scale down the full execution time on the TUH dataset from 17 hours on the lab's laptop to roughly 4 hours in our cluster.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment