YARP
Yet Another Robot Platform
 
Loading...
Searching...
No Matches
yarpmanager: edit, run and manage multiple programs on a set of machines

The command-line utility "yarpmanager-console" and its graphical companions "yarpmanager" are tools for running and managing multiple programs on a set of machines.

+ Collaboration diagram for yarpmanager: edit, run and manage multiple programs on a set of machines:

The command-line utility "yarpmanager-console" and its graphical companions "yarpmanager" are tools for running and managing multiple programs on a set of machines.

yarpmanager-console and yarpmanager

Libraries

Features

Some Concepts

See application, module and resource for more information on how to use XML model to describe each of these concepts.

Configuration

Default configuration file known as "ymanager.ini" can be automatically loaded by yarpmanager from the same folder running the manager; or it can be indicated by parameter "--from" and "--context", following YARP Resource Finder policies (see The ResourceFinder Class (basic)). Using the configuration files, one can explicitly indicates the paths where yarpmanager should search for applications, resources or modules. If the configuration file is not provided, then yarpmanager will search for the corresponding XML files in the YARP data directories (see YARP data directories).

$ yarpmanager --from <path to my_config_file>

The below example shows the syntax of configuration file:

# Path to the folder which contains application description files
apppath =  "./xml/applications"
load_subfolders = yes

# Path to the folder which contains module description files
modpath =  "./xml/modules"

# Path to the folder which contains module description files
respath =  "./xml/resources"

# Fault tolerance
# parameters: yes|No
watchdog = no

# Module failure awareness (if watchdog is enabled)
# parameters: Ignore|terminate|prompt|recover
module_failure = prompt

# Connection failure awareness (if watchdog is enabled)
# parameters: Ignore|terminate|prompt
connection_failure = prompt

# Automatically establish all connections
# parameters: Yes|no
auto_connect = no

# Appearance (for yarpmanager)
# parameters: No|dark|light
color_theme = dark

# External editor (e.g. gedit, notepad)
external_editor = gvim

# Disable autoreload
disable_autoreload

Items 'apppath', 'modpath' and 'respath' respectively point to the folder where the XML description files of Applications, Modules and Resources can be found. All description file names must have ".xml" extension. If 'load_subfolders=yes', applications description files will be recursively loaded from sub folders of 'apppath'. Set 'external_editor' to your preferable text editor. It will be used by yarpmanager for editing XML file.

Cluster management

When a "cluster-config.xml" file is available through the YARP resource finder, yarpmanager enables the cluster panel. The cluster panel lists the configured nodes, checks whether the YARP name server and the node yarprun servers are running, and provides buttons to start, stop or kill selected yarprun servers over ssh.

The cluster file has a "cluster" root element, a "nameserver" element and one or more "node" elements:

<?xml version="1.0" encoding="UTF-8"?>
<cluster name="my-cluster" user="robot">
<nameserver namespace="/yarp" node="pc104" ssh-options="-t"/>
<node address="pc104">/pc104</node>
<node address="desktop" user="developer" docker="robot-container">/desktop</node>
<node address="gpu01" conda="robot-env">/gpu01</node>
<node address="lab01" pixi="/home/robot/workspace/my-pixi-project">/lab01</node>
</cluster>

Supported attributes are:

Only one runtime attribute among "docker", "conda" and "pixi" can be used for a node.

Starting yarprun in a runtime

For each node, the cluster table shows the node status, YARP node name, display, ssh user, address, runtime, runtime target and log option. The runtime and target columns can be edited before pressing "Run selected".

If no runtime is selected, yarpmanager starts:

yarprun --server /node

If "docker" is selected, yarpmanager first checks that the configured container is running on the node host, then starts yarprun inside it:

docker exec <container> sh -lc 'yarprun --server /node'

If "conda" is selected, yarpmanager opens a login shell, sources "conda.sh" from one of the standard installation paths, activates the configured environment and starts yarprun inside it.

If "pixi" is selected, yarpmanager checks that the configured directory exists, contains a "pixi.toml", and that pixi is available on the remote host. It then runs yarprun with:

pixi run yarprun --server /node

When the log checkbox is enabled, "yarprun --log" is used. Startup output is stored on the remote host in "/tmp/yarpmanager_yarprun_<node>.log". If startup fails, yarpmanager reports the tail of that log to help diagnose the problem.

Docker container controls

The cluster panel also contains a "Docker containers" table. It lists the Docker containers referenced by the cluster nodes, grouped by host and ssh settings, and shows their current status and container id.

The available actions are:

Docker-backed yarprun nodes are highlighted in the cluster table when their target container is running. A Docker-backed node cannot be started until its container is running.

Name server and node management

The name server controls start or stop yarpserver on the configured "nameserver\@node". The cluster panel checks the configured namespace to decide whether the server is available.

The node controls operate on the selected cluster rows:

The command execution area can be used to run an arbitrary command on a selected configured node through ssh.

Running examples

First make sure you have a name server running. Open a terminal and type:

$ yarpserver

Using yarpmanager

Open another terminal and try this:

$ cd <path to YARP local repository>/data/yarpmanager/tests
$ yarpmanager-console
>> help
     # a list of yarpmanager-console keywords
>> list app
     # you should see EyesViewer-Localhost, FakeEyes, ...
>> load app EyesViewer-Localhost
>> run
>> connect
     # you should see moving red bar
>> disconnect
>> stop
>> exit

Using yarpmanager

Open another terminal and try this:

$ cd <path to YARP local repository>/data/yarpmanager/tests
$ yarpmanager

To run "EyesViewer-Cluster" using load balancer you need to have yarprun running as server. Open a terminal and type:

$ yarprun --server /node1

Open another one and type:

$ yarprun --server /node2

Then try this:

$ cd <path to YARP local repository>/data/yarpmanager/tests
$ yarpmanager
Note
You can use "Assign hosts" whenever all modules are stopped and connections are disconnected.

Application description file

Application description file is a collection of modules (programs) or other applications and connections.

Here is an example of application description file in XML format:

<application>
<!-- Generic information-->
<name>ApplicationName</name>
<description>Template Application</description>
<version></version>
<authors>
<author email=""> first author.name </author>
<author email=""> second author.name </author>
...
</authors>
<!-- Required modules. -->
<module>
<name></name> <!-- module/command name -->
<parameters></parameters> <!-- parameters -->
<node></node> <!-- hostname to lunch the module e.g. localhost, /node1 -->
<workdir></workdir> <!-- work directory -->
<environment></environment> <!-- set some environment variables for the module. e.g. 'VAR1=val; VAR2=val' -->
<deployer> </deployer> <!-- use a third-party program to run the module. e.g. 'yarpdev' for YARP devices/plugins -->
<display></display>
<stdio></stdio> <!-- redirect the stdio to another host. (Obsolete!) -->
<prefix></prefix> <!-- prefix name will be added to all the ports open by module. e.g. /my-prefix -->
<dependencies> <!-- module execution will be postponed until the dependencies are satisfied -->
<port timeout="5.0"></port>
<port timeout="5.0" request="ready" reply="yes"></port>
...
</dependencies>
<ensure>
<wait when="start">10.0</wait> <!-- wait for 10.0 sec after running the module -->
<wait when="stop">5.0</wait> <!-- wait for 5.0 sec before stopping the module -->
</ensure>
</module>
...
<!-- Embedded applications -->
<application>
<name>Application_Name</name>
<prefix></prefix>
</application>
...
<!-- Required connections -->
<connection persist="true">
<from qos="level:high; priority:10; policy:1"></from> <!-- setting the channel Quality of Service -->
<to></to>
<protocol></protocol> <!-- YARP carrier e.g. tcp/udp -->
</connection>
<connection>
<from external="true"></from> <!-- external means that the port is open by a module external to this application -->
<to></to>
<protocol></protocol>
</connection>
...
</application>

Notes

Resource description file

The resource description file can be used to provide a general information of the machines that are part of the cluster. In very simple form, the resource description file is a list of the names of machines in the cluster.

Here is an example of resource description file in XML format:

<resources>
<!-- Multiple computers can be introduced -->
<computer>
<name>hostname1</name>
<description>Template resource</description>
<!-- A disabled resource will not be used by load balancer. -->
<disable>no</disable>
</computer>
<computer>
<name>hostname2</name>
<description>a computer with GPU device</description>
<disable>no</disable>
<!-- GPU description section. -->
<gpu>
<name>Tesla C1060</name>
<cores>240</cores>
<frequency>1600</frequency>
<capability>1.3</capability>
<registers_block>128</registers_block>
<thread_block>512</thread_block>
<overlap>yes</overlap>
<global_memory>
<total_space>4000000</total_space>
</global_memory>
<shared_memory>
<total_space>128</total_space>
</shared_memory>
<constant_memory>
<total_space>16</total_space>
</constant_memory>
</gpu>
</computer>
...
</resources>

Notes

Module description file

Module description file can be used to provide general information of module and its dependencies. It is mainly used by load balancer to automatically assign machines to module which satisfy the dependencies. By default, if there is no module description, load balancer assumes that the module can be executed on every available machines. Despite it is not mandatory, it is always useful to have XML description of each module which shows how it should be used for application building (e.g parameters, inputs, outputs).

Here is an example of module description file in XML format:

<module>
<name>module_name</name>
<description>Template module</description>
<version>1.0</version>
<!-- <arguments> can have multiple <param> tags-->
<arguments>
<param desc="first parameter"> param1 </param>
</arguments>
<!-- <authors> can have multiple <author> tags. -->
<authors>
<author email=""> author1 </author>
</authors>
<!-- <data> can have multiple <input> or <output> tags. -->
<data>
<!-- input data if available-->
<input>
<type>DataType</type>
<!-- input port which receives the data. -->
<port carrier="tcp">/portname</port>
<required>yes</required>
<priority>no</priority>
<description></description>
</input>
...
<!-- output data if available -->
<output>
<type>DataType</type>
<port carrier="tcp">/portname</port>
<description></description>
</output>
...
</data>
<dependencies>
<computer>
<gpu>
<capability>1.3</capability>
</gpu>
...
</computer>
...
</dependencies>
<!-- specific libraries or header files which are used for development -->
<development>
<header></header>
<library></library>
</development>
</module>

Notes

Resource conceptional model which can be used as module's dependency:

<platform>
<name></name>
<distribution></distribution>
<release></release>
</platform>
<yarp_port>
<name></name> <!-- YARP port name -->
<yarp_port>
<memory>
<free_space></free_space> <!-- in KB -->
</memory>
<storage>
<free_space></free_space> <!-- in KB -->
</storage>
<processor>
<architecture></architecture>
<cores></cores>
<frequency></frequency>
<model></model>
</processor>
<gpu>
<name></name>
<cores></cores>
<capability></capability>
<registers_block></registers_block>
<thread_block></thread_block>
<overlap></overlap>
<global_memory>
<total_space></total_space> <!-- in KB -->
</global_memory>
<shared_memory>
<total_space></total_space> <!-- in KB -->
</shared_memory>
<constant_memory>
<total_space></total_space> <!-- in KB -->
</constant_memory>
</gpu>

Create new XML description file

yarpmanager allows for creating new application, module and resource description file using predefined templates. To create a new XML file, first make sure you set 'external_editor' to your preferable editor (e.g. gedit, notepad) then

Notes

Generate application dependency graph

The arrangement of modules and resources within an application can be visualized as dependency graph. For Instance, dependency graph of application "EyesViewer-Localhost" used in above example is shown below.

EyesViewer-Localhost

As shown in the graph, application "EyesViewer-Localhost" consists of two instances of "FakeEyes" with their relevant modules. Each module is associated with a node marked as "MultipleResource". It is linked to machines which satisfy its requirements. Notice that here, "localhost" is specifically assigned by user to each modules; Thus there is no link to machines "node1" and "node2". The graph is generated in DOT format which can be visualized using Graphviz tools. To generate the graph:

Or open an application using yarpmanager (See above example example_y) and type :

>> export MyApp.dot

To visualize the graph, Linux users can open a terminal and try this:

$ sudo apt-get install graphviz
$ dot -Tpdf -o MyApp.pdf MyApp.dot

Windows users can download Graphviz tools from http://www.graphviz.org/Download_windows.php and in windows console type :

 C:\...\> dot -Tpdf -o MyApp.pdf MyApp.dot

Frequently asked questions

How to postpone launching a module?

It can very often happen that you need to launch a module after some other modules already launched and are running. This is usually the case where a module needs to access some ports which should be open by other modules. To achieve this behavior, you simply needs to indicate which ports your module needs to be available using <dependencies> tag of the corresponding module in the application description file (see app_notes).

How to quickly launch a single application in command line?

Imagine you need to run a single application (e.g., eyesviewer-localhost.xml) without using Guis. All you need is to use 'yarpmanager-console', give your application description file as its parameter and ask it to run the modules and connect them:

     yarpmanager-console --application my-app.xml --run --connect

If you want to run the application without getting blocked in yarpmanager-console, you can ask it to quit after running the application.

     yarpmanager-console --application my-app.xml --run --connect --exit --silent

The '–silent' parameter avoids showing the verbose messages. Try 'yarpmanager-console –help' for further information.