Dev update: MGM API and Architecture

Dev update: MGM API and Architecture

The system consists of the following elements:
– Central server
– Docker registry compatible image registry managed by Megamind
– Computing nodes
– Developers
– End users

  • Central server. Megamind Central server consists of the following parts:
    • Server REST API. Provides a top-level API for task management and connecting computing nodes. The task in this context is a docker image deployed to container(s) and run on a computing node. REST API server consists of an industrial-grade web server (apache / nginx) with https configured and accounts proxying calls to Megamind REST API server. The server supports the following commands:
    • Public part
      1. get a list of free computing nodes (id, hardware characteristics, computation cost)
      2. create a task by running an image from the registry on a particular node
      3. stop the task by id, destroying it and removing it from the list. It removes port forwarding from NAT and from the list.
      4. get a list of running tasks for the account (with id) and their state (started/pending/completed)
      5. pause / resume a task
    • Non-public part
      1. get parameters to connect the node to the system (parameters for entering swarm – token and ip)
      2. node registration: transfer to the server of its characteristics, ip, user:password
  • The server also contains a manager node, a null / manager node docker swarm. The REST API server executes requests by managing the internal infrastructure using docker swarm (currently using parser of command line output, in the future versions – through the Docker API):
    1. getting a list of worker nodes: sudo docker node ls

For for more information on node:

  1. task creation: sudo docker service create
  2. deleting the task: sudo docker service rm
  3. get a list of working tasks for the account: for each node getting all tasks via docker node ps and filtering by id for of this account
  4. pause / resume task

With port-forwarding:

  • it performs port forwarding to each created container (command for iptables);
  • keeps track of already forwarded ports;
  • removes port forwarding if necessary;

Or (option with direct channels to each node):

If your nodes are on a white IP and accessible from the Internet, then you need to open a port in the firewall (and don’t forget port forwarding in docker)

  • Server of the web interface for setting tasks;

  • Registry of images managed by Megamind GMbH

Fully compatible with standard docker registry. Needed authentication with htpasswd – every account registration on server from item 1 should add an entry to htpasswd. Allows to:

  • unload (push) images
  • download (pull) images
  • get a list of images associated with this account
  • delete images

All this using the standard docker registry interfaces.

  • Nodes with computing equipment. High performance computers running MindOS.

Docker is installed on the node and the node is added into the swarm.

  • Developers

They work under operating systems that support docker images, creating images for the Image Registry and load them into the Registry. They don’t work with Megamind-specific code or

Software — all work is done using docker tools.

  • End users

They perform task management using direct API requests from clause 1.1 or using web interface: start new tasks, view running ones, stop, etc. They can also administer the counterpart of the computing Software (what requests / receives results from the nodes). Interface for accessing the task is ssh. If necessary, port forwarding is also done using ssh.

Get in touch

MegaMiner GmbH 26 Sumpfstrasse, 6312 Steinhausen, ZG, Switzerland

+44 751 708 14 51