Modules, Pipelines, and Applications

Module CRD

Module is a namespaced resource representing an atomic unit of execution within ECiDA. The following specification ignores some important aspects (e.g., rich types, preconditions and effects):

metadata:
  name: [module-name as name.project.dnssubname], e.g., lr.ml.ecida.org (short name vs. full name?)
  version: [module-version]
  component: external - e.g., cassandra, kafka, mongodb; (unmanaged by ecida) 
           | datasource - contains data, atm means nothing apart from different color in UI 
           | service - accessible from the outside, e.g., UI
spec:
  ports:
  - list of ports:
      kind: signal | hyperparameter | in | out
      bind: to_ports_in_container OR ENV_VARS OR to something 
  containers
  - list of container(s) which need to be run for this module
      name, image, ports, service?
  
  dependencies: TBD

Pipeline CRD

Pipeline is a namespaced resource representing a description of one executable workflow.

metadata:
  name: [pipeline-name]
spec:
  modules:
    - list of module references
      module: module-name
      version: module-version  // optional?
      as: module-ref  // might even be optional, and then module-ref = module-name
- links:
    - list of links connecting out-to-in ports of modules, each link:
        from: 
          module: [module-ref]
          port-name: [out-port-name]
        to: 
          module: [module-ref]
          port-name: [in-port-name]

Application CRD

Application is a namespaced resource representing a collection of pipelines.

metadata:
  name: [application-name]
spec:
  pipelines:
    - list of pipeline references

Each application should be deployed into a separate namespace according to the Environments and Namespaces rules.