资料上传资料上传资料上传资料上传

# JobFlow ## Introduction In order to solve the problem of inter-job dependencies. We need many VCJobs to cooperate each other and orchestrate them manually or by another Job Orchestration Platform to get the job done finally.We present an new way of orchestrating VCJobs called JobFlow. We proposed two concepts to running multiple batch jobs automatically named JobTemplate and JobFlow so end users can easily declare their jobs and run them using complex controlling primitives, for example, sequential or parallel executing, if-then-else statement, switch-case statement, loop executing and so on. JobFlow helps migrating AI, BigData, HPC workloads to the cloud-native world. Though there are already some workload flow engines, they are not designed for batch job workloads. Those jobs typically have a complex running dependencies and take long time to run, for example days or weeks. JobFlow helps the end users to declare their jobs as an jobTemplate and then reuse them accordingly. Also, JobFlow orchestrating those jobs using complex controlling primitives and launch those jobs automatically. This can significantly reduce the time consumption of an complex job and improve resource utilization. Finally, JobFlow is not an generally purposed workflow engine, it knows the details of VCJobs. End user can have a better understanding of their jobs, for example, job's running state, beginning and ending timestamps, the next jobs to run, pod-failure-ratio and so on. ## Scope ### In Scope - Define the API of JobFlow - Define the behaviour of JobFlow - Start sequence between multiple jobs - Dependency completion state of the job start sequence - DAG-based job dependency startup ### Out of Scope - Supports other job - Achieve vcjobs level gang ## Scenarios - Some jobs need to depend on the completion of the previous job or other status when running, etc. Otherwise, the correct result cannot be calculated. - Sometimes inter-job dependencies also require diverse dependency types, such as conditional dependencies, circular dependencies, probes, and so on.  ## Design  The blue part is the components of k8s itself, the orange is the existing definition of Volcano, and the red is the new definition of JobFlow. **jobflow job submission complete process**： 1. After passing the Admission. kubectl will create JobTemplate and JobFlow (Volcano CRD) objects in kube-apiserver. 2. The JobFlowController uses the JobTemplate as a template according to the configuration of the JobFlow, and creates the corresponding VcJob according to the flow dependency rules. 3. After VcJob is created, VcJobController creates corresponding Pods and podgroups according to the configuration of VcJob. 4. After Pod and PodGroup are created, vc-scheduler will go to kube-apiserver to get Pod/PodGroup and node information. 5. After obtaining the information, vc-scheduler will select the appropriate node for each Pod according to its configured scheduling policy. 6. After assigning nodes to Pods, kubelet will get the Pod's configuration from kube-apiserver and start the corresponding containers. **update jobflow**: Currently, jobflow does not support the update operation, and the update of jobflow will be blocked through webhook. **delete jobflow**: Deleting a jobflow when the jobflow is in a non-complete state will be intercepted by the webhook. otherwise, after deleting jobflow, all vcjobs created by jobflow will be deleted directly. ### Controller  ### Webhook ``` Create a JobFlow check 1、There cannot be a template with the same name in a JobFlow dependency Such as: A->B->A->C A appears twice 2、Closed loops cannot occur in JobFlow E.g：A -> B -> C ^ / | / < - D Create a JobTemplte check (following the vcjob parameter specification) E.g: job minAvailable must be greater than or equal to zero job maxRetry must be greater than or equal to zero tasks cannot be empty, and cannot have tasks with the same name The number of task replicas cannot be less than zero task minAvailable cannot be greater than task replicas... ``` ### JobFlow #### Introduction JobFlow defines the running flow of a set of jobs. Fields in JobFlow define how jobs are orchestrated. JobFlow is abbreviated as jf, and the resource can be viewed through kubectl get jf JobFlow aims to realize job-dependent operation between vcjobs in volcano. According to the dependency between vcjob, vcjob is issued. #### Key Fields ##### Top-Level Attributes The top-level attributes of a jobflow define its apiVersion, kind, metadata and spec. | Attribute | Type | Required | Default Value | Description | | ------------ | ----------------------- | -------- | -------------------------- | ------------------------------------------------------------ | | `apiVersion` | `string` | Y | `flow.volcano.sh/v1alpha1` | A string that identifies the version of the schema the object should have. The core types uses `flow.volcano.sh/v1alpha1` in this version of documentation. | | `kind` | `string` | Y | `JobFlow` | Must be `JobFlow` | | `metadata` | [`Metadata`](#Metadata) | Y | | Information about the JobFlow resource. | | `spec` | [`Spec`](#spec) | Y | | A specification for the JobFlow resource attributes. | | `status` | [`Status`](#Status) | Y | | A specification for the JobFlow status attributes. | ##### Metadata Metadata provides basic information about the JobFlow. | Attribute | Type | Required | Default Value | Description | | ------------- | ------------------- | -------- | ------------- | ------------------------------------------------------------ | | `name` | `string` | Y | | A name for the schematic. `name` is subject to the restrictions listed beneath this table. | | `namespace` | `string` | Y | | A namespace for the schematic. `namespace` is subject to the restrictions listed beneath this table. | | `labels` | `map[string]string` | N | | A set of string key/value pairs used as arbitrary labels on this component. Labels follow the [Kubernetes specification](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). | | `annotations` | `map[string]string` | N | | A set of string key/value pairs used as arbitrary descriptive text associated with this object. Annotations follows the [Kubernetes specification](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). | ##### Spec The specification of cloud-native services defines service metadata, version list, service capabilities and plugins. | Attribute | Type | Required | Default Value | Description | | ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | | `flows` | [`Flow array`](#Flow) | Y | | Describes the dependencies between vcjobs. | | `jobRetainPolicy` | `string` | Y | retain | After JobFlow succeed, keep the generated job. Otherwise, delete it. | ##### Flow | Attribute | Type | Required | Default Value | Desc