技术背景

假设我们在一个局域网内有多台工作站（不是服务器），那么有没有一个简单的方案可以实现一个小集群，提交分布式的任务呢？Ray为我们提供了一个很好的解决方案，允许你通过conda和Python灵活的构建集群环境，并提交分布式的任务。其基本架构为：

那么本文简单的介绍一下Ray的安装与基本使用。

安装

由于是一个Python的框架，Ray可以直接使用pip进行安装和管理：

$ python3 -m pip install ray[default]

但是需要注意的是，在所有需要构建集群的设备上，需要统一Python和Ray的版本，因此建议先使用conda创建同样的虚拟环境之后，再安装统一版本的ray。否则在添加集群节点的时候就有可能出现如下问题：

RuntimeError: Version mismatch: The cluster was started with:
    Ray: 2.7.2
    Python: 3.7.13
This process on node 172.17.0.2 was started with:
    Ray: 2.7.2
    Python: 3.7.5

启动和连接服务

一般在配置集群的时候可以先配置下密钥登陆：

$ ssh-keygen -t rsa
$ ssh-copy-id user_name@ip_address

就这么两步，就可以配置远程服务器ssh免密登陆（配置的过程中有可能需要输入一次密码）。然后在主节点（配置一个master节点）启动ray服务：

$ ray start --head --dashboard-host='0.0.0.0' --dashboard-port=8265
Usage stats collection is enabled. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.

Local node IP: xxx.xxx.xxx.xxx

--------------------
Ray runtime started.
--------------------

Next steps
  To add another node to this Ray cluster, run
    ray start --address='xxx.xxx.xxx.xxx:6379'

  To connect to this Ray cluster:
    import ray
    ray.init()

  To submit a Ray job using the Ray Jobs CLI:
    RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . -- python my_script.py

  See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
  for more information on submitting Ray jobs to the Ray cluster.

  To terminate the Ray runtime, run
    ray stop

  To view the status of the cluster, use
    ray status

  To monitor and debug Ray, view the dashboard at
    xxx.xxx.xxx.xxx:8265

  If connection to the dashboard fails, check your firewall settings and network configuration.

这就启动完成了，并给你指示了下一步的操作，例如在另一个节点上配置添加到集群中，可以使用指令：

$ ray start --address='xxx.xxx.xxx.xxx:6379'

但是前面提到了，这里要求Python和Ray版本要一致，如果版本不一致就会出现这样的报错：

RuntimeError: Version mismatch: The cluster was started with:
    Ray: 2.7.2
    Python: 3.7.13
This process on node 172.17.0.2 was started with:
    Ray: 2.7.2
    Python: 3.7.5

到这里其实Ray集群就已经部署完成了，非常的简单方便。

基础使用

我们先用一个最简单的案例来测试一下：

# test_ray.py 
import os
import ray

ray.init()

print('''This cluster consists of
    {} nodes in total
    {} CPU resources in total
'''.format(len(ray.nodes()), ray.cluster_resources()['CPU']))

这个Python脚本打印了远程节点的计算资源，那么我们可以用这样的方式去提交一个本地的job：

$ RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . -- python test_ray.py 
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-27 07:35:10,751 INFO dashboard_sdk.py:338 -- Uploading package gcs://_ray_pkg_4b79155b5de665ce.zip.
2024-08-27 07:35:10,751 INFO packaging.py:518 -- Creating a file package for local directory '.'.

-------------------------------------------------------
Job 'raysubmit_7Uqy8LjP4dxjZxGa' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_7Uqy8LjP4dxjZxGa
  Query the status of the job:
    ray job status raysubmit_7Uqy8LjP4dxjZxGa
  Request the job to be stopped:
    ray job stop raysubmit_7Uqy8LjP4dxjZxGa

Tailing logs until the job exits (disable with --no-wait):
2024-08-27 15:35:14,079 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-27 15:35:14,079 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-27 15:35:14,103 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265 
This cluster consists of
    1 nodes in total
    48.0 CPU resources in total


------------------------------------------
Job 'raysubmit_7Uqy8LjP4dxjZxGa' succeeded
------------------------------------------

这里的信息说明，远程的集群只有一个节点，该节点上有48个可用的CPU核资源。这些输出信息不仅可以在终端窗口上看到，还可以从这里给出的dashboard链接里面看到更加详细的任务管理情况：

这里也顺便提交一个输出软件位置信息的指令，确认下任务是在远程执行而不是在本地执行：

import ray

ray.init()

import numpy as np
print (np.__file__)

返回的日志为：

$ RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . -- python test_ray.py 
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-27 07:46:10,645 INFO dashboard_sdk.py:338 -- Uploading package gcs://_ray_pkg_5bba1a7144beb522.zip.
2024-08-27 07:46:10,658 INFO packaging.py:518 -- Creating a file package for local directory '.'.

-------------------------------------------------------
Job 'raysubmit_kQ3XgE4Hxp3dkmuU' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_kQ3XgE4Hxp3dkmuU
  Query the status of the job:
    ray job status raysubmit_kQ3XgE4Hxp3dkmuU
  Request the job to be stopped:
    ray job stop raysubmit_kQ3XgE4Hxp3dkmuU

Tailing logs until the job exits (disable with --no-wait):
2024-08-27 15:46:12,456 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-27 15:46:12,457 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-27 15:46:12,470 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265 
/home/dechin/anaconda3/envs/mindspore-latest/lib/python3.7/site-packages/numpy/__init__.py

------------------------------------------
Job 'raysubmit_kQ3XgE4Hxp3dkmuU' succeeded
------------------------------------------

$ python3 -m pip show numpy
Name: numpy
Version: 1.21.6
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: 
License: BSD
Location: /usr/local/python-3.7.5/lib/python3.7/site-packages
Requires: 
Required-by: CyFES, h5py, hadder, matplotlib, mindinsight, mindspore, mindspore-serving, pandas, ray, scikit-learn, scipy

这里可以看到，提交的任务中numpy是保存在mindspore-latest虚拟环境中的，而本地的numpy不在虚拟环境中，说明任务确实是在远程执行的。类似的可以在dashboard上面看到提交日志：

接下来测试一下分布式框架ray的并发特性：

import ray

ray.init()

@ray.remote(num_returns=1)
def cpu_task():
    import time
    time.sleep(2)
    import numpy as np
    nums = 100000
    arr = np.random.random((2, nums))
    arr2 = arr[1]**2 + arr[0]**2
    pi = np.where(arr2<=1, 1, 0).sum() * 4 / nums
    return pi

num_conc = 10
res = ray.get([cpu_task.remote() for _ in range(num_conc)])
print (sum(res) / num_conc)

这个案例的内容是用蒙特卡洛算法计算圆周率的值，一次提交10个任务，每个任务中撒点100000个，并休眠2s。那么如果是顺序执行的话，理论上需要休眠20s。而这里提交任务之后，输出如下：

$ time RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . --entrypoint-num-cpus 10 -- python te
st_ray.py 
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-27 08:30:13,315 INFO dashboard_sdk.py:385 -- Package gcs://_ray_pkg_d66b052eb6944465.zip already exists, skipping upload.

-------------------------------------------------------
Job 'raysubmit_Ur6MAvP7DYiCT6Uz' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_Ur6MAvP7DYiCT6Uz
  Query the status of the job:
    ray job status raysubmit_Ur6MAvP7DYiCT6Uz
  Request the job to be stopped:
    ray job stop raysubmit_Ur6MAvP7DYiCT6Uz

Tailing logs until the job exits (disable with --no-wait):
2024-08-27 16:30:15,032 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-27 16:30:15,033 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-27 16:30:15,058 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265 
3.141656

------------------------------------------
Job 'raysubmit_Ur6MAvP7DYiCT6Uz' succeeded
------------------------------------------


real    0m7.656s
user    0m0.414s
sys     0m0.010s

总的运行时间在7.656秒，其中5s左右的时间是来自网络delay。所以实际上并发之后的总运行时间就在2s左右，跟单任务休眠的时间差不多。也就是说，远程提交的任务确实是并发执行的。最终返回的结果进行加和处理，得到的圆周率估计为：3.141656。而且除了普通的CPU任务之外，还可以上传GPU任务：

import ray

ray.init()

@ray.remote(num_returns=1, num_gpus=1)
def test_ms():
    import os
    os.environ['GLOG_v']='4'
    os.environ['CUDA_VISIBLE_DEVICE']='0'
    import mindspore as ms
    ms.set_context(device_target="GPU", device_id=0)
    a = ms.Tensor([1, 2, 3], ms.float32)
    return a.asnumpy().sum()

res = ray.get(test_ms.remote())
ray.shutdown()
print (res)

这个任务是用mindspore简单创建了一个Tensor，并计算了Tensor的总和返回给本地，输出内容为：

$ RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . --entrypoint-num-gpus 1 -- python test_ray.py 
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-28 01:16:38,712 INFO dashboard_sdk.py:338 -- Uploading package gcs://_ray_pkg_10019cd9fa9bdc38.zip.
2024-08-28 01:16:38,712 INFO packaging.py:518 -- Creating a file package for local directory '.'.

-------------------------------------------------------
Job 'raysubmit_RUvkEqnkjNitKmnJ' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_RUvkEqnkjNitKmnJ
  Query the status of the job:
    ray job status raysubmit_RUvkEqnkjNitKmnJ
  Request the job to be stopped:
    ray job stop raysubmit_RUvkEqnkjNitKmnJ

Tailing logs until the job exits (disable with --no-wait):
2024-08-28 09:16:41,960 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-28 09:16:41,960 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-28 09:16:41,974 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265 
6.0

------------------------------------------
Job 'raysubmit_RUvkEqnkjNitKmnJ' succeeded
------------------------------------------

返回的计算结果是6.0，那么也是正确的。

查看和管理任务

前面的任务输出信息中，都有相应的job_id，我们可以根据这个job_id在主节点上面查看相关任务的执行情况：

$ ray job status raysubmit_RUvkEqnkjNitKmnJ

可以查看该任务的输出内容：

$ ray job logs raysubmit_RUvkEqnkjNitKmnJ

还可以终止该任务的运行：

$ ray job stop raysubmit_RUvkEqnkjNitKmnJ

总结概要

本文介绍了基于Python的分布式框架Ray的基本安装与使用。Ray框架下不仅可以通过conda和Python十分方便的构建一个集群，还可以自动的对分布式任务进行并发处理，且支持GPU分布式任务的提交，极大的简化了手动分布式开发的工作量。

版权声明

本文首发链接为：https://www.cnblogs.com/dechinphy/p/ray.html

作者ID：DechinPhy

更多原著文章：https://www.cnblogs.com/dechinphy/

请博主喝咖啡：https://www.cnblogs.com/dechinphy/gallery/image/379634.html