Installing a Xeon Phi coprocessor on Debian Squeeze

Written by erik - 29 july 2013

Today I installed a Xeon Phi coprocessor in a Debian box. The software is only supported for Red Hat systems so I was a little bit afraid. Fortunately it was easy. I took most of my information from the readme files of mpssd and from that guy. Thanks to him because he did most of the job.

Ok, let's get started. Hardware wise, you need a 16x PCI-express 3.0 slot and you need the BIOS to be configured to be compatible with devices with more than 4GB of ressource. There was a NVIDIA K20 with 6GB of memory plugged on the machine I used; but still the BIOS complained. A simple BIOS configuration allowed the machine boot. (Check the BIOS configuration under PCI-express.)

From here I did everything as root. Get the MPSS archive from Intel's website. (Mine was called mpss_gold_update_3-2.1.6720-15-rhel-6.3.tar). Untar it with tar xvf mpss_gold_update_*.tar. You'll get a bunch of RPM files. Install alien if you do not already have it with aptitude install alien. Convert all the RPM files except the -kmod- one into DEB files with for file in `ls *.rpm | grep -v kmod` ; do alien --scripts $file; done. Install them all with for file in `ls *.deb | grep -v kmod`;do dpkg --install $file; done.

Up to now, we installed all the files you need beside the kernel module. The kernel module that is provided is only for Red Hat system (with a kernel 2.6.32). My machine had a kernel 2.6.39, so I needed to do a little bit more work. Fortunately, the source for the kernel are available in the src/ directory. Convert that RPM file into a tgz with alien: alien -t intel-mic-kmod-*.src.rpm. When you uncompress it with tar zxvf intel-mic-kmod-*tgz you get a .spec file you do not care about and a tar.bz2 file that contain the kernel module source code. Uncompress it with tar jxvf intel-mic-kmod-*.tar.bz2. Compile it with make and install it with make install. You should now have a kernel module in /lib/modules/`uname -a`/extra/mic.ko. Somehow, the installation script does not run depmod to declare the module so you need to run it manually with depmod.

To use a Xeon Phi you also need a service running on the host. The files are already installed, but there is no startup script for the service. Go to /etc/init.d. We will use the skeleton script as a base. cp skeleton mpssd. You need to edit that file and perform the following modifications. Change the 3rd line to Provides: mpssd. Change line 22 to DESC="MIC service deamon". Change line 23 to NAME=mpssd. Comment line 25, you don't need arguments. At the beginning of function do_start, add the following line modprobe mic; [ -d "/var/lock/subsys" ] || mkdir /var/lock/subsys. The first part makes sure the mic kernel module is loaded. That should not be necessary because a udev rule was added earlier, but somehow I needed it. The second part ensures that the directory used by mpssd to store its lock file is properly created. Save and quit. Add the proper permission to the startup script with chmod +x mpssd. And at that point the service can be started with service mpssd start. Starting and stopping the service takes time (about a minute).

Now you need to be able to use the card. Typically one logs on the card using ssh. The card automatically takes the private IP address 172.31.1.1 but it is not routable automatically from linux. So you need to add a network interface in /etc/network/interfaces:

iface mic0 inet static
  address 172.31.1.254
  netmask 255.255.255.0

Load the interface with ifup mic0. To log on the card, you will need to ssh to the card using key-based autentification. If you have local users to the machine (I do not think that NIS based ones are supported with that technique), you should make sure that you have an ssh key on the ones that want to use the MIC card. If not, just get them to run ssh-keygen. After that, you need to load the ssh keys on the MIC card but mpssd need to be off to do that. Just running service stop mpssd; micctrl --resetconfig; service start mpssd should do the trick. I am not sure at which point, but an entry to /etc/hosts was made. So ssh mic0 should just work; otherwise ssh 172.31.1.1.

From there you are pretty much all set as you have a working installation of your Xeon Phi. You will certainly need ICC to do anything useful. Some people mount the host filesystem on the device, which can be pretty useful. I won't cover these because they are not really Debian specific and other people will certainly explain these better than me. NFS mounting is explained in MPSS readme file but I haven't tested it.

Happy hacking

EDIT on 2014/01/28: it seems that to use the offload mode, one need to have proper permission on /dev/mic/*. You can edit the udev rule in /etc/udev/rules.d/50-udev-scif.rules.

Classified in : Homepage, geek, programming, en - Tags : xeonphi, linux

7 comments

monday 12 august 2013 @ 13:22 Yucheng said : #1

The "depmod" thing really helped me! Thank you very much for sharing your knowledge!

friday 16 august 2013 @ 14:53 erik said : #2

glad I can help!

monday 28 october 2013 @ 10:03 Klaus Wedel said : #3

I fount this site also very, very helpful:
http://www.theismus.de/HPCBlog/

sunday 17 november 2013 @ 17:03 erik said : #4

Klaus Wedel, interesting, thanks for the link!

friday 14 march 2014 @ 05:49 Virginie said : #5

Hi !
I need to install Xeon Phi, but my motherboard is not compatible. I will buy a new one, but I want to be sure of my choice.
Could you tell me which one you are using for Xeon Phi ?
Thanks in advance.

saturday 22 march 2014 @ 12:33 erik said : #6

The one i used was a server board from super micro. sometimes the motherboard is not incompatible but there is a bios setting to change.

friday 04 july 2014 @ 18:04 Bioman said : #7

J'ai installé une Néon en faisant Phi de tout cela. Ce n'était pas très kmod cependant.

Without jokes, Thanks for the tips !

Write a comment

What is the third letter of the word gnqp? : 

Categories

Archives

Tags

Last articles

Last comments