Jupyter Notebooks, XBox Controllers and Mathematics

At Sage Days 70, Sylvain Corlay showed some Jupyter notebooks using an XBox gamepad controller as an input device.  I couldn’t resist.

First hurdle: a wireless controller (my son’s) with an add-on USB cable is not the same thing as a “wired controller.”  So I ordered a new one.  Work says they’ll reimburse me (“instructional supplies”).

Three lines of code, plug your controller into a USB port and you’re good to go. 4 variable inputs from the two joysticks, two more variable inputs (the triggers are on 6 and 7, untested) and 13 binary buttons (10, 11 seem unused).  I think this is mostly code in your browser, Firefox or Chrome (I did this all in Chrome).  Perhaps see gamepad.js for more.
Second hurdle: Sylvain has public code showing how to connect a regular iPython slider to react to the controller, but there is some subtle issue so the iPython interact does not behave as you would like.  So finally, I figured out to (a) read the controller input values (easy), (b) display text in Text() widgets (not hard), (c) create a matplotlib PNG image as a byte string and display it in an Image() widget (StackOverflow to the rescue), (d) specify a routine to react to any change in the controller’s state (understand Sylvain’s example code).

Third hurdle: every time I changed the plot update routine, I had to restart the kernel or my changes were not evident.  I’m sure there is a better way.

As a demonstration, I created a sine curve, parameterized by amplitude, frequency, horizontal shift and vertical shift.  So the two joysticks (“four axes”) control all the parameters.  With 500 points for each curve, the display updates smoothly, though seems to lag just a bit.  The equation of the current plot is specified as LaTeX and rendered with MathJax in the notebook.
sine-four-param-demoThe notebook is here:  http://buzzard.ups.edu/demos/xbox-controller-sine.ipynb
(which you will want to download with “Save Link As…” or similar)

I had to justify missing two days of class to travel to Sage Days, so when I got home I told my classes I’d get a demo together.  I asked a student in each of my classes to control a version of the notebook, before I did anything and with no explanation of which joystick controlled which property.  It was fun to see them experiment in front of the class to see how it behaved.

All the math here is just the sine function from numpy.  I’m really looking forward to using the Jupyter server in SageMathCloud, and having all the mathematics and plotting code from Sage available to control via the ipywidgets Controller() object. (I did this with Jupyter notebook server 4.0.6 installed locally, while SageMathCloud is at 4.0.4.)

THANKS to Sylvain for several patient emails while I got this sorted.

5 thoughts on “Jupyter Notebooks, XBox Controllers and Mathematics

  1. Thanks a lot for providing this great example. However I have problems reproducing your results. I can connect the controlller successfully, also the correctly blinking/moving buttons and sliders are shown with that ‘pad’ command (like in your first picture). But the ‘pad.on_trait_change(reaction)’ or the ‘pad.observe(reaction)’ command doesn’t trigger the reaction function. So basically I can read out single values with ‘pad.buttons[15].pressed’ or ‘pad.axes[0].value’, but there are no events triggered when I press a button or move the knobs. I am working with ipywidgets 5.2.2 and jupyter_client 4.3.0. I was trying earlier versions but I unfortunately couldn’t get it to work.

    Did you experience similar things? Which versions of ipywidgets and jupyter were you using?

Leave a Reply

Your email address will not be published. Required fields are marked *