The essence of learning by a robot

First step towards a learning tool based on web standards. It is also a tutorial on cartoon in JavaScript ...

To illustrate how we are going to give the robot a learning function, we'll start with something simple: a finger.

The finger can theoretically perform four movements:

Learning is to control these movements in a graphical interface with a surface representing a portion of the real world. The overall action is recorded. By replaying the data we reproduce this action. This is the essence of learning translated into computer terms.

For this to be useful, we must convert the generated orders into commands for the elements of artificial finger, whose movement depends on a cable connected to an actuator. This corresponds to two commands to the actuator (pull, release) and two commands transmitted to the artificial hand (forward, backward). And at the interface, we only have four functions in JavaScript code.

Realization of a learning interface

Basic demonstration below. The finger of the robot is represented in a canvas (the proportions are not preserved). The user clicks on a point in the frame to move the finger. The interface records what the user does, so it stores the successive points and can restore the sequence of movements when you click Replay.

Click to move the image of the finger ...

Programming the interface

This is essentially the drawing of the finger in a canvas tag.

function phalange3(color, x1, y1, x2, y2, tilt)
{
  ctx.beginPath();
  ctx.strokeStyle = color;	
  ctx.lineWidth = 2;	
  ctx.moveTo(x1, y1);
  ctx.lineTo(x2, y1 + tilt);
  ctx.quadraticCurveTo(x2, y2 + tilt, x1, y2);
  ctx.lineTo(x1, y2);
  ctx.stroke();	
}

function phalange2(color, x1, y1, x2, y2, tilt)
{
  ctx.beginPath();
  ctx.strokeStyle = color;
  ctx.lineWidth = 2;
  ctx.moveTo(x1, y1);
  ctx.lineTo(x2, y1 + tilt);
  ctx.moveTo(x1, y2);
  ctx.lineTo(x2, y2 + tilt);
  ctx.stroke();
}

function phalange1(color, x1, y1, x2, y2, tilt)
{
  ctx.beginPath();
  ctx.strokeStyle = color;
  ctx.lineWidth = 2;
  ctx.moveTo(x1, y1);
  ctx.lineTo(x2, y1 + tilt);
  ctx.moveTo(x1, y2);
  ctx.lineTo(x2, y2 + tilt);
  ctx.stroke();
}

function finger(x, y, tilt1, tilt2, tilt3)
{
  phalange1('#666', x, y, x + 50, y + 50, tilt1);
  phalange2('#888', x + 50, y + tilt2, x + 100, y + 50 + tilt2 , tilt2);
  phalange3('#aaa', x + 100, y + tilt3 , x + 200, y + 50 + tilt3, tilt3 );
}

function redraw(x, y)
{
  var tilt = y / 10;
  ctx.fillStyle = "white";
  ctx.fillRect(0,0, 600, 400);
  finger(10, 10, tilt, tilt, tilt * 2);
}

var canvas = document.getElementById("fakeworld");
var ctx = canvas.getContext("2d");
var fakeworld = document.getElementById("fakeworld");

redraw(10, 10);

The drawing is divided into three phalanges, which would bend the finger, but we will not do all that in this demo.

Recording

To save the movements, we create an array named recording and push on the array each pair of coordinates.

We must also capture the mouse movements and user clicks.

var xMousePosition = 0;
var yMousePosition = 0;

document.onmousemove = function(e)
{
  xMousePosition = e.clientX + window.pageXOffset;
  yMousePosition = e.clientY + window.pageYOffset;
};

function display(arr)
{
  document.getElementById("controler").innerHTML = JSON.stringify(arr, null, 4);
}

var recording = [];

function update()
{
  var x = xMousePosition - fakeworld.offsetLeft;
  var y = yMousePosition - fakeworld.offsetTop;
  redraw(x, y);
  recording.push([x, y]);
  display(recording);
}

fakeworld.onclick = update;

We therefore assign the update function to the onclick event for the canvas, this function redraws the finger by a call to redraw and add a couple of coordinates. The array's content is displayed by the display function.

Replay

It mimics the movement as a cartoon, showing the successive images of the finger moves.

Ideally we should record the durations at the same time as the movement, but in this demonstration we will simplify and provide a unique delay between two single movements, which is added during replay.

var replayed = [];

function step(couple)
{
  var x = couple[0];
  var y = couple[1];
  replayed.push(couple);
  display(replayed);
  redraw(x, y);
}

function iterate(i)
{
  if(i >= recording.length) return;
  setTimeout(function() {
    step(recording[i])
    i++;
    iterate(i);
  }, 700);
}

function replay()
{
  replayed = [];
  var couple = recording[0];
  iterate(0, couple);
}

The replayed array is filled to record and display the coordinates being unwound. The iterate function is recursive. This is necessary because to create a delay between two drawings, we use setTimeout which is asynchronous. This allows chaining a correct sequence of drawings.

What is done in replay is also what will do the real robot. The same data instead of being routed to the canvas tag, will be to the actuators of the robot and used as arguments of electronic controls. Of course, this demonstration presents just the principle and is basic. For serious 3D Animation you should use a software like Anim8or, Blender, OpenFX and many others.
There are also protocols for man-machine communication called HID (Human Interface Device) which includes Bluetooth HID.

Deep learning and pattern recognition

Deep learning and pattern recognition