Category Archives: API

TypeScript and AngularJS Unification?

Because of certain obscure reasons, AngularJS needs to do very particular things with “this”. Shortly after taking up TypeScript and trying to feed in Angular ‘objects’, I learned about how “Fat Arrow Notation” (FAN) allows for a clean interface with Angular. It’s covered in earlier posts, but the essence is

// Module that uses the angular controller, directive, factory and service defined above.
module AngularApp {
   // define how this application assembles.
   class AngularMain {
      serviceModule:ng.IModule;
      appModule:ng.IModule;

      public doCreate(angular:ng.IAngularStatic, tcontroller:Function, tservice:Function, tfactory:Function, tdirective:Function) {
         this.serviceModule = angular.module('globalsApp', [])
            .factory('GlobalsFactory', [tfactory])
            .service('GlobalsService', [tservice]);

         this.appModule = angular.module('simpleApp', ['globalsApp'])
            .controller('MainCtrl', ['GlobalsFactory', 'GlobalsService', '$timeout', tcontroller])
            .directive('testWidget', ['GlobalsService', tdirective]);
      }
   }
   // instantiate Angular with the components defined in the 'InheritApp' module above. Note that the directive and the factory
   // have to be instantiated before use.
   new AngularMain().doCreate(angular,
      InheritApp.TestController2,
      InheritApp.TestService,
      new InheritApp.TestFactory().ctor,
      new InheritApp.TestDirective().ctor);
}

Basically, the Angular parts that need to new() components (Controllers and Services) get the function pointer to the class, while components that depend on the object already being created (Directives and Factories) have a function pointer passed in that returns an object that in turn points to the innards of the class.

So I refactored all my webGL code to FAN and lo, all was good. I made good progress on building my shiny 3D charts.

Well, charts are pretty similar, so I wanted to take advantage of TypeScript’s inheritance, make a BaseChart class, which I would then extend to Area, Bar, Column, Scatter, etc. What I expected to be able to do was take a base method:

public fatArrowFunction = (arg:string):void => {
   alert("It's Parent fatArrowFunction("+arg+")");
};

And extend it:

public fatArrowFunction = (arg:string):void => {
   super.fatArrowFunction(arg)
   alert("It's Child fatArrowFunction("+arg+")");
};

“Um, no.”, said the compiler. “super() cannot be used with FAN”.

“WTF?” Said I.

It turns out that this is a known not-really-a-bug, that people who are combining Angular and TypeScript run into. After casting around for a bit, I found the fix as well:

class Base {
   constructor() {
      for (var p in this) {

         if (!Object.prototype.hasOwnProperty.call(this, p) && typeof this[p] == 'function') {
            var method = this[p];
            this[p] = () => {
               method.apply(this, arguments);
            };
            // (make a prototype method bound to the instance)
         }
      }
   }
}

Basically what this does is scan through the prototype list and set a bunch of fat arrow function pointers that point back to the prototype function. It seems that there are some people that complain, but as a developer who cut his teeth on C programming, I find function pointers kind of comforting. They become a kind of abbreviation of some big complex thing.

The problem is that the example doesn’t’ quite work, at least in the browsers I’m currently using (Chrome 41, IE 11, FF 36). Instead of pointing at their respective prototypes, all the pointers appear to reference the last item of the loop. And the behavior doesn’t show up well in debuggers. I had to print the contents of the function to see that the pointer named one thing was pointing at another. And this happened in a number of contexts. For example, this[fnName] = () => {this[‘__proto__’][fnName].apply(this, arguments);} gives the same problem.

After a few days of flailing and learning a lot, I went back to basics and tried setting the function pointers explicitly in the constructor of each class. It worked, and it wasn’t horrible. Then, and pretty much just for kicks, I added the base class back in with this method:

public setFunctionPointer(self:any, fnName:string):void{
   this[fnName] = function () {
      //console.log("calling ["+fnName+"]")
      return self['__proto__'][fnName].apply(self, arguments);
   };
}

And gave it a shot. And it worked! I was pleasantly surprised. And because I’m an eternal optimist, I added the loop back, but this time using the function call:

constructor() {
   var proto:Object = this["__proto__"];
   var methodName:string;

   for (var p in proto){
         methodName = p;
         if(methodName !== 'constructor'){
            this.setFunctionPointer(this, methodName);
         }
         //console.log("\t"+methodName+" ("+typeof proto[p]+")");
      }
}

And that, my droogs, worked.

I think it’s a prototype chaining issue, but I’m not sure how. In the non-working code, we’re basically setting this[fnName] = function () { this[fnName].apply(self, arguments)}. That should chain up to the prototype and work, but I don’t think it is. Rather, all the functions wind up chaining to the same place.

function Base() {
    var _this = this;
    for (var p in this) {
        if (!Object.prototype.hasOwnProperty.call(this, p) && typeof this[p] == 'function') {
            var method = this[p];
            this[p] = function () {
                method.apply(_this, arguments);
            };
        }
    }
}

On the other hand, look at the code generated when we use the function we get the following:

var ATSBase = (function () {
    function ATSBase() {
        var proto = this["__proto__"];
        var methodName;
        for (var p in proto) {
            methodName = p;
            if (methodName !== 'constructor') {
                this.setFunctionPointer(this, methodName);
            }
        }
    }
    
    ATSBase.prototype.setFunctionPointer = function (self, fnName) {
        this[fnName] = function () {
            //console.log("calling ["+fnName+"]")
            return self['__proto__'][fnName].apply(self, arguments);
        };
    };
    return ATSBase;
})();

Now, rather than starting at the root, the actual call is done in the prototype. I think this may cause the chain to start in the prototype object, but then again, looking at the code, I don’t see why that should be the case. One clear difference is the fact that in the first version, “this” can be in two closure states (this[p] = function (){method.apply(_this, arguments);};). So it could be closure is behaving in less than obvious ways.

Unfortunately, we are at the point in development where something works, so it’s time to move on. Maybe later after the codebase is more mature, I’ll come back and examine this further. You can explore a running version here.

Typescript Headers and Browser Quirks.

It’s been a pretty good week. The WebGl graphics in the directive are connected to the user functionality in the controller, I have tooltips running, and even have raycasting working, so the 2D items appear in the overlay plane above the 3D object:

AngularJSWebGl

 

The big problem that I needed to chase down was circular references in the typescript files. TypeScript uses reference path comments to tell the compiler where to look for type and structure information. Below is the information that I need for the angular module that creates the above application

/// <reference path="../../definitelytyped/angularjs/angular.d.ts" />
/// <reference path="../controllers/WGLA1_controller.d.ts" />
/// <reference path="../directives/WGLA2_directives.d.ts" />

In this case note that there is a path for controller and directive code. In this case, pointing directly to the code file is fine, but I have a case where my WebGLCanvas has to know about WebGLComponents and vice versa. The typescript compiler (tsc) doesn’t like that, and barfs a ‘duplicate definition’ error. At this point, I was wondering why TypeScript doesn’t have a #pragma once directive that would prevent this sort of thing, or even an #ifndef capability. It’s a preprocessor after all, and it should be able to do this. Easily.

But TypeScript does have interfaces. So in this case, I put interfaces for both modules in a single file, which I could then refer to in the downstream files and avoid the circular dependency issue.

The other issue was browsers not playing well together. I kind of thought that we had gotten beyond that, but no.

I develop with IntelliJ, and their debugger plays the best with Chrome, so that’s my default browser. At the end of the day, I’ll check to see that everything runs in IE and FF. And today FF was not playing well, and the tooltips I worked so hard on were not showing. WTF, I say.

If you look at the screenshot above, you’ll see the white text at the upper left. That’s my real-time logging (it’s pointless to write to the console at 30hz). And I could see that the unit mouse values were NaN. Again, WTF.

Now FF has my favorite debugger, and it even works (generally) with typescript, as long as you have all the .ts and .map files alongside your .js files. So I stepped into the code at the handleMouseEvents() method in WebGlCanvasClasses and started looking.

I’ve been getting the mouse coordinate from MouseEvent.offsetX. That turns out be used by IE and Chrome, but not FF. so I changed

var sx:number = ev.offsetX; to var sx:number = ev.offsetX | ev.layerX;

All fixed, I thought. But wait! There’s more! It turns out that IE has both of these values, and they don’t mean the same thing. so in the end I wind up with the following monkeypatch:

handleMouseEvents = (ev:MouseEvent):void => {
    var sx = ev.layerX;
    var sy = ev.layerY;

    if(ev.offsetX < sx){
        sx = ev.offsetX;
        sy = ev.offsetY;
    }
}

This works because the smaller value has to be the coordinate of the mouse on the div I’m interested, since all screen coordinates increase from 0. So it’s quick, but jeez.

Text Overlay for ThreeJS with Angular and TypeScript

This is not my first foray into WebGL. The last time I was working on a 3D charting API using the YUI framework, which could do things like this:

Personally, I can’t do any debugging at 30fps without having a live list of debugging text that I can watch. So almost immediately after the ‘hello world’ spinning cube, I set that up. And now I’m in the middle of moving my framework over to Angular and TypeScript. For the most part, I like how things are working out, but when it comes to lining up a transparent text plane over a threeJS element, YUI gives a lot more support than Angular. The following is so brute-force that I feel like I must be doing it wrong (And there may be a jquery-lite pattern, but after trying a few StackOverflow suggestions that didn’t work), I went with the following.

First, this all happens in the directive. I try to keep that pretty clean:

// The webGL directive. Instantiates a webGlBase-derived class for each scope
export class ngWebgl {
   private myDirective:ng.IDirective;

   constructor() {
      this.myDirective = null;
   }

   private linkFn = (scope:any, element:any, attrs:any) => {
      //var rb:WebGLBaseClasses.RootBase = new WebGLBaseClasses.RootBase(scope, element, attrs);
      var rb:WebGlRoot = new WebGlRoot(scope, element, attrs);
      scope.webGlBase = rb;
      var initObj:any = {
         showStage: true
      };
      rb.initializer(initObj);
      rb.animate();
   };

   public ctor = ():ng.IDirective => {
      if (!this.myDirective) {
         this.myDirective = {
            restrict: 'AE',
            scope: {
               'width': '=',
               'height': '=',
            },
            link: this.linkFn
         }
      }
      return this.myDirective;
   }
}

The interface with all the webGL code happens in the linkFn() method. Note that the WebGLRoot class gets assigned to the scope. This allows for multiple canvases.

WebGLRoot is a class that inherits from WebGLBaseClasses.CanvasBase, which is one of the two big classes I’m currently working on. It’s mostly there to make sure that everything inherits correctly and I don’t break that without noticing:-)

Within WebGLBaseClasses.CanvasBase is the initializer() method. That in turn calls the methods that set up the WebGL and the ‘stage’ that I want to interact with. The part we’re interested for our overlay plane is the overlay canvas’ context. You’ll needthat  to draw into later:

overlayContext:CanvasRenderingContext2D;

This is set up along with the renderer. Interesting bits are in bold:

this.renderer = new THREE.WebGLRenderer({antialias: true});
this.renderer.setClearColor(this.blackColor, 1);
this.renderer.setSize(this.contW, this.contH);

// element is provided by the angular directive
this.renderer.domElement.setAttribute("class", "glContainer");
this.myElements[0].appendChild(this.renderer.domElement);

var overlayElement:HTMLCanvasElement = document.createElement("canvas");
overlayElement.setAttribute("class", "overlayContainer");
this.myElements[0].appendChild(overlayElement);
this.overlayContext = this.overlayElement.getContext("2d");

The first thing to notice is that I have to add CSS classes to the elements. These are pretty simple, just setting absolute and Z-index:

.glContainer {
    position: absolute;
    z-index: 0;
}

.overlayContainer {
    position: absolute;
    z-index: 1;
}

That forces everything to have the same upper left corner. And once that problem was solved, drawing is pretty straightforward. The way I have things set up is with an animate method that uses requestAnimationFrame() wich then calls the render() method. That draws the 3D, and then hands the 2D context off to the draw2D() method:

draw2D = (ctx:CanvasRenderingContext2D):void =>{
   var canvas:HTMLCanvasElement = ctx.canvas;
   canvas.width = this.contW;
   canvas.height = this.contH;
   ctx.clearRect(0, 0, canvas.width, canvas.height);
   ctx.font = '12px "Times New Roman"';
   ctx.fillStyle = 'rgba( 255, 255, 255, 1)'; // Set the letter color
   ctx.fillText("Hello, framecount: "+this.frameCount, 10, 20);
};

render = ():void => {
   // do the 3D rendering
   this.camera.lookAt(this.scene.position);
   this.renderer.render(this.scene, this.camera);
   this.frameCount++;

   this.draw2D(this.overlayContext);
};

I’m supplying links to to the running code and directives, but please bear in mind that this is in-process development and not an minimal application for clarity.

Milestones

The first draft of the paper is done! It comes out at about 12 pages. I’ll need to cut it down to 6 to submit for CHI 2014 WIP. Easier than writing though. Of course, that’s just the first draft. More to come, I’m guessing. Still, it’s a nice feeling, and since I’ve burned through most of my 20% time, it’s time for me to get back to actually earning my pay, so I’ll be taking a break from this blog for a while. More projects are coming up though, so stay tuned. I’ll finish up this post with some images of all the design variations that led to the final, working version:

Prototype Evolution

Prototype Evolution (click to enbiggen)

The chronological order of development is from left to right and top to bottom. Starting at the top left:

  • The first proof of concept. Originally force-input / motion – feedback. It was with this system that I discovered that all actuator motion had to be in relation to a proximal relative base.
  • The first prototype. It had 6 Degrees of freedom, allowing for a user to move a gripper within a 3D environment and grab items. It worked well enough that it led to…
  • The second prototype. A full 5-finger gripper attached to an XYZ base. I ran into problems with this one. It turned out that motion feedback required too much of a cognitive load to work. The user would loose track of where their fingers were, even with the proximal base. So that led to…
  • The third prototype. This used resistive force sensors and vibrotactile feedback. The feedback was provided using voice coils, which were capable of full audio range, which meant that all kinds of sophisticated contact and surface effects could be provided. That proved the point that 5 fingers could work with vibrotactile feedback, but the large scale motions of the base seemed to need motion (I’ve since learned that isometric devices are most effective over short ranges). This was also loaded with electronic concepts that I wanted to try out – Arduino sensing, midi synthesizers per finger, etc.
  • To explore direct motion for the base for the fourth prototype I made a 3D printing of a 5-finger Force Input / Vibrotactile Output (FS/VO) system that would sit on top of a mouse. This was a plug-and play substitution that worked with the previous electronics and worked quite nicely, though the ability to grip doesn’t give you much to do in the XY plane
  • To Get 3D interaction, I took two FS/VO modules and added them to a Phantom Omni. I also dropped the arduino and the synthesizer and the Arduino, using XAudio2 8-channel audio and a Phidgets interface card. This system worked very nicely. The FS/VO elements combined with a force feedback base turned out to be very effective. That’s what became the basis for the paper, and hopefully the basis for future work.
  • Project code is here (MD5: B32EE89CEA9C8E02E5B99BFAF24877A0).

First Results :-)

Downloaded several wav files of sine wave tones, ranging from 100hz to 1,000hz. The files are created using this generator, and are 0.5 sec in length.

Glued the tactor actuators in place, since they kept on coming loose during the testing

Fixed the file outputEach test result is now ordered

Fixed a bug where the number of targets and the number of goals were not being recorded

Added a listing of the audio files used in the experiment.

Got some initial results based on my self-testing today: firstResults
The pure haptic and tactor times to perform the task are all over the place, but it’s pretty interesting to note that Haptic/Tactor and Open Loop are probably significantly different. Hmmmm.

Packaging!

Ok, here it is, all ready to travel:

IMG_2192

It’s still a bit of a rat’s nest inside the box, but I’ll clean that up later today.

Adding a “practice mode” to the app. It will read in a setup file and allow the user to try any of the feedback modalities using srand(current milliseconds) – done

Sent an email off to these folks asking how to get their C2-transducers.

Need to look into perceptual equivalence WRT speech/nonspeech tactile interaction. Here’s one paper that might help: http://www.haskins.yale.edu/Reprints/HL0334.pdf

Fixed my truculent pressure sensor and glued the components into the enclosure. Need to order a power strip.

Life can be a drag sometimes

  • Cleaning up commands. Mostly done
  • While testing the “test” part of the app, I’m realizing that my “ratio” calculations have some issues. Before tying to fix them directly, I’m going to try just making a different gripper that has three “sensor spheres” on each finger. Then I can just let my “drag-based” physics to the whole job. Finished. That’s much better
  • Quick! What’s wrong with the following code?
	for(int i = 0; i < 3; ++i){
		position[i] += velocityVec[i];
		if(velocityVec[i] > drag*ratio){
			velocityVec[i] -= drag*ratio;
		}else{
			velocityVec[i] = 0;
		}
	}
  • Yep, the drag is only being applied for objects moving in a positive direction. This is a problem that has been driving me crazy for days. I thought is was some artifact of the communication between the Phantom control loop (1k hz) and the simulation loop (100 hz). Nope. Simple math mistake. Facepalm.
  • Pretty picture for the day. Notice that the grippers now have multiple points of contact:

BetterGripper

  • I’ve also started to notice how feedback changes the speed that you can perform the task. Haptic and tactor seem pretty close. Open loop is much worse, at least subjectively. Let’s see what the data says.

Zeno’s development schedule

Ok, it’s not as bad as that paradox, but the amount of work remaining always grows when you get closer to the end. Still, I think I’ll be ready by the end of the week.

  • Goal Box – done. I had to add collision detection for determining if a TargetSphere was touching (for setup) or inside (achieving the goal). I basically wrote an axis-aligned bounding box where the radius of the TargetSphere was either added (inside) to or subtracted from (touching) the size of the GoalBox. I’m not calculating penetration, just looking for sign change on the line segment.
  • Added <map> of UI_cmd to handle commands coming from the control system and results coming from the sim. Worked right the first time. Yay, C++ templates!
  • Enable/disable haptics/tactors – done. It’s interesting to see how the behavior and feel of the system changes with the different capabilities enabled/disabled.
  • Started working on the experiment session management

Picture for the day:GoalBox

Blew my hand off for a while

I’m in the process of turning the Phantom testbed code into a research tool. This means that a lot of items that have been #defines now need to be variables and such.

One of the mechanisms that the shared memory app uses to communicate is a char[255] message. I basically sprintf whatever I want into that, and I can then debug both applications simultaneously.

However, after checking to see that some data were coming across correctly, I took the formatting argument out of the sprintf statement and left the value in. Suddenly I was overflowing the 255 limit and causing all kinds of havoc. Took a few hours to chase that one down. That’s what you get for playing with C/C++. Moving on.

Anyway, I now have an event handling loop, and am able to load target spheres into the application and associate them with a sound file. Tommorrow we’ll try getting the sounds associated with the targets to play. There are some issues, primarily that the gripper can touch multiple targets simultaneously. Still, it looks pretty straightforward. After that I’ll start to roll in the TestManager and TestResults classes into the application.

The other thing to do for the day is to check out the headset code with Brian this evening in the lab and see if the output file bug has either disappeared or can be replicated.