Category Archives: DirectX

Milestones

Leave a reply

The first draft of the paper is done! It comes out at about 12 pages. I’ll need to cut it down to 6 to submit for CHI 2014 WIP. Easier than writing though. Of course, that’s just the first draft. More to come, I’m guessing. Still, it’s a nice feeling, and since I’ve burned through most of my 20% time, it’s time for me to get back to actually earning my pay, so I’ll be taking a break from this blog for a while. More projects are coming up though, so stay tuned. I’ll finish up this post with some images of all the design variations that led to the final, working version:

Prototype Evolution

Prototype Evolution (click to enbiggen)

The chronological order of development is from left to right and top to bottom. Starting at the top left:

  • The first proof of concept. Originally force-input / motion – feedback. It was with this system that I discovered that all actuator motion had to be in relation to a proximal relative base.
  • The first prototype. It had 6 Degrees of freedom, allowing for a user to move a gripper within a 3D environment and grab items. It worked well enough that it led to…
  • The second prototype. A full 5-finger gripper attached to an XYZ base. I ran into problems with this one. It turned out that motion feedback required too much of a cognitive load to work. The user would loose track of where their fingers were, even with the proximal base. So that led to…
  • The third prototype. This used resistive force sensors and vibrotactile feedback. The feedback was provided using voice coils, which were capable of full audio range, which meant that all kinds of sophisticated contact and surface effects could be provided. That proved the point that 5 fingers could work with vibrotactile feedback, but the large scale motions of the base seemed to need motion (I’ve since learned that isometric devices are most effective over short ranges). This was also loaded with electronic concepts that I wanted to try out – Arduino sensing, midi synthesizers per finger, etc.
  • To explore direct motion for the base for the fourth prototype I made a 3D printing of a 5-finger Force Input / Vibrotactile Output (FS/VO) system that would sit on top of a mouse. This was a plug-and play substitution that worked with the previous electronics and worked quite nicely, though the ability to grip doesn’t give you much to do in the XY plane
  • To Get 3D interaction, I took two FS/VO modules and added them to a Phantom Omni. I also dropped the arduino and the synthesizer and the Arduino, using XAudio2 8-channel audio and a Phidgets interface card. This system worked very nicely. The FS/VO elements combined with a force feedback base turned out to be very effective. That’s what became the basis for the paper, and hopefully the basis for future work.
  • Project code is here (MD5: B32EE89CEA9C8E02E5B99BFAF24877A0).

Deadlines and schedules

Leave a reply

I was just asked to see how many hours I have left for working this research. It turns out at the rate I’m going, that I can continue until mid-October. This is basically a big shout-out to Novetta, who has granted a continuation of my 20% time that was originally a hiring condition when I went to work for Edge. Thanks. And if you’d like a programming job in the DC area that supports creativity, give them a call.

I just can’t make the audio code break in writing out results. Odd. Maybe a corrupt input file can have unforeseen effects? Regardless, I’m going to stop pursuing this particular bug without more information

Fixing the state problem. Done.

Fixing the saving issue. Also changing the naming of the speakers to reflect Dolby or not. Done.

New version release built and deployed.

And back to Phantom++

TestScreenV1

I started to add in the user interface that will support experiments. Since it was already done, I pulled in most of the Fluid code from the Vibrotactile headset, which made things pretty easy. I needed to add an enclosing control system class that can move commande between the various pieces.

I’ve also decided that each sound will have an associated object with it. This allows each object to have a simple “acoustic” texture that doesn’t require any fancy data structure.

At this point, I’m estimating that the first version of the test program should be ready by Friday.

Sounds like Deja Vu.

Leave a reply

Adding custom speaker number and placement as per Dr. Kuber’s request.

Looks like dot product should do the trick: DotProduct

Done! With only a couple of string compare issues. I also had to make the speaker index jump around the subwoofer channel until I can work out how to set the EQ.

And it looks like there are bugs in the code. It seems that you cannot do zero speed sessions. And the writing out of results with multiple sound files looks pretty confused. I’m not sure if extra CRs are being put in there or if some of the data isn’t being written out. Need to run some more examples.

What you get when you combine FLTK, OpenGL, DirectX, OpenHaptics and shared memory

Leave a reply

Wow, the title sounds like a laundry list 🙂

Building a two-fingered gripper

Going to add sound class to SimpleSphere so that we know what sounds are coming from what collision. Didn’t do that’ but I’m associating the sounds by index, which is good enough for now

Need to calculate individual forces for each sphere in the Phantom and return them. Done.

To keep the oscillations at a minimum, I’m passing the offsets from the origin. That way the loop uses the device position as the basis for calculations within the haptic loop.
Here’s the result of today’s work:

Some Assembly Required

Leave a reply

Integrating all the pieces into one test platform. The test could be to move a collection of physically-based spheres (easy collision detect) from one area to another. Time would be recorded from the indication of a start and stop (spacebar, something in the sim, etc). Variations would be:

  • Open loop: Measure position and pressure, but no feedback
  • Force Feedback (Phantom) only
  • Vibrotactile feedback only
  • Both feedbacks

Probably only use two actuators for the simplicity of the test rig. It would bean that I could use the laptop’s headphone output. Need to test this by wiring up the actuators to a micro stereo plug. Radio Shack tonight.

Got two-way communication running between Phantom and sim.

Have force magnitude adjusting a volume.

Added a SimpleSphere class for most of the testing.

Enhancements

Leave a reply
  • Meeting with Dr. Kuber.
    • Add a “distance” component to the test and a multiple emitter test
    • Got a bunch of items to add actuators to: Hardhat, noise-blocking headphones, and a push-to-talk mic.
  • Added name and gender fields to the GUI and cleaned up the menus
  • Working on adding multiple sounds
    • Added a ‘next’ button. Once pushed, the sources can show until the center is clicked again.
    • I think the test should have options for how the sounds are added
      • Permutations (A, then B, then C, then AB, AC, BC, ABC)
      • All (Going to start with this)
      • Random?
  • Added variable distance

So that’s what happens when a programming language gets old…

Leave a reply
  • Continuing with the test exec. I’m also going to need a class that records the data associated with each test segment.
  • Ran into a… Well, I don’t want to call it a bug. Let’s say that C++ is showing its age. FLTK uses char*. Most of Windows uses wchar_t. They don’t play well together, so I spent about half of my time working out the best way to convert between them. It’s this:
  • void setSoundFileString(LPCWSTR wps){
	soundFileString = new wstring(wps);
	string str(soundFileString->begin(), soundFileString->end());
	sprintf_s(soundFile, "%s", str.c_str());
}
  • I mean really!? Good grief.
  • Got a lot of the exec built and running. Clicking on the center button fires the sound, and you can drag to where you think the sound is. I am not all that accurate. It could be a frequency thing though. I’m running a low 10-20 HZ signal. The test should definitely try different frequencies.

Let’s make science!

Leave a reply

Alright, so I now have my audio library. Next on the agenda is a test program that tests reactions of users to vibroacoustic input. The test needs to present randomized stimuli to users, so that they can be tested for:

  • Time to respond with a direction
  • Accuracy of direction
  • Efficacy of stimuli

Since this is probably going to be within subjects (multiple stimuli) and also between subjects (same tests on multiple users) we’re going to want to be able to present the same sequence, so we’ll need to seed the random number generator so we get the same sequence.

  • Start with the default random number generator, but maybe run through a wrapper class in case we need something like a Mersenne Twister.
  • Xml file to specify the input and output of the experiment. This library looks reasonable.
    • Input
      • Sounds to use (random distribution of sound use)
      • Test type (Accuracy, Speed, or both)
      • Attempts per test
      • Number of tests (must be even)
      • Random seed
      • min/max delay between test segments
      • output filename
    • Output
      • Test UID
      • Date
      • Time
      • Subject
      • Researcher
      • Free form note field (1024 characters?)
      • Accuracy or Reaction time test
      • Audio configuration
      • Random seed
      • Calibration results
        • Time(s) to click in response to visual cue
        • Time(s) to click in response to audio cue
      • For each played sound
        • Sequence x of total
        • Audio file(s) used (WAV)
        • Audio source position (x, y) in screen coordinates from the origin, where the user’s head is
        • Audio playback matrix (actual speaker relative volume)
        • Time to click after play start
        • Duration of sound
        • Click position (x, y) in screen coordinates from the origin, where the sound is perceived to have come from
  • App
    • Text
      • File navigator for xml file
    • Calibrate
      • Runs a sequence of tests where the user has to click the mouse as quickly as possible in response to the canvas flashing white, and then all(?) speakers in the headpiece playing the calibration sound
      • Calibration cues are have a randomly determined timing between X and Y seconds
      • Test is disabled until calibration is run. Loading a new xml document effectively resets the system, requiring a new calibration sequence
    • Test
      • Shows a label that says either “Accuracy” or “Speed” based on which test is being run. We could change the background of the display as well?
      • The graphics screen shows a circular cursor that resets to the center of the graphics screen at the beginning of each section. Once the audio cue plays, the user can move the mouse away from the center towards the direction of the sound. The circle is clamped in its motion so that the result is always a valid angle, as long as the user moves the cursor far enough away from the center (TBD). Clicking the mouse causes the clock to stop and the cursor to reset.
      • If this is not the last test segment, then a random time period between X and Y seconds elapses before the next test is run.
      • Once the test completes, the system checks to see if that is the last one. If not, a stochastic choice is made to determine if the next test should be speed or accuracy. By the time all tests have run, the number of speed and accuracy runs will be equal.
    • Output file is appended throughout the test (open, write, close? Or read in the DOM, update and write out?)

Started the FLTK wrapper, and probably saved a good deal of time by going back to Erco’s FLTK page and associated videos

Final cleanup on the proof-of concept

Leave a reply
  • Ugh.
  • rain
  • Adding comments and velocity accessors to audio code
    • Comments are done
    • Checked in project to Subversion repo, then checked out and compiled. All seems to be working fine.
  • Cleaning up #includes
    • I had been getting the following annoying warning once I started including items like <unordered_map>: 1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\cstring(21): warning C4995: ‘strcat’: name was marked as #pragma deprecated. In the cleanup, I finally got around to testing the need for includes and deleted <strsafe.h>, which seems to be the source of my grief. Yay!
  • Ready to make the audio code a library. Monday. Then time to design the experimental interface.
  • And now that I think I know what’s going on, a good article shows up…

In the end, this is how you do it:

// create the instance
BasicAudio *ba = new BasicAudio(); 

// initialize
ba->init(); 

// create a sound from a file. In this case a WAV. There can be many of these.
ba->createSound(L"music", L"Wavs\\MusicMono.wav", 0); 

// get the instance to the sound and start(), stop(), run() etc.
ba->getSoundByName(L"music")->start(); 
loop{
	// change some audio condition

	// play the voice on a specified channel or play in 3D
	ba->playOnChannelVoice(L"music", channelIndex); 
 	ba->play3DVoice(L"music");

	// optional - run periodic checks.
	ba->run() 
}
ba->destroy();

Starting to polish

Leave a reply
  • I’m going to add the ability to create and store a named sound within BasicAudio. Sounds really don’t need to be created outside of the instance.
  • Lunch meeting with Shawn about R&D. Good meeting with good chili to boot.
  • Got the createSound method working and cleaned up the Console app some more
int _tmain(int argc, _TCHAR* argv[])
{
	BasicAudio *ba = new BasicAudio();
	ba->init();

	wprintf( L"\nReady to play mono WAV PCM file(s)...\n" );

	WavSampleSound *singleSound = (WavSampleSound *)ba->createSound(L"music", L"Wavs\\MusicMono.wav", 0);
	WavSampleSound *continuousSound = (WavSampleSound *)ba->createSound(L"heli", L"Wavs\\heli.wav", XAUDIO2_LOOP_INFINITE);

	int keyIn;
	printf("Type 'x' to quit\nC start continuous\nc stop continuous\nS start single\n");
	bool doit = true;

	int channelIndex = -1;
	while(doit){
		if(kbhit()){
			keyIn = getch();
			channelIndex = -1;

			printf("key = %c\n", keyIn);
			switch(keyIn){
			case 'x' : doit = false; 

				break;
			case 'C' : continuousSound->start();                  // play sound
				break;
			case 'c' : continuousSound->stop();                  // cease sound
				break;
			case 'S' : singleSound->start();						// play sound
				break;
			case 'w' : 
				continuousSound->setEmitterZ(continuousSound->getEmitterZ() + (FLOAT32)0.5);
				break;
			case 's' : 
				continuousSound->setEmitterZ(continuousSound->getEmitterZ() - (FLOAT32)0.5);
				break;
			case 'a' : 
				continuousSound->setEmitterX(continuousSound->getEmitterX() - (FLOAT32)0.5);
				break;
			case 'd' : 
				continuousSound->setEmitterX(continuousSound->getEmitterX() + (FLOAT32)0.5);
				break;
			case 'p':
				ba->printMatrixCoefficients();
				break;
			case '0' : channelIndex = 0; break;
			case '1' : channelIndex = 1; break;
			case '2' : channelIndex = 2; break;
			case '3' : channelIndex = 3; break;
			case '4' : channelIndex = 4; break;
			case '5' : channelIndex = 5; break;
			case '6' : channelIndex = 6; break;
			case '7' : channelIndex = 7; break;

			}
			if(channelIndex == -1){
				IXAudio2SourceVoice* voice = continuousSound->getSourceVoice();
				ba->play3DVoice(continuousSound->getEmitter(), voice);
			}else{
				printf("channel = %d\n", channelIndex);
				IXAudio2SourceVoice* voice = continuousSound->getSourceVoice();
				ba->playOnChannelVoice(voice, channelIndex);
			}
		}

		ba->run(); // perform periodic sound engine tasks
		Sleep(100);
	}

	wprintf( L"\nFinished playing\n" );

	// All XAudio2 interfaces are released when the engine is destroyed, but being tidy

	ba->destroy();

	// printf("hit return to exit");
	// getchar();

	return 0;
}