Category Archives: DirectX

Let’s make science!

Alright, so I now have my audio library. Next on the agenda is a test program that tests reactions of users to vibroacoustic input. The test needs to present randomized stimuli to users, so that they can be tested for:

  • Time to respond with a direction
  • Accuracy of direction
  • Efficacy of stimuli

Since this is probably going to be within subjects (multiple stimuli) and also between subjects (same tests on multiple users) we’re going to want to be able to present the same sequence, so we’ll need to seed the random number generator so we get the same sequence.

  • Start with the default random number generator, but maybe run through a wrapper class in case we need something like a Mersenne Twister.
  • Xml file to specify the input and output of the experiment. This library looks reasonable.
    • Input
      • Sounds to use (random distribution of sound use)
      • Test type (Accuracy, Speed, or both)
      • Attempts per test
      • Number of tests (must be even)
      • Random seed
      • min/max delay between test segments
      • output filename
    • Output
      • Test UID
      • Date
      • Time
      • Subject
      • Researcher
      • Free form note field (1024 characters?)
      • Accuracy or Reaction time test
      • Audio configuration
      • Random seed
      • Calibration results
        • Time(s) to click in response to visual cue
        • Time(s) to click in response to audio cue
      • For each played sound
        • Sequence x of total
        • Audio file(s) used (WAV)
        • Audio source position (x, y) in screen coordinates from the origin, where the user’s head is
        • Audio playback matrix (actual speaker relative volume)
        • Time to click after play start
        • Duration of sound
        • Click position (x, y) in screen coordinates from the origin, where the sound is perceived to have come from
  • App
    • Text
      • File navigator for xml file
    • Calibrate
      • Runs a sequence of tests where the user has to click the mouse as quickly as possible in response to the canvas flashing white, and then all(?) speakers in the headpiece playing the calibration sound
      • Calibration cues are have a randomly determined timing between X and Y seconds
      • Test is disabled until calibration is run. Loading a new xml document effectively resets the system, requiring a new calibration sequence
    • Test
      • Shows a label that says either “Accuracy” or “Speed” based on which test is being run. We could change the background of the display as well?
      • The graphics screen shows a circular cursor that resets to the center of the graphics screen at the beginning of each section. Once the audio cue plays, the user can move the mouse away from the center towards the direction of the sound. The circle is clamped in its motion so that the result is always a valid angle, as long as the user moves the cursor far enough away from the center (TBD). Clicking the mouse causes the clock to stop and the cursor to reset.
      • If this is not the last test segment, then a random time period between X and Y seconds elapses before the next test is run.
      • Once the test completes, the system checks to see if that is the last one. If not, a stochastic choice is made to determine if the next test should be speed or accuracy. By the time all tests have run, the number of speed and accuracy runs will be equal.
    • Output file is appended throughout the test (open, write, close? Or read in the DOM, update and write out?)

Started the FLTK wrapper, and probably saved a good deal of time by going back to Erco’s FLTK page and associated videos

Final cleanup on the proof-of concept

  • Ugh.
  • rain
  • Adding comments and velocity accessors to audio code
    • Comments are done
    • Checked in project to Subversion repo, then checked out and compiled. All seems to be working fine.
  • Cleaning up #includes
    • I had been getting the following annoying warning once I started including items like <unordered_map>: 1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\cstring(21): warning C4995: ‘strcat’: name was marked as #pragma deprecated. In the cleanup, I finally got around to testing the need for includes and deleted <strsafe.h>, which seems to be the source of my grief. Yay!
  • Ready to make the audio code a library. Monday. Then time to design the experimental interface.
  • And now that I think I know what’s going on, a good article shows up…

In the end, this is how you do it:

// create the instance
BasicAudio *ba = new BasicAudio(); 

// initialize
ba->init(); 

// create a sound from a file. In this case a WAV. There can be many of these.
ba->createSound(L"music", L"Wavs\\MusicMono.wav", 0); 

// get the instance to the sound and start(), stop(), run() etc.
ba->getSoundByName(L"music")->start(); 
loop{
	// change some audio condition

	// play the voice on a specified channel or play in 3D
	ba->playOnChannelVoice(L"music", channelIndex); 
 	ba->play3DVoice(L"music");

	// optional - run periodic checks.
	ba->run() 
}
ba->destroy();

Starting to polish

  • I’m going to add the ability to create and store a named sound within BasicAudio. Sounds really don’t need to be created outside of the instance.
  • Lunch meeting with Shawn about R&D. Good meeting with good chili to boot.
  • Got the createSound method working and cleaned up the Console app some more
int _tmain(int argc, _TCHAR* argv[])
{
	BasicAudio *ba = new BasicAudio();
	ba->init();

	wprintf( L"\nReady to play mono WAV PCM file(s)...\n" );

	WavSampleSound *singleSound = (WavSampleSound *)ba->createSound(L"music", L"Wavs\\MusicMono.wav", 0);
	WavSampleSound *continuousSound = (WavSampleSound *)ba->createSound(L"heli", L"Wavs\\heli.wav", XAUDIO2_LOOP_INFINITE);

	int keyIn;
	printf("Type 'x' to quit\nC start continuous\nc stop continuous\nS start single\n");
	bool doit = true;

	int channelIndex = -1;
	while(doit){
		if(kbhit()){
			keyIn = getch();
			channelIndex = -1;

			printf("key = %c\n", keyIn);
			switch(keyIn){
			case 'x' : doit = false; 

				break;
			case 'C' : continuousSound->start();                  // play sound
				break;
			case 'c' : continuousSound->stop();                  // cease sound
				break;
			case 'S' : singleSound->start();						// play sound
				break;
			case 'w' : 
				continuousSound->setEmitterZ(continuousSound->getEmitterZ() + (FLOAT32)0.5);
				break;
			case 's' : 
				continuousSound->setEmitterZ(continuousSound->getEmitterZ() - (FLOAT32)0.5);
				break;
			case 'a' : 
				continuousSound->setEmitterX(continuousSound->getEmitterX() - (FLOAT32)0.5);
				break;
			case 'd' : 
				continuousSound->setEmitterX(continuousSound->getEmitterX() + (FLOAT32)0.5);
				break;
			case 'p':
				ba->printMatrixCoefficients();
				break;
			case '0' : channelIndex = 0; break;
			case '1' : channelIndex = 1; break;
			case '2' : channelIndex = 2; break;
			case '3' : channelIndex = 3; break;
			case '4' : channelIndex = 4; break;
			case '5' : channelIndex = 5; break;
			case '6' : channelIndex = 6; break;
			case '7' : channelIndex = 7; break;

			}
			if(channelIndex == -1){
				IXAudio2SourceVoice* voice = continuousSound->getSourceVoice();
				ba->play3DVoice(continuousSound->getEmitter(), voice);
			}else{
				printf("channel = %d\n", channelIndex);
				IXAudio2SourceVoice* voice = continuousSound->getSourceVoice();
				ba->playOnChannelVoice(voice, channelIndex);
			}
		}

		ba->run(); // perform periodic sound engine tasks
		Sleep(100);
	}

	wprintf( L"\nFinished playing\n" );

	// All XAudio2 interfaces are released when the engine is destroyed, but being tidy

	ba->destroy();

	// printf("hit return to exit");
	// getchar();

	return 0;
}

There are days things just work

  • Making nice clean classes for audio
  • Emitter goes into the SampledSound base class – done. Almost easy, though I had to go and relearn how C++ constructors work. Gawd, it’s been a while…
  • Everything else goes into BasicAudio – done. Had some problems with frequency until I realized that I hadn’t zeroed out the listener velocity.
  • And actually, I need to be able to set velocity for doppler effects. Need to add methods like the position methods tomorrow.
  • So the classes are built, and my main() test loop is down to this:
int _tmain(int argc, _TCHAR* argv[])
{
	BasicAudio *ba = new BasicAudio();
	ba->init();
	IXAudio2* pXAudio2 = ba->getXaudioPtr();
	wprintf( L"\nReady to play mono WAV PCM file(s)...\n" );

	WavSampleSound *continuousSound = new WavSampleSound();
	WavSampleSound *singleSound = new WavSampleSound();

	continuousSound->initPCM(pXAudio2, L"Wavs\\heli.wav", XAUDIO2_LOOP_INFINITE );
	singleSound->initPCM(pXAudio2, L"Wavs\\MusicMono.wav", 0 );

	ba->addSampleSound(continuousSound);
	ba->addSampleSound(singleSound);

	int keyIn;
	printf("Type 'x' to quit\nC start continuous\nc stop continuous\nS start single\n");
	bool doit = true;

	int channelIndex = -1;
	while(doit){
		if(kbhit()){
			keyIn = getch();
			channelIndex = -1;

			printf("key = %c\n", keyIn);
			switch(keyIn){
				case 'x' : doit = false;

				break;
				case 'C' : continuousSound->start(); // play sound
				break;
				case 'c' : continuousSound->stop(); // cease sound
				break;
				case 'S' : singleSound->start(); // play sound
				break;
				case 'w' :
				continuousSound->setEmitterZ(continuousSound->getEmitterZ() + (FLOAT32)0.5);
				break;
				case 's' :
				continuousSound->setEmitterZ(continuousSound->getEmitterZ() - (FLOAT32)0.5);
				break;
				case 'a' :
				continuousSound->setEmitterX(continuousSound->getEmitterX() - (FLOAT32)0.5);
				break;
				case 'd' :
				continuousSound->setEmitterX(continuousSound->getEmitterX() + (FLOAT32)0.5);
				break;
				case 'p':
				ba->printMatrixCoefficients();
				break;
				case '0' : channelIndex = 0; break;
				case '1' : channelIndex = 1; break;
				case '2' : channelIndex = 2; break;
				case '3' : channelIndex = 3; break;
				case '4' : channelIndex = 4; break;
				case '5' : channelIndex = 5; break;
				case '6' : channelIndex = 6; break;
				case '7' : channelIndex = 7; break;
			}
			if(channelIndex == -1){
				IXAudio2SourceVoice* voice = continuousSound->getSourceVoice();
				ba->play3DVoice(continuousSound->getEmitter(), voice);
			}else{
				printf("channel = %d\n", channelIndex);
				IXAudio2SourceVoice* voice = continuousSound->getSourceVoice();
				ba->playOnChannelVoice(voice, channelIndex);
			}
		}
		ba->run(); // perform periodic sound engine tasks
		Sleep(100);
	}

	wprintf( L"\nFinished playing\n" );

	// All XAudio2 interfaces are released when the engine is destroyed, but being tidy

	ba->destroy();

	return 0;
}

Ye gods, we might actually have something to use here…

  • Well, after two days of not being able to work on this due to Comps prep and trouble with a significant fraction of my vehicles, maybe I can get something working today.
  • Added everything from How to: Integrate X3DAudio with XAudio2, and definitely got a result – when the emitter position moved off of center, the sound cut out. Now you might think that’s a bad thing, but my guess is that I have some variable that either being set with zero or junk, which is messing up the sound matrix calculations.
  • Using the XAudio2Sound3D project as a source of code to mine, I pulled over the (much more extensive) emitter setup and now have things working, although not as clearly as I’d like. Next is to clean up and package the code into the SampleSound-derived classes.
  • And by the way, this might be a good hard hat to try the rig on: http://www.amazon.com/ERB-19224-Americana-Full-Ratchet/dp/B001LYB7JW
  • Cleared a lot of nonessential (i.e. listening cone)code out of the emitter
  • Realized that I could probably use voice->SetOutputMatrix() to set the channels directly. I added a method that cleans out the dspSettings.matrixCoefficients and then sets a single value in the matrix. Whadaya know – it worked 🙂
  • Need to do some packaging of the code, but I now have enough to do tests for the situational awareness system and to get rid of the MIDI boards for the vibrotactile controller.

Now that XAudio2 is clear, time to be confused with X3DAudio

  • Finished rolling up the SampleSounds(s) into BasicAudio.
  • Starting on 3D audio, using How to: Integrate X3DAudio with XAudio2
    • Adding the 3D calls to the main() in ConsoleSound2, just after the BasicAudio class has been initialized.
    • Had to add x3daudio.lib to the project and <x3daudio.h> to the #includes.
    • This looks to be the meat of the calculation. It looks like X3DCalculate() works on the relative position of an Emitter and the Listener and uses that to populate a DSP. Once that’s done, then the Audio2 source is manipulated appropriately.
    •         X3DAudioCalculate( g_audioState.x3DInstance, &g_audioState.listener, &g_audioState.emitter, dwCalcFlags,
                                 &g_audioState.dspSettings );
      
              IXAudio2SourceVoice* voice = g_audioState.pSourceVoice;
              if( voice )
              {
                  // Apply X3DAudio generated DSP settings to XAudio2
                  voice->SetFrequencyRatio( g_audioState.dspSettings.DopplerFactor );
                  voice->SetOutputMatrix( g_audioState.pMasteringVoice, INPUTCHANNELS, g_audioState.nChannels,
                                          g_audioState.matrixCoefficients );
      
                  voice->SetOutputMatrix(g_audioState.pSubmixVoice, 1, 1, &g_audioState.dspSettings.ReverbLevel);
      
                  XAUDIO2_FILTER_PARAMETERS FilterParametersDirect = { LowPassFilter, 2.0f * sinf(X3DAUDIO_PI/6.0f * g_audioState.dspSettings.LPFDirectCoefficient), 1.0f }; // see XAudio2CutoffFrequencyToRadians() in XAudio2.h for more information on the formula used here
                  voice->SetOutputFilterParameters(g_audioState.pMasteringVoice, &FilterParametersDirect);
                  XAUDIO2_FILTER_PARAMETERS FilterParametersReverb = { LowPassFilter, 2.0f * sinf(X3DAUDIO_PI/6.0f * g_audioState.dspSettings.LPFReverbCoefficient), 1.0f }; // see XAudio2CutoffFrequencyToRadians() in XAudio2.h for more information on the formula used here
                  voice->SetOutputFilterParameters(g_audioState.pSubmixVoice, &FilterParametersReverb);
              }

Boring progress. But hey! Progress :-)

Today’s work. Just solid satisfying programming, though I am playing fast and loose with pointers because of headaches getting const to behave properly.

  • Working on getting multi buffer support. Done! Also got repeating sound working.
  • Going to put a basic Audio class together to manage SampleSounds.
  • Done. BasicAudio (awesome name, huh?) is pretty wrapped up. Once all this is clean and commented, I’ll post the files.
  • Start on 3D/multichannel tomorrow?