Monthly Archives: June 2013

Infrastructure

Leave a reply

So the testing part of the code is done and working. Yes, there are bugs, and some cases where exceptions are thrown that should be handled, but if you color inside of the lines, everything works. Of course, now I need to be able to record the output, so it’s time for some classes to handle the work of saving results for analysis.

  • Creating a TestResult class with the following information
    • session number
    • test number
    • test type (speed or accuracy)
    • time to lift
    • source position
    • cursor position
    • angle difference
    • speaker volume matrix
    • Also, there will be a toString(), toCsvString(), and  toXmlString() method for output;
  • The TestManager will instance and store TestResults in a container (vector?), which will have the additional fields
    • Researcher name
    • Subject name
    • Sound file
    • Total sessions
    • Speed tests per session
    • Accuracy tests per session

So that’s what happens when a programming language gets old…

Leave a reply
  • Continuing with the test exec. I’m also going to need a class that records the data associated with each test segment.
  • Ran into a… Well, I don’t want to call it a bug. Let’s say that C++ is showing its age. FLTK uses char*. Most of Windows uses wchar_t. They don’t play well together, so I spent about half of my time working out the best way to convert between them. It’s this:
  • void setSoundFileString(LPCWSTR wps){
	soundFileString = new wstring(wps);
	string str(soundFileString->begin(), soundFileString->end());
	sprintf_s(soundFile, "%s", str.c_str());
}
  • I mean really!? Good grief.
  • Got a lot of the exec built and running. Clicking on the center button fires the sound, and you can drag to where you think the sound is. I am not all that accurate. It could be a frequency thing though. I’m running a low 10-20 HZ signal. The test should definitely try different frequencies.

FLTK and FLUID. Simple, Good Stuff

Leave a reply
  • Starting to put together the actual test framework. Found a good open source synthesizer (ZynAddSubFX) that I used to create a pure tone that I then cut down to one second with Audacity. It’s important to note that this app only works with MONO sounds.
  • Building up the class that will handle running multiple sessions.
  • Just a quick shout out to FLTK. I have been adding and adjusting the GUI all day long as I figure out how to run the tests. To add fields, adjust positions and just generally futz around, all I have to do is use the FLUID gui IDE, export the layout as C++ code, add in stdafx.h and compile. It all just works. A great piece of code. FLTK_rocks

Closing in on something useful

Leave a reply
  • Put all the projects into SVN, checked them out and did a clean build. Everything still works.
  • Starting on capturing mouse events. Done. Capturing Left, Middle, Right, Wheel and Drag. I think I want to have it so that the user presses (holds down?) a “button” in the middle of the GL window, signifying that he’s ready for the next sound queue. Once the sound plays, he drags toward the source. This is indicated by a line indicating the vector (and a cursor?). Releasing the mouse is the event that marks and records the choice, vector, elapsed time and position (vector?) of the emitter.
  • Making a button class for OGL. Done
  • Making a line segment class so we can point to where the sound is. Done.
  • And just to show that I’ve been paying attention in class, the user doesn’t have to worry about hitting a particular length when dragging towards the sound. Since there is essentially no source (since the test starts after the subject clicks) and no target, we don’t have to worry about any Fitts’ Law biases 🙂
  • Progress for today:

  • vth

    Start Button (red), Sound Vector and Emitter

Audio is in and synchronized to the video

Leave a reply

Well that was a good day.

  • Spent a good deal of time trying to figure out the best way for the GUI and the Exec to communicate. Originally, I wanted to be able to pass a pointer to the GUI from the exec so that user actions in the GUI could be executed in a more reasonable place. Due to header conflicts, I couldn’t manage to get that to work, so I put together a UI_cmd class that is set in the UI and read in the Exec. That seems to be working pretty well, though I may want to put a queue in there and turn it more into a message bus/event pump. That level of sophistication isn’t needed yet though.
  • Integrated the sound library that I wrote. I still have to reference the D3D audio library in the main application which I think is a bit odd, but I think it may be because I’m incorrectly exporting the symbol table from the static library. Again, that’s a refinement for later.
  • At this point, the 3D position of my OGL shape and the 3D postion of my continuous sound (2D actually, Y = 0) are running in an infinite circle. It’s pretty cool to hear the audio track to the image. I’m uploading a video of the running system, and although it won’t be in surround, you can hear the flanging effects from the sound moving around the helmet.

Not bad for 90 minutes worth of work…

Leave a reply

I’m busy doing demos and presentations in my day job, so this has been suffering. Nonetheless, here’s the progress for today:

  • Added a fine-grained timer callback to the main app
  • Added an OpenGL window, set to Ortho2, and with pixel-accurate dimensions
  • Connected the timer to the OpenGL, and set the position of what will be the emitter. We won’t see this during the actual test, but it will be good for debugging.
  • I need to track mouse clicks and motion in the GL window. That will come tomorrow, and then I’ll work on integrating the audio library. That’s the basics for running the experiments. After that, I’ll work on reading and writing the input and result files.
  • Pix for today: AppProgress6.18.13

Let’s make science!

Leave a reply

Alright, so I now have my audio library. Next on the agenda is a test program that tests reactions of users to vibroacoustic input. The test needs to present randomized stimuli to users, so that they can be tested for:

  • Time to respond with a direction
  • Accuracy of direction
  • Efficacy of stimuli

Since this is probably going to be within subjects (multiple stimuli) and also between subjects (same tests on multiple users) we’re going to want to be able to present the same sequence, so we’ll need to seed the random number generator so we get the same sequence.

  • Start with the default random number generator, but maybe run through a wrapper class in case we need something like a Mersenne Twister.
  • Xml file to specify the input and output of the experiment. This library looks reasonable.
    • Input
      • Sounds to use (random distribution of sound use)
      • Test type (Accuracy, Speed, or both)
      • Attempts per test
      • Number of tests (must be even)
      • Random seed
      • min/max delay between test segments
      • output filename
    • Output
      • Test UID
      • Date
      • Time
      • Subject
      • Researcher
      • Free form note field (1024 characters?)
      • Accuracy or Reaction time test
      • Audio configuration
      • Random seed
      • Calibration results
        • Time(s) to click in response to visual cue
        • Time(s) to click in response to audio cue
      • For each played sound
        • Sequence x of total
        • Audio file(s) used (WAV)
        • Audio source position (x, y) in screen coordinates from the origin, where the user’s head is
        • Audio playback matrix (actual speaker relative volume)
        • Time to click after play start
        • Duration of sound
        • Click position (x, y) in screen coordinates from the origin, where the sound is perceived to have come from
  • App
    • Text
      • File navigator for xml file
    • Calibrate
      • Runs a sequence of tests where the user has to click the mouse as quickly as possible in response to the canvas flashing white, and then all(?) speakers in the headpiece playing the calibration sound
      • Calibration cues are have a randomly determined timing between X and Y seconds
      • Test is disabled until calibration is run. Loading a new xml document effectively resets the system, requiring a new calibration sequence
    • Test
      • Shows a label that says either “Accuracy” or “Speed” based on which test is being run. We could change the background of the display as well?
      • The graphics screen shows a circular cursor that resets to the center of the graphics screen at the beginning of each section. Once the audio cue plays, the user can move the mouse away from the center towards the direction of the sound. The circle is clamped in its motion so that the result is always a valid angle, as long as the user moves the cursor far enough away from the center (TBD). Clicking the mouse causes the clock to stop and the cursor to reset.
      • If this is not the last test segment, then a random time period between X and Y seconds elapses before the next test is run.
      • Once the test completes, the system checks to see if that is the last one. If not, a stochastic choice is made to determine if the next test should be speed or accuracy. By the time all tests have run, the number of speed and accuracy runs will be equal.
    • Output file is appended throughout the test (open, write, close? Or read in the DOM, update and write out?)

Started the FLTK wrapper, and probably saved a good deal of time by going back to Erco’s FLTK page and associated videos

Building the Audio Library

Leave a reply
  • Making sure that the release version of the code runs as well – it does. And a massive 19kb of executable. Of course, it’s not statically linked… That makes it 6,234kb
    • C/C++
      • General
        • Additional Include Directories: $(DXSDK_DIR)\Include
      • Language
        • Treat WChar_t as Built in Type: Yes (/Zc:wchar_t)
    • Linker
      • General
        • Additional Library Directories: C:\Program Files %28×86%29\Microsoft DirectX SDK %28June 2010%29\Lib\x86
      • Input
        • Additional Dependencies:
          • winmm.lib
          • dxerr.lib
          • X3DAudio.lib (only needed for DLL compilation)
  • Making library
    • Following directions here: http://msdn.microsoft.com/en-us/library/vstudio/ms235627.aspx
    • For a DLL library, you *must* select the when creating your console application’s (that is actually not a console application, but never mind that…) “export variables” checkbox option. Still figuring out how to use this.
    • For my first attempt in accessing the DLL, I’ve created the following method (The #defines are generated, I’ve included them for all to marvel at): 
      • #ifdef DXAUDIOINTERFACEDLL_EXPORTS
#define DXAUDIOINTERFACEDLL_API __declspec(dllexport)
#else
#define DXAUDIOINTERFACEDLL_API __declspec(dllimport)
#endif
DXAUDIOINTERFACEDLL_API BasicAudio* getBasicAudio(void);
      • I think this produces an exported method that returns a pointer to the BasicAudio class. Not sure though. It may, though now I’m getting the following link errors:
        • 1>DynamicLibConsoleTest.obj : error LNK2019: unresolved external symbol “public: void __thiscall BasicAudio::destroy(void)” (?destroy@BasicAudio@@QAEXXZ) referenced in function _wmain
        • etc…
        • 1>C:\Phil\MSVC Dev\MultichannelOr3dAudio\DxAudioInterfaceLibrary\Debug\DynamicLibConsoleTest.exe : fatal error LNK1120: 6 unresolved externals
      • Which means the public inner methods are not available. Maybe the answer is to make a wrapper class for all the methods? Ugly, but something to try..
      • Building wrapper class DXAUDIOINTERFACEDLL_API CDxAudioInterfaceDLL
      • Done. Compiles and links now, but doesn’t run. Ah. I was declaring a local pointer to the internal BasicAudio class. Not pretty, but it does seem to work…
  • Testing against Static console app – check
  • Testing against Dynamic console app – check
  • Switching to Release build
  • Testing against Static console app – check
  • Testing against Dynamic console app – check
  • Checked everything into Subversion. Calling it a day.

Final cleanup on the proof-of concept

Leave a reply
  • Ugh.
  • rain
  • Adding comments and velocity accessors to audio code
    • Comments are done
    • Checked in project to Subversion repo, then checked out and compiled. All seems to be working fine.
  • Cleaning up #includes
    • I had been getting the following annoying warning once I started including items like <unordered_map>: 1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\cstring(21): warning C4995: ‘strcat’: name was marked as #pragma deprecated. In the cleanup, I finally got around to testing the need for includes and deleted <strsafe.h>, which seems to be the source of my grief. Yay!
  • Ready to make the audio code a library. Monday. Then time to design the experimental interface.
  • And now that I think I know what’s going on, a good article shows up…

In the end, this is how you do it:

// create the instance
BasicAudio *ba = new BasicAudio(); 

// initialize
ba->init(); 

// create a sound from a file. In this case a WAV. There can be many of these.
ba->createSound(L"music", L"Wavs\\MusicMono.wav", 0); 

// get the instance to the sound and start(), stop(), run() etc.
ba->getSoundByName(L"music")->start(); 
loop{
	// change some audio condition

	// play the voice on a specified channel or play in 3D
	ba->playOnChannelVoice(L"music", channelIndex); 
 	ba->play3DVoice(L"music");

	// optional - run periodic checks.
	ba->run() 
}
ba->destroy();