My shoot-only-when-eyes-closed-doom-mod already works!
The eye detection software usually picks up both eyes and works decently if you stay still while playing. At the moment half of the screen is dimmed to black on the side of the respective closed eye. In the image the left eye is closed.
Technically speaking, this is what happens:
- the eye detector picks up the webcam image
- finds a face
- finds an eyepair in that face
- saves the eyepair image in memory, this is used for faster lookup on consecutive frames
- tries to find two open eyes in that eyepair
- if found, then sends keyboard events to the operating system
- ZDoom gets these events and an ACS script reacts accordingly
- player desperately blinks her eyes when meeting a cacodemon
Amazingly, it almost works.
So the game just basically shows a sprite on screen when a key is pressed, and won't allow shooting if not both of the keys are pressed simulatenously.
Some problems I still have to solve:
Should the player view be dimmed only when both eyes are closed? Now each eye does half. The detection is still pretty fiddly, the eye states often flicker quickly on/off when the classifier isn't 100% sure and webcam noise distorts the results.
Weapon damage feels painfully low now that aiming is harder, some changes are needed to make it fun.
To do a standalone release or just keep it as a ZDoom pk3 mod?
How to make eye detection more robust, even just moving your eyes quickly is enough to confuse it sometimes.
I tried training my own eye detection model using the OpenCV traincascade utility with some free pictures of eyes I found. The images were 24x24 px pictures of random eyes, both closed and open like the following.
The first results weren't too pleasing; the model was probably overfitted and it had trouble classifying live webcam image often misjudging an open eye to be closed. I made some changes to the training parameters, but the computation process is still yet to complete. These things take time, sometimes multiple days
. Luckily, the bundled haarcascade_eye.xml
classifier seems to work quite well too, I doubt my custom models will be much better than it.
If you're interested, this was the command I used to train the custom Haar Cascade.
./traincascade -vec eyes.vec -bg negative.txt -w 24 -h 24 -data training/strict -numStages 15 -numPos 2100 -numNeg 3576 -mem 1024 -mode ALL -minhitrate 0.995 -maxfalsealarm 0.5
I couldn't automatically evaluate the model since the program bundled with OpenCV to do that just didn't work. Never do user input validation with asserts, kids!