Regarding the menu entries, I personally never used them to enable/disable layers, but they were handy in some cases to indicate whether the various layers were being displayed or not, since it was sometimes hard to tell after playing with them for a while.
As for what I'm asking for, I must first admit that I'm not entirely sure if it's VBA or the game that's the problem. Alright, let's use a practical example: layer 5 (called OBJ in VBA) is (typically I assume) the sprite layer. Let's say that I turn off every layer except that one; in many games like, say, Metroid Fusion, I would see only the various sprites over a black screen. But in Castlevania: Harmony Of Dissonance, instead of black there's usually another color, like green for the title screen or blue for the introduction story, and it's still there even if you disable all eight layers.
*checks something*
Well, what do you know? After checking with "Castlevania: Harmony Of Dissonance", it would appear that this background color that shows up even with all layers turned off is the first color in the background palette.
I don't know if the implications of this would make my request harder to implement, but in theory, you'd have to change VBA-M's behaviour so that instead of defaulting to the first background palette entry as an "empty" or "transparent" color, you'd instead employ a user-defined color. Anyway, I appreciate your continued interest