Regarding what layers are displayed, it's not that important (to me at least), so don't feel like you have to waste time on that.

Sometimes, I'll use the Map Browser instead of taking screenshots if, for some reason, the map in-game is affected by some color/transparency effect that can't simply be disabled with layers, or if I want to include an area that is slightly off-screen but that I know is there. But usually, I simply capture from the game while playing since it's usually easier like that.

I'll admit that I don't know anything about alpha channel mask since the program I use the most for mapping remains goold old Paint. It doesn't have many features (read: none) but has one advantage: basic map assembly can be done easier and quicker with simpler keyboard shortcuts than with Paint Shop Pro which I also have but only use for transparencies. A simple cut-and-paste with the color I want to remove selected as the background one and presto! I can put a level's foreground over the background and that's that. I've tried using PSP but it remains clumsier and less intuitive to use for basic screenshot assembly.

Paint's simplicity I guess is also it's weakness. Whatever an alpha channel mask is, Paint doesn't know what to do with it I'm sure. The "Shape Opaqueness" choice box is all you have; either you move/paste an image "as is", or else you select a color that will be removed. This is why I'm interested in being able to either change the color that VBA-M displays by default with every layer disabled (in case said color is used by the game's graphics), or instead use a color that no GB/GBC/GBA game can use based on how the palette is calculated.

A friend, Maxim, did this with Gens, and you can change the background color to an RGB value that is outside the Genesis palette as rendered by the emulator, thus removing any fear of ever unknowingly removing a single pixel from a screenshot. For example, #FF00FF as the background while Gens cannot render anything "pinkier" than #F800F8. Does that make any sense?