In PS do CTRL/K, then select performance. The RAM figure is in MB. Mine is set to 4000 (4GB) which is 75% of my memory (6GB). Try yours with different figures.
When I stack I always do it from RAW files. Even with the 5DMK2 (21MP) I can easily load about eight RAWS in RAW editor, select them all, make changes, synchronize them, then load them all (together to CS5) as 16bit files. I then do the stacking, and as I said before, it may sometimes slow down a bit, but I never run out of RAM. I find it can slow down if I have previuosly been doing a lot of other processing. I don't see why that should happen, as I would have thought any memory used would get freed up. But maybe it doesn't. I went for a 14 mile hike yesterday, and after a good night's sleep I should be refreshed. I'm not.