From what I've read, detail brushes are meant to be used for brushes which don't block the players view. For example trim around a door or colums against a wall, etc. So... maybe give those brushes the detail tag as suggested first.
For Draw.bsp, I had to increase the max patch size from 32 to 64. 64 might be higher than I needed but it seems to work. Draw.bsp has 8005 numportals last compile and vis seemed to poop out with much more than that, around 8100 or so. 22,000 numportals is alot, must be a huge map or have alot of detail and stuff in it. I haven't tried func_wall, only know it's sometimes used for sealing off areas when a team isn't playing, like on a 4 team map or whatever.
Maybe elimate and/or combine some brushes into one to decrease the count?
It won't compile with 5K numportals? Does it error out or are you just impatient and stop the compile?