I was able to figure out what was dipping the performance issues in the game. It was the lighting inside the houses, their lights had too much of a wide angle. That was it. After fixing it, the game runs so so much better. Now I can't blame my amateurish oclussion culling and fustrum culling skills hahahahaha.