I was frequently getting too many files open compilation error in jsp in one of my portlet factory project.
OS: AIX
The server i used was Webphere Application Server 6.0.19, Websphere Portal Server 6.0.1.1

The problem was occuring in a totally weird manner.
When the first user accesses the application, it works completely fine.
The problem starts when the second user also logs in.
It gave "too many files open".

We tried hard to find a clue on this. Since this is totally abnormal behaviour of a web application server.

The mistake we did was not reading the stacktrace properly.
In the stack trace it clearly mentioned that it occured in Clone_Portal.

Bingo, we found atlast.
The problem is, when the application runs for the first time, all jsp files are loaded properly.
When the second user logs in, the server search for the files in the Clone portal, but the application is installed in the normal portal.

So we stopped the clone portal.
And it started workign fine.

Conclusion:
It is a problem in the clustered environment when the clone portal is running.