I don't think you will fix this by changing environmental variables like you attempted. I think you need to recompile the whole program to allow for larger number of sequences. You get the package from here and inside the file
coffee_defines.h change the
260000 to something larger, say
500000. Run the
./install script after that and it should create a new
t_coffee that will be able to work with larger number of sequences. There is a more detailed description on how to do all of this on their website.
I will mention that aligning >400000 sequences in any program will take a very long time, and it may not be practical to wait for it. You may need to consider using fewer sequences.