Custom Libraries and Includes

If I'm wanting to build a custom toolchain and libraries and includes on a different path, how do I add the libraries and includes?

I tried setting the libraries in /etc/ld.so.conf.d/*.conf files but it doesn't seem to work. Also, I'm having to manually specify where the include files are for any software I build that has a prerequisite that I've just built. ie gcc needing gmp and mpfr. or cloog needing gmp.