I started a math library here: https://github.com/ChaiScript/ChaiScript_extras. It would be great if you wanted to contribute to it.
I don't recommend performing lots of heavy math inside of ChaiScript, because you could get much better performance by implementing your math-heavy code in C++ and then calling it from ChaiScript, this way you get the best of both worlds, performance from C++ and flexibility from scripting.
Re: running "hundreds" of instances simultaneously: they will not interact with each other, so you will not have performance issues. Memory will grow, certainly. You can reduce the per-instance overhead by instantiating each library only once. Like:
auto stdlib = chaiscript::Std_Lib::library();
auto mathlib = chaiscript::extras::math::bootstrap();
Hopefully that helps. Running 100's of instances at once has not been tested as far as I know, but I have tried to reduce the overhead (CPU time) of creating each instance, which should help.