Hi Annes,
your report is very detailed and clear. I appreciate your precision.
Your scenario is really complex and the issue is very hard to find.
NCryptoki works as a wrapper that gets the calls from .NET, forwards them to the undelying PKCS#11 module that calls the HSM and finally the result is returned back to .NET.
I guess the issue is related to memory allocation. It seems that somewhere in the calling flow the allocated memory is rewritten with something else causing CKR_BUFFER_TOO_SMALL and CKR_SESSION_HANDLE_INVALID.
Because the issue happens so few times over 50000 attempts, there are a lot of memory allocation/deallocation, both in .NET side (when NCryptoki allocates the memory to pass to the underlying PKCS#11) as well as inside the underlying PKCS#11. For this reason it is very hard for us to understand where is it.
Do you have access to the PKCS#11 log? This may help us to understand what is happening.
Anyway, we'll try to reproduce in some way your scenario with another PKCS#11 module in order to check if the problem is inside NCryptoki or not