The tokens used for CSRF protection are meant to be different for every user and they are stored in the session. That's why a session is started automatically as soon as you render a form with CSRF protection.
Moreover, this means that you cannot fully cache pages that include CSRF protected forms. As an alternative, you can:
Embed the form inside an uncached ESI fragment and cache the rest of the page contents;
Cache the entire page and load the form via an uncached AJAX request;
Cache the entire page and use hinclude.js to load the CSRF token with an uncached AJAX request and replace the form field value with it.