> For the complete documentation index, see [llms.txt](https://martin-github07.gitbook.io/tokenization-layer/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://martin-github07.gitbook.io/tokenization-layer/master.md).

# Introduction

This is the documentation for a package about an experimental "tokenization layer", a tokenization algorithm that is a neural network layer, training as part of a model trying to solve some NLP task, to make tokens that are best for the task.

However note, this is simply a *concept*, and in it's current state, this layer should not be used in any real tasks.