Register Now

    Tokens

    Tokens are specific units of data that artificial intelligence (AI) technologies use for training and completing tasks.

    What Are Tokens?

    When training an AI, developers input large amounts of data that the technology then uses to find patterns within. Each individual unit of that data is called a token. Tokens can be letters, numbers, words, or short phrases. AI models perform a process of tokenization to break down larger data sets into these small tokens, which are each represented by a numerical value. An AI solution might break the term “ecommerce” into “e” and “commerce.” Then, it would learn the associated meanings of each term and assign a numerical token to each term. By finding patterns across examples, the AI would learn that “e” is both a letter and an indicator of an online presence. When the same characters and associated tokens pop up in other data, the AI can quickly make the connection — and the meaning — between the patterned terms. This process enables the tech to learn at scale.

    See Salsify in Action

    Bring your team, tools, and data together to launch high-impact inbound marketing campaigns in less time.

    Request a demo