# EXSLT - str:tokenize - Implementer Page

Version: 1 Status: new User Page: index.html XML Definition: str.tokenize.xml Function Package: str.tokenize.zip

#### Function Syntax

    node-set str:tokenize(string, string?)

#### Template Syntax

    <xsl:call-template name="str:tokenize">
       <xsl:with-param name="string" select="string" />
       <xsl:with-param name="delimiters" select="string" />?
    </xsl:call-template>

The `str:tokenize` function splits up a string and returns a node set of `token` elements, each containing one token from the string.

The first argument is the string to be tokenized. The second argument is a string consisting of a number of characters. Each character in this string is taken as a delimiting character. The string given by the first argument is split at any occurrence of any of these characters. For example:

    str:tokenize('2001-06-03T11:40:23', '-T:')

Gives the node set consisting of:

    <token>2001</token>
    <token>06</token>
    <token>03</token>
    <token>11</token>
    <token>40</token>
    <token>23</token>

If the second argument is omitted, the default is the string `'&#x9;&#xA;&#xD;&#x20;'` (i.e. whitespace characters). Thus:

    str:tokenize('date math str')

Gives the node set consisting of:

    <token>date</token>
    <token>math</token>
    <token>str</token>

If the second argument is an empty string, the function returns a set of `token` elements, each of which holds a single character. Thus:

    str:tokenize('foo', '')

Gives the node set consisting of:

    <token>f</token>
    <token>o</token>
    <token>o</token>

The template version of this function returns a result tree fragment consisting of a number of `token` elements.

An implementation of this extension function in the EXSLT str namespace must conform to the behaviour described in this document.

## Implementations

Built-in support for `str:tokenize` is available in the following XSLT processors:

Processor | Processor Version | Implemented Version
--- | --- | ---
4XSLT, from 4Suite. | 0.12.0a3 | 1
libxslt from Daniel Veillard et al. | 1.0.19 | 1

The following implementations of `str:tokenize` are available:

Language | Implemented Version | Creator | Date | Download
--- | --- | --- | --- | ---
EXSLT Function | 1 | Jeni Tennison | 2001-06-03 | str.tokenize.function.xsl
XSLT Template | 1 | Jeni Tennison | 2001-06-03 | str.tokenize.template.xsl
Javascript | 1 | Chris Bayes | 2001-06-21 | str.tokenize.js
Msxsl | 1 | Chris Bayes | 2001-06-21 | str.tokenize.msxsl.xsl

## Use Cases

Use Case Package: str.tokenize.use-cases.zip

The following use cases illustrate the functionality of `str:tokenize`.

Details | Calling Stylesheet | XML Data | Expected Result | Category | Template
--- | --- | --- | --- | --- | ---
1 | str.tokenize.1.xsl | str.tokenize.data.1.xml | - | example | N

## Change History

Submitted: 2001-06-03
Creator: Jeni Tennison(http://www.jenitennison.com/)

The str:tokenize function splits a string at the occurrences of given characters.

Version | Modified | By | Details
--- | --- | --- | ---
1.1 | 2001-06-21 | Jeni Tennison | Added Javascript and MSXML implementations.
1.2 | 2001-11-18 | Jeni Tennison | Indicated behaviour when second argument is an empty string; str:tokenize() should return a list of tokens, each containing a single character.
1.3 | 2002-08-21 | Craig Stewart | Added 4XSLT and libxslt implementation to the list.
1.4 | 2002-11-12 | Craig Stewart | Updated 4XSLT version to 0.12.0a3.

http://www.exslt.org/str/functions/tokenize/str.tokenize.html last modified 2002-11-12