PHP Tokenization

Tokenization is the process of breaking up a long string into words. PHP offers a special funtion for this purpose, called strtok()

strtok

strtok() function takes two arguments, the string or text to be broken up into tokens AND string containing all the delimeters.
- in strtok() delimiters are characters that count as boundary between tokens e.g blank space " " , comma"," , semi-colon";" , hyphen "-", a, i, s, 2, 7 etc.
Sytax
strtok("text to be broken up","delimeter-character")

strtok() example

<?php
//space " " in-between each word is use as a delimiter to break the text
$stoken = strtok("Tokenization is the process of breaking up a long string"," ");
while($stoken){
  print($stoken."<br/>");
  //"this help us to exit the loop"
  $stoken = strtok(" ");
}
?>

Output

Tokenization
is
the
process
of
breaking
up
a
long
string

Explode

explode() function takes two arguments, string containing all the delimeters AND a string or text to be broken up into tokens and the string to be separated.
- in explode() delimiters are full-fledged string e.g AND, WHERE, HOW, A, WHEN etc.
- in explode() the substring or broken text are returned in an array.
Sytax
explode("delimeter-string", "text to be broken up")

explode() example

<?php
$expl_token = explode("big","some big boys and big girls are not rich");
$array_len = count($expl_token);
for($i=0; $i<$array_len; $i++){
  print $expl_token[$i]."<br/>";
}
?>

Output

some
boys and
girls are not rich

Since the result are kept in array, you can equally access individual element as well, instead of accessing all the array element using loop.
<?php
print $expl_token[1];
?>

Output

boys and

Next »