Tokens are a common concept in programming languages such as C , C++ , and Java . Tokens are the smallest individual units of a C++ program, which are used by the compiler during lexical analysis to analyze and interpret the code. Using these tokens, we can recognize, understand, process, and form the syntax of the programming language. The compiler breaks the code into tokens so as to process it further.
Types of Tokens
Although, tokens are the smallest individual unit useful for a compiler. There are mainly six types of tokens available in C++ programming language and each serves a specific goal in the syntax and semantics of C++ language.
Now, we will discuss these tokens one by one.
1) Identifiers
In C++, an identifier is a name given to a variable, function, or other objects (like variables, functions, arrays, etc.) in the code. These are used to refer to these entities in the program, and they can consist of letters, numbers, and underscores. We should always use valid and meaningful names of identifiers to write understandable code.
While creating an identifier, there are some rules which we should always keep in mind :
- It should either start with any alphabetic letter (uppercase or lowercase) or underscore, but it cannot start with digits from 0 to 9.
- We should not use the white spaces or any special character as an identifier.
- We should not use the C++ keywords as an identifier name because they are already reserved for performing their specific tasks.
- The name of the identifier must be unique in the code.
- We should always keep in mind that C++ is a case-sensitive programming language, so sum and Sum are considered two different identifiers.
C++ Identifier Example:
Let us take an example to illustrate the identifier in C++.
Example
#include <iostream>
using namespace std; //using standard namespace
int main() //Main Function
{
int age = 25; //initializing Age
string name = "John Doe"; //Define variable name
cout << "My name is " << name << " and I am " << age << " years old." << endl; //result
return 0;
}
Output:
My name is John Doe and I am 25 years old.
Explanation
In this example, we have taken a variable's age and name as identifiers. The age identifier is used to store a person's age, and the name is used to store the person's name.
To Read More: C++ Identifier
2) Keywords
In C++, keywords are the reserved tokens and are defined to perform their assigned tasks. We should never use keywords as an identifier; otherwise, the compiler will get confused and throw an error.
In C++20, we have 95 keywords available, including int, float, class, short, char, delete, and many more, and each one is used as per the requirement.
C++ Keywords Example
Let us take an example to illustrate the keywords in C++.
Example
#include <iostream>
using namespace std; //using standard namespace
int main() //main function
{
int int = 10; // Define a variable named int
cout << "The value of the variable " << int << " is " << int << "." << endl;
return 0;
}
Output:
error: expected unqualified-id before numeric constant
Explanation
In this example, when we compile the code, the compiler will generate an error. This error occurs because the int keyword cannot be used as an identifier. The int is a reserved keyword, so using it as a variable name causes a compilation error. It would help if we chose a different name for our variable to avoid this error.
To Read More: C++ keywords
3) Constants
In C++, constants are the tokens that are used to initialize the variables during initialization. Constants are fixed values that do not change during program execution. They can be defined using 'const', 'constexpr', or '#define'. Once it is initialized, we are not allowed to change the value.
C++ Constants Example
Let us take an example to illustrate the constants in C++.
Example
#include <iostream>
using namespace std; //using standard namespace
int main() //Main Function
{
const double PI = 3.14159; // Define a constant named "PI" of type double
double radius = 10.0; // Define a variable named "radius" of type double
double area = PI * radius * radius; //calculate the area of circle
cout << "The area of the circle is " << area << "." << endl;
return 0;
}
Output:
The area of the circle is 314.159.
Explanation
In this example, the constant PI is used to represent the value of pi. The value of PI is defined using the const keyword and is initialized with the value of pi. After that, the constant PI is used in calculating the area of a circle, which is multiplied by the radius of the circle squared to determine the area.
4) Strings
In C++, a string token is a class that we can find in the Standard Template Library (STL), which contains the functionality to deal with a sequence of characters together to create a string. It simply means that when we use the string token, we are actually defining an object that represents a group of characters.
The string class consists of several methods that we can use over the given string to perform different actions, such as length, popback , pushback method, etc.
Syntax
It has the following syntax:
string var_name;
string name= "It is string";
C++ String Example
Let us take an example to illustrate string in C++.
Example
#include <iostream>
#include <string>
using namespace std; //using standard namespace
int main() //Main Function
{
string message = "Hello, This is Cpp Tutorial world!"; //Define a string variable name message
cout << message << endl;
return 0;
}
Output:
Hello, This is Cpp Tutorial world!
Explanation
In this example, the string class creates a string variable named message. After that, the message variable is initialized with the string "Hello, This is Cpp Tutorial world!", and then printed to the screen using the cout object.
To Read More: C++ String
5) Punctuators
Punctuators tokens have specific meanings within the syntax used for the C++ programming language. These are the token characters, and we can use punctuators in different variety of methods that include separating the items, ending the statements, and other tasks. Although we have several types of tokens which are available, some of them are as follows:
- Square Brackets : When we create an array, we store the elements of the array inside the square brackets. For example, [2, 3, 4, 6, 0] is an array having five elements.
- Single Quote (' '): When we initialize any character value, we enclose the characters within the single quotes. For example, 'a' is a character value.
- Semicolon (;): Whenever we complete any line of code in our C++ program, we terminate that line by putting a semicolon at the end. For example, int x = 10; where after initializing the value, we terminated the line with semicolon.
- Curly Braces ({ }): Whenever we write our code, it is enclosed within the curly braces always, and it represents the block of code. For example,
int main () {
// write the code
}
Here, we have put the code of the main method within these curly braces.
- Double Quotes (" "): When we initialize any string literal, we enclose the string within the double quotes. For example, "CppTutorial Tech is the best learning website" is the string value enclosed in the double quotes.
C++ Punctuators Example
Let us take an example to illustrate the Punctuators in C++.
Example
#include <iostream>
using namespace std;
int main() { //Main Function
int a = 5; //define integer variable
int b = 10;
int c = a * b; // Using the asterisk
cout << "The product of " << a << " and " << b << " is " << c << endl;
int *ptr = &a; // Use of the ampersand (&)
cout << "The address of a is " << ptr << endl;
int d = ~a; // Use of the tilde (~)
cout << "The bitwise NOT of " << a << " is " << d << endl;
return 0;
}
Output:
The product of 5 and 10 is 50
The address of a is 0x7ffca52b0640
The bitwise NOT of 5 is -6
Explanation
In this example, we have taken the asterisk (*) as the multiplication operator to calculate the product of two variables, a and b. The ampersand (&) is used to get the address of the 'a' variable, which is printed to the console. Finally, the tilde (~) is used as the bitwise NOT operator to negate the value of a.
6) Operators
These tokens are used for performing operations on different operands, including variables, expressions, and constants. In C++ programming language, we have a variety of operators that are used to perform specific operations, such as logical operations, arithmetic operations, relational operations, etc. For example, (X && Y) is a logical operation where X and Y are the operands, and the operation performed is an AND operation.
Generally, there are three types of operators available:
- Unary Operator: These are the operators that are used with single operands and perform specific unary operations on a single variable. Some of the unary operations include increment ( ++ ) and decrement ( -- ) operators, i.e. to increase the operand value by 1, use the increment operator, and to decrease the operand value by 1, use the decrement operator.
- Binary Operator: These are the operators that are used with two operands and perform specific binary operations on two variables. Some of the binary operations include arithmetic operators, logical operators, assignment operators, and many more.
- Ternary Operator: This operator is a type of operator that works between three operands, also known as a conditional operator. It is used to create conditional expressions. It either gives a true or false output.
Syntax
It has the following syntax:
Condition 1 ? Condition 2 : Condition 3;
Here, if condition 1 is true, condition 2 gets executed; else, condition 3 will get executed.
C++ Operators Example
Let us take an example to illustrate the operators in C++.
Example
#include <iostream>
using namespace std;
int main() { //Main Function
int a = 25; //defining integer variable
int b = 8;
int c = a + b; // Using plus(+) operator
cout << "The sum of " << a << " and " << b << " is " << c << endl;
c = a - b; // Using minus (-) operator
cout << "The difference between " << a << " and " << b << " is " << c << endl;
c = a * b; // Using asterisk (*) operator
cout << "The product of " << a << " and " << b << " is " << c << endl;
c = a / b; // using forward slash (/) operator
cout << "The quotient of " << a << " and " << b << " is " << c << endl;
bool d = (a == b); // Using double equals (==) operator
cout << "Is " << a << " equal to " << b << "? " << d << endl;
d = (a != b); // Using exclamation point and equals (!=) operator
cout << "Is " << a << " not equal to " << b << "? " << d << endl;
return 0;
}
Output:
The sum of 25 and 8 is 33
The difference between 25 and 8 is 17
The product of 25 and 8 is 200
The quotient of 25 and 8 is 3
Is 25 equal to 8? 0
Is 25 not equal to 8? 1
Explanation
In this example, various operators are used to perform different operations. The + operator is used for addition, the - operator is used for subtraction, the * operator is used for multiplication, and the / operator is used for division. After that, the == and != operators are used to check for equality and inequality.
To Read More: C++ Operators
C++ Token MCQs
1) Which of the following options is the type of Tokens in C++?
- Special Symbols
- Keywords
- Operators
- All of the Above
2) Which of the following option shows the correct keyword in C++?
- main
- define
- include
3) What will be the output of the following code?
#include <iostream>
using namespace std;
int main()
{
int a=7, b=12;
const int c=a*b;
cout<<c;
return 0;
}
4) Which of the following option shows the valid identifier in C++?
- name-123
- name_123
- 123name
5) What are punctuators in C++?
- Variable names
- Special characters that mark the end of statements or group codes
- Functions
- Reserved words