Tokens In C++

Tokens are a prevalent concept in programming languages like C, C++, and Java. They represent the smallest individual elements of a C++ program and play a crucial role in the compiler's lexical analysis process for code interpretation. By utilizing tokens, we can identify, comprehend, manipulate, and structure the syntax of a programming language. Breaking down the code into tokens allows the compiler to facilitate subsequent processing steps.

Types of Tokens

While tokens are the smallest meaningful units for a compiler, C++ programming language encompasses six primary types of tokens, each designed to fulfill distinct purposes within the syntax and semantics of the C++ language.

Now, we will discuss these tokens one by one.

1) Identifiers

In C++, an identifier serves as the label for a variable, function, or other entities (such as arrays, functions, etc.) within the codebase. Identifiers are essential for referencing these elements throughout the program, and they can be composed of letters, digits, and underscores. It is crucial to employ appropriate and descriptive identifier names to ensure clarity and comprehension in the codebase.

While creating an identifier, there are some rules which we should always keep in mind :

• ✅ It should either start with any alphabetic letter (uppercase or lowercase) or underscore, but it cannot start with digits from 0 to 9.
• ✅ We should not use the white spaces or any special character as an identifier.
• ✅ We should not use the C++ keywords as an identifier name because they are already reserved for performing their specific tasks.
• ✅ The name of the identifier must be unique in the code.
• ✅ We should always keep in mind that C++ is a case-sensitive programming language, so sum and Sum are considered two different identifiers.

C++ Identifier Example:

Let's consider an example to demonstrate the concept of an identifier in the C++ programming language.

Example

#include <iostream>  

  

using namespace std;   //using standard namespace

  

int main()   //Main Function

{  

    int age = 25;   //initializing Age

  

    string name = "John Doe";  //Define variable name

  

    cout << "My name is " << name << " and I am " << age << " years old." << endl;   //result

  

    return 0;  

}

Output:

My name is John Doe and I am 25 years old.

Explanation

In this instance, we are using the variable names "age" and "name" as identifiers. The "age" identifier is employed to hold the age of an individual, while the "name" identifier is utilized to store the name of the individual.

To Read More: C++ Identifier

2) Keywords

In C++, reserved tokens known as keywords are specifically designated to carry out their predetermined functions. It is crucial to avoid utilizing keywords as identifiers to prevent compiler confusion leading to errors.

In C++20, there are a total of 95 keywords at our disposal, such as int, float, class, short, char, delete, and numerous others, each serving a specific purpose based on the need.

C++ Keywords Example

Let's consider an example to demonstrate the keywords in the C++ programming language.

Example

#include <iostream>  

  

using namespace std;  //using standard namespace

int main()  //main function

{  

     

    int int = 10;  // Define a variable named int 

  

    cout << "The value of the variable " << int << " is " << int << "." << endl;  

    return 0;  

}

Output:

error: expected unqualified-id before numeric constant

Explanation

When compiling this code snippet, an error will be produced by the compiler. This issue arises due to the fact that the int keyword is not permissible as an identifier. Since int is a reserved keyword, assigning it as a variable name leads to a compilation error. To prevent this error, selecting an alternative name for the variable is necessary.

To Read More: C++ keywords

3) Constants

In C++, constants are values used to set the initial values of variables during their initialization phase. These values remain constant throughout the program's execution and cannot be altered once defined. Constants can be declared using keywords such as 'const', 'constexpr', or through the use of the preprocessor directive '#define'. Once a constant is assigned a value, it remains immutable.

C++ Constants Example

Let's consider an example to demonstrate the constants in C++.

Example

#include <iostream>  

  

using namespace std;  //using standard namespace

  

int main()  //Main Function  

{  

    

    const double PI = 3.14159;  // Define a constant named "PI" of type double 

  

    double radius = 10.0;  // Define a variable named "radius" of type double

  

    double area = PI * radius * radius;  //calculate the area of circle

  

    cout << "The area of the circle is " << area << "." << endl;  

  

    return 0;  

}

Output:

The area of the circle is 314.159.

Explanation

In this instance, the constant PI is employed to symbolize the numerical value of pi. The constant PI is established utilizing the const keyword and is set with the value of pi. Subsequently, the constant PI is applied in the computation of the circle's area, where it is multiplied by the square of the circle's radius to ascertain the area.

4) Strings

In C++, a string token refers to a class available within the Standard Template Library (STL) that offers capabilities for managing a series of characters collectively to form a string. Essentially, utilizing the string token involves instantiating an entity that signifies a collection of characters.

The string class encompasses various methods that allow manipulation of a given string, including functions like length, popback, pushback, and more.

Syntax

It has the following syntax:

string var_name;

string name= "It is string";

C++ String Example

Let's consider an example to demonstrate string manipulation in C++.

Example

#include <iostream>  

#include <string>

  

using namespace std;  //using standard namespace

  

int main()  //Main Function

{  

    string message = "Hello, This is Cpp Tutorial world!";  //Define a string variable name message

  

    cout << message << endl;  

  

    return 0;  

}

Output:

Hello, This is Cpp Tutorial world!

Explanation

In this instance, the string class generates a string variable called message. Subsequently, the message variable is set with the string "Greetings, This is C++ Tutorial world!", and then displayed on the screen utilizing the cout object.

To Read More: C++ String

5) Punctuators

Punctuators tokens have specific meanings within the syntax used for the C++ programming language. These are the token characters, and we can use punctuators in different variety of methods that include separating the items, ending the statements, and other tasks. Although we have several types of tokens which are available, some of them are as follows:

• ✅ Square Brackets : When we create an array, we store the elements of the array inside the square brackets. For example, [2, 3, 4, 6, 0] is an array having five elements.
• ✅ Single Quote (' '): When we initialize any character value, we enclose the characters within the single quotes. For example, 'a' is a character value.
• ✅ Semicolon (;): Whenever we complete any line of code in our C++ program, we terminate that line by putting a semicolon at the end. For example, int x = 10; where after initializing the value, we terminated the line with semicolon.
• ✅ Curly Braces ({ }): Whenever we write our code, it is enclosed within the curly braces always, and it represents the block of code. For example,

int main () {

// write the code

}

Here, the main function code is enclosed within these curly brackets.

• ✅ Quotation Marks (" "): When initializing a string literal, we enclose the text within double quotation marks. For instance, "CppTutorial Tech is the best platform for learning" represents a string enclosed in double quotes.

C++ Punctuators Example

Let's consider an example to demonstrate the Punctuators in C++.

Example

#include <iostream>  

  

using namespace std;

  

int main() { //Main Function

  int a = 5;  //define integer variable

  int b = 10;  

  int c = a * b; // Using the asterisk  

  cout << "The product of " << a << " and " << b << " is " << c << endl;  

  

  int *ptr = &a; // Use of the ampersand (&)  

  cout << "The address of a is " << ptr << endl;  

  

  int d = ~a; // Use of the tilde (~)  

  cout << "The bitwise NOT of " << a << " is " << d << endl;  

  

  return 0;  

}

Output:

The product of 5 and 10 is 50

The address of a is 0x7ffca52b0640

The bitwise NOT of 5 is -6

Explanation

In this instance, we've chosen the asterisk (*) symbol to function as the multiplication operator for determining the result of the variables a and b. The ampersand (&) symbol is employed to retrieve the memory address of variable 'a', which is then displayed on the console. Lastly, the tilde (~) symbol serves as the bitwise NOT operator to invert the value of a.

6) Operators

Tokens are essential elements for executing actions on various operands like variables, expressions, and constants. Within the C++ programming language, a range of operators exists to carry out distinct operations, including logical, arithmetic, and relational tasks. An instance of a logical operation is (X && Y), where X and Y function as operands, and the specific operation executed is an AND operation.

Generally, there are three types of operators available:

• ✅ Unary Operator: These are the operators that are used with single operands and perform specific unary operations on a single variable. Some of the unary operations include increment ( ++ ) and decrement ( -- ) operators, i.e. to increase the operand value by 1, use the increment operator, and to decrease the operand value by 1, use the decrement operator.
• ✅ Binary Operator: These are the operators that are used with two operands and perform specific binary operations on two variables. Some of the binary operations include arithmetic operators, logical operators, assignment operators, and many more.
• ✅ Ternary Operator: This operator is a type of operator that works between three operands, also known as a conditional operator. It is used to create conditional expressions. It either gives a true or false output.

Syntax

It has the following syntax:

Condition 1 ? Condition 2 : Condition 3;

In this scenario, when condition 1 evaluates to true, the program will execute condition 2; otherwise, condition 3 will be executed.

C++ Operators Example

Let's consider a scenario to demonstrate the operators in C++.

Example

#include <iostream>  

  

using namespace std;  

  

int main() {  //Main Function

  int a = 25;  //defining integer variable

  int b = 8;  

  int c = a + b; // Using plus(+) operator  

  cout << "The sum of " << a << " and " << b << " is " << c << endl;  

  

  c = a - b; // Using minus (-) operator

  cout << "The difference between " << a << " and " << b << " is " << c << endl;  

  

  c = a * b; // Using asterisk (*) operator  

  cout << "The product of " << a << " and " << b << " is " << c << endl;  

  c = a / b; // using forward slash (/) operator

  cout << "The quotient of " << a << " and " << b << " is " << c << endl;  

  

  bool d = (a == b); // Using double equals (==) operator 

  cout << "Is " << a << " equal to " << b << "? " << d << endl;  

  

  d = (a != b); // Using exclamation point and equals (!=) operator  

  cout << "Is " << a << " not equal to " << b << "? " << d << endl;  

  

  return 0;  

}

Output:

The sum of 25 and 8 is 33

The difference between 25 and 8 is 17

The product of 25 and 8 is 200

The quotient of 25 and 8 is 3

Is 25 equal to 8? 0

Is 25 not equal to 8? 1

Explanation

In this instance, a variety of operators are employed to execute distinct tasks. The + operator is utilized for adding values, the - operator is employed for subtracting values, the * operator is utilized for multiplying values, and the / operator is used for dividing values. Subsequently, the == and != operators are utilized to assess for equality and inequality.

To Read More: C++ Operators

C++ Token MCQs

1) Which of the following options is the type of Tokens in C++?

• ✅ Special Symbols
• ✅ Keywords
• ✅ Operators
• ✅ All of the Above

2) Which of the following option shows the correct keyword in C++?

• ✅ main
• ✅ define
• ✅ include

3) What will be the output of the following code?

#include <iostream>

using namespace std;

int main()

{

    int a=7, b=12;

    const int c=a*b;

    cout<<c;

    return 0;

}

4) Which of the following option shows the valid identifier in C++?

• ✅ name-123
• ✅ name_123
• ✅ 123name

5) What are punctuators in C++?

• ✅ Variable names
• ✅ Special characters that mark the end of statements or group codes
• ✅ Functions
• ✅ Reserved words

b) Unique characters that indicate the completion of statements or code blocks