Python Multiprocessing Tutorial

In Python, multiprocessing refers to the system's capacity to execute multiple processes concurrently. In straightforward terms, multiprocessing utilizes two or more CPUs within a single computer system. This approach also has the capability to distribute tasks among several processes.

Processing units utilize the primary memory and peripheral devices to execute programs concurrently. A multiprocessing application is divided into smaller components that operate autonomously. The operating system assigns each process to a specific processor.

Python includes a built-in package known as multiprocessing that facilitates process swapping. Prior to utilizing multiprocessing, it is essential to understand the concept of the process object.

Why Multiprocessing?

Multiprocessing is crucial for executing multiple tasks simultaneously within a computer system. Consider a computer that lacks multiprocessing capabilities or operates with a single processor. In this scenario, we allocate different processes to the system concurrently.

Subsequently, it must pause the ongoing task and transition to a different one to ensure all operations continue smoothly. This can be likened to a chef working solo in a kitchen. He must perform various duties to prepare meals, including chopping, sanitizing, cooking, kneading dough, and baking, among others.

Consequently, multiprocessing is crucial for executing multiple tasks simultaneously without disruption. It also simplifies the monitoring of all tasks involved. This is the reason why the idea of multiprocessing emerged.

• ✅ Multiprocessing can be depicted as a computer equipped with multiple central processors.
• ✅ A multi-core processor denotes a single computing unit that contains two or more autonomous cores.

In multiprocessing, the CPU is capable of allocating several tasks simultaneously, with each task being managed by its dedicated processor.

Multiprocessing In Python

Python features the multiprocessing module, which enables the execution of several tasks concurrently on a single machine. It presents a straightforward and easy-to-use API for engaging with multiprocessing.

Python Multiprocessing Example

Let us examine a straightforward illustration of multiprocessing in Python.

from multiprocessing import Process

   def disp():

      print ('Hello !! Welcome to Python Tutorial')

      if __name__ == '__main__':

      p = Process(target=disp)

      p.start()

      p.join()

Output:

'Hello !! Welcome to Python Tutorial'

Explanation:

In the code provided, the Process class has been imported, and a Process object is instantiated within the disp function. Subsequently, the process is initiated using the start method and concluded with the join method. Additionally, we can pass arguments to the defined function by utilizing the args keyword.

Python Multprocessing Example with Arguments

Let’s examine the subsequent illustration of multiprocessing involving arguments.

# Python multiprocessing example

# importing the multiprocessing module

import multiprocessing

def cube(n):

   # This function will print the cube of the given number

   print("The Cube is: {}".format(n * n * n))

def square(n):

    # This function will print the square of the given number

   print("The Square is: {}".format(n * n))

if __name__ == "__main__":

   # creating two processes

   process1 = multiprocessing.Process(target= square, args=(5, ))

   process2 = multiprocessing.Process(target= cube, args=(5, ))

   # Here we start the process 1

   process1.start()

   # Here we start process 2

   process2.start()

   # The join() method is used to wait for process 1 to complete

   process1.join()

   # It is used to wait for process 1 to complete

   process2.join()

   # Print if both processes are completed

   print("Both processes are finished")

Output:

The Cube is: 125

The Square is: 25

Both processes are finished

Explanation -

In the preceding example, we developed two functions: the cube function, which computes the cube of a specified number, and the square function, which determines the square of the specified number.

Subsequently, we instantiated the process object from the Process class, which takes two parameters. The first parameter, target, signifies the function intended for execution, while the second parameter, args, denotes the arguments to be supplied to the function.

process1 = multiprocessing.Process(target= square, args=(5, ))

process2 = multiprocessing.Process(target= cube, args=(5, ))

The process has been initiated using the start method.

process1.start()

process2.start()

The output demonstrates that it pauses until process one is completed, followed by process two. The final statement is executed only after both processes have concluded.

Python Multiprocessing Classes

The Python multiprocessing module offers various classes that are frequently utilized for developing parallel applications. This discussion will focus on its key classes: Process, Queue, and Lock. The Process class was covered in the earlier example, and now we will turn our attention to the Queue and Lock classes.

Python Multiprocessing Classes Example

Here is a straightforward illustration of how to retrieve the count of CPUs present in the system.

import multiprocessing

print("The number of CPU currently working in system : ", multiprocessing.cpu_count())

Output:

('The number of CPU currently woking in system : ', 32)

The quantity of CPUs may differ for your personal computer. In our case, we have 32 cores.

Python Multiprocessing Using Queue Class

It is understood that the Queue is a vital component of data structures. In Python multiprocessing, the Queue functions similarly to the data structure queue, adhering to the "First-In-First-Out" principle. Typically, the Queue holds Python objects and is crucial for facilitating data sharing among processes.

Queues are provided as arguments in the target function of the Process to enable the process to access data. The Queue offers the put method for adding data and the get method for retrieving data from the queues.

Python Multiprocessing Example using Queue Class

To illustrate the concept of multiprocessing with the Queue class in Python, we will consider an example.

# Importing Queue Class

from multiprocessing import Queue

fruits = ['Apple', 'Orange', 'Guava', 'Papaya', 'Banana']

count = 1

# creating a queue object

queue = Queue()

print('pushing items to the queue:')

for fr in fruits:

    print('item no: ', count, ' ', fr)

    queue.put(fr)

    count += 1

print('\npopping items from the queue:')

count = 0

while not queue.empty():

    print('item no: ', count, ' ', queue.get())

    count += 1

Output:

pushing items to the queue:

('item no: ', 1, ' ', 'Apple')

('item no: ', 2, ' ', 'Orange')

('item no: ', 3, ' ', 'Guava')

('item no: ', 4, ' ', 'Papaya')

('item no: ', 5, ' ', 'Banana')

popping items from the queue:

('item no: ', 0, ' ', 'Apple')

('item no: ', 1, ' ', 'Orange')

('item no: ', 2, ' ', 'Guava')

('item no: ', 3, ' ', 'Papaya')

('item no: ', 4, ' ', 'Banana')

Explanation -

In the preceding code, the Queue class has been imported, and a list called fruits has been initialized. Following that, we set a variable named count to 1, which will keep track of the total number of items. Subsequently, we instantiated the queue object by invoking the Queue method. This object will facilitate operations within the Queue. Within a for loop, we added elements to the queue individually using the put function, incrementing the count by 1 with each iteration of the loop.

Python Multiprocessing Lock Class

The Lock class in multiprocessing is utilized to gain a lock on a process, preventing other processes from executing similar code until the lock is freed. This class primarily serves two functions: the first is to obtain a lock via the acquire method, and the second is to relinquish the lock using the release method.

Python Multiprocessing Lock Class Example

Let’s consider a scenario involving several tasks. We will set up two separate queues: one will hold the tasks, while the second will keep a log of completed tasks. The subsequent step involves initializing the processes required to execute the tasks. As mentioned earlier, the Queue class is inherently synchronized, which means there is no requirement to obtain a lock using the Lock class.

In the subsequent illustration, we will consolidate all the multiprocessing classes. Please refer to the example below.

from multiprocessing import Lock, Process, Queue, current_process

import time

import queue

def jobTodo(tasks_to_perform, complete_tasks):

    while True:

        try:

            # The try block to catch task from the queue.

            # The get_nowait() function is used to

            # raise queue.Empty exception if the queue is empty.

            task = tasks_to_perform.get_nowait()

        except queue.Empty:

            break

        else:

                # if no exception has been raised, the else block will execute

                # add the task completion

            print(task)

            complete_tasks.put(task + ' is done by ' + current_process().name)

            time.sleep(.5)

    return True

def main():

    total_task = 8

    total_number_of_processes = 3

    tasks_to_perform = Queue()

    complete_tasks = Queue()

    number_of_processes = []

    for i in range(total_task):

        tasks_to_perform.put("Task no " + str(i))

    # defining number of processes

    for w in range(total_number_of_processes):

        p = Process(target=jobTodo, args=(tasks_to_perform, complete_tasks))

        number_of_processes.append(p)

        p.start()

    # completing process

    for p in number_of_processes:

        p.join()

    # print the output

    while not complete_tasks.empty():

        print(complete_tasks.get())

    return True

if __name__ == '__main__':

    main()

Output:

Task no 2

Task no 5

Task no 0

Task no 3

Task no 6

Task no 1

Task no 4

Task no 7

Task no 0 is done by Process-1

Task no 1 is done by Process-3

Task no 2 is done by Process-2

Task no 3 is done by Process-1

Task no 4 is done by Process-3

Task no 5 is done by Process-2

Task no 6 is done by Process-1

Task no 7 is done by Process-3

Python Multiprocessing Pool

The Python multiprocessing pool is crucial for executing a function concurrently over various input values. Additionally, it facilitates the allocation of input data among different processes (data parallelism).

Python Multiprocessing Pool Example

Examine the subsequent illustration of a multiprocessing Pool.

from multiprocessing import Pool

import time

w = (["V", 5], ["X", 2], ["Y", 1], ["Z", 3])

def work_log(data_for_work):

    print(" Process name is %s waiting time is %s seconds" % (data_for_work[0], data_for_work[1]))

    time.sleep(int(data_for_work[1]))

    print(" Process %s Executed." % data_for_work[0])

def handler():

    p = Pool(2)

    p.map(work_log, w)

if __name__ == '__main__':

    handler()

Output:

Process name is V waiting time is 5 seconds

Process V Executed.

Process name is X waiting time is 2 seconds

Process X Executed.

Process name is Y waiting time is 1 seconds

Process Y Executed.

Process name is Z waiting time is 3 seconds

Process Z Executed.

Another Python Example for Multiprocessing Pool

Let us examine an additional instance of the multiprocessing Pool.

from multiprocessing import Pool

def fun(x):

    return x*x

if __name__ == '__main__':

    with Pool(5) as p:

        print(p.map(fun, [1, 2, 3]))

Output:

[1, 8, 27]

Proxy Objects

The objects known as proxies are categorized as shared objects that exist within a separate process. This entity is also identified as a proxy. It is possible for multiple proxy objects to reference the same referent. A proxy object includes several methods designed to call the corresponding methods of its referent.

Python Proxy Objects Example

Let's consider an example to illustrate the use of proxy objects in Python.

from multiprocessing import Manager

manager = Manager()

l = manager.list([i*i for i in range(10)])

print(l)

print(repr(l))

print(l[4])

print(l[2:5])

Output:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

<ListProxy object, typeid 'list' at 0x7f063621ea10>

16

[4, 9, 16]

The proxy objects are capable of being pickled, allowing for their transfer between processes. Additionally, these objects facilitate a degree of control regarding synchronization.

Commonly Used Functions of Multiprocessing

So far, we have discussed the basic concepts of multiprocessing using Python. Multiprocessing is a broad topic itself and essential for performing various tasks within a single system. We are defining a few essential functions that are commonly used to achieve multiprocessing.