Cara menggunakan kill process python multiprocessing

The

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

The multiprocesing module avoids the limitations of the Global Interpreter Lock (GIL) by using subprocesses instead of threads. The multiprocessed code does not execute in the same order as serial code. There is no guarantee that the first process to be created will be the first to complete.

Python GIL

A global interpreter lock (GIL) is a mechanism used in Python interpreter to synchronize the execution of threads so that only one native thread can execute at a time, even if run on a multi-core processor.

The C extensions, such as numpy, can manually release the GIL to speed up computations. Also, the GIL released before potentionally blocking I/O operations.

Note that both Jython and IronPython do not have the GIL.

Concurrency means that two or more calculations happen within the same time frame. Parallelism means that two or more calculations happen at the same moment. Parallelism is therefore a specific case of concurrency. It requires multiple CPU units or cores.

True parallelism in Python is achieved by creating multiple processes, each having a Python interpreter with its own separate GIL.

Python has three modules for concurrency:

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

Embarrassinbly parallel

The term embarrassinbly parallel is used to describe a problem or workload that can be easily run in parallel. It is important to realize that not all workloads can be divided into subtasks and run parallelly. For instance those, who need lots of communication among subtasks.

The examples of perfectly parallel computations include:

• Monte Carlo analysis
• numerical integration
• rendering of computer graphics
• brute force searches in cryptography
• genetic algorithms

Another situation where parallel computations can be applied is when we run several different computations, that is, we don't divide a problem into subtasks. For instance, we could run calculations of π using different algorithms in parallel.

Both processes and threads are independent sequences of execution. The following table summarizes the differences between a process and a thread:

Process

The

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

The

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

The Python multiprocessing style guide recommends to place the multiprocessing code inside the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

Simple process example

The following is a simple program that uses

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

#!/usr/bin/python

from multiprocessing import Process


def fun(name):
    print(f'hello {name}')

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()


if __name__ == '__main__':
    main()

We create a new process and pass a value to it.

def fun(name):
    print(f'hello {name}')

The function prints the passed parameter.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

A new process is created. The

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

#!/usr/bin/python

from multiprocessing import Process
import time

def fun(val):

    print(f'starting fun with {val} s')
    time.sleep(val)
    print(f'finishing fun with {val} s')


def main():

    p1 = Process(target=fun, args=(3, ))
    p1.start()
    # p1.join()

    p2 = Process(target=fun, args=(2, ))
    p2.start()
    # p2.join()

    p3 = Process(target=fun, args=(1, ))
    p3.start()
    # p3.join()

    p1.join()
    p2.join()
    p3.join()

    print('finished main')

if __name__ == '__main__':

    main()

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

if __name__ == '__main__':
    main()

The code is placed inside the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

Python multiprocessing join

The

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

The example calls the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

The finishing main message is printed after the child process has finished.

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

When we comment out the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

It is important to call the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

#!/usr/bin/python

from multiprocessing import Process
import time

def fun(val):

    print(f'starting fun with {val} s')
    time.sleep(val)
    print(f'finishing fun with {val} s')


def main():

    p1 = Process(target=fun, args=(3, ))
    p1.start()
    # p1.join()

    p2 = Process(target=fun, args=(2, ))
    p2.start()
    # p2.join()

    p3 = Process(target=fun, args=(1, ))
    p3.start()
    # p3.join()

    p1.join()
    p2.join()
    p3.join()

    print('finished main')

if __name__ == '__main__':

    main()

If we call the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

The

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('calling fun')
    time.sleep(2)

def main():

    print('main fun')

    p = Process(target=fun)
    p.start()
    p.join()

    print(f'Process p is alive: {p.is_alive()}')


if __name__ == '__main__':
    main()

When we wait for the child process to finish with the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

The

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('calling fun')
    time.sleep(2)

def main():

    print('main fun')

    p = Process(target=fun)
    p.start()
    p.join()

    print(f'Process p is alive: {p.is_alive()}')


if __name__ == '__main__':
    main()

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('calling fun')
    time.sleep(2)

def main():

    print('main fun')

    p = Process(target=fun)
    p.start()
    p.join()

    print(f'Process p is alive: {p.is_alive()}')


if __name__ == '__main__':
    main()

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

The example runs two child processes. It prints their Id and their parent's Id.

def fun(name):
    print(f'hello {name}')

The parent Id is the same, the process Ids are different for each child process.

With the

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

def fun(name):
    print(f'hello {name}')

In the example, we create three processes; two of them are given a custom name.

def fun(name):
    print(f'hello {name}')

Subclassing Process

When we subclass the

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

def fun(name):
    print(f'hello {name}')

We create a

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

$ ./joining.py
starting main
starting fun
finishing fun
finishing main

$ ./joining.py
starting main
finishing main
starting fun
finishing fun

The management of the worker processes can be simplified with the

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

The

#!/usr/bin/python

from multiprocessing import Process
import os

def fun():

    print('--------------------------')

    print('calling fun')
    print('parent process id:', os.getppid())
    print('process id:', os.getpid())

def main():

    print('main fun')
    print('process id:', os.getpid())

    p1 = Process(target=fun)
    p1.start()
    p1.join()

    p2 = Process(target=fun)
    p2.start()
    p2.join()


if __name__ == '__main__':
    main()

def fun(name):
    print(f'hello {name}')

def fun(name):
    print(f'hello {name}')

In the example, we create a pool of processes and apply values on the

def fun(name):
    print(f'hello {name}')

def fun(name):
    print(f'hello {name}')

def fun(name):
    print(f'hello {name}')

On a computer with four cores it took slightly more than 2 seconds to finish four computations, each lasting two seconds.

def fun(name):
    print(f'hello {name}')

When we add additional value to be computed, the time increased to over four seconds.

Multiple arguments

To pass multiple arguments to a worker function, we can use the

def fun(name):
    print(f'hello {name}')

def fun(name):
    print(f'hello {name}')

In this example, we pass two values to the

def fun(name):
    print(f'hello {name}')

def fun(name):
    print(f'hello {name}')

Multiple functions

The following example shows how to run multiple functions in a pool.

def fun(name):
    print(f'hello {name}')

We have three functions, which are run independently in a pool. We use the

def fun(name):
    print(f'hello {name}')

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

Python multiprocessing π calculation

The π is the ratio of the circumference of any circle to the diameter of the circle. The π is an irrational number whose decimal form neither ends nor becomes repetitive. It is approximately equal to 3.14159. There are several formulas to calculate π.

Calculating approximations of π can take a long time, so we can leverage the parallel computations. We use the Bailey–Borwein–Plouffe formula to calculate π.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

First, we calculate three approximations sequentially. The precision is the number of digits of the computed π.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

On our machine, it took 0.57381 seconds to compute the three approximations.

In the following example, we use a pool of processes to calculate the three approximations.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

We run the calculations in a pool of three processes and we gain some small increase in efficiency.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

When we run the calculations in parallel, it took 0.38216479 seconds.

In multiprocessing, each worker has its own memory. The memory is not shared like in threading.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

We create a worker to which we pass the global

def fun(name):
    print(f'hello {name}')

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

As we can see from the output, the two lists are separate.

Sharing state between processes

Data can be stored in a shared memory using

def fun(name):
    print(f'hello {name}')

def fun(name):
    print(f'hello {name}')

Note: It is best to avoid sharing data between processes. Message passing is preferred.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

The example creates a counter object which is shared among processes. Each of the processes increases the counter.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

Each process must acquire a lock for itself.

The message passing is the preferred way of communication among processes. Message passing avoids having to use synchronization primitives such as locks, which are difficult to use and error prone in complex situations.

To pass messages, we can utilize the pipe for the connection between two processes. The queue allows multiple producers and consumers.

def main():

    p = Process(target=fun, args=('Peter',))
    p.start()

In the example, we create four processes. Each process generates a random value and puts it into the queue. After all processes finish, we get all values from the queue.

if __name__ == '__main__':
    main()

The queue is passed as an argument to the process.

if __name__ == '__main__':
    main()

The

def fun(name):
    print(f'hello {name}')

if __name__ == '__main__':
    main()

The example generates a list of four random values.

In the following example, we put words in a queue. The created processes read the words from the queue.

if __name__ == '__main__':
    main()

Four processes are created; each of them reads a word from the queue and prints it.

if __name__ == '__main__':
    main()

Queue order

In multiprocessing, there is no guarantee that the processes finish in a certain order.

if __name__ == '__main__':
    main()

We have processes that calculate the square of a value. The input data is in certain order and we need to maintain this order. To deal with this, we keep an extra index for each input value.

if __name__ == '__main__':
    main()

To illustrate variation, we randomly slow down the calculation with the

def fun(name):
    print(f'hello {name}')

if __name__ == '__main__':
    main()

We get the results. At this moment, the tuples are in random order.

if __name__ == '__main__':
    main()

We sort the result data by their index values.

if __name__ == '__main__':
    main()

We get the square values that correspond to the initial data.

Monte Carlo methods are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be deterministic in principle.

The following formula is used to calculate the approximation of π:

The M is the number of generated points in the square and N is the total number of points.

While this method of π calculation is interesting and perfect for school examples, it is not very accurate. There are far better algorithms to get π.

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

In the example, we calculate the approximation of the π value using one hundred million generated random points.

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

It took 44.78 seconds to calculate the approximation of π

Now we divide the whole task of π computation into subtasks.

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

In the example, we find out the number of cores and divide the random sampling into subtasks. Each task will compute the random values independently.

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

Instead of calculating 100_000_000 in one go, each subtask will calculate a portion of it.

#!/usr/bin/python

from multiprocessing import Process
import time

def fun():

    print('starting fun')
    time.sleep(2)
    print('finishing fun')

def main():

    p = Process(target=fun)
    p.start()
    p.join()


if __name__ == '__main__':

    print('starting main')
    main()
    print('finishing main')

The partial calculations are passed to the

def fun(name):
    print(f'hello {name}')