Python 并發(fā)、并行、同步和異步——學(xué)習(xí)筆記

  • 串行:同一個(gè)時(shí)間段只干一件事

  • 并行:同一個(gè)時(shí)間段可以干多件事

  • 并發(fā) V.S. 并行

    • 并發(fā)是指一個(gè)時(shí)間段內(nèi),有幾個(gè)程序在同一個(gè)CPU上運(yùn)行,但是任意時(shí)刻只有一個(gè)程序在CPU上運(yùn)行。
    • 并行是指任意時(shí)刻點(diǎn)上,有多個(gè)程序同時(shí)運(yùn)行在多個(gè)CPU上,即每個(gè)CPU獨(dú)立運(yùn)行一個(gè)程序。
    • 并行的最大數(shù)量和CPU的數(shù)量是一致的。
  • 同步 V.S. 異步:

    • 同步是指代碼調(diào)用IO操作時(shí),必須等待IO操作完成返回才調(diào)用的方式
    • 異步是指代碼調(diào)用IO操作時(shí),不必等待IO操作完成返回才調(diào)用的方式
  • 多線程:交替執(zhí)行,另一種意義上的串行 \Rightarrow 比進(jìn)程的概念小,減少了進(jìn)程切換過(guò)程中的消耗

  • 多進(jìn)程:并行執(zhí)行,真正意義上的并行 \Rightarrow 通過(guò)切換不同的進(jìn)程來(lái)實(shí)現(xiàn)多任務(wù)的并行

  • 多進(jìn)程 V.S. 多線程:

    • 由于全局解釋鎖GIL的存在,Python的多線程無(wú)法利用多核優(yōu)勢(shì),所以不適合計(jì)算密集型任務(wù),適合IO密集型任務(wù)。
    • CPU密集型任務(wù)適合使用多進(jìn)程
    • 進(jìn)程切換代價(jià)要高于線程
    • 線程之間可以通過(guò)全局變量來(lái)進(jìn)行通信,但是進(jìn)程不行。進(jìn)程之間的數(shù)據(jù)是完全隔離的。
  • threading類(lèi):

    ? threading.Lock():同步鎖(互斥鎖),解決數(shù)據(jù)安全問(wèn)題

    ? threading.RLock():遞歸鎖,解決線程死鎖問(wèn)題

    ? threading.Semaphore():信號(hào)量,最多允許同時(shí)有n個(gè)線程,超過(guò)信號(hào)量的線程會(huì)被堵塞,直到有線程的信號(hào)量被釋放。

    ? threading.Event():事件,讓不同的線程保持同步而不是獨(dú)立的運(yùn)行,彼此交互:你給我發(fā)一個(gè)信號(hào),我接到了執(zhí)行一個(gè)操作,再給你發(fā)一個(gè)信號(hào),你接到了執(zhí)行操作并繼續(xù)發(fā)信號(hào)……

import threading
#生成鎖對(duì)象,全局唯一
lock = threading.Lock()
#獲取鎖
lock.acquire()
#釋放鎖
lock.release()
  • 可以使用上下文管理協(xié)議來(lái)管理鎖
lock = threading.Lock()
with lock:
    pass
#等價(jià)于:
lock.acquire()
try:
    pass
finally:
    lock.release()
  • 加鎖會(huì)影響性能,獲取鎖和釋放鎖都需要時(shí)間。
  • 使用鎖可能會(huì)引起死鎖

多線程

import threading
import time

def sleeper(n,name):
    print('Hi, I`m {}. I`m going to sleep 5 seconds'.format(name))
    time.sleep(n)
    print('{} has woken up from sleep'.format(name))
e.g.1
def main():
    t = threading.Thread(target = sleeper, args = (5,'thread1'))
    t.start()

    #調(diào)用.join()方法來(lái)阻塞當(dāng)前線程,直到該線程結(jié)束后,才跳轉(zhuǎn)執(zhí)行其余的線程
    #t.join() 如果加上這個(gè)語(yǔ)句,程序需要等整個(gè)sleeper執(zhí)行完畢后才會(huì)跳回執(zhí)行后面的兩次print

    print('hello')
    print('hello')
>>
Hi, I`m thread1. I`m going to sleep 5 seconds.
hello
hello
thread1 has woken up from sleep.
e.g.2
def main():
    start = time.time()
    thread_list = []
    for i in range(1,6):
        t = threading.thread(target=sleeper, args=(5,'thread{}'.format(i)))
        thread_list.append(t)
        t.start()
        print('{} has started.'.format(t.name)
    
    for t in thread_list:
        t.join()
    end = time.time()
    print('total time: {}'.format(end - start))
    print('aaa')  
              
>>
Hi, I`m thread1. I`m going to sleep 5 seconds
Thread-1 has started.
Hi, I`m thread2. I`m going to sleep 5 seconds
Thread-2 has started.
Hi, I`m thread3. I`m going to sleep 5 seconds
Thread-3 has started.
Hi, I`m thread4. I`m going to sleep 5 seconds
Thread-4 has started.
thread2 has woken up from sleep
thread1 has woken up from sleep
thread4 has woken up from sleep
thread3 has woken up from sleep
total time: 5.001285791397095
aaa              
e.g.3
def main():
    start = time.time()
    thread_list = []
    for i in range(1,6):
        t = threading.thread(target=sleeper, args=(5,'thread{}'.format(i)))
        thread_list.append(t)
        t.start()
        t.join() #阻塞進(jìn)程
        print('{} has started.'.format(t.name)

    end = time.time()
    print('total time: {}'.format(end - start))
    print('aaa')  
              
>>
Hi, I`m thread1. I`m going to sleep 5 seconds
thread1 has woken up from sleep
Thread-1 has started.
Hi, I`m thread2. I`m going to sleep 5 seconds
thread2 has woken up from sleep
Thread-2 has started.
Hi, I`m thread3. I`m going to sleep 5 seconds
thread3 has woken up from sleep
Thread-3 has started.
Hi, I`m thread4. I`m going to sleep 5 seconds
thread4 has woken up from sleep
Thread-4 has started.
total time: 20.00114393234253
aaa              

線程之間通信的方法

queue:

  • queue是線程安全的,其源碼中已經(jīng)使用了鎖
  • queue也是共享變量的一種,只是它自身已經(jīng)具有線程安全性。如果換成其他變量可能會(huì)出現(xiàn)不同步的問(wèn)題。
  • 常用方法:
    • from queue import Queue
    • put(self, item, block=True, timeout=None):將元素加入隊(duì)列,隊(duì)列為滿時(shí)為阻塞,可以設(shè)定timeout參數(shù)
    • get(self, block=True, timeout=None):從隊(duì)列中取出元素,隊(duì)列為空時(shí)阻塞,可以設(shè)定timeout參數(shù)
    • qsize():返回隊(duì)列的大小
    • empty():判斷是否為空
    • full():判斷是否為滿
    • join():阻塞直到隊(duì)列中的所有元素都被取出執(zhí)行。在調(diào)用該方法前要使用task_done()方法,否則隊(duì)列將一直處在阻塞狀態(tài)
    • task_done():指示之前的隊(duì)列任務(wù)已經(jīng)完成
    • put_nowait(self, item):非阻塞模式,隊(duì)列滿時(shí)直接拋出異常
    • get_nowait(self):非阻塞模式,隊(duì)列空時(shí)直接拋出異常

condition

from threading import Condition

  • 條件變量,用于復(fù)雜的線程間的同步。

  • threading中的Queuesemaphore的實(shí)現(xiàn)都使用了condition

  • Condition中定義了__enter____exit__方法,所以可以使用with語(yǔ)句實(shí)現(xiàn)上下文管理

  • 常用方法:

    • Condition中實(shí)際上有兩層鎖:

      • 初始化時(shí)會(huì)上一把默認(rèn)的遞歸鎖鎖住condition,這把鎖會(huì)在調(diào)用wait()方法時(shí)被解除,稱(chēng)為底層鎖
      • 在調(diào)用wait()方法時(shí)還會(huì)分配一把鎖將當(dāng)前進(jìn)程鎖住,并將這把鎖添加到一個(gè)雙端隊(duì)列中去。等到調(diào)用notify()方法時(shí)該鎖會(huì)被釋放掉
          def __init__(self, lock=None):
              if lock is None:
                  lock = RLock()
              self._lock = lock
              # Export the lock's acquire() and release() methods
              self.acquire = lock.acquire
              self.release = lock.release
              
          def wait(self, timeout=None):
               if not self._is_owned():
                  raise RuntimeError("cannot wait on un-acquired lock")
              waiter = _allocate_lock()
              waiter.acquire() #分配一把鎖鎖住當(dāng)前進(jìn)程
              self._waiters.append(waiter) #將鎖加入到一個(gè)雙端隊(duì)列中去
              saved_state = self._release_save() #釋放掉初始化conditon時(shí)加上的底層鎖
              
          def notify(self, n=1):
             if not self._is_owned():
                raise RuntimeError("cannot notify on un-acquired lock")
              all_waiters = self._waiters
              waiters_to_notify = _deque(_islice(all_waiters, n))
              if not waiters_to_notify:
                  return
              for waiter in waiters_to_notify:
                  waiter.release() #釋放掉雙端隊(duì)列中的鎖
                  try:
                      all_waiters.remove(waiter)
                  except ValueError:
                      pass
      
    • wait(self, predicate, timeout=None)

      • Wait until notified or until a timeout occurs.

        This method releases the underlying lock, and then blocks until it is awakened by a notify() or notify_all() for the same condition variable in another thread, or until the optional timeout occurs.

      • 釋放掉當(dāng)前線程的底層鎖,使得其他線程可以訪問(wèn)當(dāng)前線程。同時(shí)為當(dāng)前線程加上一把鎖,在收到notify()的通知之前阻塞當(dāng)前線程剩余的部分繼續(xù)執(zhí)行。

      • 即:在調(diào)用了wait()方法后,需要等待notify()的通知,才能啟動(dòng)。

    • notify(self, n=1)

      • Wake up one or more threads waiting on this condition.

      • 釋放掉當(dāng)前線程的鎖,使得其余線程的wait()方法可以獲得鎖

  • class PersonA(threading.Thread):
        def __init__(self,cond):
            super().__init__(name="A")
            self.cond = cond
            
        def run(self):
            with self.cond: #為PersonA加上一把底層鎖
                print('{}: Hello B,你在嗎?'.format(self.name))
                self.cond.notify() #給B發(fā)出通知,釋放掉B中wait加上的鎖
                self.cond.wait() #釋放掉A的底層鎖,確保下一次收到notify()通知后線程能夠切入。同時(shí)給A加上一把鎖,阻塞之后程序的運(yùn)行
                
     class PersonB(threading.Thread):
        def __init__(self,cond):
            super().__init__(name="B")
            self.cond = cond
            
        def run(self):
            with self.cond: #為PersonB加上一把底層鎖
                self.cond.wait() #使用wait方法阻塞后續(xù)程序,需要等待A發(fā)出通知,同時(shí)釋放掉B的底層鎖,保證線程可以切入進(jìn)來(lái)
                print('{}: Hello A,我在'.format(self.name))
                self.cond.notify() #給A發(fā)出一個(gè)通知,同時(shí)釋放掉A中wait加上的鎖
                
     if __name__ == '__main__':
        cond = threading.Condition()
        A = PersonA(cond)
        B = PersonB(cond)
        B.start() 
        A.start()
        #如果先調(diào)用A.start(),程序會(huì)阻塞,無(wú)法繼續(xù)進(jìn)行,原因在于先啟動(dòng)A時(shí),notify()方法已經(jīng)被發(fā)出,而此時(shí)B的wait()方法還沒(méi)有啟動(dòng),也就沒(méi)法接受到notify()的通知,所以程序無(wú)法再繼續(xù)
        
    >> 
    A: Hello B,你在嗎?
    B: Hello A,我在
    
    • 注意start()的順序很重要,錯(cuò)誤的start()順序會(huì)導(dǎo)致阻塞,程序無(wú)法繼續(xù)進(jìn)行。

信號(hào)量(semaphore)

  • 用于控制進(jìn)入數(shù)量的鎖
  • 文件的讀寫(xiě),通常情況下讀可以由多個(gè)線程來(lái)讀,而寫(xiě)僅由一個(gè)線程來(lái)寫(xiě)
  • threading.semaphore(n):可以同時(shí)允許n個(gè)線程執(zhí)行,大于n的部分必須等之前有線程釋放后才能加入繼續(xù)執(zhí)行。

concurrent.futures

  • 封裝好的方法,為多線程和多進(jìn)程提供了統(tǒng)一接口,方便調(diào)用。

  • 優(yōu)勢(shì):

    • 主線程中可以獲取某一個(gè)線程的狀態(tài)或者某一任務(wù)的狀態(tài),以及返回值
    • 當(dāng)一個(gè)線程完成的時(shí)候我們主線程能立即知道
  • from concurrent.futures import ThreadPoolExecutor
    from concurrent.futures import Future
    
    def get_html(times):
        time.sleep(times)
        print('get page {} success'.format(times))
        return times
    
    executor = ThreadPoolExecutor(max_workers=2) #最大線程數(shù)設(shè)定為2
    
    task1 = executor.submit(get_html,(3)) #使用submit方法來(lái)提交線程
    task2 = executor.submit(get_html,(2)) #executor返回的是一個(gè)Future對(duì)象
    
    task1.done() #用于判定某個(gè)任務(wù)是否完成,該方法沒(méi)有阻塞,是Future的方法
    task2.cancel() #取消線程,只能在線程還沒(méi)有開(kāi)始前取消,若線程已經(jīng)開(kāi)始執(zhí)行,則無(wú)法取消
    task1.result() #返回線程執(zhí)行的結(jié)果,有阻塞,只有在線程結(jié)束后才有返回值
    
  • 常用方法:

    • as_completed(fs, timeout=None)

      • 生成器,返回已經(jīng)完成的Future
      • 返回結(jié)果不一定與參數(shù)的傳入順序相同,誰(shuí)先完成就先返回誰(shuí)
      #上例中若只獲取已經(jīng)成功了的task的結(jié)果
      from concurrent.futures import as_completed
      urls = [3,4,2]
      all_task = [executor.submit(get_html, (url)) for url in urls]
      for future in as_completed(all_task): 
          #無(wú)阻塞的方法,一旦子線程完成執(zhí)行,主線程中即可立馬返回相應(yīng)的結(jié)果
          data = future.result()
          print("get {} page success".format(data))
      
    • .map()

      • 類(lèi)似于python中的map()函數(shù),將可迭代的參數(shù)列表作用在同一函數(shù)上,其本質(zhì)依舊是一個(gè)生成器,直接返回future.result()
      • 返回的順序與調(diào)用可迭代參數(shù)的順序一致
      #只獲取已經(jīng)成功了的task的結(jié)果的另一種方法
      urls = [3,4,2]
      for future in executor.mao(get_html, urls):
          print('get {} page success'.format(data))
      
    • wait(fs, timeout=None, return_when=ALL_COMPLETED):阻塞主線程,直到指定的某些子線程完成后才能繼續(xù)執(zhí)行主線程(在return_when參數(shù)中設(shè)定)

    • with ThreadPoolExecutor(n) as executor:使用with語(yǔ)句來(lái)進(jìn)行調(diào)用

Future

  • submit(self, fn, *args, **kwargs)

    class ThreadPoolExecutor(_base.Executor):
    
        # Used to assign unique thread names when thread_name_prefix is not supplied.
        _counter = itertools.count().__next__
    
        def __init__(self, max_workers=None, thread_name_prefix=''):
            """Initializes a new ThreadPoolExecutor instance.
    
            Args:
                max_workers: The maximum number of threads that can be used to
                    execute the given calls.
                thread_name_prefix: An optional name prefix to give our threads.
            """
            if max_workers is None:
                # Use this number because ThreadPoolExecutor is often
                # used to overlap I/O instead of CPU work.
                max_workers = (os.cpu_count() or 1) * 5
            if max_workers <= 0:
                raise ValueError("max_workers must be greater than 0")
    
            self._max_workers = max_workers
            self._work_queue = queue.Queue()
            self._threads = set()
            self._shutdown = False
            self._shutdown_lock = threading.Lock()
            self._thread_name_prefix = (thread_name_prefix or
                                        ("ThreadPoolExecutor-%d" % self._counter()))
        
        
        def submit(self, fn, *args, **kwargs):
            with self._shutdown_lock: #添加鎖,為了鎖住后面的一段代碼
                if self._shutdown:
                    raise RuntimeError('cannot schedule new futures after shutdown')
    
                f = _base.Future() #生成一個(gè)Future對(duì)象,最后這個(gè)對(duì)象會(huì)被返回
                w = _WorkItem(f, fn, args, kwargs)
                #WorkItem才是整個(gè)線程池的真正執(zhí)行單元,F(xiàn)uture對(duì)象會(huì)被傳入WorkItem中
    
                self._work_queue.put(w) #整個(gè)WorkItem實(shí)例被放入到_work_queue中
                self._adjust_thread_count()
                return f
        submit.__doc__ = _base.Executor.submit.__doc__
        
        def _adjust_thread_count(self):
            # When the executor gets lost, the weakref callback will wake up
            # the worker threads.
            def weakref_cb(_, q=self._work_queue):
                q.put(None)
            # TODO(bquinlan): Should avoid creating new threads if there are more
            # idle threads than items in the work queue.
            num_threads = len(self._threads)
            if num_threads < self._max_workers: #判斷啟動(dòng)了多少線程
                thread_name = '%s_%d' % (self._thread_name_prefix or self,
                                         num_threads)
                t = threading.Thread(name=thread_name, target=_worker,
                                     args=(weakref.ref(self, weakref_cb),
                                           self._work_queue)) 
                #線程執(zhí)行的函數(shù)是_worker,而_worker中執(zhí)行的函數(shù)來(lái)自self._work_queue
                #_worker函數(shù)的作用:運(yùn)行線程,不停的從_work_queue中取出_WorkItem
                t.daemon = True
                t.start()
                self._threads.add(t) #將啟動(dòng)了的線程數(shù)量加入到_threads中
                _threads_queues[t] = self._work_queue
                
     class _WorkItem(object):
        #設(shè)定這個(gè)類(lèi)的目的其實(shí)就是執(zhí)行函數(shù),并將執(zhí)行的結(jié)果設(shè)置到Future中
        def __init__(self, future, fn, args, kwargs):
            self.future = future
            self.fn = fn #需要執(zhí)行的函數(shù)在這里傳入
            self.args = args
            self.kwargs = kwargs
    
        def run(self):
            if not self.future.set_running_or_notify_cancel():
                return
    
            try:
                result = self.fn(*self.args, **self.kwargs)
            except BaseException as exc:
                self.future.set_exception(exc)
                # Break a reference cycle with the exception 'exc'
                self = None
            else:
                self.future.set_result(result) #將函數(shù)執(zhí)行的結(jié)果設(shè)置到Future中
    
  • Future的常用方法:

    • cancel(self):取消future

          def cancel(self):
              """Cancel the future if possible.
      
              Returns True if the future was cancelled, False otherwise. A future
              cannot be cancelled if it is running or has already completed.
              """
              with self._condition: #使用條件變量
                  if self._state in [RUNNING, FINISHED]: 
                      #判斷future的狀態(tài)使否是在執(zhí)行或已完成,若是則無(wú)法取消
                      return False
      
                  if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
                      return True
      
                  self._state = CANCELLED
                  self._condition.notify_all()
      
              self._invoke_callbacks()
              return True
      
    • cancelled(self):狀態(tài)判斷,是否已經(jīng)取消

    • running(self):狀態(tài)判斷,是否在執(zhí)行

    • done(self):狀態(tài)判斷,是否已完成或已取消

    • result(self, timeout=None)

      • 返回調(diào)用的結(jié)果,是有阻塞的方法,依舊是使用condition來(lái)實(shí)現(xiàn)
      • 其中調(diào)用了condition的wait()方法
    • set_result(self, result)

      • set the return value of work associated with the future

      • 調(diào)用了condition的notify_all()方法

多進(jìn)程

在windows下使用多進(jìn)程,不管是ProcessPoolExecutor還是multiprocessing,代碼要放到if __ name__ == '__ main __'模塊中,否則會(huì)報(bào)錯(cuò)

from concurrent.futures import ProcessPoolExecutor, as_completed
def fib(b):
    if n <= 2:
        return 1
    return fib(n-1)+fib(n-2)
if __name__ == '__main__':
    with ProcessPoolExecutor(3) as executor:
        all_task = [executor.submit(fib,(num)) for num in range(15,45)]
        for future in as_completed(all_task):
            data = future.result()
            print('result: {}'.format(data))
  • 在處理CPU密集型問(wèn)題時(shí),使用多進(jìn)程會(huì)比使用多線程快,但是很難做到翻倍,因?yàn)檫M(jìn)程的切換比線程的切換更耗時(shí)。

multiprocessing

import time
import multiprocessing

def get_html(n):
    time.sleep(n)
    return n

if __name__ == '__main__':
    progress = multiprocessing.Process(target=get_html, args=(2,))
    progress.start()
    progress.join()
    print(progress.pid) #打印進(jìn)程的ID

multiprocessing 的進(jìn)程池

pool = multiprocessing.Pool(multiprocessing.cpu_count())
result = pool.apply_async(get_html, args=(3,))#進(jìn)程的提交
pool.close() #在調(diào)用.join之前要先調(diào)用.close,關(guān)閉進(jìn)程池使其不再接受進(jìn)程,否則.join方法會(huì)出錯(cuò)
pool.join()
print(result.get())
  • 常用方法:

    • apply_async(self, func, args=(), kwds={}, callback=None, error_callback=None)apply()方法的異步版本

    • apply(self, fuc, args=(), kwds={}):添加進(jìn)程

    • imap(self, func, iterable, chunksize=1):對(duì)應(yīng)到之前executor里的map方法

      pool = multiprocessing.Pool(multiprocessing.cpu_count())
      for result in pool.imap(get_html, [1,5,3]):
          print('{} sleep success'.format(result))
      >>
      1 sleep success
      5 sleep success
      3 sleep success
      
    • imap_unordered(self, func, iterable, chunksize=1):先執(zhí)行完的先返回

      for result in pool.imap_unordered(get_html, [1,5,3]):
          print('{} sleep success'.format(result))
      >>
      1 sleep success
      3 sleep success
      5 sleep success
      

進(jìn)程之間的通信

  • 多進(jìn)程中的Queue與多線程中的不一樣,這里要使用來(lái)自maltiprocessing里的Queue

    • from multiprocessing import Process,Queue
      import multiprocessing
      
      def producer(queue):
          queue.put("a")
          time.sleep(2)
      
      def consumer(queue):
          time.sleep(2)
          data = queue.get()
          print(data)
      
      if __name__ == "__main__":
          queue = Queue(10)
          my_producer = Process(target=producer, args=(queue,))
          my_consumer = Process(target=consumer, args=(queue,))
          my_producer.start()
          my_consumer.start()
          my_producer.join()
          my_consumer.join()
      
  • maltiprocessing里創(chuàng)建的Queue不能用在Pool創(chuàng)建的進(jìn)程池中

    • from multiprocessing import Pool,Queue
      if __name__ == "__main__":
          queue = Queue(10)
          pool = Pool(2)
      
          pool.apply_async(producer, args=(queue,))
          pool.apply_async(consumer, args=(queue,))
      
          pool.close()
          pool.join()
      #沒(méi)有任何結(jié)果返回
      
  • 進(jìn)程池中進(jìn)程的通信需要用來(lái)自Manager里的Queue

    • from maltiprocessing import Manager
      if __name__ == "__main__":
          queue = Manager().Queue(10)
          pool = Pool(2)
      
          pool.apply_async(producer, args=(queue,))
          pool.apply_async(consumer, args=(queue,))
      
          pool.close()
          pool.join()
      
  • 使用pipe進(jìn)行通信

    • pipe只能適用于兩個(gè)進(jìn)程間的通信

    • pipe返回的是Connection的實(shí)例,Connection的常用方法有:send(self, obj)recv(self)

      • def Pipe(duplex=True):
            return Connection(), Connection()
        
        class Connection(object):
            def send(self, obj):
                pass
        
            def recv(self):
                pass
        
    • pipe的性能比queue要高

    • from maltiprocessing import Pipe
      def producer(pipe):
          pipe.send("bobby")
      
      def consumer(pipe):
          print(pipe.recv())
      
      if __name__ == "__main__":
          recevie_pipe, send_pipe=Pipe() #管道的兩端,發(fā)送端和接收端
      
          my_producer= Process(target=producer, args=(send_pipe, ))
          my_consumer = Process(target=consumer, args=(recevie_pipe,))
      
          my_producer.start()
          my_consumer.start()
          my_producer.join()
          my_consumer.join()
      
  • 共享全局變量在多進(jìn)程中是不適用的

def producer(a):
    a += 100
    time.sleep(2)
def consumer(a):
    time.sleep(2)
    print(a)
if __name__ == '__main__':
    a = 1
    my_producer = Process(target=producer,args=(a,))
    my_consumer = Process(target=consumer,args=(a,))
    my_producer.start()
    my_consumer.start()
    my_producer.join()
    my_consumer.join()
>> 1 #返回值還是1,不是101
  • 進(jìn)程間的同步(維護(hù)一個(gè)公共的內(nèi)存/變量)使用Manager()

    • def Manager():
          return multiprocessing.SyncManager()
      
      class SyncManager(multiprocessing.managers.BaseManager):
          def Condition(self, lock=None):
              return threading.Condition(lock)
      
          def Event(self):
              return threading.Event()
      
          def Lock(self):
              return threading.Lock()
      
          def Namespace(self):
              pass
      
          def Queue(self, maxsize=None):
              return queue.Queue()
      
          def RLock(self):
              return threading.RLock()
      
          def Semaphore(self, value=None):
              return threading.Semaphore(value)
      
          def Array(self, typecode, sequence):
              pass
      
          def Value(self, typecode, value):
              pass
      
          def dict(self, mapping_or_sequence):
              pass
      
          def list(self, sequence):
              pass
      
    • def add_data(p_dict, key, value):
          p_dict[key] = value
          
      if __name__ == '__main__':
          progress_dict = Manager().dict()
          
          first = Process(target=add_data, args=(progress_dict, 'Bobby1',22))
          second = Process(target=add_data, args=(progress_dict, 'Bobby2',23))
          
          first.start()
          second.start()
          first.join()
          second.join()
          
          print(progress_dict)
      >> {'bobby1': 22, 'bobby2': 23}
      
    • 使用時(shí)注意數(shù)據(jù)的同步,必要時(shí)需要加鎖

文檔閱讀

Thread Objects

  • Thread類(lèi):在單獨(dú)的控制線程中運(yùn)行的活動(dòng)(activity),有兩種指定活動(dòng)的方法:
    • 將可調(diào)用對(duì)象傳遞給構(gòu)造器
    • 在子類(lèi)中重寫(xiě) run() 方法
  • 通過(guò)調(diào)用thread對(duì)象的start()方法來(lái)激活對(duì)象,is_alive()方法用來(lái)判斷線程是否處在激活狀態(tài)。
  • 其他線程可以調(diào)用join()方法來(lái)阻塞正在調(diào)用的線程,直到調(diào)用join()方法的線程結(jié)束。
  • threading.Thread(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)
    • target:傳入被run()方法調(diào)用的可調(diào)用對(duì)象
    • name:線程的名字
    • args:目標(biāo)調(diào)用的參數(shù)的元組
    • kwargs:目標(biāo)調(diào)用的參數(shù)的字典

Lock Objects

  • threading.Lock() 生成一個(gè)鎖對(duì)象。一旦某個(gè)線程獲取了鎖,之后的所有獲取鎖的請(qǐng)求都會(huì)被阻塞,直到鎖被釋放。任何線程都可以釋放鎖,不一定是獲得鎖的線程。

RLock Objects

  • 可重入鎖允許同一線程多次獲取同一把鎖。

Event Objects

  • 一個(gè)事件對(duì)象管理線程內(nèi)的一個(gè)內(nèi)部標(biāo)記,可以使用set()方法將標(biāo)記設(shè)置為true,使用clear()方法將標(biāo)記重置為false。標(biāo)記為false時(shí),wait()方法被阻塞,直到標(biāo)記被設(shè)定為true。
  • threading.Event()構(gòu)造一個(gè)事件對(duì)象,內(nèi)部標(biāo)記初始化為false
  • is_set():檢查內(nèi)部標(biāo)記的狀態(tài),只有在狀態(tài)為為true時(shí)返回true。
  • set():將內(nèi)部標(biāo)記設(shè)定為true。完成設(shè)定后,所有調(diào)用了wait()方法的線程都將不再受到阻塞。
  • clear():將內(nèi)部標(biāo)記重置為false
  • wait(timeout=None):阻塞進(jìn)程直到內(nèi)部標(biāo)記設(shè)定為true或滿足選定的timeout。
    • timeout:a floating point number,以秒為單位設(shè)定超時(shí)時(shí)間

queue

  • 同步隊(duì)列:隊(duì)列模塊實(shí)現(xiàn)了多生產(chǎn)者、多消費(fèi)者隊(duì)列。當(dāng)必須在多個(gè)線程之間安全地交換信息時(shí),它在線程編程中特別有用。
  • 該模塊實(shí)現(xiàn)了三種隊(duì)列:
    • FIFO隊(duì)列:先添加的任務(wù)先讀取(先入先出)
    • LIFO隊(duì)列:后添加的任務(wù)先讀取(后入先出)
    • 優(yōu)先隊(duì)列:輸入的數(shù)據(jù)排序,值最小的數(shù)據(jù)先讀取
  • 類(lèi):
    • Queue(maxsize=0):FIFO隊(duì)列的構(gòu)造函數(shù)
    • LifoQueue(maxsize=0):LIFO隊(duì)列的構(gòu)造函數(shù)
    • PriorityQueue(maxsize=0):優(yōu)先隊(duì)列的構(gòu)造函數(shù)
    • maxsize參數(shù)用于設(shè)定可以放入隊(duì)列的項(xiàng)目數(shù)的上限。若maxsize小于或等于零,則隊(duì)列大小為無(wú)限大。

Queue Objects

  • Queue.qsize():返回隊(duì)列的大小
  • Queue.empty():檢查隊(duì)列是否為空,是則返回True
  • Queue.full():檢查隊(duì)列是否為滿,是則返回True
  • Queue.put(item, block=True, timeout=None):將元素放入隊(duì)列
    • block=True, timeout=None:在必要時(shí)阻塞,當(dāng)隊(duì)列有空位時(shí)執(zhí)行插入
    • timeout為正數(shù):阻塞最多timeout秒,若隊(duì)列還是沒(méi)有空位則拋出Full異常
    • block=False:在有空位時(shí)直接插入,否則拋出異常
  • Queue.get(block=True,timeout=None):將元素從隊(duì)列中移除并返回
    (未完)
最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
平臺(tái)聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡(jiǎn)書(shū)系信息發(fā)布平臺(tái),僅提供信息存儲(chǔ)服務(wù)。

推薦閱讀更多精彩內(nèi)容