Comprehensions and Generators（生成器）
- Item27: 用Comprehensions而不是map和filter
當我們想要計算數組的元素平方時
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
squares = []
for x in a:
squares.append(x**2)
print(squares)
>>>
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
可以用Comprehension實現一樣的結果
squares = [x**2 for x in a] # List comprehension
print(squares)
>>>
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
除非應用單參數的函數，列表comprehension比map更加清晰。
map需要構建lambda函數，視覺上復雜。
alt = map(lambda x: x ** 2, a)
當然也可以增加限定條件
even_squares = [x**2 for x in a if x % 2 == 0]
print(even_squares)
>>>
[4, 16, 36, 64, 100]
map配合filter也是可以的。
alt = map(lambda x: x**2, filter(lambda x: x % 2 == 0, a))
assert even_squares == list(alt)
字典和集合也是可以使用comprehension的。
even_squares_dict = {x: x**2 for x in a if x % 2 == 0}
threes_cubed_set = {x**3 for x in a if x % 3 == 0}
print(even_squares_dict)
print(threes_cubed_set)
>>>
{2: 4, 4: 16, 6: 36, 8: 64, 10: 100}
{216, 729, 27}
用dict和set來配合map和filter進行構建也是可以的，但是語句偏長，需要換行，顯得不優雅。
alt_dict = dict(map(lambda x: (x, x**2),
filter(lambda x: x % 2 == 0, a)))
alt_set = set(map(lambda x: x**3,
filter(lambda x: x % 3 == 0, a)))
- Item28: 避免在Comprehensions里面使用超過兩個的控制表達式
Comprehension支持多層的循環。例如一個簡單的矩陣，轉換成列表。
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = [x for row in matrix for x in row]
print(flat)
>>>
[1, 2, 3, 4, 5, 6, 7, 8, 9]
只要添加額外的[]字符，也可以構建出對應的二維矩陣：
squared = [[x**2 for x in row] for row in matrix]
print(squared)
>>>
[[1, 4, 9], [16, 25, 36], [49, 64, 81]]
如果有三層或更多層的嵌套，則不適合用Comprehension了。
my_lists = [
[[1, 2, 3], [4, 5, 6]],
...
]
flat = [x for sublist1 in my_lists
for sublist2 in sublist1
for x in sublist2]
此時，使用普通的for-loop反倒更清晰：
flat = []
for sublist1 in my_lists:
for sublist2 in sublist1:
flat.extend(sublist2)
Comprehension可以有多個判斷條件
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b = [x for x in a if x > 4 if x % 2 == 0]
c = [x for x in a if x > 4 and x % 2 == 0]
超過兩個控制語句就盡量不要用Comprehension了。以下反例就是：找到行相加不小于10且行中整除3的元素。
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
filtered = [[x for x in row if x % 3 == 0]
for row in matrix if sum(row) >= 10]
print(filtered)
>>>
[[6], [9]]
- Item29: 通過使用賦值表達式，避免Comprehensions里的重復工作
以下是一個簡單的例子：零件工廠供貨。(8個零件為一批，最后得到各個零件的批次數)
stock = {
'nails': 125,
'screws': 35,
'wingnuts': 8,
'washers': 24,
}
order = ['screws', 'wingnuts', 'clips']
def get_batches(count, size):
return count // size
result = {}
for name in order:
count = stock.get(name, 0)
batches = get_batches(count, 8)
if batches:
result[name] = batches
print(result)
>>>
{'screws': 4, 'wingnuts': 1}
可以通過Compresion的方式來緊湊這個表達，但是存在重復的計算。
found = {name: get_batches(stock.get(name, 0), 8)
for name in order
if get_batches(stock.get(name, 0), 8)}
print(found)
>>>
{'screws': 4, 'wingnuts': 1}
如果其中一個調用出錯，則整體出錯。（前后一致性差）
has_bug = {name: get_batches(stock.get(name, 0), 4)
for name in order
if get_batches(stock.get(name, 0), 8)}
print('Expected:', found)
print('Found: ', has_bug)
>>>
Expected: {'screws': 4, 'wingnuts': 1}
Found: {'screws': 8, 'wingnuts': 2}
Python3.8之后的海象賦值操作可以緩解：
found = {name: batches for name in order
if (batches := get_batches(stock.get(name, 0), 8))}
:=的順序要用對，不然會報錯：
result = {name: (tenth := count // 10)
for name, count in stock.items() if tenth > 0}
>>>
Traceback ...
NameError: name 'tenth' is not defined
result = {name: tenth for name, count in stock.items()
if (tenth := count // 10) > 0}
print(result)
>>>
{'nails': 12, 'screws': 3, 'washers': 2}
:=可能會泄露變量到外面的作用域。
half = [(last := count // 2) for count in
stock.values()]
print(f'Last item of {half} is {last}')
>>>
Last item of [62, 17, 4, 12] is 12
當然，for-loop也會
for count in stock.values(): # Leaks loop variable
pass
print(f'Last item of {list(stock.values())} is {count}')
>>>
Last item of [125, 35, 8, 24] is 24
Comprehension就不會。
half = [count // 2 for count in stock.values()]
print(half) # Works
print(count) # Exception because loop variable didn't leak
>>>
[62, 17, 4, 12]
Traceback ...
NameError: name 'count' is not defined
found = ((name, batches) for name in order
if (batches := get_batches(stock.get(name, 0), 8)))
print(next(found))
print(next(found))
>>>
('screws', 4)
('wingnuts', 1)
- Item30: 考慮使用生成器（Generators）而不是返回列表
找到下一個單詞的下標：
def index_words(text):
result = []
if text:
result.append(0)
for index, letter in enumerate(text):
if letter == ' ':
result.append(index + 1)
return result
address = 'Four score and seven years ago...'
result = index_words(address)
print(result[:10])
>>>
[0, 5, 11, 15, 21, 27, 31, 35, 43, 51]
存在兩個問題：1、代碼稍顯冗余
可以用generator來代替：
def index_words_iter(text):
if text:
yield 0
for index, letter in enumerate(text):
if letter == ' ':
yield index + 1
next可以取到下一個數值：
it = index_words_iter(address)
print(next(it))
print(next(it))
>>>
0
5
或者用list取到所有數值：
result = list(index_words_iter(address))
print(result[:10])
>>>
[0, 5, 11, 15, 21, 27, 31, 35, 43, 51]
2、可能由于過大的list導致內存不足，而generator一次一行地yield，則好一些：
def index_file(handle):
offset = 0
for line in handle:
if line:
yield offset
for letter in line:
offset += 1
if letter == ' ':
yield offset
讀取文件，利用itertools.islice獲得最后的10個結果。
with open('address.txt', 'r') as f:
it = index_file(f)
results = itertools.islice(it, 0, 10)
print(list(results))
>>>
[0, 5, 11, 15, 21, 27, 31, 35, 43, 51]
- Item31: 當迭代作為參數的時候，要防備bug發生
與list有關的應用，通常都需要多次迭代列表，比如統計數字的占比：
def normalize(numbers):
total = sum(numbers)
result = []
for value in numbers:
percent = 100 * value / total
result.append(percent)
return result
visits = [15, 35, 80]
percentages = normalize(visits)
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
比如每次一行yield這些數字，然后交給函數normalize，結果發現是空list：
def read_visits(data_path):
with open(data_path) as f:
for line in f:
yield int(line)
it = read_visits('my_numbers.txt')
percentages = normalize(it)
print(percentages)
>>>
[]
因為sum的時候已經消耗了一次，再次迭代的時候，已經是空列表了：
it = read_visits('my_numbers.txt')
print(list(it))
print(list(it)) # Already exhausted
>>>
[15, 35, 80]
[]
解決方案是：自己先手動備份一次list下來，然后用這個list，確保迭代不會遇到這個問題：
def normalize_copy(numbers):
numbers_copy = list(numbers) # Copy the iterator
total = sum(numbers_copy)
result = []
for value in numbers_copy:
percent = 100 * value / total
result.append(percent)
return result
it = read_visits('my_numbers.txt')
percentages = normalize_copy(it)
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
但是這個解決方案，可能會導致生成的list過大，導致內存不足。另一種解決方案是每次都生成一個新的迭代器：
def normalize_func(get_iter):
total = sum(get_iter()) # New iterator
result = []
for value in get_iter(): # New iterator
percent = 100 * value / total
result.append(percent)
return result
path = 'my_numbers.txt'
percentages = normalize_func(lambda: read_visits(path))
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
當Python看到比如for x in foo的語句,它實際上調用iter(foo).這個iter又調用了foo.__iter__。這個__iter__ 方法應該返回一個迭代器對象(實現了__next__ 方法)。然后循環地調用這個方法直到(拋出StopIteration異常)。
聽著很復雜，實現一個：
class ReadVisits:
def __init__(self, data_path):
self.data_path = data_path
def __iter__(self):
with open(self.data_path) as f:
for line in f:
yield int(line)
這樣可以確保每次調用ReadVisits.__iter__都會得到一個新的迭代器
visits = ReadVisits(path)
percentages = normalize(visits)
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
通過iter獲得迭代器，如果是本身的話，說明傳遞了一個迭代器進來，直接拋出TypeError：
def normalize_defensive(numbers):
if iter(numbers) is numbers: # An iterator -- bad!
raise TypeError('Must supply a container')
total = sum(numbers)
result = []
for value in numbers:
percent = 100 * value / total
result.append(percent)
return result
也可以通過collections.abc來判斷是不是迭代器：
from collections.abc import Iterator
def normalize_defensive(numbers):
if isinstance(numbers, Iterator): # Another way to check
raise TypeError('Must supply a container')
total = sum(numbers)
result = []
for value in numbers:
percent = 100 * value / total
result.append(percent)
return result
visits = [15, 35, 80]
percentages = normalize_defensive(visits)
assert sum(percentages) == 100.0
visits = ReadVisits(path)
percentages = normalize_defensive(visits)
assert sum(percentages) == 100.0
visits = [15, 35, 80]
it = iter(visits)
normalize_defensive(it)
>>>
Traceback ...
TypeError: Must supply a container
- Item32: 使用大的列表Comprohensions的時候，考慮采用生成器表達式
為了解決這個問題，比如處理文件時：
value = [len(x) for x in open('my_file.txt')]
print(value)
>>>
[100, 57, 15, 1, 12, 75, 5, 86, 89, 11]
可以使用generator表達式，通過next來訪問：
it = (len(x) for x in open('my_file.txt'))
print(it)
>>>
<generator object <genexpr> at 0x108993dd0>
print(next(it))
print(next(it))
>>>
100
57
另一個強力的輸出是，generator可以組合使用：
roots = ((x, x**0.5) for x in it)
print(next(roots))
>>>
(15, 3.872983346207417)
- Item33: 用yield from來組合多個生成器
比如需要生成動畫：
def move(period, speed):
for _ in range(period):
yield speed
def pause(delay):
for _ in range(delay):
yield 0
def animate():
for delta in move(4, 5.0):
yield delta
for delta in pause(3):
yield delta
for delta in move(2, 3.0):
yield delta
其中，以5的速度動4秒，停3秒，以3的速度動2秒。然后下面進行渲染：
def render(delta):
print(f'Delta: {delta:.1f}')
# Move the images onscreen
...
def run(func):
for delta in func():
render(delta)
run(animate)
>>>
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 0.0
Delta: 0.0
Delta: 0.0
Delta: 3.0
Delta: 3.0
但是，animate顯得冗余，可以用下面的yield from來組合動畫：
def animate_composed():
yield from move(4, 5.0)
yield from pause(3)
yield from move(2, 3.0)
run(animate_composed)
>>>
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 0.0
Delta: 0.0
Delta: 0.0
Delta: 3.0
Delta: 3.0
import timeit
def child():
for i in range(1_000_000):
yield i
def slow():
for i in child():
yield i
def fast():
yield from child()
baseline = timeit.timeit(
stmt='for _ in slow(): pass',
globals=globals(),
number=50)
print(f'Manual nesting {baseline:.2f}s')
comparison = timeit.timeit(
stmt='for _ in fast(): pass',
globals=globals(),
number=50)
print(f'Composed nesting {comparison:.2f}s')
reduction = -(comparison - baseline) / baseline
print(f'{reduction:.1%} less time')
>>>
Manual nesting 4.02s
Composed nesting 3.47s
13.5% less time
yield from當成for-yield來用，性能有提升。
- Item34: 避免用send插入數據到生成器
加入現在有一個生成sin波的函數：
import math
def wave(amplitude, steps):
step_size = 2 * math.pi / steps
for step in range(steps):
radians = step * step_size
fraction = math.sin(radians)
output = amplitude * fraction
yield output
還有發射函數：
def transmit(output):
if output is None:
print(f'Output is None')
else:
print(f'Output: {output:>5.1f}')
def run(it):
for output in it:
transmit(output)
run(wave(3.0, 8))
>>>
Output: 0.0
Output: 2.1
Output: 3.0
Output: 2.1
Output: 0.0
Output: -2.1
Output: -3.0
Output: -2.1
產生正常的sin波可以，但是如果想在中間調整振幅來達到AM波就不行。
def my_generator():
received = yield 1
print(f'received = {received}')
it = iter(my_generator())
output = next(it) # Get first generator output
print(f'output = {output}')
try:
next(it) # Run generator until it exits
except StopIteration:
pass
>>>
output = 1
received = None
send方法提供了一種雙向通信的方式
it = iter(my_generator())
output = it.send(None) # Get first generator output
print(f'output = {output}')
try:
it.send('hello!') # Send value into the generator
except StopIteration:
pass
>>>
output = 1
received = hello!
通過在yield之后接收一個數值：
def wave_modulating(steps):
step_size = 2 * math.pi / steps
amplitude = yield # Receive initial amplitude
for step in range(steps):
radians = step * step_size
fraction = math.sin(radians)
output = amplitude * fraction
amplitude = yield output # Receive next amplitude
def run_modulating(it):
amplitudes = [
None, 7, 7, 7, 2, 2, 2, 2, 10, 10, 10, 10, 10]
for amplitude in amplitudes:
output = it.send(amplitude)
transmit(output)
run_modulating(wave_modulating(12))
>>>
Output is None
Output: 0.0
Output: 3.5
Output: 6.1
Output: 2.0
Output: 1.7
Output: 1.0
Output: 0.0
Output: -5.0
Output: -8.7
Output: -10.0
Output: -8.7
Output: -5.0
當有多個波的時候，用yield from組合是部分可行的。
def complex_wave():
yield from wave(7.0, 3)
yield from wave(2.0, 4)
yield from wave(10.0, 5)
run(complex_wave())
>>>
Output: 0.0
Output: 6.1
Output: -6.1
Output: 0.0
Output: 2.0
Output: 0.0
Output: -2.0
Output: 0.0
Output: 9.5
Output: 5.9
Output: -5.9
Output: -9.5
但是如果yield from和send組合在一塊，就會顯得十分復雜，不適合閱讀。而且需要修改大量的代碼。
def complex_wave_modulating():
yield from wave_modulating(3)
yield from wave_modulating(4)
yield from wave_modulating(5)
run_modulating(complex_wave_modulating())
>>>
Output is None
Output: 0.0
Output: 6.1
Output: -6.1
Output is None
Output: 0.0
Output: 2.0
Output: 0.0
Output: -10.0
Output is None
Output: 0.0
Output: 9.5
Output: 5.9
解決方案是：從wave生成的部分進行修改：
def wave_cascading(amplitude_it, steps):
step_size = 2 * math.pi / steps
for step in range(steps):
radians = step * step_size
fraction = math.sin(radians)
amplitude = next(amplitude_it) # Get next input
output = amplitude * fraction
yield output
再和yield from進行組合，就不會有問題：
def complex_wave_cascading(amplitude_it):
yield from wave_cascading(amplitude_it, 3)
yield from wave_cascading(amplitude_it, 4)
yield from wave_cascading(amplitude_it, 5)
def run_cascading():
amplitudes = [7, 7, 7, 2, 2, 2, 2, 10, 10, 10, 10, 10]
it = complex_wave_cascading(iter(amplitudes))
for amplitude in amplitudes:
output = next(it)
transmit(output)
run_cascading()
>>>
Output: 0.0
Output: 6.1
Output: -6.1
Output: 0.0
Output: 2.0
Output: 0.0
Output: -2.0
Output: 0.0
Output: 9.5
Output: 5.9
Output: -5.9
Output: -9.5
唯一的缺點是此代碼的輸入生成器假定是完全線程安全的，但情況可能并非如此。如果需要跨越線程邊界，異步函數可能更合適。
- Item35: 在生成器中，避免使用throw來造成狀態轉換（State Transitions）
當調用throw方法時，下一次出現的 yield 表達式會在接收到其輸出后重新引發提供的 Exception 實例，而不是正常繼續。看簡單實例：
class MyError(Exception):
pass
def my_generator():
yield 1
yield 2
yield 3
it = my_generator()
print(next(it)) # Yield 1
print(next(it)) # Yield 2
print(it.throw(MyError('test error')))
>>>
1
2
Traceback ...
MyError: test error
當你調用 throw 時，生成器函數可能會使用標準的 try/except 復合語句捕獲注入的異常，該復合語句圍繞最后執行的 yield 表達式。
def my_generator():
yield 1
try:
yield 2
except MyError:
print('Got MyError!')
else:
yield 3
yield 4
it = my_generator()
print(next(it)) # Yield 1
print(next(it)) # Yield 2
print(it.throw(MyError('test error')))
>>>
1
2
Got MyError!
4
可以利用throw這一點來控制計時器的reset
class Reset(Exception):
pass
def timer(period):
current = period
while current:
current -= 1
try:
yield current
except Reset:
current = period
def check_for_reset():
# Poll for external event
...
def announce(remaining):
print(f'{remaining} ticks remaining')
def run():
it = timer(4)
while True:
try:
if check_for_reset():
current = it.throw(Reset())
else:
current = next(it)
except StopIteration:
break
else:
announce(current)
run()
>>>
3 ticks remaining
2 ticks remaining
1 ticks remaining
3 ticks remaining
2 ticks remaining
3 ticks remaining
2 ticks remaining
1 ticks remaining
0 ticks remaining
上面的代碼還是稍顯復雜，一個更簡單的方式是實現狀態閉包：
class Timer:
def __init__(self, period):
self.current = period
self.period = period
def reset(self):
self.current = self.period
def __iter__(self):
while self.current:
self.current -= 1
yield self.current
代碼更簡單易懂，盡可能不用throw方法。
def run():
timer = Timer(4)
for current in timer:
if check_for_reset():
timer.reset()
announce(current)
run()
>>>
3 ticks remaining
2 ticks remaining
1 ticks remaining
3 ticks remaining
2 ticks remaining
3 ticks remaining
2 ticks remaining
1 ticks remaining
0 ticks remaining
- Item36: 考慮用itertools來配合迭代器和生成器工作
導入包
import itertools
主要這個庫包含三大點：
- 連接迭代器們
chain：連接多個迭代器：
it = itertools.chain([1, 2, 3], [4, 5, 6])
print(list(it))
>>>
[1, 2, 3, 4, 5, 6]
用repeat來輸出單值：
it = itertools.repeat('hello', 3)
print(list(it))
>>>
['hello', 'hello', 'hello']
用cycle來重復迭代項目：
it = itertools.cycle([1, 2])
result = [next(it) for _ in range (10)]
print(result)
>>>
[1, 2, 1, 2, 1, 2, 1, 2, 1, 2]
用tee來拆分單個迭代器到多個并行迭代器。（如果迭代器沒有以相同的速度前進，則此函數的內存使用量將增加，因為將需要緩沖來排隊待處理的項目。）
it1, it2, it3 = itertools.tee(['first', 'second'], 3)
print(list(it1))
print(list(it2))
print(list(it3))
>>>
['first', 'second']
['first', 'second']
['first', 'second']
zip_longest來處理不等長的zip
keys = ['one', 'two', 'three']
values = [1, 2]
normal = list(zip(keys, values))
print('zip: ', normal)
it = itertools.zip_longest(keys, values, fillvalue='nope')
longest = list(it)
print('zip_longest:', longest)
>>>
zip: [('one', 1), ('two', 2)]
zip_longest: [('one', 1), ('two', 2), ('three', 'nope')]
- 過濾項目
islice：指定end或者start和end或者start,end,step_sizes來切分（不復制）。
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
first_five = itertools.islice(values, 5)
print('First five: ', list(first_five))
middle_odds = itertools.islice(values, 2, 8, 2)
print('Middle odds:', list(middle_odds))
>>>
First five: [1, 2, 3, 4, 5]
Middle odds: [3, 5, 7]
takewhile，直到函數返回False才停止
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
less_than_seven = lambda x: x < 7
it = itertools.takewhile(less_than_seven, values)
print(list(it))
>>>
[1, 2, 3, 4, 5, 6]
dropwhile：是takewhile的相反，跳過直到返回True
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
less_than_seven = lambda x: x < 7
it = itertools.dropwhile(less_than_seven, values)
print(list(it))
>>>
[7, 8, 9, 10]
filterfalse：filter的相反，返回所有為False的項目
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
evens = lambda x: x % 2 == 0
filter_result = filter(evens, values)
print('Filter: ', list(filter_result))
filter_false_result = itertools.filterfalse(evens, values)
print('Filter false:', list(filter_false_result))
>>>
Filter: [2, 4, 6, 8, 10]
Filter false: [1, 3, 5, 7, 9]
- 產生組合
accumulate：和reduce功能相似，如果不指定參數，就是sum。
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sum_reduce = itertools.accumulate(values)
print('Sum: ', list(sum_reduce))
def sum_modulo_20(first, second):
output = first + second
return output % 20
modulo_reduce = itertools.accumulate(values, sum_modulo_20)
print('Modulo:', list(modulo_reduce))
>>>
Sum: [1, 3, 6, 10, 15, 21, 28, 36, 45, 55]
Modulo: [1, 3, 6, 10, 15, 1, 8, 16, 5, 15]
product：產生笛卡爾積
single = itertools.product([1, 2], repeat=2)
print('Single: ', list(single))
multiple = itertools.product([1, 2], ['a', 'b'])
print('Multiple:', list(multiple))
>>>
Single: [(1, 1), (1, 2), (2, 1), (2, 2)]
Multiple: [(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')]
permutations：產生排列
it = itertools.permutations([1, 2, 3, 4], 2)
print(list(it))
>>>
[(1, 2),
(1, 3),
(1, 4),
(2, 1),
(2, 3),
(2, 4),
(3, 1),
(3, 2),
(3, 4),
(4, 1),
(4, 2),
(4, 3)]
combinations：產生組合
it = itertools.combinations([1, 2, 3, 4], 2)
print(list(it))
>>>
[(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
combinations_with_replacement：產生有放回的組合。（單個元素無限次取出）
it = itertools.combinations_with_replacement([1, 2, 3, 4], 2)
print(list(it))
>>>
[(1, 1),
(1, 2),
(1, 3),
(1, 4),
(2, 2),
(2, 3),
(2, 4),
(3, 3),
(3, 4),
(4, 4)]
