Xc.Li
Apr.2016
Python部分
應(yīng)用Python的BeautifulSoup庫和RegEx獲取圖書館網(wǎng)頁上的座位信息。連續(xù)多天在相同時間獲取數(shù)據(jù)，便于橫向比較。
簡單來說就是在每天7：00~23：00之間在每個小時的00，10，20，30，40，50分鐘獲取圖書館當(dāng)時所有樓層的上座情況，然后保存到一個文件里。
獲取數(shù)據(jù)的設(shè)計思路參考(這個是上學(xué)期編的程序，略有不同)：用python自動獲取圖書館空座信息
獲得的數(shù)據(jù)保存為csv格式，數(shù)據(jù)格式：
date.csv
'time', dec_time,一樓在用，一樓空閑，二樓在用，二樓空閑...六樓在用，六樓空閑
例如：
"D:\OneDrive\Python\data\lib\Apr13.csv"
'22:30:07', 22.5, 19, 17, 150, 172, 203, 258, 153, 219, 135, 214, 34, 22
Stata部分
csv文件導(dǎo)入到Stata：
可以生存一張描繪所有樓層各個時間的圖像，生成dta文件，便于進(jìn)一步處理
Apr13.png
change-format.do
clear
local date Apr13
local day Apr13_Wed3
infile str8 `day'time dec_time `day'l1a `day'l1b `day'l2a `day'l2b `day'l3a `day'l3b `day'l4a `day'l4b `day'l5a `day'l5b `day'l6a `day'l6b using D:\OneDrive\Python\data\lib\\`date'.csv
graph twoway line `day'l1a dec_time ||line `day'l2a dec_time || line `day'l3a dec_time || line `day'l4a dec_time || line `day'l5a dec_time || line `day'l6a dec_time, xlabel(7(1)23) ylabel(0(30)350)
save "D:\lib-data\\`date'.dta"
相關(guān)的python程序：
kol模塊：自動切換Wifi網(wǎng)絡(luò)保證在線
timer模塊：每五分鐘運行一次命令的實現(xiàn)
writocsv模塊：見下文
主程序
# get library seats
# Designed By Xc.Li
import urllib
import BeautifulSoup
import re
import time
from os import system
import kol
import writetocsv
import timer
def find_number(keyword):
for tag in tags:
s_tag = str(tag)
flag = re.match(keyword, s_tag)
if flag is not None:
return s_tag[54:-18]
while True:
timer.func(10)
kol.func()
url = 'http://lib.ecust.edu.cn:8081/gateseat/lrp.aspx'
html = urllib.urlopen(url).read()
soup = BeautifulSoup.BeautifulSoup(html)
tags = soup('span')
output = list()
output.append(time.ctime()[11:-5])
# time in decimal
hour = int(time.ctime()[11:13])
minute = int(time.ctime()[14:16])
decimal_time = round(hour + minute/60.0, 2)
output.append(decimal_time)
for floor in [1, 2, 3, 4, 5, 6]:
# print 'Floor:', floor
keyword = '^<span id="Label' + str(floor) + 'f1' + '".*>'
# print 'Used:', find_number(keyword)
output.append(int(find_number(keyword)))
keyword = '^<span id="Label' + str(floor) + 'f2' + '".*>'
# print 'Left:', find_number(keyword)
output.append(int(find_number(keyword)))
print output
str_output = str(output)[1:-1]
writetocsv.write('lib', 'default', str_output)
time.sleep(60)
system('cls')
# auto-quit
if hour == 22 and minute > 29:
print "program ends!"
writetocsv.write('lib', 'default', "program end at:%s" % time.ctime())
time.sleep(5)
break
writetocsv模塊
# Write to CSV
# Designed By Xc.Li @ Mar.2016
import time
import os
def write(dirname, filename, content):
if filename == "default":
filename = time.ctime()[4:7]+time.ctime()[8:10]
file_name = "D:\OneDrive\Python\\data\%s\%s.csv" % (dirname, filename)
dir_name = "D:\OneDrive\Python\\data\%s" % dirname
try:
tf = open(file_name, 'a')
except:
os.system("mkdir "+dir_name)
tf = open(file_name, 'a')
tf.write(content)
tf.write('\n')
tf.close()