簡(jiǎn)介
jsoup is an open-source Java library of methods designed to extract and manipulate data stored in HTML documents.
Jsoup就是一個(gè)用java語言編寫用來獲取和操控html頁面及元素的開源Jar包。
我們使用的一般是1.7的版本,博主用的是jsoup-1.7.2.jar
文檔
jsoup有官方的中文使用文檔,比較簡(jiǎn)單jsoup Cookbook(中文版)
用法
一小段代碼爬取我的博客文章標(biāo)題列表,帶您簡(jiǎn)單了解它的用法
選擇器截圖:
package com.blog.jsoup;
import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
public class BlogSpider {
public static void main(String[] args) {
try {
Document doc = Jsoup.connect("http://www.lxweimin.com/users/7a8e041a132c/latest_articles").get();
//通過選擇器獲取文章列表,選擇器dom元素直接通過Chrome開發(fā)者工具檢查元素Copy -> Copy Selector即可獲取
Elements articleList = doc.select("#list-container > ul > li > div > h4 > a");
for (int i = 0; i < articleList.size(); i++) {
System.out.println(articleList.get(i).text());
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
運(yùn)行結(jié)果如下:
細(xì)心的讀者可能會(huì)發(fā)現(xiàn)進(jìn)入我的博客主頁之后最新文章列表不止爬取顯示的九篇,而是會(huì)不斷加載更多,這是因?yàn)樵L問這個(gè)頁面時(shí)生成的靜態(tài)頁面只請(qǐng)求了第一頁的數(shù)據(jù),而隨著用戶下拉,異步加載了更多的數(shù)據(jù),這個(gè)是在瀏覽器生成靜態(tài)頁面之后做的,所以在這里沒有辦法拿到。這說明了很重要的一點(diǎn),Jsoup訪問頁面所拿到的html是我們傳統(tǒng)意義上查看網(wǎng)頁源代碼的數(shù)據(jù),而不是你在審查元素的那些數(shù)據(jù),所以,想要拿到異步加載的數(shù)據(jù)需要去解析異步訪問的js,然后通過抓包來查看訪問的servlet看看能不能模擬訪問拿到更多數(shù)據(jù)。
訪問https頁面
由于https的證書加密了,直接通過上述方法去訪問并提取html頁面也會(huì)報(bào)錯(cuò)
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching m.v.qq.com found.
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1747)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1209)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:135)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:943)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:476)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:133)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:439)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:167)
at com.blog.jsoup.BlogSpider.main(BlogSpider.java:13)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching m.v.qq.com found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:193)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:77)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:264)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:250)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1188)
... 15 more
好在java有提供相應(yīng)的方法去訪問:
為了閱讀方便,貼出整個(gè)class
package com.blog.jsoup;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.security.SecureRandom;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import javax.security.cert.CertificateException;
import javax.security.cert.X509Certificate;
import org.apache.http.conn.ssl.AllowAllHostnameVerifier;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
public class BlogSpider {
public static void main(String[] args) {
try {
Document doc = Jsoup.connect("http://www.lxweimin.com/users/7a8e041a132c/latest_articles").get();
// 通過選擇器獲取文章列表,選擇器dom元素直接通過Chrome開發(fā)者工具檢查元素Copy -> Copy Selector即可獲取
Elements articleList = doc.select("#list-container > ul > li > div > h4 > a");
for (int i = 0; i < articleList.size(); i++) {
System.out.println(articleList.get(i).text());
}
} catch (IOException e) {
e.printStackTrace();
}
try {
SSLSocketFactory.getSocketFactory().setHostnameVerifier(new AllowAllHostnameVerifier());
String https = "https://m.v.qq.com/x/page/xxxxxxxxx.html";
// Document doc = Jsoup.connect(https).get();
init();
HttpsURLConnection conn = (HttpsURLConnection) new URL(https).openConnection();
conn.setDoOutput(true);
conn.setDoInput(true);
conn.connect();
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
StringBuffer sb = new StringBuffer();
String line;
while ((line = br.readLine()) != null)
sb.append(line);
Document doc = Jsoup.parse(sb.toString());
} catch (Exception e) {
e.printStackTrace();
}
}
public static void init() {
try {
SSLContext sc = SSLContext.getInstance("TLS");
sc.init(null, new TrustManager[] { new X509TrustManager() {
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
}
public void checkServerTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
}
@Override
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
@Override
public void checkClientTrusted(java.security.cert.X509Certificate[] arg0, String arg1)
throws java.security.cert.CertificateException {
// TODO Auto-generated method stub
}
@Override
public void checkServerTrusted(java.security.cert.X509Certificate[] arg0, String arg1)
throws java.security.cert.CertificateException {
// TODO Auto-generated method stub
}
} }, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
@Override
public boolean verify(String arg0, SSLSession arg1) {
return true;
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
}