您当前的位置：首页 > Python

Python 从零开始爬虫(二)：BeautifulSoup解析网页

嗨学编程发布时间：2019-04-20 14:53:23 ，浏览量：4

学了requests，了解了伪装技巧后，终于能爬到些比较正常的网页源码（html文档）了，但这离结果还差最后和是最重要的一步——筛选。这个过程就像在泥沙中淘金一样，没有合适的筛子，你就会把有价值的漏掉了，或者做了无用功把没用的也筛了出来。淘金者看土质，做筛子。对应到爬虫领域就是观察html，定制筛选器。

稍稍了解HTML

信息都在网页源码里，浏览器通过解析源码来加载我们所看到的东西，那我们是不是也应该学下如何看源码呢？——是的

但不要方，这不是html语法课，做爬虫的，只需了解一下html的原理和标签关系就行了，这跟认亲戚一样简单，你会看家族树的话根本不成问题。





    
      Harry Potter
      J K. Rowling 
      2005
      29.99
    




有时候会堆成一行，影响观察但不影响使用，丢到排版器排一下就好了
Harry PotterJ K. Rowling


    
        
            
                
                
                    嗨学编程
                    暂无认证
                
            
            
                
                    
                        4浏览
                        0关注
                        1306博文
                        0收益
                    

                    
                        0浏览
                        0点赞
                        0打赏
                        0留言
                    
                
            
            
                私信
                关注
            

        
        
            热门博文
            
                Python：用tkinter制做一个音乐下载小软件
Python丨小学妹喜欢看漫画，于是我写了四十行代码获取了它所有漫画
女同桌找我要表情包，还好我会Python，分分钟给她下载几十个G...
为了防止这上面的文章被封，我连夜用Python获取了它所有内容，真香~
这个Python读取文件的方法，堪称天花板级别...
Python做一个通过输入bv号就能下载视频的工具，评论和弹幕也不放过
Python：50行代码实现下载小说，图片章节可自动识别转文字保存...
只用二十行代码，用Python实现获取网抑云榜单文件保存本地，非常简单...
发现一个舔狗神器，Python真的太厉害了，自动下载妹子视频...
使用Python探索四大名著【红楼梦】人物之间的关系，简直帅呆了






    [ 申请 ]友情链接：
    
        快连
        快连vpn
        搜外友链
        笔趣阁
        爱思助手
        ClashX教程
        绘画宝宝
        配音宝宝
    


    
        
            关于我们
            服务条款
            广告服务
            联系我们
            网站地图
            免责声明
            WAP
        
        技术支持：
            武汉快勤科技有限公司
            XML网站地图 
            备案号：鄂ICP备18027844号-9
            
        
    




    
        立即登录/注册
        
    
    
        
        微信扫码登录
    












	    基本
        文件
        流程
        错误
        SQL
        调试
    

		    
    
	请求信息 : 2025-08-11 01:55:00 HTTP/2.0 GET : /home/article/detail/id/226742.html
运行时间 : 0.0494s ( Load:0.0116s Init:0.0016s Exec:0.0236s Template:0.0126s )
吞吐率 : 20.24req/s
内存开销 : 1,925.29 kb
查询信息 : 18 queries 0 writes 
文件加载 : 36
缓存信息 : 5 gets 0 writes 
配置加载 : 132
会话信息 : SESSION_ID=6t1vfttptipte5i8b9hlclubjo
    
    
        
    
	/www/wwwroot/www.chaojiit.com/index.php ( 1.30 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/ThinkPHP.php ( 4.71 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Think.class.php ( 12.32 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Storage.class.php ( 1.38 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Storage/Driver/File.class.php ( 3.56 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Mode/common.php ( 2.82 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Common/functions.php ( 51.07 KB )
/www/wwwroot/www.chaojiit.com/Application/Common/Common/function.php ( 6.83 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Hook.class.php ( 4.02 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/App.class.php ( 12.44 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Dispatcher.class.php ( 15.15 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Route.class.php ( 13.38 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Controller.class.php ( 10.95 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/View.class.php ( 7.96 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/BuildLiteBehavior.class.php ( 3.69 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ParseTemplateBehavior.class.php ( 3.89 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ContentReplaceBehavior.class.php ( 1.93 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Conf/convention.php ( 11.18 KB )
/www/wwwroot/www.chaojiit.com/Application/Common/Conf/config.php ( 1.81 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Lang/zh-cn.php ( 2.57 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Conf/debug.php ( 1.51 KB )
/www/wwwroot/www.chaojiit.com/Application/Home/Conf/config.php ( 0.05 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ReadHtmlCacheBehavior.class.php ( 5.62 KB )
/www/wwwroot/www.chaojiit.com/Application/Home/Controller/ArticleController.class.php ( 6.55 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Model.class.php ( 67.27 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Db.class.php ( 5.70 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Db/Driver/Mysql.class.php ( 8.73 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Db/Driver.class.php ( 41.60 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Cache.class.php ( 3.84 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Cache/Driver/File.class.php ( 5.90 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Template.class.php ( 28.35 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Template/TagLib/Cx.class.php ( 22.62 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Template/TagLib.class.php ( 9.19 KB )
/www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php ( 14.50 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/WriteHtmlCacheBehavior.class.php ( 1.43 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ShowPageTraceBehavior.class.php ( 5.27 KB )
    
    
        
    
	[ app_init ] --START--
Run Behavior\BuildLiteBehavior [ RunTime:0.000011s ]
[ app_init ] --END-- [ RunTime:0.000049s ]
[ app_begin ] --START--
Run Behavior\ReadHtmlCacheBehavior [ RunTime:0.000902s ]
[ app_begin ] --END-- [ RunTime:0.000940s ]
[ view_parse ] --START--
[ template_filter ] --START--
Run Behavior\ContentReplaceBehavior [ RunTime:0.000068s ]
[ template_filter ] --END-- [ RunTime:0.000120s ]
Run Behavior\ParseTemplateBehavior [ RunTime:0.009207s ]
[ view_parse ] --END-- [ RunTime:0.009279s ]
[ view_filter ] --START--
Run Behavior\WriteHtmlCacheBehavior [ RunTime:0.000361s ]
[ view_filter ] --END-- [ RunTime:0.000392s ]
[ app_end ] --START--
    
    
        
    
	[2] session_save_path(): open_basedir restriction in effect. File(/var/lib/php/session) is not within the allowed path(s): (/www/wwwroot/www.chaojiit.com/:/tmp/) /www/wwwroot/www.chaojiit.com/ThinkPHP/Common/functions.php 第 1239 行.
[8192] Array and string offset access syntax with curly braces is deprecated /www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Cache/Driver/File.class.php 第 59 行.
[8] Undefined variable: user /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 37 行.
[8] Undefined variable: user /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 98 行.
[8] Trying to access array offset on value of type null /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 98 行.
[8] Undefined variable: user /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 99 行.
[8] Trying to access array offset on value of type null /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 99 行.
[8] Undefined variable: pinglun_list /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 108 行.
    
    
        
    
	SHOW COLUMNS FROM `configuration` [ RunTime:0.0005s ]
SELECT `value` FROM `configuration` WHERE `name` = 'site_name' LIMIT 1   [ RunTime:0.0002s ]
SHOW COLUMNS FROM `menu` [ RunTime:0.0006s ]
SELECT * FROM `menu` WHERE `fid` = 0 AND `status` = 1  [ RunTime:0.0002s ]
SELECT * FROM `menu` WHERE `fid` = 1 AND `status` = 1  [ RunTime:0.0001s ]
SELECT * FROM `menu` WHERE `fid` = 2 AND `status` = 1  [ RunTime:0.0001s ]
SELECT * FROM `menu` WHERE `fid` = 3 AND `status` = 1  [ RunTime:0.0001s ]
SELECT * FROM `menu` WHERE `fid` = 4 AND `status` = 1  [ RunTime:0.0001s ]
SHOW COLUMNS FROM `article` [ RunTime:0.0006s ]
SELECT * FROM `article` WHERE `id` = 226742 LIMIT 1   [ RunTime:0.0005s ]
SHOW COLUMNS FROM `bloger` [ RunTime:0.0006s ]
SELECT * FROM `bloger` WHERE `id` = 204 LIMIT 1   [ RunTime:0.0002s ]
SELECT COUNT(*) AS tp_count FROM `article` WHERE `bloger_id` = 204 LIMIT 1   [ RunTime:0.0001s ]
SHOW COLUMNS FROM `article_content` [ RunTime:0.0007s ]
SELECT `content` FROM `article_content` WHERE `article_id` = 226742 LIMIT 1   [ RunTime:0.0020s ]
SHOW COLUMNS FROM `article_cate` [ RunTime:0.0008s ]
SELECT `name` FROM `article_cate` WHERE `id` = 6 LIMIT 1   [ RunTime:0.0002s ]
SELECT * FROM `article` WHERE `bloger_id` = 204 ORDER BY view_count desc LIMIT 0,10   [ RunTime:0.0009s ]
    
    
        
    
	    
    
    



0.0494s