RYDE-WORK/MediaCrawler
1
0
Fork 0
mirror of https://github.com/RYDE-WORK/MediaCrawler.git synced 2026-01-19 13:33:27 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
434 Commits 1 Branch 0 Tags
Commit Graph

434 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
翟持江
8967d3721c
将微博评论爬取函数get_note_all_comments的max_id_type便为可变请求参数 
除了原先的max_id参数外，max_id_type参数也附加在上一次api结果的解析中，初始为0，但随着获取的评论越来越多，会更改为1。
此外，修改了WeiboClient类的request函数，将返回的ok_code异常处理进行了优化，细分为0，1，else...。这样即便获取到的max_id和max_id_type为None，也不会触发像'>' not supported between instances of 'NoneType' and 'int'这样模棱两可的异常提示，方便溯源问题所在，即api响应错误。
对于评论的数据获取不全的情况，在浏览器中获取显示的评论数量为1000+，更改此次提交前获取的个数为308条，更改后为319条，使用网页端打开手动刷评论的最后一条和程序获取的最后一条内容一致，可能是微博默认开启的精选评论功能导致无法获取所有的微博...
 2025-01-10 19:08:57 +08:00
Relakkes
5d3e12169e fix: move dirctory 2025-01-10 16:53:20 +08:00
Relakkes
b89a8e5dda docs: add github issue template 2025-01-10 16:51:30 +08:00
Relakkes
792d3bbc34 docs: update README.md 2025-01-05 19:02:10 +08:00
Relakkes
5f7c8aea5b docs: 添加11-12月份捐赠者名单 2025-01-02 20:37:28 +08:00
Relakkes
fbbead814a fix: 贴吧创作者bug修复 2025-01-02 20:29:05 +08:00
程序员阿江(Relakkes)
7ce1273386
Merge pull request #528 from HuiGitH/hui/fix-xhs-gender 
fix xhs gender
 2024-12-28 21:01:58 +08:00
HuiLong
d929ad16ae fix xhs get gender 2024-12-28 20:24:37 +08:00
Relakkes
79bf9fc05d chore: add xhs field comment for issue #526 2024-12-26 18:28:23 +08:00
Relakkes
29ab6feded fix: 修复数据库密码中有特殊符号导致db连接不上的bug 2024-12-26 18:13:13 +08:00
Relakkes
ea5223c708 feat: 知乎支持详情模式 2024-12-26 17:36:33 +08:00
程序员阿江(Relakkes)
dc9116e098
Merge pull request #513 from liudongkai/feature_xhs 
feat: xhs 非代理模式下增加随机等待间隔, db存储模式下增加存储xsec_token字段
 2024-12-06 13:01:16 +08:00
liudongkai
33e7ef016d feat: xhs 非代理模式下增加随机等待间隔, db存储模式下增加存储xsec_token字段 2024-12-05 21:10:31 +08:00
程序员阿江(Relakkes)
9c7e1d499b
Merge pull request #509 from leantli/feat/xhs_comments_upgrade 
feat: xhs comments add xsec_token
 2024-12-03 18:34:56 +08:00
leantli
e830ada574 feat: xhs comments add xsec_token 2024-12-03 18:25:21 +08:00
程序员阿江(Relakkes)
6001fc8a52
Merge pull request #506 from JianxunRao/main 
fix：微博根据creator爬取note时，爬取评论失败。原因是解析的参数key有误
 2024-11-29 11:07:58 +08:00
Trojx
f9eedc59b1 fix：微博根据creator爬取note时，爬取评论失败。原因是解析的参数key有误 2024-11-29 10:47:40 +08:00
Relakkes
453ea642fb docs: update README.md 2024-11-29 10:40:52 +08:00
Relakkes
ca9b47ef63 fix: xhs 帖子详情优化 2024-11-27 09:41:24 +08:00
Relakkes
43dffeb2d1 feat: xhs帖子详情获取优化 2024-11-26 13:37:53 +08:00
Relakkes
de32d06815 docs: update README.md 2024-11-19 12:58:28 +08:00
Relakkes
935a928f90 docs: update README.md 2024-11-17 07:00:55 +08:00
Relakkes
8ab4c67443 feat: 抖音支持评论点赞数量 #495 2024-11-16 00:37:48 +08:00
Relakkes
c12a63acb2 fix: weibo creator duplicate bug 2024-11-13 22:28:54 +08:00
Relakkes
f9e7d56d4f chore: remove sponsor 2024-11-08 22:27:03 +08:00
Relakkes
3db2edd479 fix: weibo创作者bugfix 2024-11-06 16:01:07 +08:00
程序员阿江(Relakkes)
3567193a98
Merge pull request #479 from lyx0727/main 
fix: xhs出现验证码时报错，增加用户手动验证
 2024-11-01 23:09:01 +08:00
lyx0727
705b810269 fix: xhs验证码处理，跳转链接修复 2024-11-01 22:08:46 +08:00
lyx0727
1a37df4d5e fix: xhs出现验证码时报错，增加用户手动验证 2024-11-01 18:52:16 +08:00
Relakkes
ef4eba121c fix: 兼容windows编码问题 2024-10-28 15:26:49 +08:00
Relakkes
9bf6ea3650 docs: update README.md 2024-10-24 18:18:53 +08:00
Relakkes
6a96c00b4b chore: config update & xinqqiu img update 2024-10-24 15:35:34 +08:00
程序员阿江(Relakkes)
5a27ad089c
Merge pull request #462 from FloRainRJY/xiaohongshu_comment_number_restrict 
All_platform_comments_restrict
 2024-10-24 15:31:13 +08:00
程序员阿江(Relakkes)
0cd01fbf8c
Merge pull request #463 from liugangdao/main 
fix:翻页时search id不变
 2024-10-24 15:13:05 +08:00
liugangdao
78c09c4ae1 fix:翻页时search id不变 2024-10-24 13:57:27 +08:00
unknown
7e53c4acfc All_platform_comments_restrict 2024-10-23 16:32:02 +08:00
程序员阿江(Relakkes)
fa2bcc4181
Merge pull request #461 from FloRainRJY/xiaohongshu_comment_number_restrict 
feat: xhs增加最大评论数量限制
 2024-10-23 13:35:10 +08:00
unknown
19269c66fd xiaohongshu_comment_number_restrict 2024-10-22 20:33:10 +08:00
Relakkes
0bb9298d26 docs: 添加捐赠名单 2024-10-21 21:17:28 +08:00
Relakkes
dc11d39185 docs: update README.md 2024-10-20 01:05:57 +08:00
Relakkes
03e393949a fix: xhs帖子详情问题更新 2024-10-20 00:59:08 +08:00
Relakkes
9fe3e47b0f chore: 增加代码学习声明，严格禁止非法、禁止商业、不当用途 2024-10-20 00:43:25 +08:00
程序员阿江-Relakkes
ac8537c296
Update LICENSE 2024-10-18 03:27:00 +08:00
Relakkes
aa0f920369 feat: B站搜索接口增加发布日期筛选 2024-10-17 15:11:25 +08:00
Relakkes
fa1932e105 docs: 添加14群交流二维码 2024-10-17 14:54:06 +08:00
Relakkes
96508fb050 docs: 知乎支持创作者主页数据 2024-10-16 21:03:06 +08:00
Relakkes
da8f1c62b8 feat: 知乎支持创作者主页数据爬取(回答、文章、视频) 2024-10-16 21:02:27 +08:00
Relakkes
af9d2d8e84 docs: update Sponsor 2024-10-10 11:28:37 +08:00
Relakkes
9046074909 docs: update README.md 2024-09-28 12:48:43 +08:00
Relakkes
7b5b099636 feat: update douyin abogus params 2024-09-27 14:58:10 +08:00