最近写了一个小爬虫,使用了selenium,用来爬取某作者在cnki收录的文献信息,目前已经实现了自动打开每个论文并爬取页面中的标题、期刊、摘要等
但遇到了一个问题,当作者论文较多,有很多时,需要跳转到新的一页,然后继续循环,把这一页的文献挨个打开并爬取页面中的标题、期刊、摘要等
这部分我没做好,打开新页面后就没有反应了
(base) whitewatercn@MadBookAir itutor_spider_mac % /Users/whitewatercn/miniconda3/bin/python /Users/whitewatercn/FennecNas/coding/tools/itutor_spider_mac
/main.py
Traceback (most recent call last):
File "/Users/whitewatercn/FennecNas/coding/tools/itutor_spider_mac/main.py", line 55, in <module>
button.click()
File "/Users/whitewatercn/miniconda3/lib/python3.10/site-packages/selenium/webdriver/remote/webelement.py", line 94, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/whitewatercn/miniconda3/lib/python3.10/site-packages/selenium/webdriver/remote/webelement.py", line 395, in _execute
return self._parent.execute(command, params)
File "/Users/whitewatercn/miniconda3/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 345, in execute
self.error_handler.check_response(response)
File "/Users/whitewatercn/miniconda3/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found
(Session info: chrome=113.0.5672.126); For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#stale-element-reference-exception
Stacktrace:
0 chromedriver 0x0000000100c5c2b8 chromedriver + 4260536
1 chromedriver 0x0000000100c5494c chromedriver + 4229452
2 chromedriver 0x00000001008909d4 chromedriver + 281044
3 chromedriver 0x0000000100895200 chromedriver + 299520
4 chromedriver 0x0000000100896b34 chromedriver + 305972
5 chromedriver 0x0000000100896c24 chromedriver + 306212
6 chromedriver 0x00000001008d2024 chromedriver + 548900
7 chromedriver 0x00000001008d0094 chromedriver + 540820
8 chromedriver 0x00000001008cdfac chromedriver + 532396
9 chromedriver 0x00000001008cd37c chromedriver + 529276
10 chromedriver 0x00000001008c1984 chromedriver + 481668
11 chromedriver 0x00000001008c11cc chromedriver + 479692
12 chromedriver 0x00000001009027e4 chromedriver + 747492
13 chromedriver 0x00000001008bf98c chromedriver + 473484
14 chromedriver 0x00000001008c098c chromedriver + 477580
15 chromedriver 0x0000000100c1aa6c chromedriver + 3992172
16 chromedriver 0x0000000100c1ecd0 chromedriver + 4009168
17 chromedriver 0x0000000100c24d48 chromedriver + 4033864
18 chromedriver 0x0000000100c1f940 chromedriver + 4012352
19 chromedriver 0x0000000100bf757c chromedriver + 3847548
20 chromedriver 0x0000000100c3d400 chromedriver + 4133888
21 chromedriver 0x0000000100c3d558 chromedriver + 4134232
22 chromedriver 0x0000000100c4dc3c chromedriver + 4201532
23 libsystem_pthread.dylib 0x00000001843ff034 _pthread_start + 136
24 libsystem_pthread.dylib 0x00000001843f9e3c thread_start + 8