【菜鸟历险记】今天网站被人攻击了,然后生成了70G的日志文件

今天网站无故被人攻击,我也没在意,想着攻击结束就可以正常访问了,结果,等到晚上的时候,我发现攻击结束了,但是网站却502了。

然后,我在论坛上找解决方案,有人说用“./launcher rebuild app”这个命令,我试了下;结果出现下面的问题,我一看这不是说内存不足吗?

root@kehan:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        88G   84G     0 100% /

Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)N
root@kehan:/var/discourse# 
root@kehan:/var/discourse# 
root@kehan:/var/discourse# 
root@kehan:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        88G   84G     0 100% /

Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)y
If the cleanup was successful, you may try again now
root@kehan:/var/discourse# ./launcher rebuild app
x86_64 arch detected.

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

2.0.20240825-0027: Pulling from discourse/base
Digest: sha256:6de68cb49198b5281f79ed9401b3fe818c854d220dcf0238549fe2f2adb19146
Status: Downloaded newer image for discourse/base:2.0.20240825-0027
docker.io/discourse/base:2.0.20240825-0027
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        88G   84G     0 100% /

我的网站刚建好,内存应该不到300M,怎么会有这么大,刚开始我不是这么想的,因为下午的时候,系统给我发了一封邮件内容如下:

由于达到了 download_remote_images_threshold 的磁盘空间限制,download_remote_images_to_local 设置被禁用。

我当时想的是,谁入侵了我的网站,植入了下载图片的木马,然后,我想的是,可能网站会有一个很大的图片,把那个图片删除就好。

但是我是个菜鸟,不知道用啥命令,然后就在官方论坛一顿搜,看着有用的命令就执行。

后面想到了chatgpt,在chatgpt的帮助下,我终于找到了问题的元凶,就是70G的日志文件,下面是我再chatgpt的指引下执行的命令,还有反馈的结果:

Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)y
If the cleanup was successful, you may try again now
root@kehan:/var/discourse# ./launcher rebuild app
x86_64 arch detected.

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

2.0.20240825-0027: Pulling from discourse/base
Digest: sha256:6de68cb49198b5281f79ed9401b3fe818c854d220dcf0238549fe2f2adb19146
Status: Downloaded newer image for discourse/base:2.0.20240825-0027
docker.io/discourse/base:2.0.20240825-0027
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        88G   84G     0 100% /

Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)y
If the cleanup was successful, you may try again now
root@kehan:/var/discourse# 
root@kehan:/var/discourse# ./launcher enter app
x86_64 arch detected.

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

2.0.20240825-0027: Pulling from discourse/base
Digest: sha256:6de68cb49198b5281f79ed9401b3fe818c854d220dcf0238549fe2f2adb19146
Status: Downloaded newer image for discourse/base:2.0.20240825-0027
docker.io/discourse/base:2.0.20240825-0027
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        88G   84G     0 100% /

Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)y
If the cleanup was successful, you may try again now
root@kehan:/var/discourse# 
root@kehan:/var/discourse# ^C
root@kehan:/var/discourse# ^C
root@kehan:/var/discourse# sudo apt install byobu
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  gawk libevent-core-2.1-7 libmpfr6 libsigsegv2 libutempter0 pastebinit python3-distro python3-newt tmux
Suggested packages:
  apport ccze gnome-terminal | xterm po-debconf screen speedometer ttf-ubuntu-font-family update-notifier-common vim wireless-tools gawk-doc
Recommended packages:
  run-one
The following NEW packages will be installed:
  byobu gawk libevent-core-2.1-7 libmpfr6 libsigsegv2 libutempter0 pastebinit python3-distro python3-newt tmux
0 upgraded, 10 newly installed, 0 to remove and 93 not upgraded.
Need to get 2,270 kB of archives.
After this operation, 6,915 kB of additional disk space will be used.
E: You don't have enough free space in /var/cache/apt/archives/.
root@kehan:/var/discourse# sudo purge-old-kernel
sudo: purge-old-kernel: command not found
root@kehan:/var/discourse# ./launcher rebuild app
x86_64 arch detected.

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

2.0.20240825-0027: Pulling from discourse/base
Digest: sha256:6de68cb49198b5281f79ed9401b3fe818c854d220dcf0238549fe2f2adb19146
Status: Downloaded newer image for discourse/base:2.0.20240825-0027
docker.io/discourse/base:2.0.20240825-0027
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        88G   84G     0 100% /

Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)y
If the cleanup was successful, you may try again now
root@kehan:/var/discourse# docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] N
root@kehan:/var/discourse# du -sh /* | sort -h
du: cannot access '/proc/8758/task/8758/fd/4': No such file or directory
du: cannot access '/proc/8758/task/8758/fdinfo/4': No such file or directory
du: cannot access '/proc/8758/fd/3': No such file or directory
du: cannot access '/proc/8758/fdinfo/3': No such file or directory
0	/bin
0	/dev
0	/initrd.img
0	/initrd.img.old
0	/lib
0	/lib32
0	/lib64
0	/libx32
0	/proc
0	/sbin
0	/sys
0	/vmlinuz
0	/vmlinuz.old
4.0K	/home
4.0K	/mnt
4.0K	/srv
8.0K	/media
16K	/lost+found
16K	/opt
36K	/tmp
48K	/root
568K	/run
3.8M	/etc
56M	/boot
1.4G	/usr
2.1G	/swapfile
87G	/var
root@kehan:/var/discourse# du -sh /var/* | sort -h
0	/var/lock
0	/var/run
4.0K	/var/local
4.0K	/var/mail
4.0K	/var/opt
12K	/var/spool
20K	/var/tmp
408K	/var/backups
114M	/var/log
290M	/var/cache
17G	/var/lib
70G	/var/discourse
root@kehan:/var/discourse# docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] N
root@kehan:/var/discourse# ^C
root@kehan:/var/discourse# sudo apt-get clean
root@kehan:/var/discourse# du -sh /var/www/discourse/public/uploads/*
du: cannot access '/var/www/discourse/public/uploads/*': No such file or directory
root@kehan:/var/discourse# find /var/www/discourse/public/uploads -type f -exec du -sh {} + | sort -rh
find: ‘/var/www/discourse/public/uploads’: No such file or directory
root@kehan:/var/discourse# ^C
root@kehan:/var/discourse# find /var -type d -name "uploads"
/var/discourse/shared/standalone/uploads
root@kehan:/var/discourse# find /path/to/uploads -type f -exec du -h {} + | sort -rh | head -n 10
find: ‘/path/to/uploads’: No such file or directory
root@kehan:/var/discourse# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            962M     0  962M   0% /dev
tmpfs           197M  568K  197M   1% /run
/dev/vda1        88G   83G  279M 100% /
tmpfs           984M     0  984M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
overlay          88G   83G  279M 100% /var/lib/docker/overlay2/c9d06a2d4a1af6ca0389f6a8098e2ba418373eeeefa997ecab2d4002fd60c703/merged
tmpfs           197M     0  197M   0% /run/user/0
root@kehan:/var/discourse# /var/discourse# du -sh /* | sort -h
-bash: /var/discourse#: No such file or directory
root@kehan:/var/discourse# du -sh /var/* | sort -h



0	/var/lock
0	/var/run
4.0K	/var/local
4.0K	/var/mail
4.0K	/var/opt
12K	/var/spool
20K	/var/tmp
408K	/var/backups
6.6M	/var/cache
114M	/var/log
17G	/var/lib
70G	/var/discourse
root@kehan:/var/discourse# 
root@kehan:/var/discourse# 
root@kehan:/var/discourse# 
root@kehan:/var/discourse# du -sh /var/discourse/* | sort -h
4.0K	/var/discourse/bin
4.0K	/var/discourse/cids
4.0K	/var/discourse/LICENSE
8.0K	/var/discourse/scripts
12K	/var/discourse/discourse-doctor
12K	/var/discourse/README.md
20K	/var/discourse/containers
24K	/var/discourse/launcher
24K	/var/discourse/tests
28K	/var/discourse/discourse-setup
32K	/var/discourse/samples
180K	/var/discourse/launcher_go
180K	/var/discourse/templates
232K	/var/discourse/image
70G	/var/discourse/shared
root@kehan:/var/discourse# ^C
root@kehan:/var/discourse# du -sh /var/discourse/shared/* | sort -h
70G	/var/discourse/shared/standalone
root@kehan:/var/discourse# du -sh /var/discourse/shared/standalone
70G	/var/discourse/shared/standalone
root@kehan:/var/discourse# cd /var/discourse/shared/standalone
du -sh *
5.8M	backups
1.8M	letsencrypt
70G	log
4.0K	postgres_backup
110M	postgres_data
180K	postgres_run
11M	redis_data
20K	ssl
28K	state
12K	tmp
11M	uploads

上面这一顿乱操作下来,我一刷新,网站竟然可以访问了。

但是问题来了,70G的日志文件可以删吗?

为啥会生成这么大的日志文件,如何禁止被攻击时生成这么大的日志文件?

从你贴的日志上看已经非常明确的说了没有磁盘空间了。

磁盘占有率到了 100%

因为 Linux 是文件系统,通常还需要保留不少的磁盘空间才可以。

记得我们公司以前的服务器的磁盘利用率都不会超过 50% 的,差不多到 40% 的时候要不就要开始清理数据要不就要扩容。

日志

因为你的网站受到攻击了,会产生非常多的 Access 日志。

这些日志如果你想做分析的话,就拷贝下来。

如果不想分析也懒得分析就直接删。

被攻击产生日志这个无法避免,除非不要日志。

access 日志总会在哪里的,多监控下就好了。

1 Like

感谢大佬的答复,目前已经删除了日志,然后加了cloudfare的免费cdn

有可能你机器会被同时种了木马,需要留意下。

注意多备份,准备必要的时候重装恢复。

1 Like

感谢提醒,后面我多研究下

1 Like

discourse备份一下,然后重装一下系统,打开ufw把22、80、443端口开了,剩下端口别开,然后再装discourse恢复备份,设置管理员密码随机字符避免撞密码库,套上Cloudflare CDN,基本就没问题啦

咱们小白不懂网络安全,木马更是不知道怎么查杀,重装避免以后烦恼