Skip to main content
Welcome guest. | Register | Login | Post

Some services are not working

9 replies [Last post]
Gustavo's picture
Offline
Joined: 2006-09-11

Hello, everyone.

Due to a critical bug on WebSvn, we're running out of disk and therefore you might find that some things hosted by GLM are not working (like some svn repositories).

This will take some time: We are removing ~245GB (+160000 folders and +220000 files).

Cheers.

free-zombie's picture
Offline
Joined: 2006-03-08
This is the perfect

This is the perfect opportunity for me to suggest using a distributed version control system like git (FAST and rather popular), hg/mercurial (coice of opensolaris, mozilla, xine, etc.), or bzr (Ubuntu's VCS) ;-)

Hope there was no lasting damage...

Gustavo's picture
Offline
Joined: 2006-09-11
This is a DOS attack

Correction: This is a DOS attack and we're still being attacked.

I don't wtf is going on, but one of our attackers is a Google computer (66.249.67.104). I guess that we're being attacked from a zombie network, because of the different ip addresses.

An excerpt of the logs (where {hidden} represents part of the path):

...
[Sun Jul 01 09:44:37 2007] [error] [client 74.6.28.198] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:44:37 2007] [error] [client 74.6.28.198] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:44:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:44:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:33 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:33 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:43 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:43 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:49 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:49 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:05 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:05 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:27 2007] [error] [client 74.6.18.25] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:27 2007] [error] [client 74.6.18.25] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:52 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:52 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:54 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:54 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:59 2007] [error] [client 74.6.20.81] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:59 2007] [error] [client 74.6.20.81] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:23 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:23 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:27 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:27 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:39 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:39 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:43 2007] [error] [client 80.30.151.99] File does not exist: /{hidden}/svn.getgnulinux.org/failed_auth.html
[Sun Jul 01 09:47:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:26 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:26 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:42 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:42 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:57 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:57 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:11 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:11 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:50:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:50:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:50:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:50:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:51:30 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:51:30 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:52:04 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:52:04 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
...

It looks like someone dislikes what we do.

UPDATE: Those IP addresses are obviously blocked now.

libervisco's picture
Offline
Joined: 2006-05-04
Gosh.. Why would anyone do

Gosh.. Why would anyone do this to you! Sad

I hope you guys sort it out soon enough and manage to prevent it in the future. I hate internet pests with nothing better to do than destroy and raise havoc..

Offline
Joined: 2006-03-28
Until you got it sorted out

Until you got it sorted out I suggest three steps:

  1. Stop all SVN-related services
  2. Drop (or if you have unlimited traffic you could mirror them if you like ;-) ) packets coming in on the related ports
  3. Check for updates for all related servers

After this is done and everything is sorted out it might be useful to put a connection-limit in place with IPTables, so that a DoS should get cancelled by the packet-filter.

SVN just uses HTTP, does it? So it would only communicate through TCP/80, right?
I'll have a look through the docs and see that I can get you some rules ready.

Gustavo's picture
Offline
Joined: 2006-09-11
Thanks, reptiler! We're on

Thanks, reptiler!

We're on a shared server, so I'm unable to act as root. Fortunately, our svn repos are working flawlessly. What happened is that a funny guy exploited a flaw on WebSVN.

Because the WebSVN project is death, and it has a critical bug, I'm currently installing ViewVC, which is not as cool as WebSVN, but at least it's maintained.

If we keep Trac, I'll setup a branch for these projects (the trunk is intended to the upcoming DocBook port).

I must admit that I'm happy of this attack: It means that we're doing our job well!

Cheers.

Offline
Joined: 2006-03-28
Too bad. Having the option

Too bad. Having the option to block certain traffic can sometimes be really useful.

And yes, not every attack is bad. When you notice an attack and nothing serious happens you also know that you've done a good job setting up your box. Or that you're just dealing with a stupid person who tries to get in. ;-)
I have a Debian-server running at work which runs really great. Some connection-attempts are logged, like for example for SSH, which isn't even open for externals IPs.
It's just too bad I can't use the CONNMARK-target and -match on Debian 3.1, this would make my logging-rules a bit easier and better. ;-)

tbuitenh's picture
Offline
Joined: 2005-12-21
Another of the "attackers"

Another of the "attackers" is a yahoo crawler. I suspect the googlebot and yahoo crawler IPs are spoofed in one way or another. Of course blocking google and yahoo bots is not good for your search engine rating, so it would seem someone wants getgnulinux to disappear from search results. How rude.

EDIT: another one is a DNS server owned by Telefonica. I'm not really sure what kind of attack this is, but it's definitely an indirect one.

Gustavo's picture
Offline
Joined: 2006-09-11
Yes, that's right. These

Yes, that's right.

These IPs are not allowed to access svn.getgnulinux.org, but they can access other domains hosted by GLM.

Gustavo's picture
Offline
Joined: 2006-09-11