Skip to main content
Welcome guest. | Register | Login | Post
Home » forums » Friends » GNU/Linux Matters forum

Some services are not working

9 replies [Last post]
Sat, 2007-06-30 17:31
Gustavo's picture
Gustavo
User offline. Last seen 41 weeks 5 days ago. Offline
Joined: 2006-09-11
Posts:

Hello, everyone.

Due to a critical bug on WebSvn, we're running out of disk and therefore you might find that some things hosted by GLM are not working (like some svn repositories).

This will take some time: We are removing ~245GB (+160000 folders and +220000 files).

Cheers.

Top
Sat, 2007-06-30 18:05
#1
free-zombie's picture
free-zombie
User offline. Last seen 7 weeks 3 days ago. Offline
Joined: 2006-03-08
Posts:
This is the perfect

This is the perfect opportunity for me to suggest using a distributed version control system like git (FAST and rather popular), hg/mercurial (coice of opensolaris, mozilla, xine, etc.), or bzr (Ubuntu's VCS) ;-)

Hope there was no lasting damage...

Top
Sun, 2007-07-01 17:10
#2
Gustavo's picture
Gustavo
User offline. Last seen 41 weeks 5 days ago. Offline
Joined: 2006-09-11
Posts:
This is a DOS attack

Correction: This is a DOS attack and we're still being attacked.

I don't wtf is going on, but one of our attackers is a Google computer (66.249.67.104). I guess that we're being attacked from a zombie network, because of the different ip addresses.

An excerpt of the logs (where {hidden} represents part of the path):

...
[Sun Jul 01 09:44:37 2007] [error] [client 74.6.28.198] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:44:37 2007] [error] [client 74.6.28.198] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:44:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:44:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:33 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:33 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:43 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:43 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:45:49 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:45:49 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:05 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:05 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:18 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:27 2007] [error] [client 74.6.18.25] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:27 2007] [error] [client 74.6.18.25] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:52 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:52 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:54 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:54 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:46:59 2007] [error] [client 74.6.20.81] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:46:59 2007] [error] [client 74.6.20.81] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:08 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:23 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:23 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:27 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:27 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:39 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:39 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:47:43 2007] [error] [client 80.30.151.99] File does not exist: /{hidden}/svn.getgnulinux.org/failed_auth.html
[Sun Jul 01 09:47:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:47:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:02 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:26 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:26 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:36 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:42 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:42 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:48:57 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:48:57 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:02 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:03 2007] [error] [client 82.121.144.175] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:11 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:11 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:49:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:49:46 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:50:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:50:20 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:50:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:50:55 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:51:30 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:51:30 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
[Sun Jul 01 09:52:04 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/websvn
[Sun Jul 01 09:52:04 2007] [error] [client 66.249.67.104] File does not exist: /{hidden}/svn.getgnulinux.org/missing.html
...

It looks like someone dislikes what we do.

UPDATE: Those IP addresses are obviously blocked now.

Top
Sun, 2007-07-01 17:34
#3
libervisco's picture
libervisco
User offline. Last seen 13 hours 40 min ago. Offline
Joined: 2006-05-04
Posts:
Gosh.. Why would anyone do

Gosh.. Why would anyone do this to you! Sad

I hope you guys sort it out soon enough and manage to prevent it in the future. I hate internet pests with nothing better to do than destroy and raise havoc..

Top
Mon, 2007-07-02 04:44
#4
reptiler
User offline. Last seen 9 weeks 3 days ago. Offline
Joined: 2006-03-28
Posts:
Until you got it sorted out

Until you got it sorted out I suggest three steps:

  1. Stop all SVN-related services
  2. Drop (or if you have unlimited traffic you could mirror them if you like ;-) ) packets coming in on the related ports
  3. Check for updates for all related servers

After this is done and everything is sorted out it might be useful to put a connection-limit in place with IPTables, so that a DoS should get cancelled by the packet-filter.

SVN just uses HTTP, does it? So it would only communicate through TCP/80, right?
I'll have a look through the docs and see that I can get you some rules ready.

Top
Mon, 2007-07-02 08:25
#5
Gustavo's picture
Gustavo
User offline. Last seen 41 weeks 5 days ago. Offline
Joined: 2006-09-11
Posts:
Thanks, reptiler! We're on

Thanks, reptiler!

We're on a shared server, so I'm unable to act as root. Fortunately, our svn repos are working flawlessly. What happened is that a funny guy exploited a flaw on WebSVN.

Because the WebSVN project is death, and it has a critical bug, I'm currently installing ViewVC, which is not as cool as WebSVN, but at least it's maintained.

If we keep Trac, I'll setup a branch for these projects (the trunk is intended to the upcoming DocBook port).

I must admit that I'm happy of this attack: It means that we're doing our job well!

Cheers.

Top
Mon, 2007-07-02 09:25
#6
reptiler
User offline. Last seen 9 weeks 3 days ago. Offline
Joined: 2006-03-28
Posts:
Too bad. Having the option

Too bad. Having the option to block certain traffic can sometimes be really useful.

And yes, not every attack is bad. When you notice an attack and nothing serious happens you also know that you've done a good job setting up your box. Or that you're just dealing with a stupid person who tries to get in. ;-)
I have a Debian-server running at work which runs really great. Some connection-attempts are logged, like for example for SSH, which isn't even open for externals IPs.
It's just too bad I can't use the CONNMARK-target and -match on Debian 3.1, this would make my logging-rules a bit easier and better. ;-)

Top
Tue, 2007-07-03 10:17
#7
tbuitenh's picture
tbuitenh
User offline. Last seen 1 year 11 weeks ago. Offline
Joined: 2005-12-21
Posts:
Another of the "attackers"

Another of the "attackers" is a yahoo crawler. I suspect the googlebot and yahoo crawler IPs are spoofed in one way or another. Of course blocking google and yahoo bots is not good for your search engine rating, so it would seem someone wants getgnulinux to disappear from search results. How rude.

EDIT: another one is a DNS server owned by Telefonica. I'm not really sure what kind of attack this is, but it's definitely an indirect one.

Top
Tue, 2007-07-03 12:44
#8
Gustavo's picture
Gustavo
User offline. Last seen 41 weeks 5 days ago. Offline
Joined: 2006-09-11
Posts:
Yes, that's right. These

Yes, that's right.

These IPs are not allowed to access svn.getgnulinux.org, but they can access other domains hosted by GLM.

Top
Sun, 2007-07-08 14:10
#9
Gustavo's picture
Gustavo
User offline. Last seen 41 weeks 5 days ago. Offline
Joined: 2006-09-11
Posts:
I've re-installed websvn,

I've re-installed websvn, but its cache is now disabled: http://svn.getgnulinux.org/browser/

If its cache is disabled, we have nothing to worry about.

ViewVC didn't seem to be an acceptable replacement, imo.

Cheers.

Top

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Who's online

There are currently 0 users and 40 guests online.
We have 11080 members who wrote 2192 articles and 12404 comments. Welcome to our newest member, ordeweerson!