Naturally, the geeks who designed Second Life thought of this when they put it all together….

In a nutshell, lots of calling cards you’ve collected can slow your performance down.  Of course, that was obvious, wasn’t it?  Maybe time to trim…

-Veyron

I’m beginning to see a pattern here….

1. Today:  Various Issues
2. April 1:   Various Inworld Issues
3. March 31:  Second Life In-World Service Issues
4. March 30:  In World Services Down/Website and Support
5. March 30:  Transactions Timing Out
6. March 29:  Inworld Money Transactions Problematic at the Moment
7. March 28:  Asset Server Issues
8. March 27:  Inworld Issue Affecting Transactions
9. March 26:  Obviously took the day off….
10. March 25:  In World Service Disruptions Under Investigation
11. March 24:  Disruption in inworld services

I’d keep going, but I was getting tired of cutting and pasting - there seemed to be an endless supply of entries like these.  Oh, what we really need now to help Linden Labs out is a trouble blog entry generator.  It might save them a lot of time in creating the blog entry about the problem.  It could slap some random words together for the title, like “Transactions Inworld Timing Out Affecting Servers”.  Then they could use the tech support excuse generator to follow up with it.

Maybe we need to turn this into a game.  Sort of a meta-game.  This doesn’t quite work as a drinking game, as they manage to spread out the disasters by at least a day.  At most you’d get a headache, and we already have that.  I know!  We could do something like SLBrokenDownInTheGrid Bingo.  (Now does that meet new trademark policy?)  You get a bingo card and every time they have an outage, you get the first letters of all capitalized words they use in the title.  What would make a good prize?  Hmmm…  I need to think about that.  I do have this stockade I could put the winner in.

Obviously, any members of Linden Labs’ couldn’t play because they could fix the game.  Like they COULD STOP THE GRID FROM CRATERING EVERY FRIGGIN DAY!

I’m sure they’re getting right on with solving these problems as soon as they pick the next letters to use….

-Veyron

I have been in Second Life now for some time. First off, let me confess that I am one of those special kind of users, a full on addict. So my views are bit colored. I put up with far more than an average user probably would. I use the service far more than normal people do, and I bitch probably more than other people do. I’m in that 5-10% userbase of heavy users. But you already knew that.

Next I want to be clear, I am an eternal optimist. I want things to get better and I want to see Second Life succeed. I love the place, but then with every minor and major grid implosion is is being coming more and more clear that my optimism is not going to be paid off.

The other day Tiessa and I were discussing how things were on the grid. I commented that it seems like there’s a minor disaster every day now. She said, well, it used to be a major disaster every day, so obviously, it is getting better. I would argue that it is not getting better, it is getting worse.

I have seen things pretty bad on the grid. Major outages, inventory disasters, so many client crashes in one day I couldn’t even begin to count them, yet for how much better we are now, things are getting worse. Why? It’s a time and distance thing. Things have not improved enough given the amount of time that has past. It’s a slippery slope and it’s not obvious that it is happening.

Most of us put up with a lot - a lot - of basic shit with Second Life. Why? We really do love the people, the environment, and potential the place has. We got past the hideous learning curve, the icky UI, and the crashes. I think most of us try hard to overlook a lot of the warts of Second Life except when things go really wrong.

1. Ridiculously low number of avatars a sim can support without going into the toilet.
2. The joke it is to walk or gawd forbid try to drive or fly from one sim to another.
3. Hair up your arse. Hippos on your friends list? Other ludicrous problems.
4. Lack of foresight causing no end of problems.
5. Lag me into the ground.

The sad part a few of these problems could be assaulted with some brute force. Yes, it’s not the smart way to go, but it does work. A few years ago I figured out that a faster computer is much cheaper than a few hours of programmers time, and usually guarantees results. Take the current Class 5 sim servers. They started appearing in October 2006. With 4GB of RAM and a Xeon 5148. That should have obviously made the sims faster, right? Nope, it allowed them to pile more sims onto a single server. That’s what’s really going on. Servers are now twice as fast, but not Second Life servers, and even if they were, it wouldn’t matter, because they would simply pile more sims into a single system. In fact, I would suspect that they are already doing this, which leads me to the next bad thing that’s going to happen… over-commit.

Ah yes, now when they really want to start making money, they will start to pack more sims into servers, more than they can really run all at full speed, users be damned. Like we would know the difference at this point anyway? Linden Labs gets to sell more sims without actually spending any more on infrastructure. I’m sure the temptation will be irresistible at some point if they haven’t done it already.

I play quite a bit in the Havok 4 sims, and I can honestly say they seem to have done nothing for lag improvement. So, that’s a bust. I hear concerns that the mono project will not make things even slower than they are now. One thing that is probably hurting lag is voice. I’m not sure how much it’s being used, but it has to be eating up a lot of bandwidth. That bandwidth is now unavailable to make things less laggy.

What does this all boil down to? Things are not getting better, they’re not really getting worse, but it’s moving sideways. Neither really better but not really bad enough to really raise hell. Instead of bitching about what needs to be done, let’s go over an obvious wish list:

1. More than 25 groups - duh - already said it won’t happen. It’s just *too* hard for them. Whine bitch moan.
2. More attachment points, maybe some more clothing points. (We’ll just ignore that one.)
   
3. No more inventory loss… uh - we’re working on that. We promise.
4. More than 75 avies in a sim without lagging the hell out it. You can’t tell me in 2008 it can’t and should be done. If they wanted to they could support 150. Hell - 250.
5. No more lag. Let’s try not introduce the word LAG into every language in the world.
   
6. No more crashing clients. Let’s be honest here - world peace and harmony will happen before it drops below 5% of sessions ending in a crash.

You know and I know. Linden Labs has made miles of promises and has miles to go. You know they can’t deliver on them. Let’s all be brutally honest about it. It ain’t gonna happen. Anyone who’s been dealing with Second Life and Linden Labs long enough knows none of these things they can actually deliver on. It don’t think it is for lack of brain power, I think it has been a management problem in Linden Labs. We see Cory and now Phil depart management rolls, but probably now a culture problem. They have years of bad habits that will be very hard to break.

Where are we going to be in future? We will be in the same basic place in spring of 2009 as we are right now. Bitching about lag, inventory loss, and crashes.

Where does this lead us? Ask Ultima Online what happened when World of Warcraft came along. Second Life survives because their competitors are incompetent. They limp along dragging us long with them because of that. But it won’t last. Nature and capitalism hate a vacuum. And that giant sucking sound coming from Linden Labs will soon draw - if it hasn’t already - a competentcompetitor. Think it can’t happen? Ultima Online had a huge market share of the MMORPG field, now it’s tiny.

When will it start? I think the first sign will be a change in the value of the Linden dollar. I suspect Linden Labs has is actively manipulating the currency market to keep the exchange rate more or less constant. I expect that will end at some point as it is costing Linden Labs too much money. Then inflation will start. The economy will fall apart. It will be interesting to see how they might try to control it. Price controls?

Second Life is at tipping point right now. It’s going to get better or it’s going to get worse.

-Veyron

Here’s a trick that works to de-Ruth someone you see as Ruthed.  It seems idiotic, but it works - and hey - this is Second Life, so it is apropo.  Change groups, from one to other.  Then wait a few moments, and that should do it.  Unless it’s a really laggy sim, then it might take a bit longer.  That’s all.  Go figure.  I forgot who gave me this tip.

-Veyron

Okay, I knew there was going to be a lot of chatter about this coming up.  A tool called SecondInventory is out.  On paper it allows you to make a local backup copy of your inventory.  A completely sane idea.  Something that is preached religiously by computer professionals since like the beginning of time.

I can personally attest to loosing items, and I might add some expensive items to Second Life.  Failing to rez and going poof.  I’ve seen it happen right before my eyes.  Think it won’t happen to you?  I know someone who just lost something precious today.  It failed to rez and went poof.  It’s basically irreplaceable too and it was no copy.  I’ve take losses myself of about L$15000 over the last year and half.  One item was expensive and it did disappear out of my inventory.

I have become pretty obsessed about inventory loss.  To the point that I have boxes in which I stuff most of the things I can copy into.  I keep these boxes rezzed in another location  Hopefully in case my personal inventory takes a loss, I can recover some things.

But why on Earth is it to this point?  It’s pretty much inexcusable.  If you’re going to design a system that handle items that have value - assets in Second Life, then you make damn sure you don’t loose them.  Concepts like not loosing transactions (items could be considered transactions) have been around for ages.  Consider a double entry accounting system.  Someone screwed up and Linden Labs knows it.  If they didn’t loose things, this sort of product would have no real reason to exist (other than for naughtiness).

Which goes back to the flip side.  For content creators this is not really a good thing.  It potentially allows their items to escape the content control system in Second Life.  It has of course potential for misuse.  Computers are very good at one thing, adding and copying data.  In fact when you boil down how they work, that’s all they do.  Preventing them from copying data is really quite hard - if not essentially impossible.

Now I’m a big supporter of the content creators.  They are the ones who make Second Life what it is.  I am like a shopaholic.  I think the root of the problem rests squarely at Linden Lab’s feet.  They give you no means of backing up your data.  They give you no guarantees that your data is safe.  So, this program is a solution for a known problem.  Someone was going to do it eventually.

How do I feel about it?  Personally, I would love to have a tool like this.  Why?  Because I have a - uh - well, way to much invested in my inventory.  For example, I use a Huddles EZAnimator HUD.  A cool device.  It cost me L$1500 (I think).  Okay, not too expensive and it’s also a mod/copy/no tran HUD.  You load it with animations, poses, etc.  I’ve used it for over a year.  Mine has dozens of dances and animations, and I think over a hundred poses.  Things I’ve collected over the last year.  Of course, some of them are no copy.  Thus making that copy of the HUD no copy.  My guess is that there is about L$50,000 in animations, dances, and poses in it, maybe more.  Whenever I want to make changes to it such as adding dances or changing configurations I have to rez it.  I usually start getting nervous before rezzing it.  More than once it has taken a few moments to appear and start getting that sinking feeling.

I also have things that are one of a kind items, things you can’t get anymore, from people who are not around anymore.  I personally could use the tool ethically, it would certainly make me feel less anxious about my inventory.  I suspect a few other will try to use or use the device in naughty ways.  So, if content creators want to protect themselves from this program, the solution is not to go after this program, but to go after Linden Labs.  If no one is going to loose content, no one needs this program.  If I’m possibly going to loose content, I probably need this program.

It does have some upsides to it.  Even for content creators.  It could allow content to flee Linden Lab’s  Second Life for another SL platform.  Linden Lab’s SL is only worth anything because of the content in it.  If that content could escape to another grid, they loose their “monopoly” on it.  A content creator could conceivably pick up their content - which they own - and easily move it to another grid.  And then setup shop there.  Lack of content has probably been the biggest impediment to the other grids going anywhere.  Just being able to pickup all of the SL freebies out there and move them would be a huge thing.  Granted there are a lot of stick bits I’m ignoring here, but I think you get the idea.

I’m sure the yelling has already begun, but is the solution to inventory loss completed?  As of today I can attest it’s not.

-Veyron

I can’t say I didn’t expect this, but Linden Labs has mostly ignored demands for more groups.  The best they put out was some double talk about technical issues on the back end impacting performance.  Duh, like if I drive to work every day in my car, I will wear out my car eventually….

So the Jira for the feature request is unassigned.  Closing in on 1000 residents have voted for it.  And yet, not a single Linden has said boo in Jira.  Supposedly Jira is how residents are able to communicate technical issues and desires to Linden Labs.  Clearly, this is not working.  Why doesn’t Linden Labs make a blog statement on the issue?  Take 10 minutes out of their busy day and uh - communicate?

Here’s an idea.  State the facts, and have LL close the issue.  Show some cojones and pull the trigger.  Leaving it out there and letting the customers prattle on about it is just cowardly on their part.

Here’s an even better idea, how about offering more groups to the paying residents?  The ones who are now a tiny minority.  That small minority would put a tiny load on the system and Linden Labs would make more money.  Hmm, the more money would pay for more developers and hardware to do more things.  Nahhh…  That’s too easy.  Back to the head in the sand.

-Veyron

Blogged with Flock

I want to address the discussion I have had with a few people already who think that Linden Labs should NOT change the group limit for fear of making Second Life buggier or worse than it already is.   Now, I’m not sure what the problem is, if there is any because Linden Labs is not communicating on the problem.  The premise is that the databases are too messed up to support more than 25 groups and they will not support more than 25 groups.   For example, just last week, they lost (and then recovered) close to half a million inventory items.  Thus supporting the argument that they would have serious problems supporting more than 25 groups.

Now, think carefully about this situation.  I work with programmers all the time on situations like this.  Programmers are by their very nature lazy, it’s how they work.  It’s not necessary a bad thing, but it’s always an underlying current.  Fixing the database problems is going to be a sticky and ugly problem.  Not nearly as much fun as adding more eye candy to the environment.  Never mind that the sticky and ugly database problems are the root of a lot of issues.  Now they don’t even have the excuse of an increasing population and userbase load.

What this means that if the 25 group limit is REALLY a database issue, then confess, make a something called a plan to fix the database and while you are at it, up the group limit.

-Veyron

As the grid is down right now, I took a look at the quality metrics for Second Life.  I think this graph, from the Linden’s own web site really says it all.   It really is appalling.  Over 1/5 of all Second Life sessions crash.  Maybe Cory needed to explain these numbers and couldn’t.  Notice how the trend is not really moving downward, sort of sideways.  Also look that this is over a one year time span.  One year to work on things and no substantial quality improvement.  But hey, we did get voice, and now lots of new shiny baubles to look at, um, until it crashes.  I do give Linden Labs kudos for putting up metrics like these and being open about it.

At some point, this graph will be a direct threat to Second Life’s and Linden Lab’s existence, if it is not already.  Philip Linden might think it’s a long road ahead to stability, I think it’s going to be a short road.  The number of active users has started to plateau now and now there is a risk of instability leading to disgust and then people leaving the environment.

Companies invariably do one of two things in general, they get bigger until a bigger fish eats them (or they become the top predator) or they get smaller until they disappear.  The question now is what is going to happen to Linden Labs/Second Life.

-Veyron

So with the news that Linden Labs CTO Cory Linden has left the building I have to wonder if things are going to get better or worse.  (As a side note, I personally hate to see people fail or have these sorts of things happen to them.  I do not subscribe to the theory that in order to succeed others must fail.)  I am curious to see how often avies log out of Second Life without crashing.   That means your  Second Life session ended, on your terms, not blowing up on you.  For example, if you log into Second Life for the evening, it crashes 4 times while you’re on and then you log out at the end, that is pretty close to - almost all the time.

Ask your friends to vote in the poll over in the corner and I’ll close out the poll near the end of the month.  Maybe it’s not as bad as I think it is.  Maybe it is.

-Veyron

Nicholaz - “The Mad Patcher” - has fixed not only the asstachment bugs, but the long long desired ability to mute group IM’s.  For example, if you joined the FashionConsolidated group, you joined that to get notices for the fashionistas, not to hear some prattle from some n00b begging for some Lindens.  Now you can mute it as well.

You can get the new client from Nicholaz from his blog.

-Veyron

For asstachments that is….  You know, you teleport or fly around and then all of the sudden, your hair is where it really does not belong.  And then how annoying it is to fix it.  I think once I recall almost every prim object I was wearing was suddenly attached down there.  How annoying.

Well, help is here.  It looks like my hero, Nicholaz, may have solved the asstachment bug.   The irony of this is that it will probably be months before Linden Labs actually puts the fix into the main line code.  Why?  Well, we have to improve on this voice thing and work on some more shiny objects.  (::sigh::  don’t get me started)

Now, what I do see as good things Linden Labs is working on.  Havok 4 and Mono.  Both should improve sim performance and stability.  Both worthy causes.  Neither are shiny baubles like Windlight or voice.  I would really welcome serious forward movement on Havok 4 and Mono.  Havok looks like it’s moving forward in the next few months, but I hope the mono engine gets going.  That will improve script speed substantially, and thus, make the sims faster.

-Veyron

Okay, so I’ve seen a steady stream of interesting client software come out of Linden Labs lately.  The Windlight software and the new release canidate (RC) line of software.  Both have some pretty nice features and I kinda like how they work.  However, they both have some - well - fatal problems.  One of them is essentially fatal.  It’s memory leaks.  They are serious memory leaks.  I have seen the both beta clients leak 1GB of RAM in a few minutes of use - just before crashing.

What is annoying here is that this problem is probably fixed already.  Yup, probably already fixed.  We’re just not going to get it - at least not right away.

Nicholaz Beresford has been working on a modifying (ie fixing) the Second Life client for some time now, and I am impressed.  His fixed up version is FAR more stable than what is coming out of Linden Labs these days.  This is one guy who is actually bothering to fix problems.  Don’t believe me.  The JIRA database lists 159 issues as fixed internally.  That means the Lindens have fixed the bug, but it’s not actually in the viewers we have.  159.  That is just the start.

Windlight is great, but it is total CRAP if it crashes 5 minutes after launch.  I just don’t get it.  Linden Labs talks about how they are now going to focus on stability, openness, and getting it right.

In the military when they screw up something with safety or similarly stupid, they have a stand down.   Everyone gets their arses chewed and re-explained what needs to get done and how to GET IT DONE RIGHT.  What the programmers and managers need to now is stop playing with chrome, fiddly bits, and other stupid shiny things and fix the damn bugs.

In the mean time, I recommend you all give Nicholaz’s viewer a work out.  It seems to be a lot more stable.

-Veyorn

Well, I haven’t seen it work yet, but this might be a critical app for you fashion conscious.  A de-Ruthing tool.  We all pretty much hate being Ruthed.  The Autoderuth Tool by Dougal Jacobs is a free-to-copy-and-distribute gift to the grid.  You can get the HUD at the Linden Lifestyles Headquarters.

-Veyron

Koz needs help with the Lindens in getting a “bug” fixed.  It will solve a problem with BlogHUD not being able to get the coordinates of a postcard now emailed from the browser - how you post an image to the site.  I suspect that Snapzilla is similarly broken.  You can log into Jira and vote for the Lindens to fix this problem.

Thanks!

-Veyron

So, I have a few bugs to smash on the new blog.  One of which is a highly annoying internal bug which is not allowing me to edit anything posted.  This seemed to appear at some point, as it worked while I was developing the site.  I’m working on that one.  My work around is to go into the database and hack the raw data, not an optimal solution.  If any WordPress experts out there have any ideas, let me know.  It seems from the support site, I’m not the only one who is having this problem now and then.  I’ve been tinkering turning on and off things trying to quash it.  Email me if you have ideas.

I also discovered that the site was not looking so good under Internet Explorer.  Which is notorious for having a poor CSS implementation.  I use Firefox, and I didn’t bother to check how it looked under IE (do people still use Internet Exploder?).  My bad.   I’ve also made some changes to try and make it look better under IE.  Again, let me know from the feedback form if something looks bad or wrong.  I’ll add it to my list.

Lastly, some of the old postings I imported from Blogger had the wrong URL, so I had to go into the database, and hack that up.  So, that should be right now.  At least I’m knocking a few problems off one by one.

-Veyron