Aurora-Sim WebAPI Roadmap

1.x

I’m considering our frozen branch to be the 1.x release- it gets no new features, only bug fixes at this point

2.x

This work focuses on the separation between the original PHP and the c# module should be complete, allowing for more flexible development.

  • Remove the manual SQL installation step.
  • Port all usage of the module’s API to libAurora.php

3.x

  • Switch to HTTP Authentication instead of an md5-hashed password for API usage.
  • Read-only methods switched to HTTP GET rather than having everything as POST
  • Migrating to REST rather than stuffing all request data into POST
  • Fine-grained control for other users to access methods in the API
  • API usage logging
  • API usage throttling

Update

The c# module from webui 1.x and webui 2x have been merged into a single repository, with work on WebAPI 3.x starting today.

Update

Everything on the 3.x roadmap has been implemented except for RESTification, although the read-only methods have been switched to HTTP GET so the REST step might be tabled for a point release rather than 3.0 itself.

Deploying OpenSim on Windows Server 2008

This article was written at work, but I thought it might be of use to current and potential OpenSim users :-)

Lates OSGrid Binaries are available from their website.

Requirements

  1. MySQL (ftp://ftp.mirrorservice.org/sites/ftp.mysql.com/Downloads/MySQL-5.5/mysql-5.5.8-win32.msi used)
  2. OpenSim (http://download.osgrid.org/osgrid.opensim-01262011.v0.7.1.c472764.zip used)
  3. The IP address of the machine that OpenSim is being installed on.

Installation

MySQL

  1. Start the installer
  2. Click "Next"
  3. Accept the GPL EULA, then click "Next"
  4. Select "Custom" setup type (this is a good habit to get into when installing software, to prevent useless crud being installed)
  5. Ensure all required components will be installed, then click "Next"
  6. Click "Install"
  7. System Administrator permissions may be required at this point
  8. Select "Detailed Configuration"
  9. Select "Server Machine"
  10. Select "Multifunctional Database"
  11. Specify location (default of "C:\MySQL Datafiles\" is usually fine)
  12. Unless there is going to be a large number of regions (mid-high double figures & upwards), the default "Decision Support" option should be sufficient
  13. Strict mode should remain enabled. Unless security policies dictate otherwise, specify "Enable TCP/IP Networking" and port number "3306"
  14. Specify "Best Support for Multilingualism" (UTF-8)
  15. Install as Windows Service, service name MySQL, launch automatically
  16. "Include Bin Directory in Windows PATH" is optional, but may come in handy when administering the database from the command line.
  17. Once installed, setup a user & database for OpenSim to make use of.
    1. Open a command line window
    2. Log into the MySQL shell via "mysql –user=root –password=whateverItWasYouConfiguredEarlier"
    3. CREATE DATABASE opensim;
    4. CREATE USER ‘opensim’@'localhost’ IDENTIFIED BY ‘somethingDifferentToTheEarlierPassword’;
    5. GRANT ALL PRIVILEGES ON opensim.* TO opensim@localhost IDENTIFIED BY ‘thePasswordYouJustEntered’;
    6. FLUSH PRIVILEGES
    7. quit

OpenSim

  1. Create a folder on the C drive named OpenSim
  2. copy the /bin/ folder from the OpenSim zip file to the OpenSim directory.
  3. Rename the folder according to the version of OpenSim, as this helps quickly rolling back to a previous version should there be any problems.
Robust.ini
  1. Open Robust.ini.example from the renamed /bin/ folder in a text editor such as "Notepad++", and save as "Robust.ini"
  2. In the DatabaseService section, change the "Database", "User ID" and "Password" parameters of the "ConnectionString"
  3. Optionally, change the "WelcomeMessage" string to something more appropriate.
  4. Change the values in the "GridInfoService" section to something appropriate. The port number should be set to 8002.
  5. Start Robust.exe to ensure there are no errors in configuration.
  6. If successful, type "quit" and press enter to close the process.
/config-include/GridCommon.ini
  1. Comment out the Include-Storage line for SQLite
  2. Uncomment the MySQL parameter line, using the same parameters specified in Robust.ini for the ConnectionString
  3. Change all URI parameters to http://whateverTheIPAddressIs:8003
Create a blank slate for deploying island clusters
  1. In C:\OpenSim\, create a folder named "IslandClusters"
  2. Create a folder named "blank-slate", with three sub-folders named "backup", "crashes" and "regions"
  3. Create a file named "shutdown.txt" with only the word "backup" contained within
  4. Save a copy OpenSim.ini (or OpenSim.ini.example) to C:\OpenSim\IslandClusters\blank-slate\OpenSim.ini
  5. In the "Startup" section;
    1. Specify crash_dir as "C:\OpenSim\IslandClusters\blank-slate\crashes"
    2. Specify startup_console_commands_file as "C:\OpenSim\IslandClusters\blank-slate\startup.txt"
    3. Specify shutdown_console_commands_file as "C:\OpenSim\IslandClusters\blank-slate\shutdown.txt"
    4. Specify regionload_regionsdir as "C:\OpenSim\IslandClusters\blank-slate\regions"
    5. Alter TextureOnMapTile to true
    6. Alter/uncomment see_into_this_sim_from_neighbor to true
    7. Alter/uncomment MaptileRefresh to 3600
  6. In the "Network" section, Alter/uncomment http_listener_port to 10000
  7. In the "Messaging" section, disable (comment out) all options.
  8. In the "DataSnapshot" section, disable (comment out) all options except for "gridname"- change that to the short grid name specified in Robust.ini
  9. In the "Search" section, disable (comment out) all options.
  10. In the "Profile" section, disable (comment out) all options.
  11. In the "XEngine" section, change "AllowOSFunctions" to false
  12. In the "Groups" section, disable (comment out) all options.
  13. Create a new shortcut to OpenSim.exe in the renamed /bin/ folder in "C:\OpenSim\IslandClusters\blank-slate", specifying the command line options as –inifile="C:\OpenSim\IslandClusters\blank-slate\OpenSim.ini"
  14. Zip up the folder to blank-slate.zip, then delete the folder.
Deploying an island cluster

It is important to keep track of port numbers used in islands, since they cannot be shared on the same machine.

Robust.exe should have been started prior to these steps.

  1. Extract blank-slate.zip to C:\OpenSim\IslandClusters
  2. Rename the resulting "blank-slate" folder to the name of the new island cluster
  3. Open OpenSim.ini in a text editor, and replace all instances of "blank-slate" with the name of the new island cluster folder
  4. Right-click the OpenSim shortcut, and change "blank-slate" in the command line arguments to the name of the new island cluster folder
  5. Double-click the new shortcut- if everything has been configured correctly, the OpenSim shell will eventually ask for a Region name.
  6. Most defaults can be accepted, by the co-ordinates cannot be shared by islands on the same grid, whilst port numbers cannot be shared by islands on the same server.
Adding content to the shared Library

It will be necessary to restart Robust.exe

  1. Create a folder named _libraries in C:\OpenSim\
  2. Create a sub-folder for the content.
  3. Place the folders.lib.xml, items.lib.xml and assets.lib.xml files in the new sub-folder.
  4. Create a sub-folder in the content folder named assets.
  5. Place all assets in the assets sub-folder.
  6. Open inventory/Libraries.xml from the renamed /bin/ directory.
  7. Duplicate the last <Section> block in the file, altering the duplicate’s Name attribute to correspond to the name specified in step 2
  8. Update the Value attributes in the two <Key> elements to the absolute path to the folders.lib.xml and items.lib.xml files
  9. Open assets/AssetSets.xml from the renamed /bin/ directory.
  10. Duplicate the last <Section> block in the file, altering the duplicate’s Name attribute to correspond to the name specified in step 2
  11. Update the Value attribute in the <Key> elements to the absolute path of the assets.lib.xml
Windows Firewall with Advanced Security

In order to allow people to connect to the OpenSim grid, at least 3 inbound firewall rules will need to be created.

Windows Firewall does not currently provide the rule creation window with an interface for specifying port ranges (although you can provide a comma separated list of port numbers), nor does it allow you to provide an interface for specifying multiple protocols. Rather than selecting "Any" protocol, separate rules should be created for both TCP and UDP protocols.

It is suggested that one pair of rules be created for each island cluster, to make managing the rules easier in the long-term.

  • Robust.exe needs ports 8002 & 8003 over TCP
  • For reach island, Opensim.exe needs access to the port number specified during deployment over both TCP and UDP.
  • If the Robust rest console is to be used, a TCP port needs to be opened on the number specified by "ConsolePort" in the Network section of Robust.ini

Upgrading

While the OpenSim wiki contains instructions on upgrading Opensim, our deployment process differs to the default.

Don’t forget to backup the OpenSim database by bringing up a command prompt window, and type mysqldump -h localhost -u username -p opensim > c:\OpenSim\backup\opensim-YYYY-MM-DD_HH-MM.sql, where YYYY-MM-DD_HH-MM is today’s today and current time. You will be prompted for the password.

Updating Robust.exe & OpenSim common configurations

  1. As with installing OpenSim from scratch, extract the new version of OpenSim to a folder that is version-specific
  2. Review Robust.ini.example, OpenSimDefaults.ini, OpenSim.ini, config-include/Grid.ini and config-include/GridCommon.ini for changes made since the previous version. (Notepad++ comes in handy for this task, as it has a built-in document comparison tool).
  3. If there are no unexpected changes to the files, copy the Robust.ini, OpenSim.ini and config-include/Grid.ini files from the previous deployment to the new deployment folder. If there are changes, review them for integration or overriding.
  4. Test the configuration of the newer Robust.exe by launching it and checking for unexpected errors. Quit the process if it is successful.
  5. Copy any alterations from the old inventory/Libraries.xml and assets/AssetSets.xml files to the files in the new deployment folder. Do not simply copy the files as new libraries and asset sets may have been added to a newer version of OpenSim.
  6. Start the newer Robust.exe

Updating the blank slate

Steps 3,4 and 5 need to be repeated for each island cluster, but are written in the context of the "blank-slate" archive, since the blank slate also needs to be updated.

  1. Extract the contents of C:\OpenSim\IslandClusters\blank-slate.zip to C:\OpenSim\IslandClusters\blank-slate\
  2. Rename C:\OpenSim\IslandClusters\blank-slate.zip to refer to the version of OpenSim it is meant for.
  3. Right click the shortcut to OpenSim in the blank slate folder and select properties.
  4. Update Target to reflect the location of the newer OpenSim.exe.
  5. Update Start in to reflect the directory for the newer deploy of OpenSim.
  6. Create a new zip file archiving the files in C:\OpenSim\IslandClusters\blank-slate\.
  7. Delete the blank-slate folder

Life On Mars/Ashes to Ashes: It’s over now

I’ve just finished watching the final episode of Ashes to Ashes season 3. The ending didn’t come as a complete surprise, since two years ago I wrote up my thoughts on possible explanations for what was happening. Granted, I was wrong about what if Gene Hunt is an “angel”, but I was right about a few things.

Tyler was sent to make a better man of Hunt.

The reason I’m suspect I’m right about this point- that Sam Tyler was sent to redeem Gene Hunt- is that by the time of Ashes to Ashes, Gene has forgotten who he is and why he was there. Some of the lines in Life On Mars would cast some doubt on whether Hunt had forgotten his role, but we can certainly see the character being redeemed over time. Further evidence is given in Alex Drake’s actions helping Hunt remember who he is and why he’s there.

Interestingly, Hunt ends up forcing Ray Carling to go through the events that led up to him committing suicide- namely that Carling caused a civilian to die during the course of his duty (I’ll have to re-watch Life On Mars to refresh my memory as to whether he beat up the prisoner in that episode), and his boss covered up Carling’s role in the matter. Sharon Granger goes through a similar ordeal in episode 7 season 1 of Ashes to Ashes when she gets stabbed in the stomach, almost dying in the process (and seeing the Angel of Death that had been stalking Alex Drake). Again, I’ll have to watch the DVDs to see if Chris is made to relive the manner of his death during the course of either show.

So in the final episode of Ashes to Ashes, we’ve found out that the world that the majority of both shows takes place in is actually an afterlife of sorts for police officers. In this world, Hunt is responsible for helping them work through their issues. So while he’s not an angel as had occurred to me two years ago, he’s certainly an important figure in the spirit world.

Interestingly, getting back to the references to Quantum Leap I made in my previous musings, Nelson the barman bears some subtle similarities to Al the barman in the final episode of Quantum Leap.

Ashes to Ashes ends things well, and although things were certainly left open with Hunt continuing in his role guiding the spirits of police officers, I’m hoping no further shows are made within the Life on Mars/Ashes to Ashes universe. I don’t really think there were any unanswered questions left (unlike how things ended with Firefly), so any further shows set within the same universe would likely involve entirely different characters (e.g. no Philip Glenister reprising the role of Hunt), or perhaps some plot involving Hunt finding himself a replacement which would allow him to move onto Heaven (such a plot would be unlikely to produce a show as good as either Life On Mars or Ashes to Ashes). However, a book detailing Hunt’s original death and how he comes to gain his responsibilities in dealing with the spirits of police officers wouldn’t go amiss.

Updates to agni.sl.mapapi.net

I’ve made quite a few changes to my pluggable version of slurl.com, most of which centred around replacing the hash-slurl.js plugin with hash-fu.js. Below is a summary of the changes made today.

The Map

  • Removed some CSS code left over from the original work done for Hack a Day 2009.
  • Replaced hash-slurl.js with hash-fu.js.
  • Removed the Gears plugin since that’s being phased out, with developers being encouraged to use HTML5 goodness.

UI Top Bar

  • Plugins for the UI top bar were altered so they’d report an error if the UI top bar plugin wasn’t loaded instead of throwing an exception to the JavaScript console.
  • The UI top bar plugin was tweaked so it could be hidden.
  • If the error reporting plugin is loaded, but UI top bar isn’t, the modal dialog pops up immediately instead of adding a notification to the UI top bar.
  • Implemented a feature request; When plugins with drop-down boxes are loaded into the top bar, they’re hidden until they’re populated with options.
  • A bug in the region search plugin was fixed (return false wasn’t getting passed from the search handler to the event handler).
  • Support for hash-fu.js was added to the third party services plugin. Example loading the New World Notes stories featuring SLURLs.
  • mapapi.js

    No feature alterations here, just some fixes for a couple bugs that cropped up during the course of today’s updates, one of which was in the error handler and the other was to fix how axes with co-ordinates set to 0 were handled.

    hash-fu.js

    hash-fu.js isn’t too dissimilar to hash-slurl.js, in the respect that they both monitor document.location.hash for changes, then attempt to act upon those changes.

    Where hash-slurl.js tried to cram the processing for such changes into the same file, hash-fu.js prefers to leave such logic to other plugins. Where hash-slurl.js uses an if-else structure with hard-coded feature support, hash-fu.js uses a for structure with callbacks. Additionally, it supports multiple commands being fired in the same fragment via a fake query string e.g. example.com/#?foo=bar&slurl/Ahern/.

    Second Life: Script-triggered Sounds and Viewer Audio Channels

    Second Life allows objects to be scripted to trigger the playback of sounds. These can include anything including (but not limited to) birdsong, gunshots, clicks and sound bites.

    The viewer- or at least those intended to present the full audio-visual experience (there are text-only applications for accessing Second Life)- has an audio preferences panel, giving the user control over the sound levels of various channels:

    • Music
    • Media
    • Voice
    • Sounds
    • Ambient
    • UI

    There’s also the “Master” volume slider, but that’s not a channel per se.

    Currently, all script-triggered audio gets piped through the “Sounds” channel- this allows you to mute in-world audio if you’re on a voice conference, mute the UI if you’re filming Machinima, or mute the in-world gusts of wind via the Ambient channel (though I’m not sure what else uses Ambient).

    An idea that occurred to me that seemed to gain some interest in AW Groupies (hence the blog post here fleshing out the idea a little more) was whether or not script-triggered audio could/should have the ability to be declared as belonging to a specific audio channel, i.e. set sounds like birdsong, the roar of a waterfall or the trickle of a stream to the Ambient channel, so the sound level could be reduced or muted entirely, leaving the user more able to hear sounds like footsteps, doorbells, or anything else that’s intended to provide audio feedback to the user (such as the earlier gunshots example). CodeWarrior actually suggested that some script-triggered sounds might be more suitably applied to the UI channel rather than the current global “Sounds” channel.

    This is obviously a feature that would require some server-side assistance to work as intended, though I am curious to know if it’s possible to have a viewer “intelligently” distribute sounds across the various channels, e.g. if a particular asset gets constantly looped, it should be shoved into the “Ambient” channel. Although not really suitable for long-term use (it would probably break a few intended behaviours in current content), it would certainly give a tangible demonstration of what effect any extra functions should have on the overall user experience. Then of course, there is the issue that any server-side solution would have to be done in such a way that was backwards compatible with non-supporting clients (e.g. not preventing script-triggered sounds from glooming into the global “Sounds” channel).

    Fixing a pet peeve with the display of many Twitter avatars on a single page

    So I sent Suzen Juel a tweet about Spotify, suggesting that if not for distributing her own music, she could use it to let her fans know what music she likes.

    Twitter being twitter and having the open API that it does, SpotiChart had a tweetbot that detected the presence of a link to Spotify playlist, inviting me to make use of their service (and thankfully, their coders seem to have implemented a mechanism to save their resources and the sanity of Twitter users by only sending the automated tweet directed at each user once). So I took a look around their site, specifically the “Most Tweeted Playlists” chart. Given that it was the first time I’d viewed the page, even on my fast (~8mbps) ADSL connection there was a noticeable delay as the images loaded- lots of irritating “broken image” place-holders being displayed by Firefox. Even after all the avatars had finished loading, one or two avatars had just 500′d, resulting in the display of the alt text.

    This experience agitated my web geek sensibilities, and it sparked a possible solution in my head, which I successfully tested.

    The Problem

    The problem lies in browser’s default behaviour when handling uncached images, as well as ones that have failed for whatever reason. If an alt attribute is specified on the image tag, the text gets displayed in place of where the image should be. This isn’t always desirable, since alt text can mess up the layout of a web page.

    Example of Twitter avatar display

    No special CSS close-up

    With no special CSS applied- only some basic floating, you can see that things don’t look so pretty. It looks worse in browsers such as Internet Explorer, which display a broken image icon alongside the alt text.

    Adding constraints

    constraints close-up

    When no dimensions are specified, elements are allowed to expand to accommodate the alt text, or in the case of an image containing no alt text, the elements usually shrink down to a point where they may as well not be on the page. By adding height and width, a problem is avoided where the page expands vertically (and occasionally horizontally) as the content loads. On newer computers this is problem is merely an annoyance, but on older or underpowered devices, this could be quite irritating as it causes unnecessary reflows and repaints.

    We’re still left with the alt text being displayed, as well as potentially having holes appearing in a large grid of avatars when images fail to load.

    Solution Attempt One

    clipping close-up

    The initial solution that popped into my head after visiting spotichart.com worked, but it introduced a problem: images that load will get replaced by the fallback image along with images that don’t!

    Solution Attempt Two

    jQuery solution close-up

    So we need a solution that allows us to only apply the fallback image in cases where it’s needed. Well the server serving an HTML document can’t reliably detect when an image will fail, so the detection has to occur on the client. This is where JavaScript comes to the rescue, or in this case: jQuery.

    Since JavaScript provides an “error” event, jQuery can be used to append a value to the class attribute of an image that has failed to load. When JavaScript is disabled or not available, the CSS provides a graceful fall back.

    Summary

    • Not providing image dimensions can interfere with your layout, and can cause a source of annoyance as the page is rendered.
    • jQuery can help indicate which images failed to load.
    • CSS allows for graceful degradation.

    The code for this solution- which can apply to other situations like image galleries, or anywhere that large numbers of small images are displayed together in a list or grid- is available under the public domain with all the example CSS & HTML removed to make the source code easier to read through.

    WebComic RSS

    I had a poke around my public services (the stuff on services.signpostmarv.name), cleaned up a bunch of crap that was no-longer in use, then decided to poke around the code for my WebComic RSS service. There’s a mix of UHU4 and UHU5 code in there, so I thought it was time to start upgrading the code to UHU6.

    I’m making heavy use of the namespaces & late static binding features in UHU6, so it’s easier for me to keep things organised (yay namespaces!), as well as trimming down on the code used in methods for inherited classes (yay late static binding!). This, combined with my weaning myself off Smarty templating in favour of Object API methods seems to have made the turnaround time for generating the feed much shorter.

    At the time of writing, I’ve only upgraded the feeds for , , and . Additionally, the simplified code for WebComic RSS in UHU6 eliminated a long-standing bug with the Punch an’ Pie feed which caused the strip urls to be borked.

    Object API methods

    A quick word on what I’m referring to here- basically, I’m using methods with hinted types for the arguments to restrict the input at runtime so I know that the objects going in will be of a specified interface. For instance, the API for my project makes heavy use of the vCalendar format. Rather than creating a never-ending list of parsing rules for the various classes I’ll be creating as I work more on UHU6, I’ve just written a single parsing rule for a calendar class. The calendar class implements the but can only contain objects which implement the “event” interface. Coding to interfaces instead of particular implementations of an interface keeps the code much, much leaner.

    Quick SL Achievements web client demo

    Since I’ve made some progress on the project, I decided to create this video demonstrating the web client for the SL Achievements API that I’m working on.

    There is a working HUD at the moment, but that version of the HUD doesn’t talk with the server so completing an achievement doesn’t get recorded.

    For the API, I’ve used a vCard/vCalendar-based syntax that should be easily parsed by LSL, though some API calls will have extra-condensed outputs available. With the “list all achievements” call for instance, there’s a vCard-based output that’s fairly human-readable, and the extra-condensed output designed to be used by the LSL HUD.