Friday, April 17, 2015

New client libraries, PHP client bindings, contributed applications and other significant improvements released to the beta site

A few days ago we released a substantial update to our beta site.  Many improvements have been made to the back end to make things more efficient, and a few creature comforts have been added to the front end as well.  We'll likely reflect these changes to the main site sometime this week.

I'll cover the minor changes at the end of the blog.  There are three big changes that I need to describe in more detail.  Without further ado:

Client Libraries Now Class Based

What this means is that the various client bindings we offer are now encapsulated in classes rather than exposed as top level functions which required all arguments to be passed on every call.  We made this change to make it easier to manage credentials and also in support of the new "contributed applications" functionality described below.  There is one exception: we didn't change the Java client library.  We'll do that eventually, be we left it for last because no one is really using that library at the moment.

Each client library now provides four classes:
  • CapsuleerAPI: this class exposes all methods used to access capsuleer data from a synchronized account.
  • CorporationAPI: this class exposes all methods used to access corporation data from a synchronized account.
  • ReferenceAPI: this class exposes methods for retrieving EVE server reference information (e.g. the current skill tree).
  • SDEAPI: this is actually two classes, one for each of the two latest EVE Online releases.  These classes expose methods for retrieving Static Data Export (SDE) information.
To use the new API, you first instantiate one of these classes and pass EveKit API key information into the constructor.  You can then call the methods of the class without having to present your EveKit API key on every call.  The SDE API also allows you to specify the name of the release you'd like.  If you don't specify a release, then you'll get the API for the latest release.  If you're release agnostic, then upgrading your code just got easier: just never pass a release string into the SDE API constructor and you'll always use the latest release.

NOTE: this change does not actually change the underlying REST calls.  Instead, this change repackages how the various client bindings are exposed.  If you're making direct REST calls, you won't be affected by this change.  Likewise, if you've downloaded an older version of the client, then that client will continue to work until you need a feature only present in a newer release.

Let's look at a few examples in Google App Script (essentially, Javascript without requiring callbacks) to see how your code will change.  Before the change, getting your corporation assets might look like this:

var apikey = 1234;
var apihash =  'FA8EB656E23C0AE73796F9D17E7A0290B2BA3952BF89C3B1E4A35FDE480E6E06';
var assetResponse = commGetAssets(apikey, apihash);

In the new code, there is one extra step:

var apikey = 1234;
var apihash =  'FA8EB656E23C0AE73796F9D17E7A0290B2BA3952BF89C3B1E4A35FDE480E6E06';
var corpAPI = new CorporationAPI(apikey, apihash);
var assetResponse = corpAPI.getAssets();

However, once you've created the CorporationAPI instance, you don't need to worry about API credentials any more unless someone changes the credentials on the EVE Online support page.  The same pattern works for the SDE API.  Before the change (retrieving list of agent types here):

var sdekey = 1234;
var sdehash =  'FA8EB656E23C0AE73796F9D17E7A0290B2BA3952BF89C3B1E4A35FDE480E6E06';
var agentTypesResponse = scylla_agtGetAllAgentType(sdekey, sdehash);

In the new code, you now instantiate an appropriate SDE class and make the call on that:

var sdekey = 1234;
var sdehash =  'FA8EB656E23C0AE73796F9D17E7A0290B2BA3952BF89C3B1E4A35FDE480E6E06';
var sdeAPI = new SDEAPI(sdekey, sdehash, 'scylla');
var agentTypesResponse = sdeAPI.agtGetAllAgentType();

If you leave the release name off of the call to "new SDEAPI" then you'll get the latest release.  Regardless, this normalizes the calls across versions so most of your code won't have to change when a new SDE comes out.

PHP Client API released

To accompany the change in how we present the client API, we've also released PHP client bindings.  You can download the code straight from the GitHub site or if you're using composer you can pull the package from packagist using the package name "dark-planet-labs/evekit-php".  The GitHub page contains complete instructions for how to integrate the code.

The PHP bindings are used in the same manner as the other bindings.  For example, the two code snippets above look as follows with the PHP bindings:

// See the GitHub site for details on how to ensure the bindings are in your path
use EveKit\EveKit;
// EveKit API key and hashcode.
$apikey = 1234;
$apihash = 'FA8EB656E23C0AE73796F9D17E7A0290B2BA3952BF89C3B1E4A35FDE480E6E06';
// Create an appropriate API instance
$api = EveKit::getCorporationAPI($apikey, $apihash);
// Retrieve and echo asset tree
echo $api->getAssets();
// Now let's pull something from the SDE
$sdekey = 5678;
$sdehash = 'FA8EB656E23C0AE73796F9D17E7A0290B2BA3952BF89C3B1E4A35FDE480E6E06';
// If we omit the last argument in this next call, then we'll get the API for the
// most recent release.
$sde = EveKit::getSDEAPI($sdekey, $sdehash, 'scylla');
// Retrieve and echo the list of agent types
echo $sde->getAllAgentTypes();
PHP calls return instances of EveKit\API\Response or EveKit\API\ResponseList as appropriate.  The structure of these objects follows the structure of similar objects in the other client bindings (e.g. Javascript).  One difference is that the PHP bindings throw exceptions on errors (like the Java bindings) as opposed to returning an error type (like the Javascript or GAS bindings).  Data types for API calls are defined in the EveKit\Types package.

Contributed Applications and the AppBridge

It's long been the goal to make EveKit a platform for others to develop their own third party apps.  The new "contributed application" feature is an attempt to further this goal.  A contributed application is just a third party website that you register with EveKit.  An EveKit user can then choose to "instantiate" a contributed application, in which case EveKit makes it possible for the contribute application to access EveKit data on behalf of the instantiating user.  The goal here is to simplify and standardize the way one builds applications designed to work with EveKit.

The details of this feature were described in a previous post, but here it is again with more careful explanation.  There are two logical parties involved in any contributed application: the application contributor, and the application user.  The application contributor is responsible for creating the contributed application, which consists of the following:
  • A unique name and description;
  • A URL pointing to the application web site;
  • An indication of whether the application should launch in a new page (this is needed for certain websites which don't allow authentication from a frame); 
  • A public or private designation; and,
  • One or more key-value pairs which represent the initial state of the app (these are strings).
Contributed applications are associated with the EveKit screen name you authenticated as at the time the application was contributed.  If you choose to make a contributed application public, then it can be searched and used by any other EveKit user.

An application user can choose to instantiate a contributed application.  An application instance has the following features:
  • A unique name;
  • An indication as to whether the application is allowed to use the user's SDE key; and,
  • A list of capsuleer or corporation access keys that should be made available to the instantiated application.
An instantiated application will appear in the "Apps" menu when the instantiating user is logged into EveKit.  When a user selects an instantiated application, a new frame or tab is opened using the URL of the contributed application.  The query string of this URL will contain two special parameters "evekitkey" and "evekithash" which represent a key-value pair the application can use to access the private key-value store of the application (including any API keys that have been shared with this application).  Through this mechanism, the contributed application can then access the exposed data of the application user.

The new EveKit client libraries include an "AppBridge" class which simplifies implementing contributed application websites.  This class is instantiated with the values of "evekitkey" and "evekithash", and provides convenience methods for interacting with the private key-value store of the instantiated application (including retrieving API keys).  Here's a snippet of code showing how the AppBridge class can be used in a PHP based contributed application:

// Pull in the EveKit client bindings
use EveKit\EveKit; 
// Extract evekitkey and evekithash from the URL
parse_str($_SERVER['QUERY_STRING'], $qstring);
$ekey = $qstring['evekitkey'];
$ehash = $qstring['evekithash'];
// Create the AppBridge.
$bridge = EveKit::getAppBridge($ekey, $ehash);
// Contributed applications may have a limited set of configuration data.
// You can read a configuration value as follows:
$val = $bridge->getConfig("somekey");
// You can save a configuration value like this:
$bridge->setConfig("somekey", "somevalue");
// For this example, let's assume the user assigned the SDE key and a corp
// key to this application.  You can get all the appropriate Corporation APIs
// like this:
$corpAPIs = $bridge->getCorporationAPIs();
// You can then make an API call as in the example above, e.g.
// (in this example, there is only one corp key)
$assets = $corpAPIs[0]->getAssets();
// The SDE API is equally easy to use, e.g.
$agentTypes = $bridge->getSDEAPI()->getAllAgentTypes();
// If you want a particular SDE release, you can also use this form:
$agentTypes = $bridge->getSDEAPI('scylla')->getAllAgentTypes();

The AppBridge provides raw access to the underlying API keys if needed, but in most cases you should only need to use the API convenience functions provided by this class.

Fine Print

And now some fine print around the contributed application feature:
  • Application names (both contributed and instantiated) must be alphanumeric (including underscore) and at least two characters in length;
  • Contributed application names must be unique among all applications contributed by the same user.  Likewise, instantiated application names must be unique among all applications instantiated by the same user;
  • The URL for a third party app must be less than 1000 character in length.  You're free to embed a query string but avoid the reserved strings "evekitkey" and "evekithash" which are used by EveKit to pass credentials to the instantiated application;
  • You can change most features of a contributed application at any time after it is submitted (if you were the original contributor);
  • Users "capture" a snapshot of your contributed application when they choose to use it.  This means that you can't break existing users if you decide to modify or remove a contributed application after one or more users have decided to use it.  You'll need to use other means to communicate with existing users of your application (e.g. if you need to make upgrades);
  • The private key-value store for an instantiated application has the following limits:
    • Keys must be alphanumeric (including underscore and period), must be at least one character long, and no longer than 300 characters;
    • Values must be strings no longer than 2048 characters;
    • Keys may not start with the reserved prefix 'evekit_';
    • The initial key-value store for an application may not have more than 200 key-value entries; and
    • An instantiated application may not have more than 300 key-value entries.  Note that any API keys assigned to the instantiated application consume one entry per key.

Other Improvements

There are a few other quality of life changes on the beta site.  For example, accounts are now clearly marked with red text when their XML API key has expired.  Likewise, the account synchronization history view uses a new color scheme to make it clear when certain data was not sync'd because the XML API key doesn't allow access.

Friday, April 3, 2015

100 Billion: March Income and Balance Sheet, shrinking to two accounts, and (finally) a description of what the 100 Billion Corp actually does!

March has come and gone and the 100 Billion Corporation actually made some money, but not enough to overcome the PLEX load we're charging ourselves.  It turns out we can't fill up three full EVE accounts with activity yet, so we're shrinking to two accounts to reduce the PLEX load.  More on that below.  I went to FanFest, saw some Icelandic nature, and gave a talk about development on Google App Engine.  Check out the link if you're interested, and the numerous FanFest recaps others have posted.  We finish off this entry with a description of what the six characters which make up the 100 Billion Corp actually do to make ISK.

March By The Numbers

Let's take a look at the balance sheet first.  Remember that a balance sheet is a record of a company's assets and liabilities at a given point in time.  Here's what things looked like at the end of February:

and here's where we stand after March:

This is pretty much flat month to month except that staff equity decreased by ISK 2 billion as we continue to struggle overcoming the PLEX tax.  There's one new liability here labelled "Historical".  This is how we're recording the fact that we had three PLEX'd accounts for the first three months of the company, and the sum here is exactly the cost of buying three PLEX for one account at the beginning of the first three months of the year.  For now, we'll just keep two accounts for the corporation, to the tune of about ISK 1.5 billion per month.  So that's what we need to bring in to stay even.  Staff equity wise, we would have done better if we'd never paid for that third account.  Due to company income and asset appreciation, we would have been flat across the board.  As it stands, however, our investors lost another ISK 2 billion this month.

FanFest and other real life duties kept me from finishing some of the tools I had planned to make some of this tracking easier.  However, I am planning a significant EveKit release in about two weeks (mid-cycle for EVE Online releases) which will have some changes I need to support these tools.  I'll be posting a blog entry in a few days describing the changes.

Let's now look at the asset valuation for March.  Here's the top 20 from the end of February:

and here's the top 20 at the end of March:

It's worth mentioning at this point that I had to switch asset valuation sources early in March.  Previously, I had used EVE Marketdata to retrieve asset prices in The Forge.  However, the site had a major several day outage during which I decided to switch to EVE Central.  There are some clear differences in prices which are cause for concern.  A significant "to do" is to switch to the new CREST market endpoint instead, but I haven't got around to that yet.

The most notable differences month to month are the massive increase in value of the Genolution Core Augmentation CA-3, and the complete disappearance of the Genolution Core Augmentation CA-4.  The former moved up about 20%, and the latter didn't even make the top 20.  The price of the CA-4 at the end of the month was ISK 39 million.  That's likely an anomaly due to someone making a very foolish sale of a high value item.  Since the balance sheet is a snapshot in time, it's bound to be affected by anomalies like this.  It's also notable that Toxic Metals continue to rise significantly.  We actually dumped some of our stock in March to generate some cash flow, we may do more of the same this month to lock in some profit.

Overall, the message from this balance sheet is basically the same as for the last few months:

  • We're carrying too much of a cash balance.  A goal for April is to invest at least ISK 1 billion in a long term play of some sort.  It's probably too early to buy PLEX.  Toxic metals is an interesting play, however.  Also, there was a change announced on the O7 show last night in which major materials changes were discussed.  I still need to digest, but this is another possible play although one in which many others will participate.
  • We did better this month, but we're still not making enough to keep the company afloat.

Let's move on to the income statement.  Here's where we stood at the end of February:

and here's where we are at the end of March:

What changed?
  • We're in the green from an income point of view.  With this income, we essentially paid for one of the three PLEX we were carrying into the month.
  • The vast majority of our income was from selling produced goods on the market.  More on that below.
  • We had less time for mission running this month, so that part of our income suffered a bit.
  • PI taxes were again a significant part of our expenses, which is not surprising given how much we rely on PI for our production business.
Let's look at the top 20 transactions by value.  For February:

and for March:

For March, we diversified a bit in what we sell and ended up making most of our income from tech II drones.  Our stock in Small Armor Repairer II is lower, hence less money from those sales.  Competition also picked up around the Small Armor Repairer II for reasons I don't yet understand.  When this happens, the 0.01 ISK botters show up and it takes significantly longer to sell my stock (remember: I promised not to use any bots, not only is it not legal, it's not cool for my experiment).  Tech II drones are extremely reliable income, they sell very quickly in the hub I sell them in.  Random mission loot rounded out the bottom of the list as usual.

The debit side of the ledger is dominated by pre-production items.  That is, raw materials for items used in tech II production.  The one exception was Electronic Engineering data cores, which are used for tech II blueprint invention.  Overall, the margins here were very nice although you can't see the mineral or other material costs that went into the tech II production. I'll need to post a separate spreadsheet to show that.

You'll note on the income side that I dumped some of my Toxic Metals stock this month.  The price trend around Toxic Metals suggest I should do some more of that again to lock in some profit.  We have more than enough stock for our current needs, I can easily dump half the stock and not miss it.

That's it for the 100 Billion corp financial update, now let's talk about what the company actually does.

The Business of the 100 Billion Corp

The 100 Billion Corp is (now) composed of six characters, all dedicated solely to the corporation.  The company generates income from the following activities:
  • Mission Running: Two of the characters are used in a multi-box setup to blitz level 4 missions.  Typically, one character runs the mission (in a Tengu), while the other follows up behind to collect salvage.  A small set of tougher missions require both characters to run, then we bookmark and collect the salvage later.
  • Tech II Production: The main income source of the company at the moment is tech II production of the following items: Adaptive Nano Plating II, Small Armor Repairer II, Small Capacitor Booster II, Hammerhead II, Hobgoblin II, Damage Control II, 150mm Railgun II, Light Ion Blaster II, Light Neutron Blaster II, 1MN Afterburner II, Warp Scrambler II.  I generated this list from a not very scientific scan of items which are often used for PvP fits and which have reasonable volume.
  • Planetary Interaction (PI): We use PI mainly to produce materials needed for tech II production.  We currently produce: Construction Blocks, Mechanical Parts, Miniature Electronics, Rocket Fuel, Superconductors, Transmitters, Guidance Systems, and Robotics.  Any other materials we produce by buying raw materials and running production jobs.  The list here was chosen largely to support tech II production, although the company made its first billion almost solely on the back of selling robotics on the market.
  • Market: we sell the vast majority of our tech II production on the market, as well as any "valuable" mission loot or rewards (e.g. implants given as mission rewards).  We occasionally dump some of our mineral stock as well.
  • Investments: we have a small collection of items we're holding as investments, usually awards or gifts from CCP in the past which we aren't using (e.g. certain implants).  We haven't sold any of these yet to generate cash.
Most of the six characters in the corporation are multi-taskers (meaning: they trained skills for multiple roles), but generally the characters get used as follows:
  • Character 1 (C1): mission running, mining, PI, invention, production, market
  • Character 2 (C2): mission running, mining, PI, production
  • Character 3 (C3): PI, market, production
  • Character 4 (C4): mining, PI
  • Character 5 (C5): PI
  • Character 6 (C6): PI
One thing I almost never do is mine!  I still do this occasionally, but usually as a background task while I'm doing something else (like hacking on EveKit).  Instead of mining, we salvage or re-process a fair amount of mission loot.  A typical session of mission running goes as follows:
  1. Run one or more missions with the primary character, use the secondary character to follow the primary and loot/salvage.
  2. For all mission loot:
    1. Salvage items which can be used in production are saved;
    2. Ammo we use is saved (mostly missiles);
    3. Drones we use for production are saved (e.g. Hammerheads or Hobgoblins);
    4. Items which can not be re-processed are either saved (if they are worth at least ISK 400000), or are immediately discarded or sold;
    5. Items which can be re-processed but which are worth at least ISK 400000 are saved to be sold on the market;
    6. Everything else is reprocessed into minerals and used for production.

We use the EVE client estimated value of an item when deciding what to do with mission loot.  This is the value shown in the client when hovering over an item in your inventory.  It's not perfect, but it's a reasonable estimate for quickly sifting through mission loot.  The ISK 400000 threshold is also a bit arbitrary.  It's my estimate of large enough value to justify carrying an item to a market hub for sale.

Generally speaking, we sell all production or loot items at a hub using limit orders (a sell order with a specific price).  We occasionally sell with market orders instead (meaning, an order which immediately sells at the current best bid price) when the spread (the difference between the best bid order and best ask order) is very small.  This most often happens with mission loot, where the volume is so low it's simply not worth the time to periodically reset the order to deal with competition.

The activities above consume a fair amount of my time.  A typical week looks like this:
  • Daily: log in at least once to reset competitive market orders, complete any finished production jobs, and start new production jobs.  This takes about 30 minutes a day.
  • Twice a week: reset PI jobs.  We typically run four day resource extractions, with transport of some resources to drive PI production jobs.  I've honed this down to a fairly tight procedure which I can get done in 30 minutes if there are no distractions.
  • Weekends: usually at least one mission running session lasting two hours or so.  I have to sacrifice this for vacations, family time, etc.
  • Every other month or so: invention of tech II blueprints.  I normally produce 200 run blueprint copies (or whatever the maximum is for a given blueprint original), then gather enough data cores so that I can run large batch invention jobs.  A typical invention run takes around two weeks to complete and generates enough stock to allow production for a month or two.
It's very easy to get into a rut with the above schedule, so I tend to use the mission running time to experiment or try some new things, like look for possible new production targets.  Very rarely, I'll mine instead of mission run, and use this time to hack on EveKit, write blogs, catch up on EVE news, etc.

Hacking on EveKit and keeping up with EVE news is another major time sink, but I'll leave that for another entry.

That's all for this month's income report.  Expect another entry in a few days describing the upcoming EveKit release.